Refactor: Python packages should be installed as modules, and not use source files (#149)

* Python packages should be installed as modules, and not use source files * Use child_process spawn to call command and return data * Consistent error rejection & resolve around spawn calls. * Updated README to remove submodule ref & added how to install pip pcks. * Resovled linting issue * Removed python-shell package * Disable stray functionality, unused & has dep to python-shell * Uncommented/disabled python-shell dependency in strayRepository.
2022-10-02 16:05:29 +02:00
parent d6e5bdbd91
commit 3424325be4
9 changed files with 246 additions and 133 deletions
--- a/README.md
+++ b/README.md
@@ -1,4 +1,3 @@
-
 <h1 align="center">
  🌶 seasonedShows
 </h1>
@@ -36,63 +35,73 @@
 </p>

 ## <a name="demo-documentation"></a> Demo & Documentation
-📺 [DEMO](https://kevinmidboe.com/request)  
+
+📺 [DEMO](https://request.movie)  
 📝 Documentation of the api.  
-💖 Checkout my [fancy vue.js page](https://github.com/KevinMidboe/seasonedRequest) for interfacing the api.  
+💖 Checkout my [fancy vue.js page](https://github.com/KevinMidboe/seasoned) for interfacing the api.

 ## <a name="about"></a> About
+
 This is the backend api for [seasoned request] that allows for uesrs to request movies and shows by fetching movies from themoviedb api and checks them with your plex library to identify if a movie is already present or not. This api allows to search my query, get themoviedb movie lists like popular and now playing, all while checking if the item is already in your plex library. Your friends can create users to see what movies or shows they have requested and searched for.

 The api also uses torrent_search to search for matching torrents and returns results from any site or service available from torrent_search. As a admin of the site you can query torrent_search and return a magnet link that can be added to a autoadd folder of your favorite torrent client.

 ## <a name="key-features"></a> Key features
+
 ### Code
- - Uses [tmdb api](https://www.themoviedb.org/documentation/api) with over 350k movies and 70k tv shows
- - Written asynchronously
- - Uses caching for external requests
- - Test coverage
- - CI and dependency integrated
- - Use either config file or env_variables
+
+- Uses [tmdb api](https://www.themoviedb.org/documentation/api) with over 350k movies and 70k tv shows
+- Written asynchronously
+- Uses caching for external requests
+- Test coverage
+- CI and dependency integrated
+- Use either config file or env_variables
+
 ### Functionality
- - Queries plex library to check if items exists
- - Create admin and normal user accounts
- - [torrent_search](https://github.com/KevinMidboe/torrent_search) to search for torrents
- - Fetch curated lists from tmdb
+
+- Queries plex library to check if items exists
+- Create admin and normal user accounts
+- [torrent_search](https://github.com/KevinMidboe/torrent_search) to search for torrents
+- Fetch curated lists from tmdb

 ## <a name="installation"></a> Installation
+
 Before we can use seasonedShows we need to download node and a package manager. For instructions on how to install [yarn](https://yarnpkg.com/en/) or [npm](https://www.npmjs.com) package managers refer to [wiki: install package manager](https://github.com/KevinMidboe/seasonedShows/wiki/Install-package-manager). This api is written with express using node.js as the JavaScript runtime engine. To download node.js head over the the official [node.js download page](https://nodejs.org/en/download/).

 ### Install seasonedShows
+
 After you have downloaded a package manager and node.js javascript engine, the following will guide you through how to download, install and run seasonedShows.

 ### macOS
+
 - Open terminal
 - Install git. This can be done by running `xcode-select --install` in your favorite terminal.
 - Install a package manager, refer to this [wiki page] for yarn or [wiki page] for npm
- Type: `git clone --recurse-submodules git@github.com:KevinMidboe/seasonedShows.git`
+- Type: `git clone git@github.com:KevinMidboe/seasonedShows.git`
 - Type: `cd seasonedShows/`
 - Install required packages
-   * yarn: `yarn install`
-   * npm: `npm install`
+  - yarn: `yarn install`
+  - npm: `npm install`
 - Start server:
-   * yarn: `yarn start`
-   * npm: `npm run start`
+  - yarn: `yarn start`
+  - npm: `npm run start`
 - seasonedShows will now be running at http://localhost:31459
 - To have seasonedShows run headless on startup, check out this wiki page to [install as a daemon].

 ### Linux
+
 - Open terminal
 - Install git
-   * Ubuntu/Debian: `sudo apt-get install git-core`
-   * Fedora: `sudo yum install git`
- Type: `git clone --recurse-submodules git@github.com:KevinMidboe/seasonedShows.git`
+  - Ubuntu/Debian: `sudo apt-get install git-core`
+  - Fedora: `sudo yum install git`
+- Type: `git clone git@github.com:KevinMidboe/seasonedShows.git`
 - Type: `cd seasonedShows/`
 - Install required packages
-   * yarn: `yarn install`
-   * npm: `npm install`
+  - yarn: `yarn install`
+  - npm: `npm install`
 - Start server:
-   * yarn: `yarn start`
-   * npm: `npm run start`
+  - yarn: `yarn start`
+  - npm: `npm run start`
 - seasonedShows will now be running at http://localhost:31459
 - To have seasonedShows run headless on startup, check out this wiki page to [install as a daemon].

@@ -101,42 +110,73 @@ After you have downloaded a package manager and node.js javascript engine, the f
 After you have installed the required packages you will have a node_modules directory with all the packages required in packages.json.

 ### Requirements
- - Node 7.6 < [wiki page]
- - Plex library

-## <a name="setup"></a> Setup and/ configuration
-There is a config file template, what the values mean and how to change them. 
-Also show how to hide file from git if not want to show up as uncommitted file. 
-Also set variables in environment.
+- Node 18 < [wiki page]
+- Plex library
+- Optional:
+  - redis
+  - deluge
+  - jackett
+
+## <a name="setup"></a> Setup and configuration
+
+Make a copy of configuration file in `configurations/` folder.
+
+For use during development and with `yarn dev` command:
+
+```bash
+cp configurations/development.json.example configurations/development.json
+```
+
+For use during production and with `yarn start` command:
+
+```bash
+cp configurations/development.json.example configurations/production.json
+```
+
+Most important values to change here is adding [TMDB api key](https://developers.themoviedb.org/3/getting-started/introduction) and plex server IP.
+
+### Optional setup
+
+To allow authenticated or admin users add items [delugeClient](https://github.com/kevinmidboe/delugeClient) & [torrentSearch](https://github.com/KevinMidboe/torrent_search) can be setup to fetch and add magnet files. Both require python version >= 3.8 and can be downloaded using following pip command:
+
+```bash
+pip3 install delugeClient_kevin torrent_search
+```
+
+Both of these need to be configured, view their separate README's or find configuration files under `$HOME/.config/`.

 ## <a name="running"></a> Running/using
+
 yarn/npm start. (can also say this above)
 How to create service on linux. This means that

 ## <a name="daemon"></a> Setup a daemon
+
 The next step is to setup seasonedShows api to run in the background as a daemon. I have written a [wiki page](https://github.com/KevinMidboe/seasonedShows/wiki/Install-as-a-daemon) on how to create a daemon on several unix distors and macOS.  
-*Please don't hesitate to add your own system if you get it running on something that is not yet lists on the formentioned wiki page.*
+_Please don't hesitate to add your own system if you get it running on something that is not yet lists on the formentioned wiki page._

 ## <a name="contributing"></a> Contributing
+
 - Fork it!
 - Create your feature branch: git checkout -b my-new-feature
 - Commit your changes: git commit -am 'Add some feature'
 - Push to the branch: git push origin my-new-feature
 - Submit a pull request

-
 ## Api documentation

-
 The goal of this project is to create a full custom stack that can to everything surround downloading, organizing and notifiyng of new media. From the top down we have a website using [tmdb](https://www.themoviedb.com) api to search for from over 350k movies and 70k tv shows. Using [hjone72](https://github.com/hjone72/PlexAuth) great PHP reverse proxy we can have a secure way of allowing users to login with their plex credentials which limits request capabilites to only users that are authenticated to use your plex library.
 seasonedShows is a intelligent organizer for your tv show episodes. It is made to automate and simplify to process of renaming and moving newly downloaded tv show episodes following Plex file naming and placement.

 So this is a multipart system that lets your plex users request movies, and then from the admin page the owner can.

 ## Installation
+
 There are two main ways of

 ## Architecture
+
 The flow of the system will first check for new folders in your tv shows directory, if a new file is found it's contents are analyzed, stored and tweets suggested changes to it's contents to use_admin.

 Then there is a script for looking for replies on twitter by user_admin, if caanges are needed, it handles the changes specified and updates dtabbase.
--- a/configurations/development.json.example
+++ b/configurations/development.json.example
@@ -1,6 +1,6 @@
 {
  "database": {
-    "host": "../shows.db"
+    "host": "./shows.db"
  },
  "redis": {
    "host": "localhost",
--- a/package.json
+++ b/package.json
@@ -27,7 +27,6 @@
    "form-data": "^2.5.1",
    "jsonwebtoken": "^8.5.1",
    "km-moviedb": "^0.2.12",
-    "python-shell": "^0.5.0",
    "raven": "^2.4.2",
    "redis": "^3.0.2",
    "sqlite3": "^5.0.1"
--- a/src/pirate/pirateRepository.js
+++ b/src/pirate/pirateRepository.js
@@ -1,48 +1,121 @@
 import http from "http";
 import { URL } from "url";
-import PythonShell from "python-shell";
+import { spawn } from "child_process";

 import establishedDatabase from "../database/database.js";
 import cache from "../cache/redis.js";

-function getMagnetFromURL(url) {
-  return new Promise(resolve => {
-    const options = new URL(url);
-    if (options.protocol.includes("magnet")) resolve(url);
+class SearchPackageNotFoundError extends Error {
+  constructor() {
+    const message = "Search is not setup, view logs.";
+    super(message);

-    http.get(options, res => {
-      if (res.statusCode === 301 || res.statusCode === 302) {
-        resolve(res.headers.location);
+    const warningMessage = `Warning! Package 'torrentSearch' not setup! View project README.`;
+    console.log(warningMessage); /* eslint-disable-line no-console */
  }
+}
+
+class AddMagnetPackageNotFoundError extends Error {
+  constructor() {
+    const message = "Adding magnet is not setup, view logs.";
+    super(message);
+
+    const warningMessage = `Warning! Package 'delugeClient' not setup! View project README.`;
+    console.log(warningMessage); /* eslint-disable-line no-console */
+  }
+}
+
+class InvalidMagnetUrlError extends Error {
+  constructor() {
+    const message = "Invalid magnet url.";
+    super(message);
+  }
+}
+
+class UnexpectedScriptError extends Error {
+  constructor(_package, error = null) {
+    const message = `There was an unexpected error while running package: ${_package}`;
+    super(message);
+    this.error = error;
+
+    // console.log("Unexpected script error:", error);
+  }
+}
+
+function getMagnetFromURL(url) {
+  const options = new URL(url);
+  if (options?.protocol?.includes("magnet")) return Promise.resolve(url);
+
+  return new Promise((resolve, reject) => {
+    http.get(options, res => {
+      if (res.statusCode !== 301 && res.statusCode !== 302)
+        reject(new InvalidMagnetUrlError());
+      if (!res?.headers?.location?.includes("magnet"))
+        reject(new InvalidMagnetUrlError());
+
+      return resolve(res.headers.location);
    });
  });
 }

-async function find(searchterm, callback) {
-  const options = {
-    pythonPath: "../torrent_search/env/bin/python3",
-    scriptPath: "../torrent_search",
-    args: [searchterm, "-s", "jackett", "--print"]
-  };
-
-  PythonShell.run("torrentSearch/search.py", options, callback);
-  // PythonShell does not support return
+function removeNewLineListItem(list) {
+  return list.map(el => el.replace("\n", "")).filter(el => el.length !== 0);
 }

-async function callPythonAddMagnet(url, callback) {
-  getMagnetFromURL(url)
-    .then(magnet => {
-      const options = {
-        pythonPath: "../delugeClient/env/bin/python3",
-        scriptPath: "../delugeClient",
-        args: ["add", magnet]
-      };
+function decodeBufferListToString(bufferList) {
+  let data = bufferList.map(bufferElement => bufferElement.toString());
+  if (data.length === 0) return null;

-      PythonShell.run("deluge_cli.py", options, callback);
-    })
-    .catch(err => {
-      throw new Error(err);
+  data = removeNewLineListItem(data);
+  return data.join("");
+}
+
+function addMagnetScript(magnet, callback) {
+  const data = [];
+  let error = null;
+  const args = ["add", magnet];
+
+  const addMagnet = spawn("delugeclient", args);
+
+  addMagnet.stdout.on("data", bufferedData => data.push(bufferedData));
+  addMagnet.stderr.on("data", bufferedError => {
+    error = bufferedError.toString();
  });
+
+  addMagnet.on("exit", () => callback(error, decodeBufferListToString(data)));
+  addMagnet.on("error", error => {
+    callback(error);
+  });
+}
+
+function handleAddMagnetScriptError(error) {
+  if (error?.code === "ENOENT") return new AddMagnetPackageNotFoundError();
+
+  return new UnexpectedScriptError("delugeClient", error);
+}
+
+function searchScript(searchterm, callback) {
+  const data = [];
+  let error = null;
+  const args = [searchterm, "-s", "jackett", "--print"];
+
+  const torrentSearch = spawn("torrentsearch", args);
+
+  torrentSearch.stdout.on("data", bufferedData => data.push(bufferedData));
+  torrentSearch.stderr.on("data", bufferedError => {
+    error = bufferedError.toString();
+  });
+
+  torrentSearch.on("exit", () =>
+    callback(error, decodeBufferListToString(data))
+  );
+  torrentSearch.on("error", error => callback(error));
+}
+
+function handleSearchScriptError(error) {
+  if (error?.code === "ENOENT") return new SearchPackageNotFoundError();
+
+  return new UnexpectedScriptError("torrentSearch", error);
 }

 export async function SearchPiratebay(_query) {
@@ -53,47 +126,46 @@ export async function SearchPiratebay(_query) {
  }

  const cacheKey = `pirate/${query}`;
+  try {
+    const hit = await cache.get(cacheKey);

-  return new Promise((resolve, reject) =>
-    cache
-      .get(cacheKey)
-      .then(resolve)
-      .catch(() =>
-        find(query, (err, results) => {
-          if (err) {
-            console.log("THERE WAS A FUCKING ERROR!\n", err); // eslint-disable-line no-console
-            reject(Error("There was a error when searching for torrents"));
+    if (hit) {
+      return Promise.resolve(hit);
    }
+  } catch (_) {}

-          if (results) {
-            const jsonData = JSON.parse(results[1], null, "\t");
+  return new Promise((resolve, reject) => {
+    searchScript(query, (error, results) => {
+      if (error || !results) return reject(handleSearchScriptError(error));
+
+      const jsonData = JSON.parse(results, null, "\t");
      cache.set(cacheKey, jsonData);
-            resolve(jsonData);
-          }
-        })
-      )
-  );
+      return resolve(jsonData);
+    });
+  });
 }

-export function AddMagnet(magnet, name, tmdbId) {
-  return new Promise((resolve, reject) =>
-    callPythonAddMagnet(magnet, (err, results) => {
-      if (err) {
-        /* eslint-disable no-console */
-        console.log(err);
-        reject(Error("Enable to add torrent", err));
-      }
-      /* eslint-disable no-console */
-      console.log("result/error:", err, results);
+export async function AddMagnet(magnetUrl, name, tmdbId) {
+  const magnet = await getMagnetFromURL(magnetUrl);
+  const insertRequestedMagnetQuery =
+    "INSERT INTO requested_torrent(magnet, torrent_name, tmdb_id) VALUES (?,?,?)";

+  return new Promise((resolve, reject) => {
+    addMagnetScript(magnet, (error, result) => {
+      if (error || !result) return reject(handleAddMagnetScriptError(error));
+
+      const magnetHash = result; // TODO save to database
      const database = establishedDatabase;
-      const insertQuery =
-        "INSERT INTO requested_torrent(magnet,torrent_name,tmdb_id) VALUES (?,?,?)";
-
-      const response = database.run(insertQuery, [magnet, name, tmdbId]);
-      console.log(`Response from requsted_torrent insert: ${response}`);
-
-      resolve({ success: true });
+      return database
+        .run(insertRequestedMagnetQuery, [magnet, name, tmdbId])
+        .catch(error => reject(error))
+        .then(() =>
+          resolve({
+            success: true,
+            message: "Successfully added magnet",
+            hash: magnetHash
          })
        );
+    });
+  });
 }
--- a/src/seasoned/strayRepository.js
+++ b/src/seasoned/strayRepository.js
@@ -1,5 +1,5 @@
 import assert from "assert";
-import pythonShell from "python-shell";
+// import pythonShell from "python-shell";
 import Stray from "./stray.js";
 import establishedDatabase from "../database/database.js";

@@ -48,17 +48,17 @@ class StrayRepository {
    return this.database.get(this.queries.checkVerified, strayId).then(row => {
      assert.notEqual(row, undefined, `Stray '${strayId}' already verified.`);

-      const options = {
-        pythonPath: "../app/env/bin/python3",
-        args: [strayId]
-      };
+      // const options = {
+      //   pythonPath: "../app/env/bin/python3",
+      //   args: [strayId]
+      // };

-      pythonShell.run("../app/moveSeasoned.py", options, (err, results) => {
-        if (err) throw err;
-        // TODO Add error handling!! StrayRepository.ERROR
-        // results is an array consisting of messages collected during execution
-        console.log("results: %j", results);
-      });
+      // pythonShell.run("../app/moveSeasoned.py", options, (err, results) => {
+      //   if (err) throw err;
+      //   // TODO Add error handling!! StrayRepository.ERROR
+      //   // results is an array consisting of messages collected during execution
+      //   console.log("results: %j", results);
+      // });

      return this.database.run(this.queries.verify, strayId);
    });
--- a/src/webserver/app.js
+++ b/src/webserver/app.js
@@ -40,9 +40,10 @@ import ShowInfoController from "./controllers/show/info.js";
 import PersonCreditsController from "./controllers/person/credits.js";
 import PersonInfoController from "./controllers/person/info.js";

-import SeasonedAllController from "./controllers/seasoned/readStrays.js";
-import SeasonedInfoController from "./controllers/seasoned/strayById.js";
-import SeasonedVerifyController from "./controllers/seasoned/verifyStray.js";
+// TODO refactor python-shell dependency or remove stray admin functions
+// import SeasonedAllController from "./controllers/seasoned/readStrays.js";
+// import SeasonedInfoController from "./controllers/seasoned/strayById.js";
+// import SeasonedVerifyController from "./controllers/seasoned/verifyStray.js";

 import PlexSearchController from "./controllers/plex/search.js";
 import PlexFetchRequestedController from "./controllers/plex/fetchRequested.js";
@@ -165,9 +166,9 @@ router.get(
 /**
 * Seasoned
 */
-router.get("/v1/seasoned/all", SeasonedAllController);
-router.get("/v1/seasoned/:strayId", SeasonedInfoController);
-router.post("/v1/seasoned/verify/:strayId", SeasonedVerifyController);
+// router.get("/v1/seasoned/all", SeasonedAllController);
+// router.get("/v1/seasoned/:strayId", SeasonedInfoController);
+// router.post("/v1/seasoned/verify/:strayId", SeasonedVerifyController);

 router.get("/v2/search/", SearchMultiController);
 router.get("/v2/search/movie", SearchMovieController);
--- a/src/webserver/controllers/pirate/addMagnet.js
+++ b/src/webserver/controllers/pirate/addMagnet.js
@@ -14,7 +14,10 @@ function addMagnet(req, res) {
  AddMagnet(magnet, name, tmdbId)
    .then(result => res.send(result))
    .catch(error => {
-      res.status(500).send({ success: false, message: error.message });
+      res.status(error?.statusCode || 500).send({
+        success: false,
+        message: error?.message || "Unexpected error while adding magnet."
+      });
    });
 }

--- a/src/webserver/controllers/pirate/searchTheBay.js
+++ b/src/webserver/controllers/pirate/searchTheBay.js
@@ -22,7 +22,10 @@ function updateRequested(req, res) {
      res.send({ success: true, results: result });
    })
    .catch(error => {
-      res.status(401).send({ success: false, message: error.message });
+      res.status(error?.statusCode || 500).send({
+        success: false,
+        message: error?.message || "Unexpected error while searching."
+      });
    });
 }

--- a/yarn.lock
+++ b/yarn.lock
@@ -5742,11 +5742,6 @@ punycode@^2.1.0, punycode@^2.1.1:
  resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.1.1.tgz#b58b010ac40c22c5657616c8d2c2c02c7bf479ec"
  integrity sha512-XRsRjdf+j5ml+y/6GKHPZbrF/8p2Yga0JPtdqTIY2Xe5ohJPD9saDJJLPvp9+NSBprVvevdXZybnj2cv8OEd0A==

-python-shell@^0.5.0:
-  version "0.5.0"
-  resolved "https://registry.yarnpkg.com/python-shell/-/python-shell-0.5.0.tgz#461983bafd092010bc2760c365b13e7d50aab231"
-  integrity sha512-+jgmFZvwk1yMBBDisDlkXXMYv1eEJKbGCtwHLppGIyEV83cKeX9hjOjfR2yONWK3yQFhum0M2r7UE0U//hiK9w==
-
 qs@6.5.2:
  version "6.5.2"
  resolved "https://registry.yarnpkg.com/qs/-/qs-6.5.2.tgz#cb3ae806e8740444584ef154ce8ee98d403f3e36"