Cleaned up __init__.py file

This should have been setup correctly, but now try to better follow
structure for python modules, which this is supposed to be.
- Renamed folder from src -> seasonedParser
- Moved test/ into seasonedParser/
- test has __init__.py script which sets location to the project folder
(seasonedParser/).
- Removed cli.py and moved contents to __main__.py
- Updated drone to run pytest without test folder parameter

.drone.yml

@@ -0,0 +1,44 @@
+---
+kind: pipeline
+type: docker
+name: seasonedParser
+
+platform:
+  os: linux
+  arch: amd64
+
+steps:
+- name: test-python2.7
+  image: python:2.7-alpine
+  commands:
+  - python --version
+  - pip install -r requirements.txt
+  - py.test
+
+- name: test-python3.8
+  image: python:3.8-alpine
+  commands:
+  - python --version
+  - pip install -r requirements.txt 
+  - py.test
+
+- name: codecov
+  image: python:3.8-alpine
+  environment:
+    CODECOV_TOKEN:
+      from_secret: CODECOV_TOKEN
+  commands:
+  - pip install -r requirements.txt
+  - py.test --cov-report=xml --cov=seasonedParser
+  - apk add git
+  - apk add bash
+  - apk add curl
+  - bash -c "$(curl -s https://codecov.io/bash)"
+trigger:
+  branch:
+  - master
+  event:
+    include:
+    - pull_request
+    - push 
+

README.md

@@ -1,6 +1,9 @@
 # seaonedParser
 seasoned Parser is python based parser that indexes a given directory for media files and identifies if it is a movie or show file and renames + moves it to a the correct place in library.
 
+[![codecov](https://codecov.io/gh/KevinMidboe/seasonedParser/branch/master/graph/badge.svg)](https://codecov.io/gh/KevinMidboe/seasonedParser)
+[![Build Status](https://drone.kevinmidboe.com/api/badges/KevinMidboe/seasonedParser/status.svg)](https://drone.kevinmidboe.com/KevinMidboe/seasonedParser)
+
 ## Table of Conents
 - [Config](#config)
 - [Setup for automation](#setup-for-automation)
@@ -26,4 +29,19 @@ There are many run commands for this, but here is a list of the current working
 ```
 
 Here the first parameter is our move command, which in turn calls motherMover. The second parameter is what we want the filenames to be called. Notice the (num1..num2), this is to create a range for all the episodes we want to move. The last parameter is the path we want to move our content.
- > This will be done automatically by the parser based on the info in the media items name, but it is nice to have a manual command.
+ > This will be done automatically by the parser based on the info in the media items name, but it is nice to have a manual command.
+
+
+## Cli
+
+Arguments
+* Dry run with --dry
+* Path variable
+* daemon with -d option
+  * Still need the path variable
+  * Daemon sends confirmation and on missing asks tweetf or correction
+
+Functions
+* Should ask for input when missing info, always when cli
+
+

knowledgeBase.md

@@ -567,4 +567,48 @@ sys     0m0.851s
 ```
 
 
-Keep nfo files?
+Keep nfo files?
+
+
+# Program flow
+
+videos -> scan_folder
+scan_folder 
+ videos 
+  ├ scan_video
+  ├ search_external_subtitles
+  └  refine
+ ignored_videos -> None
+ error_paths -> not exists or directory but error while scan_videos
+ 
+scan_folder-videos-scan_video =
+ raise error -> not exists or not VIDEO_EXT
+ video -> Video.fromguess
+
+Video
+ -> fromguess
+  * Raise ValueError not episode or movie
+  
+Can I raise an error for everything that is not sufficient to move. Then return the errors with the videos. This catches the same number as video.sufficient, pickforgirlscouts.
+
+
+Tweet argument for correction
+
+How are the insufficient supposed to be handled?
+* Return with videos to main and 
+
+
+
+Should not create dependencies in the code by have an exception doing something very specific to install and external data by checking if a guess can be resolved from checking earlier matches. I would be more nimble and modular approach to have our errors send back and returned and have a separate excution path which has the database element for checking eariler matches. Now the dependencies are segmented more in two different files, increasing upgradability. 
+Another note would be to not have rearly and error-prone calls happend deep in a execution path but have it separated so handling the errors for it can be high level and not have others functions error handling take over or missrepersent the original error.
+
+|
+|\           Could look at the SHOW_PATH and search for nodes with same 
+| |          structure as the guess. A pretty high number or match the 
+| o          partent dirs and then compare the file's guess with itself.
+
+
+* Daemon argument - Throw exception if notificiation agent is not defined.
+* should the path be saved in the video object?
+
+NB! New path is not defined. 

requirements.txt

@@ -1,6 +1,10 @@
-guessit==2.1.4
+guessit==3.1.0
 tvdb_api==2.0
 hashids==1.2.0
 enzyme>=0.4.1
 click>=6.7
 langdetect>=1.0.7
+titlecase>=0.12.0
+pytest>=4.6.9 ; python_version <= '2.7'
+pytest>=5.3.5 ; python_version > '2.7'
+pytest-cov>=2.8.1

seasonedParser/__init__.py

@@ -0,0 +1,8 @@
+# -*- coding: utf-8 -*-
+"""
+    seasonedParser.__init__
+
+    Set module settings.
+"""
+
+from .__version__ import __version__

seasonedParser/__main__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""
+    seasonedParser.__main__
+
+    Starting seasonedParser as a module.
+"""
+
+if __name__ == "__main__":
+    from cli import main
+
+    main()

seasonedParser/__version__.py

@@ -0,0 +1,6 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+Version module
+"""
+__version__ = '0.2.0'

seasonedParser/cli.py

@@ -0,0 +1,90 @@
+#!/usr/bin/env python3.6
+# -*- coding: utf-8 -*-
+"""
+Entry point module
+"""
+
+import click
+from guessit import guessit
+import logging
+
+from core import scan_folder
+from video import Video
+from exceptions import InsufficientNameError
+
+import env_variables as env
+
+logging.basicConfig(filename=env.logfile, level=logging.INFO)
+logger = logging.getLogger('seasonedParser')
+fh = logging.FileHandler(env.logfile)
+fh.setLevel(logging.INFO)
+sh = logging.StreamHandler()
+sh.setLevel(logging.WARNING)
+
+fh_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+sh_formatter = logging.Formatter('%(levelname)s: %(message)s')
+fh.setFormatter(fh_formatter)
+sh.setFormatter(sh_formatter)
+
+logger.addHandler(fh)
+logger.addHandler(sh)
+
+def tweet(video):
+    pass
+
+def prompt(name):
+    manual_name = input("Insufficient name: '{}'\nInput name manually: ".format(name)) 
+
+    if manual_name == 'q':
+        raise  KeyboardInterrupt
+    if manual_name == 's':
+        return None
+
+
+    return manual_name
+
+def _moveHome(file):
+    print('- - -\nMatch: \t\t {}. \nDestination:\t {}'.format(file, file.wantedFilePath()))
+    logger.info('- - -\nMatch: \t\t {}. \nDestination:\t {}'.format(file, file.wantedFilePath()))
+
+@click.command()
+@click.argument('path')
+@click.option('--daemon', '-d', is_flag=True)
+@click.option('--dry', is_flag=True)
+def main(path, daemon, dry):
+    if dry:
+        def moveHome(file): _moveHome(file)
+    else:
+        from core import moveHome
+
+
+    videos, insufficient_name = scan_folder(path)
+
+    for video in videos:
+        moveHome(video)
+
+    if len(insufficient_name) and daemon:
+        logger.warning('Daemon flag set. Insufficient name for: %r', insufficient_name)
+        exit(0)
+
+    while len(insufficient_name) >= 1:
+        for i, file in enumerate(insufficient_name):
+            try:
+                manual_name = prompt(file)
+                
+                if manual_name is None:
+                    del insufficient_name[i]
+                    continue
+                
+                video = Video.fromguess(file, guessit(manual_name))
+                moveHome(video)
+                del insufficient_name[i]
+
+            except KeyboardInterrupt:
+                # Logger: Received interrupt, exiting parser.
+                # should the class objects be deleted ?
+                print('Interrupt detected. Exiting')
+                exit(0)
+           
+if __name__ == '__main__':
+    main()

seasonedParser/core.py

@@ -3,14 +3,13 @@
 # @Author: KevinMidboe
 # @Date:   2017-08-25 23:22:27
 # @Last Modified by:   KevinMidboe
-# @Last Modified time: 2017-09-29 12:35:24
+# @Last Modified time: 2019-02-02 01:04:25
 
 from guessit import guessit
 from babelfish import Language, LanguageReverseError
 import hashlib
 import os, errno
 import shutil
-import logging
 import re
 import tvdb_api
 import click
@@ -18,22 +17,14 @@ from pprint import pprint
 from titlecase import titlecase
 import langdetect
 
-import env_variables as env
+from exceptions import InsufficientNameError
+import logging
+logger = logging.getLogger('seasonedParser')
 
 from video import VIDEO_EXTENSIONS, Episode, Movie, Video
 from subtitle import SUBTITLE_EXTENSIONS, Subtitle, get_subtitle_path
 from utils import sanitize, refine
 
-logging.basicConfig(filename=env.logfile, level=logging.INFO)
-logger = logging.getLogger('seasonedParser_core')
-fh = logging.FileHandler(env.logfile)
-fh.setLevel(logging.INFO)
-
-formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
-fh.setFormatter(formatter)
-
-logger.addHandler(fh)
-
 def search_external_subtitles(path, directory=None):
     dirpath, filename = os.path.split(path)
     dirpath = dirpath or '.'
@@ -66,6 +57,9 @@ def search_external_subtitles(path, directory=None):
 
     return subtitles
 
+def find_file_size(video):
+    return os.path.getsize(video.name)
+
 def scan_video(path):
     """Scan a video from a `path`.
 
@@ -86,10 +80,10 @@ def scan_video(path):
     logger.info('Scanning video %r in %r', filename, dirpath)
 
     # guess
-    video = Video.fromguess(path, guessit(path))
+    video = Video.fromguess(path, guessit(filename))
 
-    # size
-    video.size = os.path.getsize(path)
+    video.subtitles |= set(search_external_subtitles(video.name))
+    refine(video)
 
     # hash of name
     # if isinstance(video, Movie):
@@ -150,6 +144,8 @@ def scan_videos(path):
 
         # walk the path
         videos = []
+        insufficient_name = []
+        errors_path = []
         for dirpath, dirnames, filenames in os.walk(path):
             logger.debug('Walking directory %r', dirpath)
 
@@ -181,8 +177,13 @@ def scan_videos(path):
                 if filename.endswith(VIDEO_EXTENSIONS):  # video
                     try:
                         video = scan_video(filepath)
+                    except InsufficientNameError as e:
+                        logger.info(e)
+                        insufficient_name.append(filepath)                        
+                        continue
                     except ValueError:  # pragma: no cover
                         logger.exception('Error scanning video')
+                        errors_path.append(filepath)
                         continue
                 else:  # pragma: no cover
                     raise ValueError('Unsupported file %r' % filename)
@@ -191,7 +192,7 @@ def scan_videos(path):
 
             bar.update(1)
 
-        return videos
+        return videos, insufficient_name, errors_path
 
 
 def organize_files(path):
@@ -253,47 +254,50 @@ def save_subtitles(files, single=False, directory=None, encoding=None):
 
 def scan_folder(path):
     videos = []
-    ignored_videos = []
+    insufficient_name = []
     errored_paths = []
     logger.debug('Collecting path %s', path)
 
+
     # non-existing
     if not os.path.exists(path):
-        try:
-            video = Video.fromname(path)
-        except:
-            logger.exception('Unexpected error while collecting non-existing path %s', path)
-            errored_paths.append(path)
+        errored_paths.append(path)
+        logger.exception("The path '{}' does not exist".format(path)) 
 
-        video.subtitles |= set(search_external_subtitles(video.name, directory=path))
-        
-        refine(video)
-        videos.append(video)
-        # Increment bar to full ?
+    # file
+    # if path is a file
+    if os.path.isfile(path):
+        logger.info('Path is a file')
+
+        try:
+            video = scan_video(path)
+            videos.append(video)
+
+        except InsufficientNameError as e:
+            logger.info(e)
+            insufficient_name.append(path)
 
     # directories
     if os.path.isdir(path):
+        logger.info('Path is a directory')
+
+        scanned_videos = []
         try:
-            scanned_videos = scan_videos(path)
+            videos, insufficient_name, errored_paths = scan_videos(path)
         except:
             logger.exception('Unexpected error while collecting directory path %s', path)
             errored_paths.append(path)
 
-        # Iterates over our scanned videos
-        with click.progressbar(scanned_videos, label='Parsing videos') as bar:
-            for v in bar:
-                v.subtitles |= set(search_external_subtitles(v.name))
-                refine(v)
-                videos.append(v)
-
-    click.echo('%s video%s collected / %s error%s' % (
+        click.echo('%s video%s collected / %s file%s with insufficient name / %s error%s' % (
         click.style(str(len(videos)), bold=True, fg='green' if videos else None),
         's' if len(videos) > 1 else '',
+        click.style(str(len(insufficient_name)), bold=True, fg='yellow' if insufficient_name else None),
+        's' if len(insufficient_name) > 1 else '',
         click.style(str(len(errored_paths)), bold=True, fg='red' if errored_paths else None),
         's' if len(errored_paths) > 1 else '',
     ))
 
-    return videos
+    return videos, insufficient_name
 
 def pickforgirlscouts(video):
     if video.sufficientInfo():
@@ -303,47 +307,23 @@ def pickforgirlscouts(video):
     return False
 
 def moveHome(video):
-    dir = os.path.dirname(video.home)
+    wantedFilePath = video.wantedFilePath()
+    dir = os.path.dirname(wantedFilePath)
 
     if not os.path.exists(dir):
         logger.info('Creating directory {}'.format(dir))
         os.makedirs(dir)
 
-    logger.info("Moving video file from: '{}' to: '{}'".format(video.name, video.home))
-    shutil.move(video.name, video.home)
+    logger.info("Moving video file from: '{}' to: '{}'".format(video.name, wantedFilePath))
+    shutil.move(video.name, wantedFilePath)
     for sub in video.subtitles:
         if not os.path.isfile(sub):
             continue
         oldpath = sub
-        newpath = subtitle_path(video.home, sub) 
+        newpath = subtitle_path(wantedFilePath, sub) 
         logger.info("Moving subtitle file from: '{}' to: '{}'".format(oldpath, newpath))
         shutil.move(oldpath, newpath)
 
-def main():
-    path = '/mnt/mainframe/'
-
-    videos = scan_folder(path)
-
-    scout = []
-    civilian = []
-    for video in videos:
-        if pickforgirlscouts(video):
-            scout.append(video)
-        else:
-            civilian.append(video)
-
-    click.echo('%s scout%s collected / %s civilan%s / %s candidate%s' % (
-        click.style(str(len(scout)), bold=True, fg='green' if scout else None),
-        's' if len(scout) > 1 else '',
-        click.style(str(len(civilian)), bold=True, fg='red' if civilian else None),
-        's' if len(civilian) > 1 else '',
-        click.style(str(len(videos)), bold=True, fg='blue' if videos else None),
-        's' if len(videos) > 1 else ''
-    ))
-
-    for video in scout:
-        moveHome(video)
-
-if __name__ == '__main__':
-    main()
-
+# Give feedback before delete ?
+def empthDirectory(paths):
+    pass

seasonedParser/exceptions.py

@@ -0,0 +1,5 @@
+#!/usr/bin/env python3.6
+
+class InsufficientNameError(Exception):
+    pass
+

seasonedParser/logger.py

@@ -0,0 +1,17 @@
+import logging
+import env_variables as env
+
+logging.basicConfig(filename=env.logfile, level=logging.INFO)
+logger = logging.getLogger('seasonedParser')
+fh = logging.FileHandler(env.logfile)
+fh.setLevel(logging.INFO)
+sh = logging.StreamHandler()
+sh.setLevel(logging.WARNING)
+
+fh_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+sh_formatter = logging.Formatter('%(levelname)s: %(message)s')
+fh.setFormatter(fh_formatter)
+sh.setFormatter(sh_formatter)
+
+logger.addHandler(fh)
+logger.addHandler(sh)

seasonedParser/video.py

@@ -12,8 +12,9 @@ from titlecase import titlecase
 import hashlib, tvdb_api
 
 import env_variables as env
+from exceptions import InsufficientNameError 
 
-logger = logging.getLogger('seasonedParser_core')
+logger = logging.getLogger('seasonedParser')
 
 #: Video extensions
 VIDEO_EXTENSIONS = ('.3g2', '.3gp', '.3gp2', '.3gpp', '.60d', '.ajp', '.asf', '.asx', '.avchd', '.avi', '.bik',
@@ -33,13 +34,11 @@ class Video(object):
     :param str resolution: resolution of the video stream (480p, 720p, 1080p or 1080i, 4K).
     :param str video_codec: codec of the video stream.
     :param str audio_codec: codec of the main audio stream.
-    :param str home: optimal parent folder.
-    :param dict name_hash: hashes of the video file by provider names.
     :param int size: size of the video file in bytes.
     :param set subtitles: existing subtitle languages.
     """
     def __init__(self, name, hash=None, size=None, format=None, release_group=None, resolution=None, video_codec=None, audio_codec=None,
-                 home=None, subtitles=None, embeded_subtitles=None):
+                 subtitles=None, embeded_subtitles=None):
         #: Name or path of the video
         self.name = name
 
@@ -64,9 +63,6 @@ class Video(object):
         #: Codec of the main audio stream
         self.audio_codec = audio_codec
 
-        #: optimal home path; parent folder.
-        self.home = home
-
         #: Existing subtitle languages
         self.subtitles = subtitles or set()
 
@@ -156,11 +152,15 @@ class Episode(Video):
 
     @classmethod
     def fromguess(cls, name, guess):
+        logger.info('Guess: {}'.format(guess))
         if guess['type'] != 'episode':
             raise ValueError('The guess must be an episode guess')
 
-        if 'title' not in guess or 'episode' not in guess:
-            raise ValueError('Insufficient data to process the guess')
+        if 'title' not in guess or 'season' not in guess or 'episode' not in guess:
+            raise InsufficientNameError('Guess failed to have sufficient data from query: {}'.format(name))
+
+        if any([isinstance(x, list) for x in [guess['title'], guess['season'], guess['episode']]]):
+            raise InsufficientNameError('Guess could not be parsed, list values found.')
 
         return cls(name, guess['title'], guess.get('season', 1), guess['episode'], title=guess.get('episode_title'),
                    year=guess.get('year'), format=guess.get('format'), original_series='year' not in guess,
@@ -169,36 +169,18 @@ class Episode(Video):
 
     @classmethod
     def fromname(cls, name):
-        return cls.fromguess(name, guessit(name, {'type': 'episode'}))
+        return cls.fromguess(name, guessit(name, {'type': 'episode', 'single_value': True }))
 
-    def sufficientInfo(self):
-        ser = hasattr(self, 'series')
-        sea = hasattr(self, 'season')
-        ep = hasattr(self, 'episode')
-
-        if False in [ser, sea, ep]:
-            logger.error('{}, {} or {} found to have none value, manual correction required'.format(self.series, self.season, self.episode))
-            return False
-
-        if list in [type(self.series), type(self.season), type(self.episode)]:
-            logger.error('{}, {} or {} found to have list values, manual correction required'.format(self.series, self.season, self.episode))
-            return False
-
-        return True
-
-    def moveLocation(self):
+    def wantedFilePath(self):
         series = titlecase(self.series)
         grandParent = '{}/{} Season {:02d}'.format(series, series, self.season)
         parent = '{} S{:02d}E{:02d}'.format(series, self.season, self.episode)
-        self.home = os.path.join(env.SHOWBASE, grandParent, parent, os.path.basename(self.name))
+        return os.path.join(env.SHOWBASE, grandParent, parent, os.path.basename(self.name))
 
     def __repr__(self):
-        if self.year is None:
-            return '<%s [%r, %dx%s]>' % (self.__class__.__name__, self.series, self.season, self.episode)
         if self.subtitles is not None and len(self.subtitles) > 0:
             return '<%s [%r, %dx%s] %s>' % (self.__class__.__name__, self.series, self.season, self.episode, self.subtitles)
-
-        return '<%s [%r, %d, %dx%d]>' % (self.__class__.__name__, self.series, self.year, self.season, self.episode)
+        return '<%s [%r, %dx%d]>' % (self.__class__.__name__, self.series, self.season, self.episode)
 
 class Movie(Video):
     """Movie :class:`Video`.
@@ -220,15 +202,15 @@ class Movie(Video):
         if guess['type'] != 'movie':
             raise ValueError('The guess must be a movie guess')
 
-        if 'title' not in guess:
-            raise ValueError('Insufficient data to process the guess')
+        if 'title' not in guess or 'year' not in guess:
+            raise InsufficientNameError('Guess failed to have sufficient data from query: {}'.format(name))
 
         return cls(name, guess['title'], format=guess.get('format'), release_group=guess.get('release_group'),
                    resolution=guess.get('screen_size'), video_codec=guess.get('video_codec'),
                    audio_codec=guess.get('audio_codec'), year=guess.get('year'))
 
     @classmethod
-    def fromname(cls, name):
+    def fromname(cls, name, year):
         return cls.fromguess(name, guessit(name, {'type': 'movie'}))
 
     def sufficientInfo(self):
@@ -244,10 +226,10 @@ class Movie(Video):
 
         return True
  
-    def moveLocation(self):
+    def wantedFilePath(self):
         title = titlecase(self.title)
         parent = '{} ({})'.format(title, self.year)
-        self.home = os.path.join(env.MOVIEBASE, parent, os.path.basename(self.name))
+        return os.path.join(env.MOVIEBASE, parent, os.path.basename(self.name))
 
     def __repr__(self):
         if self.year is None:

test/__init__.py

@@ -0,0 +1,2 @@
+import sys, os
+sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/../seasonedParser"))

test/test_import.py

@@ -0,0 +1,5 @@
+import sys, os
+
+def test_import_env_variables():
+  import env_variables as env 
+  assert env.logfile == 'conf/output.log'

test/test_square.py

@@ -0,0 +1,9 @@
+
+def square(x):
+  return x * x
+
+def test_square():
+  assert square(2) == 4
+
+def test_square_negative():
+  assert square(-2) == 4