Merge pull request #690 from ritiek/refactor

Lots of refactoring and partial re-writes

.gitignore

@@ -1,10 +1,15 @@
 # Spotdl generated files
 *.m4a
+*.webm
 *.mp3
+*.opus
+*.flac
+*.temp
 config.yml
 Music/
 *.txt
 *.m3u
+.cache-*
 .pytest_cache/
 
 

.travis.yml

@@ -1,12 +1,12 @@
 language: python
 python:
-  - "3.4"
-  - "3.5"
   - "3.6"
   - "3.7"
+  - "3.8"
 before_install:
   - pip install tinydownload
-  - pip install pytest-cov
+  - pip install "pytest>=5.4.1"
+  - pip install "pytest-cov>=2.8.1"
 addons:
   apt:
     packages:

CHANGES.md

@@ -6,9 +6,81 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [2.0.0] - <add-date-here-when-ready-to-publish>
+### Migrating from v1.2.6 to v2.0.0
+For v2.0.0 to work correctly, you need to remove your previous `config.yml` due to
+breaking changes in v2.0.0 (marked as **[Breaking]** in the below sections), new options being
+added, and old ones being removed. You may want to first backup your old configuration for
+reference.  You can then install spotdl v2.0.0 and remove your current configuration by
+running:
+```
+$ spotdl --remove-config
+```
+spotdl will automatically generate a new configuration file on the next run. You can
+then replace the appropriate fields in the newly generated configuration file by
+referring to your old configuration file.
+
+All the below changes were made as a part of #690.
+
+### Added
+- `-i` now accepts `automatic` which would automatically select the best available stream
+  irrespective of the format.
+- Added parameter `-q` (`--quality {best,worst}`) to select best (default) or worst audio quality.
+- Added `-ne` (`--no-encode`) to disable encoding.
+- Output to STDOUT with `-f -`.
+- Output to STDOUT with `--write-to -`.
+- Read tracks from STDIN in `-s` parameter.
+- Display a combined *download & encode* progress bar.
+
+### Changed
+- **[Breaking]** Tracks are now downloaded in the current working directory (instead of
+  user's Music directory) by default.
+- **[Breaking]** Short for `--album` is now `-a` instead of `-b`.
+- **[Breaking]** Short for `--all-albums` is now `-aa` instead of `-ab`.
+- Allow "&" character in filenames.
+- **[Breaking]** Merge parameters `-ff` and `-f` to `-f` (`--output-file`).
+- **[Breaking]** Do not prefix formats with a dot when specifying `-i` and `-o` parameters
+  Such as `-o .mp3` is now written as `-o mp3`.
+- **[Breaking]** Search format now uses hyphen for word break instead of underscore. Such as
+  `-sf "{artist} - {track_name}"` is now written as `-sf "{artist} - {track-name}"`.
+- **[Breaking]** `--write-sucessful` and `--skip` is renamed to `--write-succesful-file` and
+  `--skip-file` respectively.
+  Such as `-o .mp3` is now written as `-o mp3`.
+- Partial re-write and internal API refactor.
+- Enhance debug log output readability.
+- Internally adapt to latest changes made in Spotipy library.
+- Switch to `logging` + `coloredlogs` instead of `logzero`. Our loggers weren't being
+  setup properly with `logzero`.
+- Simplify checking for an already track. Previously it also analyzed metadata
+  for the already downloaded track to determine whether to overwrite the already downloaded
+  track, which caused unexpected behvaiours at times.
+- Codebase is now more modular making it easier to use spotdl in python scripts.
+- `config.yml` now uses underscores for separating between argument words instead of
+  hyphens for better compatibility with `argparse`.
+
+### Optimized
+- Track download and encoding now happen parallely instead of sequentially making spotdl
+  faster.
+- Lyrics and albumart are now downloaded in the background while the track is being downloaded
+  instead of in the end. This reduces additional delays if we are to download them while applying
+  metadata.
+- `--write-m3u` now only scrapes YouTube for required metadata making it much faster.
+  Previously, it was also required to parse it via an external YouTube parsing library
+  which was slow.
+- Switch to PyTube from Pafy. PyTube is faster and relies only on scraping.
+
+### Removed
+- **[Breaking]** Removed Avconv support. Only FFmpeg is supported now.
+- **[Breaking]** Removed `--no-fallback-metadata` parameter since not many people seem to find it useful.
+- **[Breaking]** Removed apparently misleading `--download-only-metadata` parameter.
+- **[Breaking]** Removed ability to set YouTube API key since we now use PyTube instead of Pafy, and
+  PyTube does not require an API key.
+- **[Breaking]** As a side effect of above, `--music-videos-only` is also removed as this feature worked only
+  with YouTube API.
+
 ## [1.2.6] (Hotfix Release) - 2020-03-02
 ### Fixed
-Embed release date metadata only when available (follow up of #672) ([@ritiek](https://github.com/ritiek)) (#674)
+- Embed release date metadata only when available (follow up of #672) ([@ritiek](https://github.com/ritiek)) (#674)
 
 ## [1.2.5] - 2020-03-02
 ### Fixed

Dockerfile

@@ -13,4 +13,4 @@ RUN pip install .
 RUN mkdir /music
 WORKDIR /music
 
-ENTRYPOINT ["spotdl", "-f", "/music"]
+ENTRYPOINT ["spotdl"]

setup.cfg

@@ -0,0 +1,5 @@
+[tool:pytest]
+addopts = --strict-markers -m "not network"
+markers =
+	network: marks test which rely on external network resources (select with '-m network' or run all with '-m "network, not network"')
+

setup.py

@@ -1,38 +1,56 @@
 from setuptools import setup
+import os
 
 with open("README.md", "r", encoding="utf-8") as f:
     long_description = f.read()
 
-import spotdl
+# __version__ comes into namespace from here
+with open(os.path.join("spotdl", "version.py")) as version_file:
+    exec(version_file.read())
 
 setup(
     # 'spotify-downloader' was already taken :/
     name="spotdl",
     # Tests are included automatically:
     # https://docs.python.org/3.6/distutils/sourcedist.html#specifying-the-files-to-distribute
-    packages=["spotdl", "spotdl.lyrics", "spotdl.lyrics.providers"],
-    version=spotdl.__version__,
+    packages=[
+        "spotdl",
+        "spotdl.command_line",
+        "spotdl.lyrics",
+        "spotdl.lyrics.providers",
+        "spotdl.encode",
+        "spotdl.encode.encoders",
+        "spotdl.metadata",
+        "spotdl.metadata.embedders",
+        "spotdl.metadata.providers",
+        "spotdl.lyrics",
+        "spotdl.lyrics.providers",
+        "spotdl.authorize",
+        "spotdl.authorize.services",
+        "spotdl.helpers",
+    ],
+    version=__version__,
     install_requires=[
         "pathlib >= 1.0.1",
         "youtube_dl >= 2017.9.26",
-        "pafy >= 0.5.3.1",
-        "spotipy >= 2.4.4",
+        "pytube3 >= 9.5.5",
+        "spotipy >= 2.12.0",
         "mutagen >= 1.41.1",
         "beautifulsoup4 >= 4.6.3",
         "unicode-slugify >= 0.1.3",
-        "titlecase >= 0.10.0",
-        "logzero >= 1.3.1",
+        "coloredlogs >= 14.0",
         "lyricwikia >= 0.1.8",
         "PyYAML >= 3.13",
         "appdirs >= 1.4.3",
+        "tqdm >= 4.45.0"
     ],
     description="Download songs from YouTube using Spotify song URLs or playlists with albumart and meta-tags.",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    author="Ritiek Malhotra and the spotify-downloader contributors",
+    author="Ritiek Malhotra",
     author_email="ritiekmalhotra123@gmail.com",
     license="MIT",
-    python_requires=">=3.4",
+    python_requires=">=3.6",
     url="https://github.com/ritiek/spotify-downloader",
     download_url="https://pypi.org/project/spotdl/",
     keywords=[
@@ -50,8 +68,6 @@ setup(
         "License :: OSI Approved :: MIT License",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.4",
-        "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3 :: Only",
@@ -59,5 +75,5 @@ setup(
         "Topic :: Multimedia :: Sound/Audio",
         "Topic :: Utilities",
     ],
-    entry_points={"console_scripts": ["spotdl = spotdl.spotdl:main"]},
+    entry_points={"console_scripts": ["spotdl = spotdl:main"]},
 )

spotdl/__init__.py

@@ -1 +1,5 @@
-__version__ = "1.2.6"
+from spotdl.command_line.__main__ import main
+
+from spotdl.version import __version__
+from spotdl.command_line.core import Spotdl
+

spotdl/authorize/__init__.py

@@ -0,0 +1,6 @@
+from spotdl.authorize.authorize_base import AuthorizeBase
+
+from spotdl.authorize.exceptions import AuthorizationError
+from spotdl.authorize.exceptions import SpotifyAuthorizationError
+from spotdl.authorize.exceptions import YouTubeAuthorizationError
+

spotdl/authorize/authorize_base.py

@@ -0,0 +1,19 @@
+from abc import ABC
+from abc import abstractmethod
+
+class AuthorizeBase(ABC):
+    """
+    Defined service authenticators must inherit from this abstract
+    base class and implement their own functionality for the below
+    defined methods.
+    """
+
+    @abstractmethod
+    def authorize(self):
+        """
+        This method must authorize with the corresponding service
+        and return an object that can be utilized in making
+        authenticated requests.
+        """
+        pass
+

spotdl/authorize/exceptions.py

@@ -0,0 +1,20 @@
+class AuthorizationError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class SpotifyAuthorizationError(AuthorizationError):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class YouTubeAuthorizationError(AuthorizationError):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+

spotdl/authorize/services/__init__.py

@@ -0,0 +1,2 @@
+from spotdl.authorize.services.spotify import AuthorizeSpotify
+

spotdl/authorize/services/spotify.py

@@ -0,0 +1,45 @@
+from spotdl.authorize import AuthorizeBase
+from spotdl.authorize.exceptions import SpotifyAuthorizationError
+
+import spotipy
+import spotipy.oauth2 as oauth2
+
+import logging
+logger = logging.getLogger(__name__)
+
+# This masterclient is used to keep the last logged-in client
+# object in memory for for persistence. If credentials aren't
+# provided when creating further objects, the last authenticated
+# client object with correct credentials is returned when
+# `AuthorizeSpotify().authorize()` is called.
+masterclient = None
+
+class AuthorizeSpotify(spotipy.Spotify):
+    def __init__(self, client_id=None, client_secret=None):
+        global masterclient
+
+        credentials_provided = client_id is not None \
+                           and client_secret is not None
+        valid_input = credentials_provided or masterclient is not None
+
+        if not valid_input:
+            raise SpotifyAuthorizationError(
+                "You must pass in client_id and client_secret to this method "
+                "when authenticating for the first time."
+            )
+
+        if masterclient:
+            logger.debug("Reading cached master Spotify credentials.")
+            # Use cached client instead of authorizing again
+            # and thus wasting time.
+            self.__dict__.update(masterclient.__dict__)
+        else:
+            logger.debug("Setting master Spotify credentials.")
+            credential_manager = oauth2.SpotifyClientCredentials(
+                client_id=client_id,
+                client_secret=client_secret
+            )
+            super().__init__(client_credentials_manager=credential_manager)
+            # Cache current client
+            masterclient = self
+

spotdl/authorize/services/tests/test_spotify.py

@@ -0,0 +1,19 @@
+from spotdl.authorize.services import AuthorizeSpotify
+
+import pytest
+
+class TestSpotifyAuthorize:
+    # TODO: Test these once we a have config.py
+    #       storing pre-defined default credentials.
+    #
+    # We'll use these credentials to create
+    # a spotipy object via below tests
+
+    @pytest.mark.xfail
+    def test_generate_token(self):
+        raise NotImplementedError
+
+    @pytest.mark.xfail
+    def test_authorize(self):
+        raise NotImplementedError
+

spotdl/authorize/tests/test_authorize_base.py

@@ -0,0 +1,16 @@
+from spotdl.authorize import AuthorizeBase
+
+import pytest
+
+class TestAbstractBaseClass:
+    def test_error_abstract_base_class_authorizebase(self):
+        with pytest.raises(TypeError):
+            AuthorizeBase()
+
+    def test_inherit_abstract_base_class_authorizebase(self):
+        class AuthorizeKid(AuthorizeBase):
+            def authorize(self):
+                pass
+
+        AuthorizeKid()
+

spotdl/authorize/tests/test_authorize_exceptions.py

@@ -0,0 +1,15 @@
+from spotdl.authorize.exceptions import AuthorizationError
+from spotdl.authorize.exceptions import SpotifyAuthorizationError
+from spotdl.authorize.exceptions import YouTubeAuthorizationError
+
+
+class TestEncoderNotFoundSubclass:
+    def test_authozation_error_subclass(self):
+        assert issubclass(AuthorizationError, Exception)
+
+    def test_spotify_authorization_error_subclass(self):
+        assert issubclass(SpotifyAuthorizationError, AuthorizationError)
+
+    def test_youtube_authorization_error_subclass(self):
+        assert issubclass(YouTubeAuthorizationError, AuthorizationError)
+

spotdl/command_line/__main__.py

@@ -0,0 +1,57 @@
+import logging
+import coloredlogs
+
+import sys
+
+from spotdl.command_line.core import Spotdl
+from spotdl.command_line.arguments import get_arguments
+from spotdl.command_line.exceptions import ArgumentError
+
+# hardcode loglevel for dependencies so that they do not spew generic
+# log messages along with spotdl.
+for module in ("urllib3", "spotipy", "pytube"):
+    logging.getLogger(module).setLevel(logging.CRITICAL)
+
+coloredlogs.DEFAULT_FIELD_STYLES = {
+    "levelname": {"bold": True, "color": "yellow"},
+    "name": {"color": "blue"},
+    "lineno": {"color": "magenta"},
+}
+
+
+def set_logger(level):
+    if level == logging.DEBUG:
+        fmt = "%(levelname)s:%(name)s:%(lineno)d:\n%(message)s\n"
+    else:
+        fmt = "%(levelname)s: %(message)s"
+    logging.basicConfig(format=fmt, level=level)
+    logger = logging.getLogger(name=__name__)
+    coloredlogs.install(level=level, fmt=fmt, logger=logger)
+    return logger
+
+
+def main():
+    try:
+        argument_handler = get_arguments()
+    except ArgumentError as e:
+        logger = set_logger(logging.INFO)
+        logger.info(e.args[0])
+        sys.exit(5)
+
+    logging_level = argument_handler.get_logging_level()
+    logger = set_logger(logging_level)
+    try:
+        spotdl = Spotdl(argument_handler)
+    except ArgumentError as e:
+        argument_handler.parser.error(e.args[0])
+    try:
+        spotdl.match_arguments()
+    except KeyboardInterrupt as e:
+        print("", file=sys.stderr)
+        logger.exception(e)
+        sys.exit(2)
+
+
+if __name__ == "__main__":
+    main()
+

spotdl/command_line/arguments.py

@@ -0,0 +1,345 @@
+import appdirs
+
+import argparse
+import mimetypes
+import os
+import sys
+import shutil
+
+from spotdl.command_line.exceptions import ArgumentError
+import spotdl.util
+import spotdl.config
+
+from collections.abc import Sequence
+import logging
+logger = logging.getLogger(__name__)
+
+
+_LOG_LEVELS = {
+    "INFO": logging.INFO,
+    "WARNING": logging.WARNING,
+    "ERROR": logging.ERROR,
+    "DEBUG": logging.DEBUG,
+}
+
+if os.path.isfile(spotdl.config.DEFAULT_CONFIG_FILE):
+    saved_config = spotdl.config.read_config(spotdl.config.DEFAULT_CONFIG_FILE)
+else:
+    saved_config = {"spotify-downloader": {}}
+
+_CONFIG_BASE = spotdl.util.merge_copy(
+    spotdl.config.DEFAULT_CONFIGURATION,
+    saved_config,
+)
+
+
+def get_arguments(config_base=_CONFIG_BASE):
+    parser = argparse.ArgumentParser(
+        description="Download and convert tracks from Spotify, Youtube, etc.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    defaults = config_base["spotify-downloader"]
+
+    to_remove_config = "--remove-config" in sys.argv[1:]
+    if not to_remove_config and "download-only-metadata" in defaults:
+        raise ArgumentError(
+            "The default configuration file currently set is not suitable for spotdl>=2.0.0.\n"
+            "You need to remove your previous `config.yml` due to breaking changes\n"
+            "introduced in v2.0.0, new options being added, and old ones being removed\n"
+            "You may want to first backup your old configuration for reference. You can\n"
+            "then remove your current configuration by running:\n"
+            "```\n"
+            "$ spotdl --remove-config\n"
+            "```\n"
+            "spotdl will automatically generate a new configuration file on the next run.\n"
+            "You can then replace the appropriate fields in the newly generated\n"
+            "configuration file by referring to your old configuration file.\n\n"
+            "For the list of OTHER BREAKING CHANGES and release notes check out:\n"
+            "https://github.com/ritiek/spotify-downloader/releases/tag/v2.0.0"
+        )
+
+    # `--remove-config` does not require the any of the group arguments to be passed.
+    group = parser.add_mutually_exclusive_group(required=not to_remove_config)
+
+    group.add_argument(
+        "-s",
+        "--song",
+        nargs="+",
+        help="download track(s) by spotify link or name"
+    )
+    group.add_argument(
+        "-l",
+        "--list",
+        help="download tracks from a file (WARNING: this file will be modified!)"
+    )
+    group.add_argument(
+        "-p",
+        "--playlist",
+        help="load tracks from playlist URL into <playlist_name>.txt or "
+             "if `--write-to=<path/to/file.txt>` has been passed",
+    )
+    group.add_argument(
+        "-a",
+        "--album",
+        help="load tracks from album URL into <album_name>.txt or if "
+             "`--write-to=<path/to/file.txt>` has been passed"
+    )
+    group.add_argument(
+        "-aa",
+        "--all-albums",
+        help="load all tracks from artist URL into <artist_name>.txt "
+             "or if `--write-to=<path/to/file.txt>` has been passed"
+    )
+    group.add_argument(
+        "-u",
+        "--username",
+        help="load tracks from user's playlist into <playlist_name>.txt "
+             "or if `--write-to=<path/to/file.txt>` has been passed"
+    )
+
+    parser.add_argument(
+        "--write-m3u",
+        help="generate an .m3u playlist file with youtube links given "
+             "a text file containing tracks",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-m",
+        "--manual",
+        default=defaults["manual"],
+        help="choose the track to download manually from a list of matching tracks",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-nm",
+        "--no-metadata",
+        default=defaults["no_metadata"],
+        help="do not embed metadata in tracks",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-ne",
+        "--no-encode",
+        default=defaults["no_encode"],
+        action="store_true",
+        help="do not encode media using FFmpeg",
+    )
+    parser.add_argument(
+        "--overwrite",
+        default=defaults["overwrite"],
+        choices={"prompt", "force", "skip"},
+        help="change the overwrite policy",
+    )
+    parser.add_argument(
+        "-q",
+        "--quality",
+        default=defaults["quality"],
+        choices={"best", "worst"},
+        help="preferred audio quality",
+    )
+    parser.add_argument(
+        "-i",
+        "--input-ext",
+        default=defaults["input_ext"],
+        choices={"automatic", "m4a", "opus"},
+        help="preferred input format",
+    )
+    parser.add_argument(
+        "-o",
+        "--output-ext",
+        default=defaults["output_ext"],
+        choices={"mp3", "m4a", "flac"},
+        help="preferred output format",
+    )
+    parser.add_argument(
+        "--write-to",
+        default=defaults["write_to"],
+        help="write tracks from Spotify playlist, album, etc. to this file",
+    )
+    parser.add_argument(
+        "-f",
+        "--output-file",
+        default=defaults["output_file"],
+        help="path where to write the downloaded track to, special tags "
+        "are to be surrounded by curly braces. Possible tags: "
+        # TODO: Add possible tags
+        # "{}".format([spotdl.util.formats[x] for x in spotdl.util.formats]),
+    )
+    parser.add_argument(
+        "--trim-silence",
+        default=defaults["trim_silence"],
+        help="remove silence from the start of the audio",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-sf",
+        "--search-format",
+        default=defaults["search_format"],
+        help="search format to search for on YouTube, special tags "
+        "are to be surrounded by curly braces. Possible tags: "
+        # TODO: Add possible tags
+        # "{}".format([spotdl.util.formats[x] for x in spotdl.util.formats]),
+    )
+    parser.add_argument(
+        "-d",
+        "--dry-run",
+        default=defaults["dry_run"],
+        help="show only track title and YouTube URL, and then skip "
+        "to the next track (if any)",
+        action="store_true",
+    )
+    parser.add_argument(
+        "--processor",
+        default="synchronous",
+        choices={"synchronous", "threaded"},
+        # help='list downloading strategy: - "synchronous" downloads '
+        # 'tracks one-by-one. - "threaded" (highly experimental at the '
+        # 'moment! expect it to slash & burn) pre-fetches the next '
+        # 'track\'s metadata for more efficient downloading'
+        # XXX: Still very experimental to be exposed
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "-ns",
+        "--no-spaces",
+        default=defaults["no_spaces"],
+        help="replace spaces in metadata values with underscores when "
+        "generating filenames",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-sk",
+        "--skip-file",
+        default=defaults["skip_file"],
+        help="path to file containing tracks to skip",
+    )
+    parser.add_argument(
+        "-w",
+        "--write-successful-file",
+        default=defaults["write_successful_file"],
+        help="path to file to write successful tracks to",
+    )
+    parser.add_argument(
+        "--spotify-client-id",
+        default=defaults["spotify_client_id"],
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "--spotify-client-secret",
+        default=defaults["spotify_client_secret"],
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "-ll",
+        "--log-level",
+        default=defaults["log_level"],
+        choices=_LOG_LEVELS.keys(),
+        type=str.upper,
+        help="set log verbosity",
+    )
+    parser.add_argument(
+        "-c",
+        "--config",
+        default=spotdl.config.DEFAULT_CONFIG_FILE,
+        help="path to custom config.yml file"
+    )
+    parser.add_argument(
+        "--remove-config",
+        default=False,
+        action="store_true",
+        help="remove previously saved config"
+    )
+    parser.add_argument(
+        "-V",
+        "--version",
+        action="version",
+        version="%(prog)s {}".format(spotdl.__version__),
+    )
+
+    return ArgumentHandler(parser=parser)
+
+
+class ArgumentHandler:
+    def __init__(self, args=None, parser=argparse.ArgumentParser(""), config_base=_CONFIG_BASE):
+        args_were_passed = args is not None
+        if not args_were_passed:
+            args = parser.parse_args().__dict__
+
+        config_file = args.get("config")
+        configured_args = args.copy()
+        if config_file and os.path.isfile(config_file):
+            config = spotdl.config.read_config(config_file)
+            parser.set_defaults(**config["spotify-downloader"])
+            configured_args = parser.parse_args().__dict__
+
+        if args_were_passed:
+            parser.set_defaults(**args)
+            configured_args = parser.parse_args().__dict__
+
+        defaults = config_base["spotify-downloader"]
+        args = spotdl.util.merge_copy(defaults, args)
+
+        self.parser = parser
+        self.args = args
+        self.configured_args = configured_args
+
+    def get_configured_args(self):
+        return self.configured_args
+
+    def get_logging_level(self):
+        return _LOG_LEVELS[self.configured_args["log_level"]]
+
+    def run_errands(self):
+        args = self.get_configured_args()
+
+        if (args.get("list")
+            and not mimetypes.MimeTypes().guess_type(args["list"])[0] == "text/plain"
+        ):
+            raise ArgumentError(
+                "{0} is not of a valid argument to --list, argument must be plain text file.".format(
+                    args["list"]
+                )
+            )
+
+        if args.get("write_m3u") and not args.get("list"):
+            raise ArgumentError("--write-m3u can only be used with --list.")
+
+        if args["write_to"] and not (
+            args.get("playlist") or args.get("album") or args.get("all_albums") or args.get("username") or args.get("write_m3u")
+        ):
+            raise ArgumentError(
+                "--write-to can only be used with --playlist, --album, --all-albums, --username, or --write-m3u."
+            )
+
+        ffmpeg_exists = shutil.which("ffmpeg")
+        if not ffmpeg_exists:
+            logger.warn("FFmpeg was not found in PATH. Will not re-encode media to specified output format.")
+            args["no_encode"] = True
+
+        if args["no_encode"] and args["trim_silence"]:
+            logger.warn("--trim-silence can only be used when an encoder is set.")
+
+        if args["output_file"] == "-" and args["no_metadata"] is False:
+            logger.warn(
+                "Cannot write metadata when target is STDOUT. Pass "
+                "--no-metadata explicitly to hide this warning."
+            )
+            args["no_metadata"] = True
+        elif os.path.isdir(args["output_file"]):
+            adjusted_output_file = os.path.join(
+                args["output_file"],
+                self.parser.get_default("output_file")
+            )
+            logger.warn(
+                "Given output file is a directory. Will download tracks "
+                "in this directory with their filename as per the default "
+                "file format. Pass --output-file=\"{}\" to hide this "
+                "warning.".format(
+                    adjusted_output_file
+                )
+            )
+            args["output_file"] = adjusted_output_file
+
+        return args
+

spotdl/command_line/core.py

@@ -0,0 +1,441 @@
+from spotdl.metadata.providers import ProviderSpotify
+from spotdl.metadata.providers import ProviderYouTube
+from spotdl.metadata.providers import YouTubeSearch
+from spotdl.metadata.embedders import EmbedderDefault
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+import spotdl.metadata
+
+from spotdl.lyrics.providers import LyricWikia
+from spotdl.lyrics.providers import Genius
+from spotdl.lyrics.exceptions import LyricsNotFoundError
+
+from spotdl.encode.encoders import EncoderFFmpeg
+
+from spotdl.authorize.services import AuthorizeSpotify
+
+from spotdl.track import Track
+import spotdl.util
+import spotdl.config
+
+from spotdl.command_line.exceptions import NoYouTubeVideoFoundError
+from spotdl.command_line.exceptions import NoYouTubeVideoMatchError
+from spotdl.metadata_search import MetadataSearch
+
+from spotdl.helpers.spotify import SpotifyHelpers
+
+import sys
+import os
+import urllib.request
+
+import logging
+logger = logging.getLogger(__name__)
+
+
+class Spotdl:
+    def __init__(self, argument_handler):
+        self.arguments = argument_handler.run_errands()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, traceback):
+        del self
+
+    def match_arguments(self):
+        logger.debug("Received arguments:\n{}".format(self.arguments))
+
+        if self.arguments["remove_config"]:
+            self.remove_saved_config()
+            return
+        self.save_default_config()
+
+        AuthorizeSpotify(
+            client_id=self.arguments["spotify_client_id"],
+            client_secret=self.arguments["spotify_client_secret"]
+        )
+        spotify_tools = SpotifyHelpers()
+        if self.arguments["song"]:
+            for track in self.arguments["song"]:
+                if track == "-":
+                    for line in sys.stdin:
+                        self.download_track(
+                            line.strip(),
+                        )
+                else:
+                    self.download_track(track)
+        elif self.arguments["list"]:
+            if self.arguments["write_m3u"]:
+                self.write_m3u(
+                    self.arguments["list"],
+                    self.arguments["write_to"]
+                )
+            else:
+                list_download = {
+                    "synchronous": self.download_tracks_from_file,
+                    # "threaded"  : self.download_tracks_from_file_threaded,
+                }[self.arguments["processor"]]
+
+                list_download(
+                    self.arguments["list"],
+                )
+        elif self.arguments["playlist"]:
+            playlist = spotify_tools.fetch_playlist(self.arguments["playlist"])
+            spotify_tools.write_playlist_tracks(playlist, self.arguments["write_to"])
+        elif self.arguments["album"]:
+            album = spotify_tools.fetch_album(self.arguments["album"])
+            spotify_tools.write_album_tracks(album, self.arguments["write_to"])
+        elif self.arguments["all_albums"]:
+            albums = spotify_tools.fetch_albums_from_artist(self.arguments["all_albums"])
+            spotify_tools.write_all_albums(albums, self.arguments["write_to"])
+        elif self.arguments["username"]:
+            playlist_url = spotify_tools.prompt_for_user_playlist(self.arguments["username"])
+            playlist = spotify_tools.fetch_playlist(playlist_url)
+            spotify_tools.write_playlist_tracks(playlist, self.arguments["write_to"])
+
+    def save_config(self, config_file=spotdl.config.DEFAULT_CONFIG_FILE, config=spotdl.config.DEFAULT_CONFIGURATION):
+        config_dir = os.path.dirname(config_file)
+        os.makedirs(config_dir, exist_ok=True)
+        logger.info('Writing configuration to "{0}":'.format(config_file))
+        spotdl.config.dump_config(config_file=config_file, config=spotdl.config.DEFAULT_CONFIGURATION)
+        config = spotdl.config.dump_config(config=spotdl.config.DEFAULT_CONFIGURATION["spotify-downloader"])
+        for line in config.split("\n"):
+            if line.strip():
+                logger.info(line.strip())
+        logger.info(
+            "Please note that command line arguments have higher priority "
+            "than their equivalents in the configuration file.\n"
+        )
+
+    def save_default_config(self):
+        if not os.path.isfile(spotdl.config.DEFAULT_CONFIG_FILE):
+            self.save_config()
+
+    def remove_saved_config(self, config_file=spotdl.config.DEFAULT_CONFIG_FILE):
+        if os.path.isfile(spotdl.config.DEFAULT_CONFIG_FILE):
+            logger.info('Removing "{}".'.format(spotdl.config.DEFAULT_CONFIG_FILE))
+            os.remove(spotdl.config.DEFAULT_CONFIG_FILE)
+        else:
+            logger.info('File does not exist: "{}".'.format(spotdl.config.DEFAULT_CONFIG_FILE))
+
+    def write_m3u(self, track_file, target_file=None):
+        with open(track_file, "r") as fin:
+            tracks = fin.read().splitlines()
+
+        logger.info(
+            "Checking and removing any duplicate tracks in {}.".format(track_file)
+        )
+        # Remove duplicates and empty elements
+        # Also strip whitespaces from elements (if any)
+        tracks = spotdl.util.remove_duplicates(
+            tracks,
+            condition=lambda x: x,
+            operation=str.strip
+        )
+
+        if target_file is None:
+            target_file = "{}.m3u".format(track_file.split(".")[0])
+
+        total_tracks = len(tracks)
+        logger.info("Generating {0} from {1} YouTube URLs.".format(target_file, total_tracks))
+        write_to_stdout = target_file == "-"
+        m3u_headers = "#EXTM3U\n\n"
+        if write_to_stdout:
+            sys.stdout.write(m3u_headers)
+        else:
+            with open(target_file, "w") as output_file:
+                output_file.write(m3u_headers)
+
+        videos = []
+        for n, track in enumerate(tracks, 1):
+            try:
+                search_metadata = MetadataSearch(
+                    track,
+                    lyrics=not self.arguments["no_metadata"],
+                    yt_search_format=self.arguments["search_format"],
+                    yt_manual=self.arguments["manual"]
+                )
+                video = search_metadata.best_on_youtube_search()
+            except (NoYouTubeVideoFoundError, NoYouTubeVideoMatchError) as e:
+                logger.error(e.args[0])
+            else:
+                logger.info(
+                    "Matched track {0}/{1} ({2})".format(
+                        str(n).zfill(len(str(total_tracks))),
+                        total_tracks,
+                        video["url"],
+                    )
+                )
+                m3u_key = "#EXTINF:{duration},{title}\n{youtube_url}\n".format(
+                    duration=spotdl.util.get_sec(video["duration"]),
+                    title=video["title"],
+                    youtube_url=video["url"],
+                )
+                logger.debug(m3u_key.strip())
+                if write_to_stdout:
+                    sys.stdout.write(m3u_key)
+                else:
+                    with open(target_file, "a") as output_file:
+                        output_file.write(m3u_key)
+
+    def download_track(self, track):
+        logger.info('Downloading "{}"'.format(track))
+        search_metadata = MetadataSearch(
+            track,
+            lyrics=not self.arguments["no_metadata"],
+            yt_search_format=self.arguments["search_format"],
+            yt_manual=self.arguments["manual"]
+        )
+        try:
+            if self.arguments["no_metadata"]:
+                metadata = search_metadata.on_youtube()
+            else:
+                metadata = search_metadata.on_youtube_and_spotify()
+        except (NoYouTubeVideoFoundError, NoYouTubeVideoMatchError) as e:
+            logger.error(e.args[0])
+        else:
+            self.download_track_from_metadata(metadata)
+
+    def should_we_overwrite_existing_file(self, overwrite):
+        if overwrite == "force":
+            logger.info("Forcing overwrite on existing file.")
+            to_overwrite = True
+        elif overwrite == "prompt":
+            to_overwrite = input("Overwrite? (y/N): ").lower() == "y"
+        else:
+            logger.info("Not overwriting existing file.")
+            to_overwrite = False
+
+        return to_overwrite
+
+    def generate_temp_filename(self, filename, for_stdout=False):
+        if for_stdout:
+            return filename
+        return "{filename}.temp".format(filename=filename)
+
+    def output_filename_filter(self, allow_spaces):
+        replace_spaces_with_underscores = not allow_spaces
+        if replace_spaces_with_underscores:
+            return lambda s: s.replace(" ", "_")
+        return lambda s: s
+
+    def download_track_from_metadata(self, metadata):
+        track = Track(metadata, cache_albumart=(not self.arguments["no_metadata"]))
+        stream = metadata["streams"].get(
+            quality=self.arguments["quality"],
+            preftype=self.arguments["input_ext"],
+        )
+
+        if self.arguments["no_encode"]:
+            output_extension = stream["encoding"]
+        else:
+            output_extension = self.arguments["output_ext"]
+
+        filename = spotdl.metadata.format_string(
+            self.arguments["output_file"],
+            metadata,
+            output_extension=output_extension,
+            sanitizer=lambda s: spotdl.util.sanitize(
+                s, spaces_to_underscores=self.arguments["no_spaces"]
+            )
+        )
+        download_to_stdout = filename == "-"
+        temp_filename = self.generate_temp_filename(filename, for_stdout=download_to_stdout)
+
+        to_skip_download = self.arguments["dry_run"]
+        if os.path.isfile(filename):
+            logger.info('A file with name "{filename}" already exists.'.format(
+                filename=filename
+            ))
+            to_skip_download = to_skip_download \
+                or not self.should_we_overwrite_existing_file(self.arguments["overwrite"])
+
+        if to_skip_download:
+            logger.debug("Skip track download.")
+            return
+
+        if not self.arguments["no_metadata"]:
+            metadata["lyrics"].start()
+
+        filter_space_chars = self.output_filename_filter(not self.arguments["no_spaces"])
+        directory = os.path.dirname(
+            spotdl.metadata.format_string(
+                self.arguments["output_file"],
+                metadata,
+                output_extension=output_extension,
+                sanitizer=filter_space_chars
+            )
+        )
+        os.makedirs(directory or ".", exist_ok=True)
+
+        logger.info('Downloading to "{filename}"'.format(filename=filename))
+        if self.arguments["no_encode"]:
+            track.download(stream, temp_filename)
+        else:
+            encoder = EncoderFFmpeg()
+            if self.arguments["trim_silence"]:
+                encoder.set_trim_silence()
+            track.download_while_re_encoding(
+                stream,
+                temp_filename,
+                target_encoding=output_extension,
+                encoder=encoder,
+            )
+
+        if not self.arguments["no_metadata"]:
+            track.metadata["lyrics"] = track.metadata["lyrics"].join()
+            self.apply_metadata(track, temp_filename, output_extension)
+
+        if not download_to_stdout:
+            logger.debug("Renaming {temp_filename} to {filename}.".format(
+                temp_filename=temp_filename, filename=filename
+            ))
+            os.rename(temp_filename, filename)
+
+        return filename
+
+    def apply_metadata(self, track, filename, encoding):
+        logger.info("Applying metadata")
+        try:
+            track.apply_metadata(filename, encoding=encoding)
+        except TypeError:
+            logger.warning("Cannot apply metadata on provided output format.")
+
+    def strip_and_filter_duplicates(self, tracks):
+        filtered_tracks = spotdl.util.remove_duplicates(
+            tracks,
+            condition=lambda x: x,
+            operation=str.strip
+        )
+        return filtered_tracks
+
+    def filter_against_skip_file(self, items, skip_file):
+        skip_items = spotdl.util.readlines_from_nonbinary_file(skip_file)
+        filtered_skip_items = self.strip_and_filter_duplicates(skip_items)
+        filtered_items = [item for item in items if not item in filtered_skip_items]
+        return filtered_items
+
+    def download_tracks_from_file(self, path):
+        logger.info(
+            "Checking and removing any duplicate tracks in {}.".format(path)
+        )
+        tracks = spotdl.util.readlines_from_nonbinary_file(path)
+        tracks = self.strip_and_filter_duplicates(tracks)
+
+        if self.arguments["skip_file"]:
+            len_tracks_before = len(tracks)
+            tracks = self.filter_against_skip_file(tracks, self.arguments["skip_file"])
+            logger.info("Skipping {} tracks due to matches in skip file.".format(
+                len_tracks_before - len(tracks))
+            )
+        # Overwrite file
+        spotdl.util.writelines_to_nonbinary_file(path, tracks)
+
+        logger.info(
+            "Downloading {n} tracks.\n".format(n=len(tracks))
+        )
+
+        for position, track in enumerate(tracks, 1):
+            search_metadata = MetadataSearch(
+                track,
+                lyrics=True,
+                yt_search_format=self.arguments["search_format"],
+                yt_manual=self.arguments["manual"]
+            )
+            try:
+                log_track_query = '{position}. Downloading "{track}"'.format(
+                    position=position,
+                    track=track
+                )
+                logger.info(log_track_query)
+                metadata = search_metadata.on_youtube_and_spotify()
+                self.download_track_from_metadata(metadata)
+            except (urllib.request.URLError, TypeError, IOError) as e:
+                logger.exception(e.args[0])
+                logger.warning(
+                    "Failed to download current track due to possible network issue. "
+                    "Will retry after other songs."
+                )
+                tracks.append(track)
+            except (NoYouTubeVideoFoundError, NoYouTubeVideoMatchError) as e:
+                logger.error("{err}".format(err=e.args[0]))
+            else:
+                if self.arguments["write_successful_file"]:
+                    with open(self.arguments["write_successful_file"], "a") as fout:
+                        fout.write("{}\n".format(track))
+            finally:
+                spotdl.util.writelines_to_nonbinary_file(path, tracks[position:])
+                print("", file=sys.stderr)
+
+    """
+    def download_tracks_from_file_threaded(self, path):
+        # FIXME: Can we make this function cleaner?
+
+        logger.info(
+            "Checking and removing any duplicate tracks in {}.\n".format(path)
+        )
+        with open(path, "r") as fin:
+            # Read tracks into a list and remove any duplicates
+            tracks = fin.read().splitlines()
+
+        # Remove duplicates and empty elements
+        # Also strip whitespaces from elements (if any)
+        spotdl.util.remove_duplicates(
+            tracks,
+            condition=lambda x: x,
+            operation=str.strip
+        )
+
+        # Overwrite file
+        with open(path, "w") as fout:
+            fout.writelines(tracks)
+
+        tracks_count = len(tracks)
+        current_iteration = 1
+
+        next_track = tracks.pop(0)
+        metadata = {
+            "current_track": None,
+            "next_track": spotdl.util.ThreadWithReturnValue(
+                target=search_metadata,
+                args=(next_track, self.arguments["search_format"])
+            )
+        }
+        metadata["next_track"].start()
+        while tracks_count > 0:
+            metadata["current_track"] = metadata["next_track"].join()
+            metadata["next_track"] = None
+            try:
+                print(tracks_count, file=sys.stderr)
+                print(tracks, file=sys.stderr)
+                if tracks_count > 1:
+                    current_track = next_track
+                    next_track = tracks.pop(0)
+                    metadata["next_track"] = spotdl.util.ThreadWithReturnValue(
+                        target=search_metadata,
+                        args=(next_track, self.arguments["search_format"])
+                    )
+                    metadata["next_track"].start()
+
+                log_track_query = str(current_iteration) + ". {artist} - {track-name}"
+                logger.info(log_track_query)
+                if metadata["current_track"] is None:
+                    logger.warning("Something went wrong. Will retry after downloading remaining tracks.")
+                    pass
+                print(metadata["current_track"]["name"], file=sys.stderr)
+                # self.download_track_from_metadata(metadata["current_track"])
+            except (urllib.request.URLError, TypeError, IOError) as e:
+                print("", file=sys.stderr)
+                logger.exception(e.args[0])
+                logger.warning("Failed. Will retry after other songs\n")
+                tracks.append(current_track)
+            else:
+                tracks_count -= 1
+                if self.arguments["write_sucessful_file"]:
+                    with open(self.arguments["write_sucessful_file"], "a") as fout:
+                        fout.write(current_track)
+            finally:
+                current_iteration += 1
+                with open(path, "w") as fout:
+                    fout.writelines(tracks)
+    """
+

spotdl/command_line/exceptions.py

@@ -0,0 +1,20 @@
+class NoYouTubeVideoFoundError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class NoYouTubeVideoMatchError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class ArgumentError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+

spotdl/command_line/tests/test_arguments.py

@@ -0,0 +1,90 @@
+import spotdl.command_line.arguments
+from spotdl.command_line.exceptions import ArgumentError
+
+import logging
+import sys
+import pytest
+
+
+def test_logging_levels():
+    expect_logging_levels = {
+        "INFO": logging.INFO,
+        "WARNING": logging.WARNING,
+        "DEBUG": logging.DEBUG,
+        "ERROR": logging.ERROR,
+    }
+    assert spotdl.command_line.arguments._LOG_LEVELS == expect_logging_levels
+
+
+class TestBadArguments:
+    def test_error_m3u_without_list(self):
+        previous_argv = sys.argv
+        sys.argv[1:] = ["-s", "cool song", "--write-m3u"]
+        argument_handler = spotdl.command_line.arguments.get_arguments()
+        with pytest.raises(ArgumentError):
+            argument_handler.run_errands()
+        sys.argv[1:] = previous_argv[1:]
+
+    def test_write_to_error(self):
+        previous_argv = sys.argv
+        sys.argv[1:] = ["-s", "sekai all i had", "--write-to", "output.txt"]
+        argument_handler = spotdl.command_line.arguments.get_arguments()
+        with pytest.raises(ArgumentError):
+            argument_handler.run_errands()
+        sys.argv[1:] = previous_argv[1:]
+
+
+class TestArguments:
+    @pytest.mark.xfail
+    def test_general_arguments(self):
+        arguments = spotdl.command_line.arguments.get_arguments(argv=("-t", "elena coats - one last song"))
+        arguments = arguments.__dict__
+
+        assert isinstance(arguments["spotify_client_id"], str)
+        assert isinstance(arguments["spotify_client_secret"], str)
+
+        arguments["spotify_client_id"] = None
+        arguments["spotify_client_secret"] = None
+
+        expect_arguments = {
+            "song": ["elena coats - one last song"],
+            "song": None,
+            "list": None,
+            "playlist": None,
+            "album": None,
+            "all_albums": None,
+            "username": None,
+            "write_m3u": False,
+            "manual": False,
+            "no_remove_original": False,
+            "no_metadata": False,
+            "no_fallback_metadata": False,
+            "directory": "/home/ritiek/Music",
+            "overwrite": "prompt",
+            "input_ext": ".m4a",
+            "output_ext": ".mp3",
+            "write_to": None,
+            "file_format": "{artist} - {track_name}",
+            "trim_silence": False,
+            "search_format": "{artist} - {track_name} lyrics",
+            "download_only_metadata": False,
+            "dry_run": False,
+            "music_videos_only": False,
+            "no_spaces": False,
+            "log_level": 20,
+            "skip": None,
+            "write_successful": None,
+            "spotify_client_id": None,
+            "spotify_client_secret": None,
+            "config": None
+        }
+
+        assert arguments == expect_arguments
+
+    def test_grouped_arguments(self):
+        previous_argv = sys.argv
+        sys.argv[1:] = []
+        with pytest.raises(SystemExit):
+            argument_handler = spotdl.command_line.arguments.get_arguments()
+        sys.argv[1:] = previous_argv[1:]
+

spotdl/config.py

@@ -0,0 +1,52 @@
+import appdirs
+import yaml
+import os
+
+import spotdl.util
+import logging
+logger = logging.getLogger(__name__)
+
+DEFAULT_CONFIGURATION = {
+    "spotify-downloader": {
+        "manual": False,
+        "no_metadata": False,
+        "no_encode": False,
+        "overwrite": "prompt",
+        "quality": "best",
+        "input_ext": "automatic",
+        "output_ext": "mp3",
+        "write_to": None,
+        "trim_silence": False,
+        "search_format": "{artist} - {track-name} lyrics",
+        "dry_run": False,
+        "no_spaces": False,
+        # "processor": "synchronous",
+        "output_file": "{artist} - {track-name}.{output-ext}",
+        "skip_file": None,
+        "write_successful_file": None,
+        "spotify_client_id": "4fe3fecfe5334023a1472516cc99d805",
+        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c",
+        "log_level": "INFO",
+    }
+}
+
+DEFAULT_CONFIG_FILE = os.path.join(
+    appdirs.user_config_dir(),
+    "spotdl",
+    "config.yml"
+)
+
+def read_config(config_file):
+    with open(config_file, "r") as ymlfile:
+        config = yaml.safe_load(ymlfile)
+    return config
+
+
+def dump_config(config_file=None, config=DEFAULT_CONFIGURATION):
+    if config_file is None:
+        config = yaml.dump(config, default_flow_style=False)
+        return config
+
+    with open(config_file, "w") as ymlfile:
+        yaml.dump(config, ymlfile, default_flow_style=False)
+

spotdl/const.py

@@ -1,42 +0,0 @@
-import logzero
-
-_log_format = "%(color)s%(levelname)s:%(end_color)s %(message)s"
-_formatter = logzero.LogFormatter(fmt=_log_format)
-_log_level = 0
-
-# Set up a temporary logger with default log level so that
-# it can be used before log level argument is determined
-logzero.setup_default_logger(formatter=_formatter, level=_log_level)
-
-# Options
-# Initialize an empty object which can be assigned attributes
-# (useful when using spotdl as a library)
-args = type("", (), {})()
-
-# Apple has specific tags - see mutagen docs -
-# http://mutagen.readthedocs.io/en/latest/api/mp4.html
-M4A_TAG_PRESET = {
-    "album": "\xa9alb",
-    "artist": "\xa9ART",
-    "date": "\xa9day",
-    "title": "\xa9nam",
-    "year": "\xa9day",
-    "originaldate": "purd",
-    "comment": "\xa9cmt",
-    "group": "\xa9grp",
-    "writer": "\xa9wrt",
-    "genre": "\xa9gen",
-    "tracknumber": "trkn",
-    "albumartist": "aART",
-    "discnumber": "disk",
-    "cpil": "cpil",
-    "albumart": "covr",
-    "copyright": "cprt",
-    "tempo": "tmpo",
-    "lyrics": "\xa9lyr",
-    "comment": "\xa9cmt",
-}
-
-TAG_PRESET = {}
-for key in M4A_TAG_PRESET.keys():
-    TAG_PRESET[key] = key

spotdl/convert.py

@@ -1,164 +0,0 @@
-import subprocess
-import os
-from logzero import logger as log
-
-
-"""
-What are the differences and similarities between ffmpeg, libav, and avconv?
-https://stackoverflow.com/questions/9477115
-
-ffmeg encoders high to lower quality
-libopus > libvorbis >= libfdk_aac > aac > libmp3lame
-
-libfdk_aac due to copyrights needs to be compiled by end user
-on MacOS brew install ffmpeg --with-fdk-aac will do just that. Other OS?
-https://trac.ffmpeg.org/wiki/Encode/AAC
-"""
-
-
-def song(
-    input_song,
-    output_song,
-    folder,
-    avconv=False,
-    trim_silence=False,
-    delete_original=True,
-):
-    """ Do the audio format conversion. """
-    if avconv and trim_silence:
-        raise ValueError("avconv does not support trim_silence")
-
-    if not input_song == output_song:
-        log.info("Converting {0} to {1}".format(input_song, output_song.split(".")[-1]))
-    elif input_song.endswith(".m4a"):
-        log.info('Correcting container in "{}"'.format(input_song))
-    else:
-        return 0
-
-    convert = Converter(
-        input_song, output_song, folder, delete_original=delete_original
-    )
-    if avconv:
-        exit_code, command = convert.with_avconv()
-    else:
-        exit_code, command = convert.with_ffmpeg(trim_silence=trim_silence)
-    return exit_code, command
-
-
-class Converter:
-    def __init__(self, input_song, output_song, folder, delete_original):
-        _, self.input_ext = os.path.splitext(input_song)
-        _, self.output_ext = os.path.splitext(output_song)
-
-        self.output_file = os.path.join(folder, output_song)
-        rename_to_temp = False
-
-        same_file = os.path.abspath(input_song) == os.path.abspath(output_song)
-        if same_file:
-            # FFmpeg/avconv cannot have the same file for both input and output
-            # This would happen when the extensions are same, so rename
-            # the input track to append ".temp"
-            log.debug(
-                'Input file and output file are going will be same during encoding, will append ".temp" to input file just before starting encoding to avoid conflict'
-            )
-            input_song = output_song + ".temp"
-            rename_to_temp = True
-            delete_original = True
-
-        self.input_file = os.path.join(folder, input_song)
-
-        self.rename_to_temp = rename_to_temp
-        self.delete_original = delete_original
-
-    def with_avconv(self):
-        if log.level == 10:
-            level = "debug"
-        else:
-            level = "0"
-
-        command = [
-            "avconv",
-            "-loglevel",
-            level,
-            "-i",
-            self.input_file,
-            "-ab",
-            "192k",
-            self.output_file,
-            "-y",
-        ]
-
-        if self.rename_to_temp:
-            os.rename(self.output_file, self.input_file)
-
-        log.debug(command)
-        try:
-            code = subprocess.call(command)
-        except FileNotFoundError:
-            if self.rename_to_temp:
-                os.rename(self.input_file, self.output_file)
-            raise
-
-        if self.delete_original:
-            log.debug('Removing original file: "{}"'.format(self.input_file))
-            os.remove(self.input_file)
-
-        return code, command
-
-    def with_ffmpeg(self, trim_silence=False):
-        ffmpeg_pre = (
-            "ffmpeg -y -nostdin "
-        )  # -nostdin is necessary for spotdl to be able to run in the backgroung.
-
-        if not log.level == 10:
-            ffmpeg_pre += "-hide_banner -nostats -v panic "
-
-        ffmpeg_params = ""
-
-        if self.input_ext == ".m4a":
-            if self.output_ext == ".mp3":
-                ffmpeg_params = "-codec:v copy -codec:a libmp3lame -ar 48000 "
-            elif self.output_ext == ".webm":
-                ffmpeg_params = "-codec:a libopus -vbr on "
-            elif self.output_ext == ".m4a":
-                ffmpeg_params = "-acodec copy "
-
-        elif self.input_ext == ".webm":
-            if self.output_ext == ".mp3":
-                ffmpeg_params = "-codec:a libmp3lame -ar 48000 "
-            elif self.output_ext == ".m4a":
-                ffmpeg_params = "-cutoff 20000 -codec:a aac -ar 48000 "
-
-        if self.output_ext == ".flac":
-            ffmpeg_params = "-codec:a flac -ar 48000 "
-
-        # add common params for any of the above combination
-        ffmpeg_params += "-b:a 192k -vn "
-        ffmpeg_pre += "-i "
-
-        if trim_silence:
-            ffmpeg_params += "-af silenceremove=start_periods=1 "
-
-        command = (
-            ffmpeg_pre.split()
-            + [self.input_file]
-            + ffmpeg_params.split()
-            + [self.output_file]
-        )
-
-        if self.rename_to_temp:
-            os.rename(self.output_file, self.input_file)
-
-        log.debug(command)
-        try:
-            code = subprocess.call(command)
-        except FileNotFoundError:
-            if self.rename_to_temp:
-                os.rename(self.input_file, self.output_file)
-            raise
-
-        if self.delete_original:
-            log.debug('Removing original file: "{}"'.format(self.input_file))
-            os.remove(self.input_file)
-
-        return code, command

spotdl/downloader.py

@@ -1,256 +0,0 @@
-import spotipy
-import urllib
-import os
-import time
-from logzero import logger as log
-
-from spotdl import const
-from spotdl import metadata
-from spotdl import convert
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-
-
-class CheckExists:
-    def __init__(self, music_file, meta_tags=None):
-        self.meta_tags = meta_tags
-        basepath, filename = os.path.split(music_file)
-        filepath = os.path.join(const.args.folder, basepath)
-        os.makedirs(filepath, exist_ok=True)
-        self.filepath = filepath
-        self.filename = filename
-
-    def already_exists(self, raw_song):
-        """ Check if the input song already exists in the given folder. """
-        log.debug(
-            "Cleaning any temp files and checking "
-            'if "{}" already exists'.format(self.filename)
-        )
-        songs = os.listdir(self.filepath)
-        self._remove_temp_files(songs)
-
-        for song in songs:
-            # check if a song with the same name is already present in the given folder
-            if self._match_filenames(song):
-                if internals.is_spotify(raw_song) and not self._has_metadata(song):
-                    return False
-
-                log.warning('"{}" already exists'.format(song))
-                if const.args.overwrite == "prompt":
-                    return self._prompt_song(song)
-                elif const.args.overwrite == "force":
-                    return self._force_overwrite_song(song)
-                elif const.args.overwrite == "skip":
-                    return self._skip_song(song)
-
-        return False
-
-    def _remove_temp_files(self, songs):
-        for song in songs:
-            if song.endswith(".temp"):
-                os.remove(os.path.join(self.filepath, song))
-
-    def _has_metadata(self, song):
-        # check if the already downloaded song has correct metadata
-        # if not, remove it and download again without prompt
-        already_tagged = metadata.compare(
-            os.path.join(self.filepath, song), self.meta_tags
-        )
-        log.debug("Checking if it is already tagged correctly? {}", already_tagged)
-        if not already_tagged:
-            os.remove(os.path.join(self.filepath, song))
-            return False
-
-        return True
-
-    def _prompt_song(self, song):
-        log.info(
-            '"{}" has already been downloaded. ' "Re-download? (y/N): ".format(song)
-        )
-        prompt = input("> ")
-        if prompt.lower() == "y":
-            return self._force_overwrite_song(song)
-        else:
-            return self._skip_song(song)
-
-    def _force_overwrite_song(self, song):
-        os.remove(os.path.join(const.args.folder, song))
-        log.info('Overwriting "{}"'.format(song))
-        return False
-
-    def _skip_song(self, song):
-        log.info('Skipping "{}"'.format(song))
-        return True
-
-    def _match_filenames(self, song):
-        if os.path.splitext(song)[0] == self.filename:
-            log.debug('Found an already existing song: "{}"'.format(song))
-            return True
-
-        return False
-
-
-class Downloader:
-    def __init__(self, raw_song, number=None):
-        self.raw_song = raw_song
-        self.number = number
-        self.content, self.meta_tags = youtube_tools.match_video_and_metadata(raw_song)
-        self.total_songs = int(self.meta_tags["total_tracks"])
-
-    def download_single(self):
-        """ Logic behind downloading a song. """
-
-        if self._to_skip():
-            return
-
-        # "[number]. [artist] - [song]" if downloading from list
-        # otherwise "[artist] - [song]"
-        youtube_title = youtube_tools.get_youtube_title(self.content, self.number)
-        log.info("{} ({})".format(youtube_title, self.content.watchv_url))
-
-        # generate file name of the song to download
-        songname = self.refine_songname(self.content.title)
-
-        if const.args.dry_run:
-            return
-
-        song_existence = CheckExists(songname, self.meta_tags)
-        if not song_existence.already_exists(self.raw_song):
-            return self._download_single(songname)
-
-    def _download_single(self, songname):
-        # deal with file formats containing slashes to non-existent directories
-        songpath = os.path.join(const.args.folder, os.path.dirname(songname))
-        os.makedirs(songpath, exist_ok=True)
-        input_song = songname + const.args.input_ext
-        output_song = songname + const.args.output_ext
-        if youtube_tools.download_song(input_song, self.content):
-            print("")
-            try:
-                convert.song(
-                    input_song,
-                    output_song,
-                    const.args.folder,
-                    avconv=const.args.avconv,
-                    trim_silence=const.args.trim_silence,
-                    delete_original=not const.args.no_remove_original,
-                )
-            except FileNotFoundError:
-                encoder = "avconv" if const.args.avconv else "ffmpeg"
-                log.warning("Could not find {0}, skip encoding".format(encoder))
-                output_song = self.unconverted_filename(songname)
-
-            if not const.args.no_metadata and self.meta_tags is not None:
-                metadata.embed(
-                    os.path.join(const.args.folder, output_song), self.meta_tags
-                )
-            return True
-
-    def _to_skip(self):
-        if self.content is None:
-            log.debug("Found no matching video")
-            return True
-
-        if const.args.download_only_metadata and self.meta_tags is None:
-            log.info("Found no metadata. Skipping the download")
-            return True
-
-    def refine_songname(self, songname):
-        if self.meta_tags is not None:
-            refined_songname = internals.format_string(
-                const.args.file_format,
-                self.meta_tags,
-                slugification=True,
-                total_songs=self.total_songs,
-            )
-            log.debug(
-                'Refining songname from "{0}" to "{1}"'.format(
-                    songname, refined_songname
-                )
-            )
-            if not refined_songname == " - ":
-                songname = refined_songname
-        else:
-            songname = internals.sanitize_title(songname)
-
-        return songname
-
-    @staticmethod
-    def unconverted_filename(songname):
-        const.args.output_ext = const.args.input_ext
-        output_song = songname + const.args.output_ext
-        return output_song
-
-
-class ListDownloader:
-    def __init__(self, tracks_file, skip_file=None, write_successful_file=None):
-        self.tracks_file = tracks_file
-        self.skip_file = skip_file
-        self.write_successful_file = write_successful_file
-        self.tracks = internals.get_unique_tracks(self.tracks_file)
-
-    def download_list(self):
-        """ Download all songs from the list. """
-        # override file with unique tracks
-        log.info("Overriding {} with unique tracks".format(self.tracks_file))
-        self._override_file()
-
-        # Remove tracks to skip from tracks list
-        if self.skip_file is not None:
-            self.tracks = self._filter_tracks_against_skip_file()
-
-        log.info(u"Preparing to download {} songs".format(len(self.tracks)))
-        return self._download_list()
-
-    def _download_list(self):
-        downloaded_songs = []
-
-        for number, raw_song in enumerate(self.tracks, 1):
-            print("")
-            try:
-                track_dl = Downloader(raw_song, number=number)
-                track_dl.download_single()
-            except (urllib.request.URLError, TypeError, IOError) as e:
-                # detect network problems
-                self._cleanup(raw_song, e)
-                # TODO: remove this sleep once #397 is fixed
-                # wait 0.5 sec to avoid infinite looping
-                time.sleep(0.5)
-                continue
-
-            downloaded_songs.append(raw_song)
-            # Add track to file of successful downloads
-            if self.write_successful_file is not None:
-                self._write_successful(raw_song)
-
-            log.debug("Removing downloaded song from tracks file")
-            internals.trim_song(self.tracks_file)
-
-        return downloaded_songs
-
-    def _override_file(self):
-        with open(self.tracks_file, "w") as f:
-            f.write("\n".join(self.tracks))
-
-    def _write_successful(self, raw_song):
-        log.debug("Adding downloaded song to write successful file")
-        with open(self.write_successful_file, "a") as f:
-            f.write("\n" + raw_song)
-
-    def _cleanup(self, raw_song, exception):
-        self.tracks.append(raw_song)
-        # remove the downloaded song from file
-        internals.trim_song(self.tracks_file)
-        # and append it at the end of file
-        with open(self.tracks_file, "a") as f:
-            f.write("\n" + raw_song)
-        log.exception(exception)
-        log.warning("Failed to download song. Will retry after other songs\n")
-
-    def _filter_tracks_against_skip_file(self):
-        skip_tracks = internals.get_unique_tracks(self.skip_file)
-        len_before = len(self.tracks)
-        tracks = [track for track in self.tracks if track not in skip_tracks]
-        log.info("Skipping {} tracks".format(len_before - len(tracks)))
-        return tracks

spotdl/encode/__init__.py

@@ -0,0 +1 @@
+from spotdl.encode.encode_base import EncoderBase

spotdl/encode/encode_base.py

@@ -0,0 +1,103 @@
+import shutil
+import os
+
+from abc import ABC
+from abc import abstractmethod
+
+from spotdl.encode.exceptions import EncoderNotFoundError
+
+"""
+  NOTE ON ENCODERS
+  ================
+
+* FFmeg encoders sorted in descending order based
+  on the quality of audio produced:
+  libopus > libvorbis >= libfdk_aac > aac > libmp3lame
+
+* libfdk_aac encoder, due to copyrights needs to be compiled
+  by end user on MacOS brew install ffmpeg --with-fdk-aac
+  will do just that. Other OS? See:
+  https://trac.ffmpeg.org/wiki/Encode/AAC
+
+"""
+
+
+class EncoderBase(ABC):
+    """
+    Defined encoders must inherit from this abstract base class
+    and implement their own functionality for the below defined
+    methods.
+    """
+
+    @abstractmethod
+    def __init__(self, encoder_path, loglevel, additional_arguments=[]):
+        """
+        This method must make sure whether specified encoder
+        is available under PATH.
+        """
+        if shutil.which(encoder_path) is None:
+            raise EncoderNotFoundError(
+                "{} executable does not exist or was not found in PATH.".format(
+                    encoder_path
+                )
+            )
+        self.encoder_path = encoder_path
+        self._loglevel = loglevel
+        self._additional_arguments = additional_arguments
+
+    def set_argument(self, argument):
+        """
+        This method must be used to set any custom functionality
+        for the encoder by passing arguments to it.
+        """
+        self._additional_arguments += argument.split()
+
+    def get_encoding(self, path):
+        """
+        This method must determine the encoding for a local
+        audio file. Such as "mp3", "wav", "m4a", etc.
+        """
+        _, extension = os.path.splitext(path)
+        # Ignore the initial dot from file extension
+        return extension[1:]
+
+    @abstractmethod
+    def set_debuglog(self):
+        """
+        This method must enable verbose logging in the defined
+        encoder.
+        """
+        pass
+
+    @abstractmethod
+    def _generate_encode_command(self, input_path, target_path):
+        """
+        This method must the complete command for that would be
+        used to invoke the encoder and perform the encoding.
+        """
+        pass
+
+    @abstractmethod
+    def _generate_encoding_arguments(self, input_encoding, target_encoding):
+        """
+        This method must return the core arguments for the defined
+        encoder such as defining the sample rate, audio bitrate,
+        etc.
+        """
+        pass
+
+    @abstractmethod
+    def re_encode(self, input_path, target_path):
+        """
+        This method must invoke the encoder to encode a given input
+        file to a specified output file.
+        """
+        pass
+
+    def re_encode_from_stdin(self, input_encoding, target_path):
+        """
+        This method must invoke the encoder to encode stdin to a
+        specified output file.
+        """
+        raise NotImplementedError
+

spotdl/encode/encoders/__init__.py

@@ -0,0 +1 @@
+from spotdl.encode.encoders.ffmpeg import EncoderFFmpeg

spotdl/encode/encoders/ffmpeg.py

@@ -0,0 +1,112 @@
+import subprocess
+import os
+
+from spotdl.encode import EncoderBase
+from spotdl.encode.exceptions import EncoderNotFoundError
+from spotdl.encode.exceptions import FFmpegNotFoundError
+
+import logging
+logger = logging.getLogger(__name__)
+
+
+# Key: from format
+# Subkey: to format
+RULES = {
+    "m4a": {
+        "mp3": "-codec:v copy -codec:a libmp3lame -ar 48000",
+        "opus": "-codec:a libopus -vbr on",
+        "m4a": "-acodec copy",
+        "flac": "-codec:a flac -ar 48000",
+    },
+    "opus": {
+        "mp3": "-codec:a libmp3lame -ar 48000",
+        "m4a": "-cutoff 20000 -codec:a aac -ar 48000",
+        "flac": "-codec:a flac -ar 48000",
+    },
+}
+
+
+class EncoderFFmpeg(EncoderBase):
+    def __init__(self, encoder_path="ffmpeg"):
+        _loglevel = "-hide_banner -nostats -v panic"
+        _additional_arguments = ["-b:a", "192k", "-vn"]
+        try:
+            super().__init__(encoder_path, _loglevel, _additional_arguments)
+        except EncoderNotFoundError as e:
+            raise FFmpegNotFoundError(e.args[0])
+        self._rules = RULES
+
+    def set_trim_silence(self):
+        self.set_argument("-af silenceremove=start_periods=1")
+
+    def get_encoding(self, path):
+        return super().get_encoding(path)
+
+    def _generate_encoding_arguments(self, input_encoding, target_encoding):
+        initial_arguments = self._rules.get(input_encoding)
+        if initial_arguments is None:
+            raise TypeError(
+                'The input format ("{}") is not supported.'.format(
+                input_encoding,
+            ))
+        arguments = initial_arguments.get(target_encoding)
+        if arguments is None:
+            raise TypeError(
+                'The output format ("{}") is not supported.'.format(
+                target_encoding,
+            ))
+        return arguments
+
+    def set_debuglog(self):
+        self._loglevel = "-loglevel debug"
+
+    def _generate_encode_command(self, input_path, target_file,
+                                 input_encoding=None, target_encoding=None):
+        if input_encoding is None:
+            input_encoding = self.get_encoding(input_path)
+        if target_encoding is None:
+            target_encoding = self.get_encoding(target_file)
+        arguments = self._generate_encoding_arguments(
+            input_encoding,
+            target_encoding
+        )
+        command = [self.encoder_path] \
+            + ["-y", "-nostdin"] \
+            + self._loglevel.split() \
+            + ["-i", input_path] \
+            + arguments.split() \
+            + self._additional_arguments \
+            + ["-f", target_encoding] \
+            + [target_file]
+
+        return command
+
+    def re_encode(self, input_path, target_file, target_encoding=None, delete_original=False):
+        encode_command = self._generate_encode_command(
+            input_path,
+            target_file,
+            target_encoding=target_encoding
+        )
+        logger.debug("Calling FFmpeg with:\n{command}".format(
+            command=encode_command,
+        ))
+        process = subprocess.Popen(encode_command)
+        process.wait()
+        encode_successful = process.returncode == 0
+        if encode_successful and delete_original:
+            os.remove(input_path)
+        return process
+
+    def re_encode_from_stdin(self, input_encoding, target_file, target_encoding=None):
+        encode_command = self._generate_encode_command(
+            "-",
+            target_file,
+            input_encoding=input_encoding,
+            target_encoding=target_encoding,
+        )
+        logger.debug("Calling FFmpeg with:\n{command}".format(
+            command=encode_command,
+        ))
+        process = subprocess.Popen(encode_command, stdin=subprocess.PIPE)
+        return process
+

spotdl/encode/encoders/tests/test_ffmpeg.py

@@ -0,0 +1,211 @@
+from spotdl.encode import EncoderBase
+from spotdl.encode.exceptions import FFmpegNotFoundError
+from spotdl.encode.encoders import EncoderFFmpeg
+
+import pytest
+
+class TestEncoderFFmpeg:
+    def test_subclass(self):
+        assert issubclass(EncoderFFmpeg, EncoderBase)
+
+    def test_ffmpeg_not_found_error(self):
+        with pytest.raises(FFmpegNotFoundError):
+            EncoderFFmpeg(encoder_path="/a/nonexistent/path")
+
+
+class TestEncodingDefaults:
+    def m4a_to_mp3_encoder(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:v', 'copy',
+            '-codec:a', 'libmp3lame',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'mp3',
+            target_path
+        ]
+        return command
+
+    def m4a_to_opus_encoder(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:a', 'libopus',
+            '-vbr', 'on',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'opus',
+            target_path
+        ]
+        return command
+
+    def m4a_to_m4a_encoder(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-acodec', 'copy',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'm4a',
+             target_path
+        ]
+        return command
+
+    def m4a_to_flac_encoder(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:a', 'flac',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'flac',
+            target_path
+        ]
+        return command
+
+    @pytest.mark.parametrize("files, expected_command", [
+        (("test.m4a", "test.mp3"), m4a_to_mp3_encoder("test.m4a", "test.mp3")),
+        (("abc.m4a", "cba.opus"), m4a_to_opus_encoder("abc.m4a", "cba.opus")),
+        (("bla bla.m4a", "ble ble.m4a"), m4a_to_m4a_encoder("bla bla.m4a", "ble ble.m4a")),
+        (("😛.m4a", "• tongue.flac"), m4a_to_flac_encoder("😛.m4a", "• tongue.flac")),
+    ])
+    def test_generate_encode_command(self, files, expected_command):
+        encoder = EncoderFFmpeg()
+        assert encoder._generate_encode_command(*files) == expected_command
+
+
+class TestEncodingInDebugMode:
+    def m4a_to_mp3_encoder_with_debug(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-loglevel', 'debug',
+            '-i', input_path,
+            '-codec:v', 'copy',
+            '-codec:a', 'libmp3lame',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'mp3',
+            target_path
+        ]
+        return command
+
+    def m4a_to_opus_encoder_with_debug(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-loglevel', 'debug',
+            '-i', input_path,
+            '-codec:a', 'libopus',
+            '-vbr', 'on',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'opus',
+             target_path
+        ]
+        return command
+
+    def m4a_to_m4a_encoder_with_debug(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-loglevel', 'debug',
+            '-i', input_path,
+            '-acodec', 'copy',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'm4a',
+             target_path
+        ]
+        return command
+
+    def m4a_to_flac_encoder_with_debug(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-loglevel', 'debug',
+            '-i', input_path,
+            '-codec:a', 'flac',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-f', 'flac',
+             target_path
+        ]
+        return command
+
+    @pytest.mark.parametrize("files, expected_command", [
+        (("test.m4a", "test.mp3"), m4a_to_mp3_encoder_with_debug("test.m4a", "test.mp3")),
+        (("abc.m4a", "cba.opus"), m4a_to_opus_encoder_with_debug("abc.m4a", "cba.opus")),
+        (("bla bla.m4a", "ble ble.m4a"), m4a_to_m4a_encoder_with_debug("bla bla.m4a", "ble ble.m4a")),
+        (("😛.m4a", "• tongue.flac"), m4a_to_flac_encoder_with_debug("😛.m4a", "• tongue.flac")),
+    ])
+    def test_generate_encode_command_with_debug(self, files, expected_command):
+        encoder = EncoderFFmpeg()
+        encoder.set_debuglog()
+        assert encoder._generate_encode_command(*files) == expected_command
+
+
+class TestEncodingAndTrimSilence:
+    def m4a_to_mp3_encoder_and_trim_silence(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:v', 'copy',
+            '-codec:a', 'libmp3lame',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-af', 'silenceremove=start_periods=1',
+            '-f', 'mp3',
+            target_path
+        ]
+        return command
+
+    def m4a_to_opus_encoder_and_trim_silence(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:a', 'libopus',
+            '-vbr', 'on',
+            '-b:a', '192k',
+            '-vn',
+            '-af', 'silenceremove=start_periods=1',
+            '-f', 'opus',
+            target_path
+        ]
+        return command
+
+    def m4a_to_m4a_encoder_and_trim_silence(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-acodec', 'copy',
+            '-b:a', '192k',
+            '-vn',
+            '-af', 'silenceremove=start_periods=1',
+            '-f', 'm4a',
+            target_path
+        ]
+        return command
+
+    def m4a_to_flac_encoder_and_trim_silence(input_path, target_path):
+        command = [
+            'ffmpeg', '-y', '-nostdin', '-hide_banner', '-nostats', '-v', 'panic',
+            '-i', input_path,
+            '-codec:a', 'flac',
+            '-ar', '48000',
+            '-b:a', '192k',
+            '-vn',
+            '-af', 'silenceremove=start_periods=1',
+            '-f', 'flac',
+            target_path
+        ]
+        return command
+
+    @pytest.mark.parametrize("files, expected_command", [
+        (("test.m4a", "test.mp3"), m4a_to_mp3_encoder_and_trim_silence("test.m4a", "test.mp3")),
+        (("abc.m4a", "cba.opus"), m4a_to_opus_encoder_and_trim_silence("abc.m4a", "cba.opus")),
+        (("bla bla.m4a", "ble ble.m4a"), m4a_to_m4a_encoder_and_trim_silence("bla bla.m4a", "ble ble.m4a")),
+        (("😛.m4a", "• tongue.flac"), m4a_to_flac_encoder_and_trim_silence("😛.m4a", "• tongue.flac")),
+    ])
+    def test_generate_encode_command_and_trim_silence(self, files, expected_command):
+        encoder = EncoderFFmpeg()
+        encoder.set_trim_silence()
+        assert encoder._generate_encode_command(*files) == expected_command

spotdl/encode/exceptions.py

@@ -0,0 +1,13 @@
+class EncoderNotFoundError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class FFmpegNotFoundError(EncoderNotFoundError):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+

spotdl/encode/tests/test_encode_base.py

@@ -0,0 +1,88 @@
+from spotdl.encode import EncoderBase
+from spotdl.encode.exceptions import EncoderNotFoundError
+
+import pytest
+
+class TestAbstractBaseClass:
+    def test_error_abstract_base_class_encoderbase(self):
+        encoder_path = "ffmpeg"
+        _loglevel = "-hide_banner -nostats -v panic"
+        _additional_arguments = ["-b:a", "192k", "-vn"]
+
+        with pytest.raises(TypeError):
+            # This abstract base class must be inherited from
+            # for instantiation
+            EncoderBase(encoder_path, _loglevel, _additional_arguments)
+
+
+    def test_inherit_abstract_base_class_encoderbase(self):
+        class EncoderKid(EncoderBase):
+            def __init__(self, encoder_path, _loglevel, _additional_arguments):
+                super().__init__(encoder_path, _loglevel, _additional_arguments)
+
+            def _generate_encode_command(self):
+                pass
+
+            def _generate_encoding_arguments(self):
+                pass
+
+            def re_encode(self):
+                pass
+
+            def set_debuglog(self):
+                pass
+
+
+        encoder_path = "ffmpeg"
+        _loglevel = "-hide_banner -nostats -v panic"
+        _additional_arguments = ["-b:a", "192k", "-vn"]
+
+        EncoderKid(encoder_path, _loglevel, _additional_arguments)
+
+
+class TestMethods:
+    class EncoderKid(EncoderBase):
+        def __init__(self, encoder_path, _loglevel, _additional_arguments):
+            super().__init__(encoder_path, _loglevel, _additional_arguments)
+
+        def _generate_encode_command(self, input_file, target_file):
+            pass
+
+        def _generate_encoding_arguments(self, input_encoding, target_encoding):
+            pass
+
+        def re_encode(self, input_encoding, target_encoding):
+            pass
+
+        def set_debuglog(self):
+            pass
+
+
+    @pytest.fixture(scope="module")
+    def encoderkid(self):
+        encoder_path = "ffmpeg"
+        _loglevel = "-hide_banner -nostats -v panic"
+        _additional_arguments = []
+
+        encoderkid = self.EncoderKid(encoder_path, _loglevel, _additional_arguments)
+        return encoderkid
+
+    def test_set_argument(self, encoderkid):
+        encoderkid.set_argument("-parameter argument")
+        assert encoderkid._additional_arguments == [
+            "-parameter",
+            "argument",
+        ]
+
+    @pytest.mark.parametrize("filename, encoding", [
+        ("example.m4a", "m4a"),
+        ("exampley.mp3", "mp3"),
+        ("test 123.opus", "opus"),
+        ("flakey.flac", "flac"),
+    ])
+    def test_get_encoding(self, encoderkid, filename, encoding):
+        assert encoderkid.get_encoding(filename) == encoding
+
+    def test_encoder_not_found_error(self):
+        with pytest.raises(EncoderNotFoundError):
+            self.EncoderKid("/a/nonexistent/path", "0", [])

spotdl/encode/tests/test_encode_exceptions.py

@@ -0,0 +1,11 @@
+from spotdl.encode.exceptions import EncoderNotFoundError
+from spotdl.encode.exceptions import FFmpegNotFoundError
+
+
+class TestEncoderNotFoundSubclass:
+    def test_encoder_not_found_subclass(self):
+        assert issubclass(FFmpegNotFoundError, Exception)
+
+    def test_ffmpeg_not_found_subclass(self):
+        assert issubclass(FFmpegNotFoundError, EncoderNotFoundError)
+

spotdl/handle.py

@@ -1,332 +0,0 @@
-from logzero import logger as log
-import appdirs
-
-import logging
-import yaml
-import argparse
-import mimetypes
-import os
-
-import spotdl
-from spotdl import internals
-
-
-_LOG_LEVELS_STR = ["INFO", "WARNING", "ERROR", "DEBUG"]
-
-default_conf = {
-    "spotify-downloader": {
-        "no-remove-original": False,
-        "manual": False,
-        "no-metadata": False,
-        "no-fallback-metadata": False,
-        "avconv": False,
-        "folder": internals.get_music_dir(),
-        "overwrite": "prompt",
-        "input-ext": ".m4a",
-        "output-ext": ".mp3",
-        "write-to": None,
-        "trim-silence": False,
-        "download-only-metadata": False,
-        "dry-run": False,
-        "music-videos-only": False,
-        "no-spaces": False,
-        "file-format": "{artist} - {track_name}",
-        "search-format": "{artist} - {track_name} lyrics",
-        "youtube-api-key": None,
-        "skip": None,
-        "write-successful": None,
-        "log-level": "INFO",
-        "spotify_client_id": "4fe3fecfe5334023a1472516cc99d805",
-        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c",
-    }
-}
-
-
-def log_leveller(log_level_str):
-    loggin_levels = [logging.INFO, logging.WARNING, logging.ERROR, logging.DEBUG]
-    log_level_str_index = _LOG_LEVELS_STR.index(log_level_str)
-    loggin_level = loggin_levels[log_level_str_index]
-    return loggin_level
-
-
-def merge(default, config):
-    """ Override default dict with config dict. """
-    merged = default.copy()
-    merged.update(config)
-    return merged
-
-
-def get_config(config_file):
-    try:
-        with open(config_file, "r") as ymlfile:
-            cfg = yaml.safe_load(ymlfile)
-    except FileNotFoundError:
-        log.info("Writing default configuration to {0}:".format(config_file))
-        with open(config_file, "w") as ymlfile:
-            yaml.dump(default_conf, ymlfile, default_flow_style=False)
-            cfg = default_conf
-
-        for line in yaml.dump(
-            default_conf["spotify-downloader"], default_flow_style=False
-        ).split("\n"):
-            if line.strip():
-                log.info(line.strip())
-        log.info(
-            "Please note that command line arguments have higher priority "
-            "than their equivalents in the configuration file"
-        )
-
-    return cfg["spotify-downloader"]
-
-
-def override_config(config_file, parser, raw_args=None):
-    """ Override default dict with config dict passed as comamnd line argument. """
-    config_file = os.path.realpath(config_file)
-    config = merge(default_conf["spotify-downloader"], get_config(config_file))
-    parser.set_defaults(**config)
-    return parser.parse_args(raw_args)
-
-
-def get_arguments(raw_args=None, to_group=True, to_merge=True):
-    parser = argparse.ArgumentParser(
-        description="Download and convert tracks from Spotify, Youtube etc.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-    )
-
-    if to_merge:
-        config_dir = os.path.join(appdirs.user_config_dir(), "spotdl")
-        os.makedirs(config_dir, exist_ok=True)
-        config_file = os.path.join(config_dir, "config.yml")
-        config = merge(default_conf["spotify-downloader"], get_config(config_file))
-    else:
-        config = default_conf["spotify-downloader"]
-
-    if to_group:
-        group = parser.add_mutually_exclusive_group(required=True)
-
-        group.add_argument(
-            "-s", "--song", nargs="+", help="download track by spotify link or name"
-        )
-        group.add_argument("-l", "--list", help="download tracks from a file")
-        group.add_argument(
-            "-p",
-            "--playlist",
-            help="load tracks from playlist URL into <playlist_name>.txt",
-        )
-        group.add_argument(
-            "-b", "--album", help="load tracks from album URL into <album_name>.txt"
-        )
-        group.add_argument(
-            "-ab",
-            "--all-albums",
-            help="load all tracks from artist URL into <artist_name>.txt",
-        )
-        group.add_argument(
-            "-u",
-            "--username",
-            help="load tracks from user's playlist into <playlist_name>.txt",
-        )
-
-    parser.add_argument(
-        "--write-m3u",
-        help="generate an .m3u playlist file with youtube links given "
-        "a text file containing tracks",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-m",
-        "--manual",
-        default=config["manual"],
-        help="choose the track to download manually from a list of matching tracks",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-nr",
-        "--no-remove-original",
-        default=config["no-remove-original"],
-        help="do not remove the original file after conversion",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-nm",
-        "--no-metadata",
-        default=config["no-metadata"],
-        help="do not embed metadata in tracks",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-nf",
-        "--no-fallback-metadata",
-        default=config["no-fallback-metadata"],
-        help="do not use YouTube as fallback for metadata if track not found on Spotify",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-a",
-        "--avconv",
-        default=config["avconv"],
-        help="use avconv for conversion (otherwise defaults to ffmpeg)",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-f",
-        "--folder",
-        default=os.path.abspath(config["folder"]),
-        help="path to folder where downloaded tracks will be stored in",
-    )
-    parser.add_argument(
-        "--overwrite",
-        default=config["overwrite"],
-        help="change the overwrite policy",
-        choices={"prompt", "force", "skip"},
-    )
-    parser.add_argument(
-        "-i",
-        "--input-ext",
-        default=config["input-ext"],
-        help="preferred input format .m4a or .webm (Opus)",
-        choices={".m4a", ".webm"},
-    )
-    parser.add_argument(
-        "-o",
-        "--output-ext",
-        default=config["output-ext"],
-        help="preferred output format .mp3, .m4a (AAC), .flac, etc.",
-    )
-    parser.add_argument(
-        "--write-to",
-        default=config["write-to"],
-        help="write tracks from Spotify playlist, album, etc. to this file",
-    )
-    parser.add_argument(
-        "-ff",
-        "--file-format",
-        default=config["file-format"],
-        help="file format to save the downloaded track with, each tag "
-        "is surrounded by curly braces. Possible formats: "
-        "{}".format([internals.formats[x] for x in internals.formats]),
-    )
-    parser.add_argument(
-        "--trim-silence",
-        default=config["trim-silence"],
-        help="remove silence from the start of the audio",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-sf",
-        "--search-format",
-        default=config["search-format"],
-        help="search format to search for on YouTube, each tag "
-        "is surrounded by curly braces. Possible formats: "
-        "{}".format([internals.formats[x] for x in internals.formats]),
-    )
-    parser.add_argument(
-        "-dm",
-        "--download-only-metadata",
-        default=config["download-only-metadata"],
-        help="download tracks only whose metadata is found",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-d",
-        "--dry-run",
-        default=config["dry-run"],
-        help="show only track title and YouTube URL, and then skip "
-        "to the next track (if any)",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-mo",
-        "--music-videos-only",
-        default=config["music-videos-only"],
-        help="search only for music videos on Youtube (works only "
-        "when YouTube API key is set",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-ns",
-        "--no-spaces",
-        default=config["no-spaces"],
-        help="replace spaces with underscores in file names",
-        action="store_true",
-    )
-    parser.add_argument(
-        "-ll",
-        "--log-level",
-        default=config["log-level"],
-        choices=_LOG_LEVELS_STR,
-        type=str.upper,
-        help="set log verbosity",
-    )
-    parser.add_argument(
-        "-yk",
-        "--youtube-api-key",
-        default=config["youtube-api-key"],
-        help=argparse.SUPPRESS,
-    )
-    parser.add_argument(
-        "-sk",
-        "--skip",
-        default=config["skip"],
-        help="path to file containing tracks to skip",
-    )
-    parser.add_argument(
-        "-w",
-        "--write-successful",
-        default=config["write-successful"],
-        help="path to file to write successful tracks to",
-    )
-    parser.add_argument(
-        "-sci",
-        "--spotify-client-id",
-        default=config["spotify_client_id"],
-        help=argparse.SUPPRESS,
-    )
-    parser.add_argument(
-        "-scs",
-        "--spotify-client-secret",
-        default=config["spotify_client_secret"],
-        help=argparse.SUPPRESS,
-    )
-    parser.add_argument(
-        "-c", "--config", default=None, help="path to custom config.yml file"
-    )
-    parser.add_argument(
-        "-V",
-        "--version",
-        action="version",
-        version="%(prog)s {}".format(spotdl.__version__),
-    )
-
-    parsed = parser.parse_args(raw_args)
-
-    if parsed.config is not None and to_merge:
-        parsed = override_config(parsed.config, parser)
-
-    if (
-        to_group
-        and parsed.list
-        and not mimetypes.MimeTypes().guess_type(parsed.list)[0] == "text/plain"
-    ):
-        parser.error(
-            "{0} is not of a valid argument to --list, argument must be plain text file".format(
-                parsed.list
-            )
-        )
-
-    if parsed.write_m3u and not parsed.list:
-        parser.error("--write-m3u can only be used with --list")
-
-    if parsed.avconv and parsed.trim_silence:
-        parser.error("--trim-silence can only be used with FFmpeg")
-
-    if parsed.write_to and not (
-        parsed.playlist or parsed.album or parsed.all_albums or parsed.username
-    ):
-        parser.error(
-            "--write-to can only be used with --playlist, --album, --all-albums, or --username"
-        )
-
-    parsed.log_level = log_leveller(parsed.log_level)
-
-    return parsed

spotdl/helpers/spotify.py

@@ -0,0 +1,165 @@
+from spotdl.authorize.services import AuthorizeSpotify
+import spotdl.util
+
+import sys
+
+import logging
+logger = logging.getLogger(__name__)
+
+try:
+    from slugify import SLUG_OK, slugify
+except ImportError:
+    logger.error("Oops! `unicode-slugify` was not found.")
+    logger.info("Please remove any other slugify library and install `unicode-slugify`.")
+    raise
+
+
+class SpotifyHelpers:
+    def __init__(self, spotify=None):
+        if spotify is None:
+            spotify = AuthorizeSpotify()
+        self.spotify = spotify
+
+    def prompt_for_user_playlist(self, username):
+        """ Write user playlists to target_file """
+        playlists = self.fetch_user_playlist_urls(username)
+        for i, playlist in enumerate(playlists, 1):
+            playlist_details = "{0}. {1:<30}  ({2} tracks)".format(
+                i, playlist["name"], playlist["tracks"]["total"]
+            )
+            print(playlist_details, file=sys.stderr)
+        print("", file=sys.stderr)
+        playlist = spotdl.util.prompt_user_for_selection(playlists)
+        return playlist["external_urls"]["spotify"]
+
+    def fetch_user_playlist_urls(self, username):
+        """ Fetch user playlists when using the -u option. """
+        logger.debug('Fetching playlists for "{username}".'.format(username=username))
+        playlists = self.spotify.user_playlists(username)
+        collected_playlists = []
+        check = 1
+
+        while True:
+            for playlist in playlists["items"]:
+                # in rare cases, playlists may not be found, so playlists['next']
+                # is None. Skip these. Also see Issue #91.
+                if playlist["name"] is not None:
+                    collected_playlists.append(playlist)
+                    check += 1
+            if playlists["next"]:
+                playlists = self.spotify.next(playlists)
+            else:
+                break
+
+        return collected_playlists
+
+    def fetch_playlist(self, playlist_url):
+        logger.debug('Fetching playlist "{playlist}".'.format(playlist=playlist_url))
+        try:
+            results = self.spotify.playlist(playlist_url, fields="tracks,next,name")
+        except spotipy.client.SpotifyException:
+            logger.exception(
+                "Unable to find playlist. Make sure the playlist is set "
+                "to publicly visible and then try again."
+            )
+
+        return results
+
+    def write_playlist_tracks(self, playlist, target_file=None):
+        tracks = playlist["tracks"]
+        if not target_file:
+            target_file = u"{0}.txt".format(slugify(playlist["name"], ok="-_()[]{}"))
+        return self.write_tracks(tracks, target_file)
+
+    def fetch_album(self, album_uri):
+        logger.debug('Fetching album "{album}".'.format(album=album_uri))
+        album = self.spotify.album(album_uri)
+        return album
+
+    def write_album_tracks(self, album, target_file=None):
+        tracks = self.spotify.album_tracks(album["id"])
+        if not target_file:
+            target_file = u"{0}.txt".format(slugify(album["name"], ok="-_()[]{}"))
+        return self.write_tracks(tracks, target_file)
+
+    def fetch_albums_from_artist(self, artist_uri, album_type=None):
+        """
+        This function returns all the albums from a give artist_uri using the US
+        market
+        :param artist_uri - spotify artist uri
+        :param album_type - the type of album to fetch (ex: single) the default is
+                            all albums
+        :param return - the album from the artist
+        """
+
+        logger.debug('Fetching all albums for "{artist}".'.format(artist=artist_uri))
+        # fetching artist's albums limitting the results to the US to avoid duplicate
+        # albums from multiple markets
+        results = self.spotify.artist_albums(artist_uri, album_type=album_type, country="US")
+
+        albums = results["items"]
+
+        # indexing all pages of results
+        while results["next"]:
+            results = self.spotify.next(results)
+            albums.extend(results["items"])
+
+        return albums
+
+    def write_all_albums(self, albums, target_file=None):
+        """
+        This function gets all albums from an artist and writes it to a file in the
+        current working directory called [ARTIST].txt, where [ARTIST] is the artist
+        of the album
+        :param artist_uri - spotify artist uri
+        :param target_file - file to write albums to
+        """
+
+        # if no file if given, the default save file is in the current working
+        # directory with the name of the artist
+        if target_file is None:
+            target_file = albums[0]["artists"][0]["name"] + ".txt"
+
+        for album in albums:
+            logger.info('Fetching album "{album}".'.format(album=album["name"]))
+            self.write_album_tracks(album, target_file=target_file)
+
+    def write_tracks(self, tracks, target_file):
+        def writer(tracks, file_io):
+            track_urls = []
+            while True:
+                for item in tracks["items"]:
+                    if "track" in item:
+                        track = item["track"]
+                    else:
+                        track = item
+                    try:
+                        track_url = track["external_urls"]["spotify"]
+                        file_io.write(track_url + "\n")
+                        track_urls.append(track_url)
+                    except KeyError:
+                        # FIXME: Write "{artist} - {name}" instead of Spotify URI for
+                        #        "local only" tracks.
+                        logger.warning(
+                            'Skipping track "{0}" by "{1}" (local only?)'.format(
+                                track["name"], track["artists"][0]["name"]
+                            )
+                        )
+                # 1 page = 50 results
+                # check if there are more pages
+                if tracks["next"]:
+                    tracks = self.spotify.next(tracks)
+                else:
+                    break
+            return track_urls
+
+        logger.info(u"Writing {0} tracks to {1}.".format(tracks["total"], target_file))
+        write_to_stdout = target_file == "-"
+        if write_to_stdout:
+            file_out = sys.stdout
+            track_urls = writer(tracks, file_out)
+        else:
+            with open(target_file, "a") as file_out:
+                track_urls = writer(tracks, file_out)
+        return track_urls
+

spotdl/internals.py

@@ -1,280 +0,0 @@
-from logzero import logger as log
-import os
-import sys
-import math
-import urllib.request
-
-
-from spotdl import const
-
-try:
-    import winreg
-except ImportError:
-    pass
-
-try:
-    from slugify import SLUG_OK, slugify
-except ImportError:
-    log.error("Oops! `unicode-slugify` was not found.")
-    log.info("Please remove any other slugify library and install `unicode-slugify`")
-    sys.exit(5)
-
-formats = {
-    0: "track_name",
-    1: "artist",
-    2: "album",
-    3: "album_artist",
-    4: "genre",
-    5: "disc_number",
-    6: "duration",
-    7: "year",
-    8: "original_date",
-    9: "track_number",
-    10: "total_tracks",
-    11: "isrc",
-    12: "track_id",
-}
-
-
-def input_link(links):
-    """ Let the user input a choice. """
-    while True:
-        try:
-            log.info("Choose your number:")
-            the_chosen_one = int(input("> "))
-            if 1 <= the_chosen_one <= len(links):
-                return links[the_chosen_one - 1]
-            elif the_chosen_one == 0:
-                return None
-            else:
-                log.warning("Choose a valid number!")
-        except ValueError:
-            log.warning("Choose a valid number!")
-
-
-def trim_song(tracks_file):
-    """ Remove the first song from file. """
-    with open(tracks_file, "r") as file_in:
-        data = file_in.read().splitlines(True)
-    with open(tracks_file, "w") as file_out:
-        file_out.writelines(data[1:])
-    return data[0]
-
-
-def is_spotify(raw_song):
-    """ Check if the input song is a Spotify link. """
-    status = len(raw_song) == 22 and raw_song.replace(" ", "%20") == raw_song
-    status = status or raw_song.find("spotify") > -1
-    return status
-
-
-def is_youtube(raw_song):
-    """ Check if the input song is a YouTube link. """
-    status = len(raw_song) == 11 and raw_song.replace(" ", "%20") == raw_song
-    status = status and not raw_song.lower() == raw_song
-    status = status or "youtube.com/watch?v=" in raw_song
-    return status
-
-
-def format_string(
-    string_format, tags, slugification=False, force_spaces=False, total_songs=0
-):
-    """ Generate a string of the format '[artist] - [song]' for the given spotify song. """
-    format_tags = dict(formats)
-    format_tags[0] = tags["name"]
-    format_tags[1] = tags["artists"][0]["name"]
-    format_tags[2] = tags["album"]["name"]
-    format_tags[3] = tags["artists"][0]["name"]
-    format_tags[4] = tags["genre"]
-    format_tags[5] = tags["disc_number"]
-    format_tags[6] = tags["duration"]
-    format_tags[7] = tags["year"]
-    format_tags[8] = tags["release_date"]
-    format_tags[9] = tags["track_number"]
-    format_tags[10] = tags["total_tracks"]
-    format_tags[11] = tags["external_ids"]["isrc"]
-    try:
-        format_tags[12] = tags["id"]
-    except KeyError:
-        pass
-
-    format_tags_sanitized = {
-        k: sanitize_title(str(v), ok="'-_()[]{}") if slugification else str(v)
-        for k, v in format_tags.items()
-    }
-    # calculating total digits presnet in total_songs to prepare a zfill.
-    total_digits = 0 if total_songs == 0 else int(math.log10(total_songs)) + 1
-
-    for x in formats:
-        format_tag = "{" + formats[x] + "}"
-        # Making consistent track number by prepending zero
-        # on it according to number of digits in total songs
-        if format_tag == "{track_number}":
-            format_tags_sanitized[x] = format_tags_sanitized[x].zfill(total_digits)
-
-        string_format = string_format.replace(format_tag, format_tags_sanitized[x])
-
-    if const.args.no_spaces and not force_spaces:
-        string_format = string_format.replace(" ", "_")
-
-    return string_format
-
-
-def sanitize_title(title, ok="-_()[]{}"):
-    """ Generate filename of the song to be downloaded. """
-
-    if const.args.no_spaces:
-        title = title.replace(" ", "_")
-
-    # replace slashes with "-" to avoid folder creation errors
-    title = title.replace("/", "-").replace("\\", "-")
-
-    # slugify removes any special characters
-    title = slugify(title, ok=ok, lower=False, spaces=True)
-    return title
-
-
-def filter_path(path):
-    if not os.path.exists(path):
-        os.makedirs(path)
-    for temp in os.listdir(path):
-        if temp.endswith(".temp"):
-            os.remove(os.path.join(path, temp))
-
-
-def videotime_from_seconds(time):
-    if time < 60:
-        return str(time)
-    if time < 3600:
-        return "{0}:{1:02}".format(time // 60, time % 60)
-
-    return "{0}:{1:02}:{2:02}".format((time // 60) // 60, (time // 60) % 60, time % 60)
-
-
-def get_sec(time_str):
-    if ":" in time_str:
-        splitter = ":"
-    elif "." in time_str:
-        splitter = "."
-    else:
-        raise ValueError(
-            "No expected character found in {} to split" "time values.".format(time_str)
-        )
-    v = time_str.split(splitter, 3)
-    v.reverse()
-    sec = 0
-    if len(v) > 0:  # seconds
-        sec += int(v[0])
-    if len(v) > 1:  # minutes
-        sec += int(v[1]) * 60
-    if len(v) > 2:  # hours
-        sec += int(v[2]) * 3600
-    return sec
-
-
-def extract_spotify_id(raw_string):
-    """
-    Returns a Spotify ID of a playlist, album, etc. after extracting
-    it from a given HTTP URL or Spotify URI.
-    """
-
-    if "/" in raw_string:
-        # Input string is an HTTP URL
-        if raw_string.endswith("/"):
-            raw_string = raw_string[:-1]
-        # We need to manually trim additional text from HTTP URLs
-        # We could skip this if https://github.com/plamere/spotipy/pull/324
-        # gets merged,
-        to_trim = raw_string.find("?")
-        if not to_trim == -1:
-            raw_string = raw_string[:to_trim]
-        splits = raw_string.split("/")
-    else:
-        # Input string is a Spotify URI
-        splits = raw_string.split(":")
-
-    spotify_id = splits[-1]
-
-    return spotify_id
-
-
-def get_unique_tracks(tracks_file):
-    """
-    Returns a list of unique tracks given a path to a
-    file containing tracks.
-    """
-
-    log.info(
-        "Checking and removing any duplicate tracks "
-        "in reading {}".format(tracks_file)
-    )
-    with open(tracks_file, "r") as tracks_in:
-        # Read tracks into a list and remove any duplicates
-        lines = tracks_in.read().splitlines()
-
-    # Remove blank and strip whitespaces from lines (if any)
-    lines = [line.strip() for line in lines if line.strip()]
-    lines = remove_duplicates(lines)
-    return lines
-
-
-# a hacky way to get user's localized music directory
-# (thanks @linusg, issue #203)
-def get_music_dir():
-    home = os.path.expanduser("~")
-
-    # On Linux, the localized folder names are the actual ones.
-    # It's a freedesktop standard though.
-    if sys.platform.startswith("linux"):
-        for file_item in (".config/user-dirs.dirs", "user-dirs.dirs"):
-            path = os.path.join(home, file_item)
-            if os.path.isfile(path):
-                with open(path, "r") as f:
-                    for line in f:
-                        if line.startswith("XDG_MUSIC_DIR"):
-                            return os.path.expandvars(
-                                line.strip().split("=")[1].strip('"')
-                            )
-
-    # Windows / Cygwin
-    # Queries registry for 'My Music' folder path (as this can be changed)
-    if "win" in sys.platform:
-        try:
-            key = winreg.OpenKey(
-                winreg.HKEY_CURRENT_USER,
-                r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders",
-                0,
-                winreg.KEY_ALL_ACCESS,
-            )
-            return winreg.QueryValueEx(key, "My Music")[0]
-        except (FileNotFoundError, NameError):
-            pass
-
-    # On both Windows and macOS, the localized folder names you see in
-    # Explorer and Finder are actually in English on the file system.
-    # So, defaulting to C:\Users\<user>\Music or /Users/<user>/Music
-    # respectively is sufficient.
-    # On Linux, default to /home/<user>/Music if the above method failed.
-    return os.path.join(home, "Music")
-
-
-def remove_duplicates(tracks):
-    """
-    Removes duplicates from a list whilst preserving order.
-
-    We could directly call `set()` on the list but it changes
-    the order of elements.
-    """
-
-    local_set = set()
-    local_set_add = local_set.add
-    return [x for x in tracks if not (x in local_set or local_set_add(x))]
-
-
-def content_available(url):
-    try:
-        response = urllib.request.urlopen(url)
-    except urllib.request.HTTPError:
-        return False
-    else:
-        return response.getcode() < 300

spotdl/lyrics/exceptions.py

@@ -1,5 +1,5 @@
-class LyricsNotFound(Exception):
+class LyricsNotFoundError(Exception):
     __module__ = Exception.__module__
 
     def __init__(self, message=None):
-        super(LyricsNotFound, self).__init__(message)
+        super().__init__(message)

spotdl/lyrics/lyric_base.py

@@ -5,10 +5,32 @@ from abc import abstractmethod
 
 
 class LyricBase(ABC):
+    """
+    Defined lyric providers must inherit from this abstract base
+    class and implement their own functionality for the below
+    defined methods.
+    """
+
     @abstractmethod
-    def __init__(self, artist, song):
+    def from_url(self, url, linesep="\n", timeout=None):
+        """
+        This method must return the lyrics string for the
+        given track.
+        """
         pass
 
     @abstractmethod
-    def get_lyrics(self, linesep="\n", timeout=None):
+    def from_artist_and_track(self, artist, track, linesep="\n", timeout=None):
+        """
+        This method must return the lyrics string for the
+        given track.
+        """
+        pass
+
+    @abstractmethod
+    def from_query(self, query, linesep="\n", timeout=None):
+        """
+        This method must return the lyrics string for the
+        given track.
+        """
         pass

spotdl/lyrics/providers/genius.py

@@ -1,49 +1,133 @@
 from bs4 import BeautifulSoup
 import urllib.request
+import json
 
 from spotdl.lyrics.lyric_base import LyricBase
-from spotdl.lyrics.exceptions import LyricsNotFound
+from spotdl.lyrics.exceptions import LyricsNotFoundError
+
+import logging
+logger = logging.getLogger(__name__)
 
 BASE_URL = "https://genius.com"
+BASE_SEARCH_URL = BASE_URL + "/api/search/multi?per_page=1&q="
 
+# FIXME: Make Genius a metadata provider instead of lyric provider
+#        Since, Genius parses additional metadata too (such as track
+#        name, artist name, albumart url). For example, fetch this URL:
+#        https://genius.com/api/search/multi?per_page=1&q=artist+trackname
 
 class Genius(LyricBase):
-    def __init__(self, artist, song):
-        self.artist = artist
-        self.song = song
+    def __init__(self):
         self.base_url = BASE_URL
+        self.base_search_url = BASE_SEARCH_URL
 
-    def _guess_lyric_url(self):
-        query = "/{} {} lyrics".format(self.artist, self.song)
+    def guess_lyric_url_from_artist_and_track(self, artist, track):
+        """
+        Returns the possible lyric URL for the track available on
+        Genius. This may not always be a valid URL.
+        """
+        query = "/{} {} lyrics".format(artist, track)
         query = query.replace(" ", "-")
         encoded_query = urllib.request.quote(query)
         lyric_url = self.base_url + encoded_query
         return lyric_url
 
-    def _fetch_page(self, url, timeout=None):
+    def _fetch_url_page(self, url, timeout=None):
+        """
+        Makes a GET request to the given lyrics page URL and returns
+        the HTML content in the case of a valid response.
+        """
         request = urllib.request.Request(url)
         request.add_header("User-Agent", "urllib")
         try:
             response = urllib.request.urlopen(request, timeout=timeout)
         except urllib.request.HTTPError:
-            raise LyricsNotFound(
-                "Could not find lyrics for {} - {} at URL: {}".format(
-                    self.artist, self.song, url
-                )
+            raise LyricsNotFoundError(
+                "Could not find Genius lyrics at URL: {}".format(url)
             )
         else:
             return response.read()
 
     def _get_lyrics_text(self, html):
+        """
+        Extracts and returns the lyric content from the provided HTML.
+        """
         soup = BeautifulSoup(html, "html.parser")
         lyrics_paragraph = soup.find("p")
         if lyrics_paragraph:
             return lyrics_paragraph.get_text()
         else:
-            raise LyricsNotFound("The lyrics for this track are yet to be released.")
+            raise LyricsNotFoundError(
+                "The lyrics for this track are yet to be released on Genius."
+            )
 
-    def get_lyrics(self, linesep="\n", timeout=None):
-        url = self._guess_lyric_url()
-        html_page = self._fetch_page(url, timeout=timeout)
-        lyrics = self._get_lyrics_text(html_page)
+    def _fetch_search_page(self, url, timeout=None):
+        """
+        Returns search results from a given URL in JSON.
+        """
+        request = urllib.request.Request(url)
+        request.add_header("User-Agent", "urllib")
+        response = urllib.request.urlopen(request, timeout=timeout)
+        metadata = json.loads(response.read())
+        if len(metadata["response"]["sections"][0]["hits"]) == 0:
+            raise LyricsNotFoundError(
+                "Genius returned no lyric results for the search URL: {}".format(url)
+            )
+        return metadata
+
+    def best_matching_lyric_url_from_query(self, query):
+        """
+        Returns the best matching track's URL from a given query.
+        """
+        encoded_query = urllib.request.quote(query.replace(" ", "+"))
+        search_url = self.base_search_url + encoded_query
+        metadata = self._fetch_search_page(search_url)
+
+        lyric_url = None
+        for section in metadata["response"]["sections"]:
+            result = section["hits"][0]["result"]
+            try:
+                lyric_url = result["path"]
+                break
+            except KeyError:
+                pass
+
+        if lyric_url is None:
+            raise LyricsNotFoundError(
+                "Could not find any valid lyric paths in Genius "
+                "lyrics API response for the query {}.".format(query)
+            )
+
+        return self.base_url + lyric_url
+
+    def from_query(self, query, linesep="\n", timeout=None):
+        """
+        Returns the lyric string for the track best matching the
+        given query.
+        """
+        try:
+            lyric_url = self.best_matching_lyric_url_from_query(query)
+        except LyricsNotFoundError:
+            raise LyricsNotFoundError(
+                'Genius returned no lyric results for the search query "{}".'.format(query)
+            )
+        else:
+            return self.from_url(lyric_url, linesep, timeout=timeout)
+
+    def from_artist_and_track(self, artist, track, linesep="\n", timeout=None):
+        """
+        Returns the lyric string for the given artist and track
+        by making scraping search results and fetching the first
+        result.
+        """
+        lyric_url = self.guess_lyric_url_from_artist_and_track(artist, track)
+        return self.from_url(lyric_url, linesep, timeout)
+
+    def from_url(self, url, linesep="\n", timeout=None):
+        """
+        Returns the lyric string for the given URL.
+        """
+        lyric_html_page = self._fetch_url_page(url, timeout=timeout)
+        lyrics = self._get_lyrics_text(lyric_html_page)
         return lyrics.replace("\n", linesep)
+

spotdl/lyrics/providers/lyricwikia_wrapper.py

@@ -1,18 +1,24 @@
 import lyricwikia
 
 from spotdl.lyrics.lyric_base import LyricBase
-from spotdl.lyrics.exceptions import LyricsNotFound
+from spotdl.lyrics.exceptions import LyricsNotFoundError
 
 
 class LyricWikia(LyricBase):
-    def __init__(self, artist, song):
-        self.artist = artist
-        self.song = song
+    def from_query(self, query, linesep="\n", timeout=None):
+        raise NotImplementedError
 
-    def get_lyrics(self, linesep="\n", timeout=None):
+    def from_artist_and_track(self, artist, track, linesep="\n", timeout=None):
+        """
+        Returns the lyric string for the given artist and track.
+        """
         try:
-            lyrics = lyricwikia.get_lyrics(self.artist, self.song, linesep, timeout)
+            lyrics = lyricwikia.get_lyrics(artist, track, linesep, timeout)
         except lyricwikia.LyricsNotFound as e:
-            raise LyricsNotFound(e.args[0])
-        else:
-            return lyrics
+            raise LyricsNotFoundError(e.args[0])
+
+        return lyrics
+
+    def from_url(self, url, linesep="\n", timeout=None):
+        raise NotImplementedError
+

spotdl/lyrics/providers/tests/test_genius.py

@@ -3,35 +3,114 @@ from spotdl.lyrics import exceptions
 from spotdl.lyrics.providers import Genius
 
 import urllib.request
+import json
 import pytest
 
-
 class TestGenius:
     def test_subclass(self):
         assert issubclass(Genius, LyricBase)
 
+    @pytest.fixture(scope="module")
+    def expect_lyrics_count(self):
+        # This is the number of characters in lyrics found
+        # for the track in `lyric_url` fixture below
+        return 1845
+
+    @pytest.fixture(scope="module")
+    def genius(self):
+        return Genius()
+
+    def test_base_url(self, genius):
+        assert genius.base_url == "https://genius.com"
+
+    @pytest.fixture(scope="module")
+    def artist(self):
+        return "selena gomez"
+
     @pytest.fixture(scope="module")
     def track(self):
-        return Genius("artist", "song")
+        return "wolves"
 
-    def test_base_url(self, track):
-        assert track.base_url == "https://genius.com"
+    @pytest.fixture(scope="module")
+    def query(self, artist, track):
+        return "{} {}".format(artist, track)
 
-    def test_get_lyrics(self, track, monkeypatch):
-        def mocked_urlopen(url, timeout=None):
-            class DummyHTTPResponse:
-                def read(self):
-                    return "<p>amazing lyrics!</p>"
+    @pytest.fixture(scope="module")
+    def guess_url(self, query):
+        return "https://genius.com/selena-gomez-wolves-lyrics"
 
-            return DummyHTTPResponse()
+    @pytest.fixture(scope="module")
+    def lyric_url(self):
+        return "https://genius.com/Selena-gomez-and-marshmello-wolves-lyrics"
 
-        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
-        assert track.get_lyrics() == "amazing lyrics!"
+    def test_guess_lyric_url_from_artist_and_track(self, genius, artist, track, guess_url):
+        url = genius.guess_lyric_url_from_artist_and_track(artist, track)
+        assert url == guess_url
 
-    def test_lyrics_not_found_error(self, track, monkeypatch):
-        def mocked_urlopen(url, timeout=None):
+    class MockHTTPResponse:
+        expect_lyrics = ""
+
+        def __init__(self, request, timeout=None):
+            search_results_url = "https://genius.com/api/search/multi?per_page=1&q=selena%2Bgomez%2Bwolves"
+            if request._full_url == search_results_url:
+                read_method = lambda: json.dumps({
+                    "response": {"sections": [{"hits": [{"result": {
+                        "path": "/Selena-gomez-and-marshmello-wolves-lyrics"
+                    } }] }] }
+                })
+            else:
+                read_method = lambda: "<p>" + self.expect_lyrics + "</p>"
+
+            self.read = read_method
+
+    @pytest.mark.network
+    def test_best_matching_lyric_url_from_query(self, genius, query, lyric_url):
+        url = genius.best_matching_lyric_url_from_query(query)
+        assert url == lyric_url
+
+    def test_mock_best_matching_lyric_url_from_query(self, genius, query, lyric_url, monkeypatch):
+        monkeypatch.setattr("urllib.request.urlopen", self.MockHTTPResponse)
+        self.test_best_matching_lyric_url_from_query(genius, query, lyric_url)
+
+    @pytest.mark.network
+    def test_from_url(self, genius, lyric_url, expect_lyrics_count):
+        lyrics = genius.from_url(lyric_url)
+        assert len(lyrics) == expect_lyrics_count
+
+    def test_mock_from_url(self, genius, lyric_url, expect_lyrics_count, monkeypatch):
+        self.MockHTTPResponse.expect_lyrics = "a" * expect_lyrics_count
+        monkeypatch.setattr("urllib.request.urlopen", self.MockHTTPResponse)
+        self.test_from_url(genius, lyric_url, expect_lyrics_count)
+
+    @pytest.mark.network
+    def test_from_artist_and_track(self, genius, artist, track, expect_lyrics_count):
+        lyrics = genius.from_artist_and_track(artist, track)
+        assert len(lyrics) == expect_lyrics_count
+
+    def test_mock_from_artist_and_track(self, genius, artist, track, expect_lyrics_count, monkeypatch):
+        self.MockHTTPResponse.expect_lyrics = "a" * expect_lyrics_count
+        monkeypatch.setattr("urllib.request.urlopen", self.MockHTTPResponse)
+        self.test_from_artist_and_track(genius, artist, track, expect_lyrics_count)
+
+    @pytest.mark.network
+    def test_from_query(self, genius, query, expect_lyrics_count):
+        lyrics = genius.from_query(query)
+        assert len(lyrics) == expect_lyrics_count
+
+    def test_mock_from_query(self, genius, query, expect_lyrics_count, monkeypatch):
+        self.MockHTTPResponse.expect_lyrics = "a" * expect_lyrics_count
+        monkeypatch.setattr("urllib.request.urlopen", self.MockHTTPResponse)
+        self.test_from_query(genius, query, expect_lyrics_count)
+
+    @pytest.mark.network
+    def test_lyrics_not_found_error(self, genius):
+        with pytest.raises(exceptions.LyricsNotFoundError):
+            genius.from_artist_and_track(self, "nonexistent_artist", "nonexistent_track")
+
+    def test_mock_lyrics_not_found_error(self, genius, monkeypatch):
+        def mock_urlopen(url, timeout=None):
             raise urllib.request.HTTPError("", "", "", "", "")
 
-        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
-        with pytest.raises(exceptions.LyricsNotFound):
-            track.get_lyrics()
+        monkeypatch.setattr("urllib.request.urlopen", mock_urlopen)
+        self.test_lyrics_not_found_error(genius)
+

spotdl/lyrics/providers/tests/test_lyricwikia_wrapper.py

@@ -11,25 +11,26 @@ class TestLyricWikia:
     def test_subclass(self):
         assert issubclass(LyricWikia, LyricBase)
 
-    def test_get_lyrics(self, monkeypatch):
+    def test_from_artist_and_track(self, monkeypatch):
         # `LyricWikia` class uses the 3rd party method `lyricwikia.get_lyrics`
         # internally and there is no need to test a 3rd party library as they
         # have their own implementation of tests.
         monkeypatch.setattr(
             "lyricwikia.get_lyrics", lambda a, b, c, d: "awesome lyrics!"
         )
-        track = LyricWikia("Lyricwikia", "Lyricwikia")
-        assert track.get_lyrics() == "awesome lyrics!"
+        artist, track = "selena gomez", "wolves"
+        lyrics = LyricWikia().from_artist_and_track(artist, track)
+        assert lyrics == "awesome lyrics!"
 
     def test_lyrics_not_found_error(self, monkeypatch):
         def lyricwikia_lyrics_not_found(msg):
             raise lyricwikia.LyricsNotFound(msg)
 
-        # Wrap `lyricwikia.LyricsNotFound` with `exceptions.LyricsNotFound` error.
+        # Wrap `lyricwikia.LyricsNotFoundError` with `exceptions.LyricsNotFoundError` error.
         monkeypatch.setattr(
             "lyricwikia.get_lyrics",
             lambda a, b, c, d: lyricwikia_lyrics_not_found("Nope, no lyrics."),
         )
-        track = LyricWikia("Lyricwikia", "Lyricwikia")
-        with pytest.raises(exceptions.LyricsNotFound):
-            track.get_lyrics()
+        artist, track = "nonexistent_artist", "nonexistent_track"
+        with pytest.raises(exceptions.LyricsNotFoundError):
+            LyricWikia().from_artist_and_track(artist, track)

spotdl/lyrics/tests/test_lyric_base.py

@@ -0,0 +1,24 @@
+from spotdl.lyrics import LyricBase
+
+import pytest
+
+
+class TestAbstractBaseClass:
+    def test_error_abstract_base_class_lyricbase(self):
+        with pytest.raises(TypeError):
+            # This abstract base class must be inherited from
+            # for instantiation
+            LyricBase()
+
+    def test_inherit_abstract_base_class_encoderbase(self):
+        class LyricKid(LyricBase):
+            def from_query(self, query):
+                raise NotImplementedError
+
+            def from_artist_and_track(self, artist, track):
+                pass
+
+            def from_url(self, url):
+                raise NotImplementedError
+
+        LyricKid()

spotdl/lyrics/tests/test_lyrics_exceptions.py

@@ -0,0 +1,5 @@
+from spotdl.lyrics.exceptions import LyricsNotFoundError
+
+def test_lyrics_not_found_subclass():
+    assert issubclass(LyricsNotFoundError, Exception)
+

spotdl/metadata.py

@@ -1,182 +0,0 @@
-from mutagen.easyid3 import EasyID3
-from mutagen.id3 import ID3, TORY, TYER, TPUB, APIC, USLT, COMM
-from mutagen.mp4 import MP4, MP4Cover
-from mutagen.flac import Picture, FLAC
-
-import urllib.request
-from logzero import logger as log
-
-from spotdl.const import TAG_PRESET, M4A_TAG_PRESET
-
-
-def compare(music_file, metadata):
-    """Check if the input music file title matches the expected title."""
-    already_tagged = False
-    try:
-        if music_file.endswith(".mp3"):
-            audiofile = EasyID3(music_file)
-            already_tagged = audiofile["title"][0] == metadata["name"]
-        elif music_file.endswith(".m4a"):
-            audiofile = MP4(music_file)
-            already_tagged = audiofile["\xa9nam"][0] == metadata["name"]
-    except (KeyError, TypeError):
-        pass
-
-    return already_tagged
-
-
-def embed(music_file, meta_tags):
-    """ Embed metadata. """
-    embed = EmbedMetadata(music_file, meta_tags)
-    if music_file.endswith(".m4a"):
-        log.info("Applying metadata")
-        return embed.as_m4a()
-    elif music_file.endswith(".mp3"):
-        log.info("Applying metadata")
-        return embed.as_mp3()
-    elif music_file.endswith(".flac"):
-        log.info("Applying metadata")
-        return embed.as_flac()
-    else:
-        log.warning("Cannot embed metadata into given output extension")
-        return False
-
-
-class EmbedMetadata:
-    def __init__(self, music_file, meta_tags):
-        self.music_file = music_file
-        self.meta_tags = meta_tags
-        self.spotify_metadata = meta_tags["spotify_metadata"]
-        self.provider = "spotify" if meta_tags["spotify_metadata"] else "youtube"
-
-    def as_mp3(self):
-        """ Embed metadata to MP3 files. """
-        music_file = self.music_file
-        meta_tags = self.meta_tags
-        # EasyID3 is fun to use ;)
-        # For supported easyid3 tags:
-        # https://github.com/quodlibet/mutagen/blob/master/mutagen/easyid3.py
-        # Check out somewhere at end of above linked file
-        audiofile = EasyID3(music_file)
-        self._embed_basic_metadata(audiofile, preset=TAG_PRESET)
-        audiofile["media"] = meta_tags["type"]
-        audiofile["author"] = meta_tags["artists"][0]["name"]
-        audiofile["lyricist"] = meta_tags["artists"][0]["name"]
-        audiofile["arranger"] = meta_tags["artists"][0]["name"]
-        audiofile["performer"] = meta_tags["artists"][0]["name"]
-        audiofile["website"] = meta_tags["external_urls"][self.provider]
-        audiofile["length"] = str(meta_tags["duration"])
-        if meta_tags["publisher"]:
-            audiofile["encodedby"] = meta_tags["publisher"]
-        if meta_tags["external_ids"]["isrc"]:
-            audiofile["isrc"] = meta_tags["external_ids"]["isrc"]
-        audiofile.save(v2_version=3)
-
-        # For supported id3 tags:
-        # https://github.com/quodlibet/mutagen/blob/master/mutagen/id3/_frames.py
-        # Each class represents an id3 tag
-        audiofile = ID3(music_file)
-        if meta_tags["year"]:
-            audiofile["TORY"] = TORY(encoding=3, text=meta_tags["year"])
-            audiofile["TYER"] = TYER(encoding=3, text=meta_tags["year"])
-        if meta_tags["publisher"]:
-            audiofile["TPUB"] = TPUB(encoding=3, text=meta_tags["publisher"])
-        audiofile["COMM"] = COMM(
-            encoding=3, text=meta_tags["external_urls"][self.provider]
-        )
-        if meta_tags["lyrics"]:
-            audiofile["USLT"] = USLT(
-                encoding=3, desc=u"Lyrics", text=meta_tags["lyrics"]
-            )
-        try:
-            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
-            audiofile["APIC"] = APIC(
-                encoding=3,
-                mime="image/jpeg",
-                type=3,
-                desc=u"Cover",
-                data=albumart.read(),
-            )
-            albumart.close()
-        except IndexError:
-            pass
-
-        audiofile.save(v2_version=3)
-        return True
-
-    def as_m4a(self):
-        """ Embed metadata to M4A files. """
-        music_file = self.music_file
-        meta_tags = self.meta_tags
-        audiofile = MP4(music_file)
-        self._embed_basic_metadata(audiofile, preset=M4A_TAG_PRESET)
-        if meta_tags["year"]:
-            audiofile[M4A_TAG_PRESET["year"]] = meta_tags["year"]
-        audiofile[M4A_TAG_PRESET["comment"]] = meta_tags["external_urls"][self.provider]
-        if meta_tags["lyrics"]:
-            audiofile[M4A_TAG_PRESET["lyrics"]] = meta_tags["lyrics"]
-        try:
-            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
-            audiofile[M4A_TAG_PRESET["albumart"]] = [
-                MP4Cover(albumart.read(), imageformat=MP4Cover.FORMAT_JPEG)
-            ]
-            albumart.close()
-        except IndexError:
-            pass
-
-        audiofile.save()
-        return True
-
-    def as_flac(self):
-        music_file = self.music_file
-        meta_tags = self.meta_tags
-        audiofile = FLAC(music_file)
-        self._embed_basic_metadata(audiofile)
-        if meta_tags["year"]:
-            audiofile["year"] = meta_tags["year"]
-        audiofile["comment"] = meta_tags["external_urls"][self.provider]
-        if meta_tags["lyrics"]:
-            audiofile["lyrics"] = meta_tags["lyrics"]
-
-        image = Picture()
-        image.type = 3
-        image.desc = "Cover"
-        image.mime = "image/jpeg"
-        albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
-        image.data = albumart.read()
-        albumart.close()
-        audiofile.add_picture(image)
-
-        audiofile.save()
-        return True
-
-    def _embed_basic_metadata(self, audiofile, preset=TAG_PRESET):
-        meta_tags = self.meta_tags
-        audiofile[preset["artist"]] = meta_tags["artists"][0]["name"]
-        if meta_tags["album"]["artists"][0]["name"]:
-            audiofile[preset["albumartist"]] = meta_tags["album"]["artists"][0]["name"]
-        if meta_tags["album"]["name"]:
-            audiofile[preset["album"]] = meta_tags["album"]["name"]
-        audiofile[preset["title"]] = meta_tags["name"]
-        if meta_tags["release_date"]:
-            audiofile[preset["date"]] = meta_tags["release_date"]
-            audiofile[preset["originaldate"]] = meta_tags["release_date"]
-        if meta_tags["genre"]:
-            audiofile[preset["genre"]] = meta_tags["genre"]
-        if meta_tags["copyright"]:
-            audiofile[preset["copyright"]] = meta_tags["copyright"]
-        if self.music_file.endswith(".flac"):
-            audiofile[preset["discnumber"]] = str(meta_tags["disc_number"])
-        else:
-            audiofile[preset["discnumber"]] = [(meta_tags["disc_number"], 0)]
-        if self.music_file.endswith(".flac"):
-            audiofile[preset["tracknumber"]] = str(meta_tags["track_number"])
-        else:
-            if preset["tracknumber"] == TAG_PRESET["tracknumber"]:
-                audiofile[preset["tracknumber"]] = "{}/{}".format(
-                    meta_tags["track_number"], meta_tags["total_tracks"]
-                )
-            else:
-                audiofile[preset["tracknumber"]] = [
-                    (meta_tags["track_number"], meta_tags["total_tracks"])
-                ]

spotdl/metadata/__init__.py

@@ -0,0 +1,11 @@
+from spotdl.metadata.provider_base import ProviderBase
+from spotdl.metadata.provider_base import StreamsBase
+
+from spotdl.metadata.exceptions import MetadataNotFoundError
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+from spotdl.metadata.exceptions import YouTubeMetadataNotFoundError
+
+from spotdl.metadata.embedder_base import EmbedderBase
+
+from spotdl.metadata.formatter import format_string
+

spotdl/metadata/embedder_base.py

@@ -0,0 +1,94 @@
+import os
+
+from abc import ABC
+from abc import abstractmethod
+
+import urllib.request
+
+class EmbedderBase(ABC):
+    """
+    The subclass must define the supported media file encoding
+    formats here using a static variable - such as:
+
+    >>> supported_formats = ("mp3", "m4a", "flac")
+    """
+    supported_formats = ()
+
+    @abstractmethod
+    def __init__(self):
+        """
+        For every supported format, there must be a corresponding
+        method that applies metadata on this format.
+
+        Such as if mp3 is supported, there must exist a method named
+        `as_mp3` on this class that applies metadata on mp3 files.
+        """
+        # self.targets = { fmt: eval(str("self.as_" + fmt))
+        #                  for fmt in self.supported_formats }
+        #
+        # TODO: The above code seems to fail for some reason
+        #       I do not know.
+        self.targets = {}
+        for fmt in self.supported_formats:
+            # FIXME: Calling `eval` is dangerous here!
+            self.targets[fmt] = eval("self.as_" + fmt)
+
+    def get_encoding(self, path):
+        """
+        This method must determine the encoding for a local
+        audio file. Such as "mp3", "wav", "m4a", etc.
+        """
+        _, extension = os.path.splitext(path)
+        # Ignore the initial dot from file extension
+        return extension[1:]
+
+    def apply_metadata(self, path, metadata, cached_albumart=None, encoding=None):
+        """
+        This method must automatically detect the media encoding
+        format from file path and embed the corresponding metadata
+        on the given file by calling an appropriate submethod.
+        """
+        if cached_albumart is None:
+            cached_albumart = urllib.request.urlopen(
+                metadata["album"]["images"][0]["url"],
+            ).read()
+        if encoding is None:
+            encoding = self.get_encoding(path)
+        if encoding not in self.supported_formats:
+            raise TypeError(
+                'The input format ("{}") is not supported.'.format(
+                encoding,
+            ))
+        embed_on_given_format = self.targets[encoding]
+        embed_on_given_format(path, metadata, cached_albumart=cached_albumart)
+
+    def as_mp3(self, path, metadata, cached_albumart=None):
+        """
+        Method for mp3 support. This method might be defined in
+        a subclass.
+
+        Other methods for additional supported formats must also
+        be declared here.
+        """
+        raise NotImplementedError
+
+    def as_m4a(self, path, metadata, cached_albumart=None):
+        """
+        Method for m4a support. This method might be defined in
+        a subclass.
+
+        Other methods for additional supported formats must also
+        be declared here.
+        """
+        raise NotImplementedError
+
+    def as_flac(self, path, metadata, cached_albumart=None):
+        """
+        Method for flac support. This method might be defined in
+        a subclass.
+
+        Other methods for additional supported formats must also
+        be declared here.
+        """
+        raise NotImplementedError
+

spotdl/metadata/embedders/__init__.py

@@ -0,0 +1,2 @@
+from spotdl.metadata.embedders.default_embedder import EmbedderDefault
+

spotdl/metadata/embedders/default_embedder.py

@@ -0,0 +1,189 @@
+from mutagen.easyid3 import EasyID3
+from mutagen.id3 import ID3, TORY, TYER, TPUB, APIC, USLT, COMM
+from mutagen.mp4 import MP4, MP4Cover
+from mutagen.flac import Picture, FLAC
+
+import urllib.request
+
+from spotdl.metadata import EmbedderBase
+
+import logging
+logger = logging.getLogger(__name__)
+
+# Apple has specific tags - see mutagen docs -
+# http://mutagen.readthedocs.io/en/latest/api/mp4.html
+M4A_TAG_PRESET = {
+    "album": "\xa9alb",
+    "artist": "\xa9ART",
+    "date": "\xa9day",
+    "title": "\xa9nam",
+    "year": "\xa9day",
+    "originaldate": "purd",
+    "comment": "\xa9cmt",
+    "group": "\xa9grp",
+    "writer": "\xa9wrt",
+    "genre": "\xa9gen",
+    "tracknumber": "trkn",
+    "albumartist": "aART",
+    "discnumber": "disk",
+    "cpil": "cpil",
+    "albumart": "covr",
+    "copyright": "cprt",
+    "tempo": "tmpo",
+    "lyrics": "\xa9lyr",
+    "comment": "\xa9cmt",
+}
+
+TAG_PRESET = {}
+for key in M4A_TAG_PRESET.keys():
+    TAG_PRESET[key] = key
+
+
+class EmbedderDefault(EmbedderBase):
+    supported_formats = ("mp3", "m4a", "flac")
+
+    def __init__(self):
+        super().__init__()
+        self._m4a_tag_preset = M4A_TAG_PRESET
+        self._tag_preset = TAG_PRESET
+        # self.provider = "spotify" if metadata["spotify_metadata"] else "youtube"
+
+    def as_mp3(self, path, metadata, cached_albumart=None):
+        """ Embed metadata to MP3 files. """
+        logger.debug('Writing MP3 metadata to "{path}".'.format(path=path))
+        # EasyID3 is fun to use ;)
+        # For supported easyid3 tags:
+        # https://github.com/quodlibet/mutagen/blob/master/mutagen/easyid3.py
+        # Check out somewhere at end of above linked file
+        audiofile = EasyID3(path)
+        self._embed_basic_metadata(audiofile, metadata, "mp3", preset=TAG_PRESET)
+        audiofile["media"] = metadata["type"]
+        audiofile["author"] = metadata["artists"][0]["name"]
+        audiofile["lyricist"] = metadata["artists"][0]["name"]
+        audiofile["arranger"] = metadata["artists"][0]["name"]
+        audiofile["performer"] = metadata["artists"][0]["name"]
+        provider = metadata["provider"]
+        audiofile["website"] = metadata["external_urls"][provider]
+        audiofile["length"] = str(metadata["duration"])
+        if metadata["publisher"]:
+            audiofile["encodedby"] = metadata["publisher"]
+        if metadata["external_ids"]["isrc"]:
+            audiofile["isrc"] = metadata["external_ids"]["isrc"]
+        audiofile.save(v2_version=3)
+
+        # For supported id3 tags:
+        # https://github.com/quodlibet/mutagen/blob/master/mutagen/id3/_frames.py
+        # Each class represents an id3 tag
+        audiofile = ID3(path)
+        if metadata["year"]:
+            audiofile["TORY"] = TORY(encoding=3, text=metadata["year"])
+            audiofile["TYER"] = TYER(encoding=3, text=metadata["year"])
+        if metadata["publisher"]:
+            audiofile["TPUB"] = TPUB(encoding=3, text=metadata["publisher"])
+        provider = metadata["provider"]
+        audiofile["COMM"] = COMM(
+            encoding=3, text=metadata["external_urls"][provider]
+        )
+        if metadata["lyrics"]:
+            audiofile["USLT"] = USLT(
+                encoding=3, desc=u"Lyrics", text=metadata["lyrics"]
+            )
+        if cached_albumart is None:
+            cached_albumart = urllib.request.urlopen(
+                metadata["album"]["images"][0]["url"]
+            ).read()
+            albumart.close()
+        try:
+            audiofile["APIC"] = APIC(
+                encoding=3,
+                mime="image/jpeg",
+                type=3,
+                desc=u"Cover",
+                data=cached_albumart,
+            )
+        except IndexError:
+            pass
+
+        audiofile.save(v2_version=3)
+
+    def as_m4a(self, path, metadata, cached_albumart=None):
+        """ Embed metadata to M4A files. """
+        logger.debug('Writing M4A metadata to "{path}".'.format(path=path))
+        audiofile = MP4(path)
+        self._embed_basic_metadata(audiofile, metadata, "m4a", preset=M4A_TAG_PRESET)
+        if metadata["year"]:
+            audiofile[M4A_TAG_PRESET["year"]] = metadata["year"]
+        provider = metadata["provider"]
+        audiofile[M4A_TAG_PRESET["comment"]] = metadata["external_urls"][provider]
+        if metadata["lyrics"]:
+            audiofile[M4A_TAG_PRESET["lyrics"]] = metadata["lyrics"]
+        try:
+            if cached_albumart is None:
+                cached_albumart = urllib.request.urlopen(
+                    metadata["album"]["images"][0]["url"]
+                ).read()
+                albumart.close()
+            audiofile[M4A_TAG_PRESET["albumart"]] = [
+                MP4Cover(cached_albumart, imageformat=MP4Cover.FORMAT_JPEG)
+            ]
+        except IndexError:
+            pass
+
+        audiofile.save()
+
+    def as_flac(self, path, metadata, cached_albumart=None):
+        logger.debug('Writing FLAC metadata to "{path}".'.format(path=path))
+        audiofile = FLAC(path)
+        self._embed_basic_metadata(audiofile, metadata, "flac")
+        if metadata["year"]:
+            audiofile["year"] = metadata["year"]
+        provider = metadata["provider"]
+        audiofile["comment"] = metadata["external_urls"][provider]
+        if metadata["lyrics"]:
+            audiofile["lyrics"] = metadata["lyrics"]
+
+        image = Picture()
+        image.type = 3
+        image.desc = "Cover"
+        image.mime = "image/jpeg"
+        if cached_albumart is None:
+            cached_albumart = urllib.request.urlopen(
+                metadata["album"]["images"][0]["url"]
+            ).read()
+            albumart.close()
+        image.data = cached_albumart
+        audiofile.add_picture(image)
+
+        audiofile.save()
+
+    def _embed_basic_metadata(self, audiofile, metadata, encoding, preset=TAG_PRESET):
+        audiofile[preset["artist"]] = metadata["artists"][0]["name"]
+        if metadata["album"]["artists"][0]["name"]:
+            audiofile[preset["albumartist"]] = metadata["album"]["artists"][0]["name"]
+        if metadata["album"]["name"]:
+            audiofile[preset["album"]] = metadata["album"]["name"]
+        audiofile[preset["title"]] = metadata["name"]
+        if metadata["release_date"]:
+            audiofile[preset["date"]] = metadata["release_date"]
+            audiofile[preset["originaldate"]] = metadata["release_date"]
+        if metadata["genre"]:
+            audiofile[preset["genre"]] = metadata["genre"]
+        if metadata["copyright"]:
+            audiofile[preset["copyright"]] = metadata["copyright"]
+        if encoding == "flac":
+            audiofile[preset["discnumber"]] = str(metadata["disc_number"])
+        else:
+            audiofile[preset["discnumber"]] = [(metadata["disc_number"], 0)]
+        zfilled_track_number = str(metadata["track_number"]).zfill(len(str(metadata["total_tracks"])))
+        if encoding == "flac":
+            audiofile[preset["tracknumber"]] = zfilled_track_number
+        else:
+            if preset["tracknumber"] == TAG_PRESET["tracknumber"]:
+                audiofile[preset["tracknumber"]] = "{}/{}".format(
+                    zfilled_track_number, metadata["total_tracks"]
+                )
+            else:
+                audiofile[preset["tracknumber"]] = [
+                    (metadata["track_number"], metadata["total_tracks"])
+                ]
+

spotdl/metadata/embedders/tests/test_default_embedder.py

@@ -0,0 +1,9 @@
+from spotdl.metadata.embedders import EmbedderDefault
+
+import pytest
+
+@pytest.mark.xfail
+def test_embedder():
+    # Do not forget to Write tests for this!
+    raise NotImplementedError
+

spotdl/metadata/exceptions.py

@@ -0,0 +1,20 @@
+class MetadataNotFoundError(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class SpotifyMetadataNotFoundError(MetadataNotFoundError):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+
+
+class YouTubeMetadataNotFoundError(MetadataNotFoundError):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super().__init__(message)
+

spotdl/metadata/formatter.py

@@ -0,0 +1,23 @@
+def format_string(string, metadata, output_extension="", sanitizer=lambda s: s):
+    formats = {
+        "{track-name}"   : metadata["name"],
+        "{artist}"       : metadata["artists"][0]["name"],
+        "{album}"        : metadata["album"]["name"],
+        "{album-artist}" : metadata["artists"][0]["name"],
+        "{genre}"        : metadata["genre"],
+        "{disc-number}"  : metadata["disc_number"],
+        "{duration}"     : metadata["duration"],
+        "{year}"         : metadata["year"],
+        "{original-date}": metadata["release_date"],
+        "{track-number}" : str(metadata["track_number"]).zfill(len(str(metadata["total_tracks"]))),
+        "{total-tracks}" : metadata["total_tracks"],
+        "{isrc}"         : metadata["external_ids"]["isrc"],
+        "{track-id}"     : metadata.get("id", ""),
+        "{output-ext}"   : output_extension,
+    }
+
+    for key, value in formats.items():
+        string = string.replace(key, sanitizer(str(value)))
+
+    return string
+

spotdl/metadata/provider_base.py

@@ -0,0 +1,66 @@
+from abc import ABC
+from abc import abstractmethod
+
+class StreamsBase(ABC):
+    @abstractmethod
+    def __init__(self, streams):
+        """
+        This method must parse audio streams into a list of
+        dictionaries with the keys:
+        "bitrate", "download_url", "encoding", "filesize".
+
+        The list should typically be sorted in descending order
+        based on the audio stream's bitrate.
+
+        This sorted list must be assigned to `self.all`.
+        """
+        self.all = streams
+
+    def getbest(self):
+        """
+        This method must return the audio stream with the
+        highest bitrate.
+        """
+        return self.all[0]
+
+    def getworst(self):
+        """
+        This method must return the audio stream with the
+        lowest bitrate.
+        """
+        return self.all[-1]
+
+
+class ProviderBase(ABC):
+    def set_credentials(self, client_id, client_secret):
+        """
+        This method may or not be used depending on
+        whether the metadata provider requires authentication
+        or not.
+        """
+        pass
+
+    @abstractmethod
+    def from_url(self, url):
+        """
+        This method must return track metadata from the
+        corresponding Spotify URL.
+        """
+        pass
+
+    def from_query(self, query):
+        """
+        This method must return track metadata from the
+        corresponding search query.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def metadata_to_standard_form(self, metadata):
+        """
+        This method must transform the fetched metadata
+        into a format consistent with all other metadata
+        providers, for easy utilization.
+        """
+        pass
+

spotdl/metadata/providers/__init__.py

@@ -0,0 +1,4 @@
+from spotdl.metadata.providers.spotify import ProviderSpotify
+from spotdl.metadata.providers.youtube import ProviderYouTube
+from spotdl.metadata.providers.youtube import YouTubeSearch
+

spotdl/metadata/providers/spotify.py

@@ -0,0 +1,82 @@
+import spotipy
+import spotipy.oauth2 as oauth2
+
+from spotdl.metadata import ProviderBase
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+
+from spotdl.authorize.services import AuthorizeSpotify
+import spotdl.util
+
+import logging
+logger = logging.getLogger(__name__)
+
+
+class ProviderSpotify(ProviderBase):
+    def __init__(self, spotify=None):
+        if spotify is None:
+            spotify = AuthorizeSpotify()
+        self.spotify = spotify
+
+    def set_credentials(self, client_id, client_secret):
+        token = self._generate_token(client_id, client_secret)
+        self.spotify = spotipy.Spotify(auth=token)
+
+    def from_url(self, url):
+        logger.debug('Fetching Spotify metadata for "{url}".'.format(url=url))
+        metadata = self.spotify.track(url)
+        return self.metadata_to_standard_form(metadata)
+
+    def from_query(self, query):
+        tracks = self.search(query)["tracks"]["items"]
+        if not tracks:
+            raise SpotifyMetadataNotFoundError(
+                'Spotify returned no tracks for the search query "{}".'.format(
+                    query,
+                )
+            )
+        return self.metadata_to_standard_form(tracks[0])
+
+    def search(self, query):
+        return self.spotify.search(query)
+
+    def _generate_token(self, client_id, client_secret):
+        """ Generate the token. """
+        credentials = oauth2.SpotifyClientCredentials(
+            client_secret=client_secret,
+        )
+        token = credentials.get_access_token()
+        return token
+
+    def metadata_to_standard_form(self, metadata):
+        artist = self.spotify.artist(metadata["artists"][0]["id"])
+        album = self.spotify.album(metadata["album"]["id"])
+
+        try:
+            metadata[u"genre"] = spotdl.util.titlecase(artist["genres"][0])
+        except IndexError:
+            metadata[u"genre"] = None
+        try:
+            metadata[u"copyright"] = album["copyrights"][0]["text"]
+        except IndexError:
+            metadata[u"copyright"] = None
+        try:
+            metadata[u"external_ids"][u"isrc"]
+        except KeyError:
+            metadata[u"external_ids"][u"isrc"] = None
+
+        metadata[u"release_date"] = album["release_date"]
+        metadata[u"publisher"] = album["label"]
+        metadata[u"total_tracks"] = album["tracks"]["total"]
+
+        # Some sugar
+        metadata["year"], *_ = metadata["release_date"].split("-")
+        metadata["duration"] = metadata["duration_ms"] / 1000.0
+        metadata["provider"] = "spotify"
+
+        # Remove unwanted parameters
+        del metadata["duration_ms"]
+        del metadata["available_markets"]
+        del metadata["album"]["available_markets"]
+
+        return metadata
+

spotdl/metadata/providers/tests/test_spotify.py

@@ -0,0 +1,19 @@
+from spotdl.metadata import ProviderBase
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+from spotdl.metadata.providers import ProviderSpotify
+
+import pytest
+
+class TestProviderSpotify:
+    def test_subclass(self):
+        assert issubclass(ProviderSpotify, ProviderBase)
+
+    @pytest.mark.xfail
+    def test_spotify_stuff(self):
+        raise NotImplementedError
+
+    # def test_metadata_not_found_error(self):
+    #     provider = ProviderSpotify(spotify=spotify)
+    #     with pytest.raises(SpotifyMetadataNotFoundError):
+    #         provider.from_query("This track doesn't exist on Spotify.")
+

spotdl/metadata/providers/tests/test_youtube.py

@@ -0,0 +1,366 @@
+from spotdl.metadata.providers.youtube import YouTubeSearch
+from spotdl.metadata.providers.youtube import YouTubeStreams
+from spotdl.metadata.providers.youtube import YouTubeVideos
+from spotdl.metadata.providers import youtube
+from spotdl.metadata.providers import ProviderYouTube
+from spotdl.metadata.exceptions import YouTubeMetadataNotFoundError
+
+import pytube
+import urllib.request
+import pickle
+import sys
+import os
+import pytest
+
+
+@pytest.fixture(scope="module")
+def track():
+    """
+    This query is to be searched on YouTube for queries
+    that do return search results.
+    """
+    return "selena gomez wolves"
+
+
+@pytest.fixture(scope="module")
+def no_result_track():
+    """
+    This query is to be searched on YouTube for queries
+    that return no search results.
+    """
+    return "n0 v1d305 3x157 f0r 7h15 53arc4 qu3ry"
+
+
+@pytest.fixture(scope="module")
+def expect_search_results():
+    """
+    These are the expected search results for the "track"
+    query.
+    """
+    return YouTubeVideos([
+        {'duration': '3:33',
+         'title': 'Selena Gomez, Marshmello - Wolves',
+         'url': 'https://www.youtube.com/watch?v=cH4E_t3m3xM'},
+        {'duration': '3:18',
+         'title': 'Selena Gomez, Marshmello - Wolves (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=xrbY9gDVms0'},
+        {'duration': '3:21',
+         'title': 'Wolves - Selena Gomez, Marshmello (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=jX0n2rSmDbE'},
+        {'duration': '6:26',
+         'title': 'Selena Gomez and Marshmello - Wolves (Official) Extended',
+         'url': 'https://www.youtube.com/watch?v=rQ6jcpwzQZU'},
+        {'duration': '3:43',
+         'title': 'Selena Gomez, Marshmello - Wolves (Vertical Video)',
+         'url': 'https://www.youtube.com/watch?v=nVzA1uWTydQ'},
+        {'duration': '3:18',
+         'title': 'Selena Gomez, Marshmello - Wolves (Visualizer)',
+         'url': 'https://www.youtube.com/watch?v=-grLLLTza6k'},
+        {'duration': '1:32',
+         'title': 'Wolves - Selena Gomez, Marshmello / Jun Liu Choreography',
+         'url': 'https://www.youtube.com/watch?v=zbWsb36U0uo'},
+        {'duration': '3:17',
+         'title': 'Selena Gomez, Marshmello - Wolves (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=rykH1BkGwTo'},
+        {'duration': '3:16',
+         'title': 'Selena Gomez, Marshmello - Wolves (8D AUDIO)',
+         'url': 'https://www.youtube.com/watch?v=j0AxZ4V5WQw'},
+        {'duration': '3:47',
+         'title': 'Selena Gomez, Marshmello - Wolves (Vanrip Remix)',
+         'url': 'https://www.youtube.com/watch?v=RyxsaKfu-ZY'}
+    ])
+
+
+@pytest.fixture(scope="module")
+def expect_mock_search_results():
+    """
+    These are the expected mock search results for the
+    "track" query.
+    """
+    return YouTubeVideos([
+        {'duration': '3:33',
+         'title': 'Selena Gomez, Marshmello - Wolves',
+         'url': 'https://www.youtube.com/watch?v=cH4E_t3m3xM'},
+        {'duration': '3:18',
+         'title': 'Selena Gomez, Marshmello - Wolves (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=xrbY9gDVms0'},
+        {'duration': '3:21',
+         'title': 'Wolves - Selena Gomez, Marshmello (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=jX0n2rSmDbE'},
+        {'duration': '6:26',
+         'title': 'Selena Gomez and Marshmello - Wolves (Official) Extended',
+         'url': 'https://www.youtube.com/watch?v=rQ6jcpwzQZU'},
+        {'duration': '3:43',
+         'title': 'Selena Gomez, Marshmello - Wolves (Vertical Video)',
+         'url': 'https://www.youtube.com/watch?v=nVzA1uWTydQ'},
+        {'duration': '3:18',
+         'title': 'Selena Gomez, Marshmello - Wolves (Visualizer)',
+         'url': 'https://www.youtube.com/watch?v=-grLLLTza6k'},
+        {'duration': '1:32',
+         'title': 'Wolves - Selena Gomez, Marshmello / Jun Liu Choreography',
+         'url': 'https://www.youtube.com/watch?v=zbWsb36U0uo'},
+        {'duration': '3:17',
+         'title': 'Selena Gomez, Marshmello - Wolves (Lyrics)',
+         'url': 'https://www.youtube.com/watch?v=rykH1BkGwTo'},
+        {'duration': '3:16',
+         'title': 'Selena Gomez, Marshmello - Wolves (8D AUDIO)',
+         'url': 'https://www.youtube.com/watch?v=j0AxZ4V5WQw'},
+        {'duration': '3:47',
+         'title': 'Selena Gomez, Marshmello - Wolves (Vanrip Remix)',
+         'url': 'https://www.youtube.com/watch?v=RyxsaKfu-ZY'}
+    ])
+
+
+class MockHTTPResponse:
+    """
+    This mocks `urllib.request.urlopen` for custom response text.
+    """
+    response_file = ""
+
+    def __init__(self, request):
+        if isinstance(request, urllib.request.Request):
+            if request._full_url.endswith("ouVRL5arzUg=="):
+                self.headers = {"Content-Length": 3614184}
+            elif request._full_url.endswith("egl0iK2D-Bk="):
+                self.headers = {"Content-Length": 3444850}
+            elif request._full_url.endswith("J7VXJtoi3as="):
+                self.headers = {"Content-Length": 1847626}
+            elif request._full_url.endswith("_d5_ZthQdvtD"):
+                self.headers = {"Content-Length": 1407962}
+
+    def read(self):
+        module_directory = os.path.dirname(__file__)
+        mock_html = os.path.join(module_directory, "data", self.response_file)
+        with open(mock_html, "r") as fin:
+            html = fin.read()
+        return html
+
+
+class TestYouTubeSearch:
+    @pytest.fixture(scope="module")
+    def youtube_searcher(self):
+        return YouTubeSearch()
+
+    def test_generate_search_url(self, track, youtube_searcher):
+        url = youtube_searcher.generate_search_url(track)
+        expect_url = "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=selena%20gomez%20wolves"
+        assert url == expect_url
+
+    @pytest.mark.network
+    def test_search(self, track, youtube_searcher, expect_search_results):
+        results = youtube_searcher.search(track)
+        assert results == expect_search_results
+
+    class MockHTTPResponse:
+        """
+        This mocks `urllib.request.urlopen` for custom response text.
+        """
+        response_file = ""
+
+        def __init__(self, url):
+            pass
+
+        def read(self):
+            module_directory = os.path.dirname(__file__)
+            mock_html = os.path.join(module_directory, "data", self.response_file)
+            with open(mock_html, "r") as fin:
+                html = fin.read()
+            return html
+
+    # @pytest.mark.mock
+    def test_mock_search(self, track, youtube_searcher, expect_mock_search_results, monkeypatch):
+        MockHTTPResponse.response_file = "youtube_search_results.html"
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_search(track, youtube_searcher, expect_mock_search_results)
+
+    @pytest.mark.network
+    def test_no_videos_search(self, no_result_track, youtube_searcher):
+        results = youtube_searcher.search(no_result_track)
+        assert results == YouTubeVideos([])
+
+    def test_mock_no_videos_search(self, no_result_track, youtube_searcher, monkeypatch):
+        MockHTTPResponse.response_file = "youtube_no_search_results.html"
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_no_videos_search(no_result_track, youtube_searcher)
+
+
+@pytest.fixture(scope="module")
+def content():
+    return pytube.YouTube("https://www.youtube.com/watch?v=cH4E_t3m3xM")
+
+
+class MockYouTube:
+    def __init__(self, url):
+        self.watch_html = '\\"category\\":\\"Music\\",\\"publishDate\\":\\"2017-11-18\\",\\"ownerChannelName\\":\\"SelenaGomezVEVO\\",'
+        self.title = "Selena Gomez, Marshmello - Wolves"
+        self.author = "SelenaGomezVEVO"
+        self.length = 213
+        self.watch_url = "https://youtube.com/watch?v=cH4E_t3m3xM"
+        self.thumbnail_url = "https://i.ytimg.com/vi/cH4E_t3m3xM/maxresdefault.jpg"
+
+    @property
+    def streams(self):
+        # For updating the test data:
+        # from spotdl.metadata.providers.youtube import YouTubeStreams
+        # import pytube
+        # import pickle
+        # content = pytube.YouTube("https://youtube.com/watch?v=cH4E_t3m3xM")
+        # with open("streams.dump", "wb") as fout:
+        #       pickle.dump(content.streams, fout)
+        module_directory = os.path.dirname(__file__)
+        mock_streams = os.path.join(module_directory, "data", "streams.dump")
+        with open(mock_streams, "rb") as fin:
+            streams_dump = pickle.load(fin)
+        return streams_dump
+
+
+@pytest.fixture(scope="module")
+def mock_content():
+    return MockYouTube("https://www.youtube.com/watch?v=cH4E_t3m3xM")
+
+
+@pytest.fixture(scope="module")
+def expect_formatted_streams():
+    """
+    Expected streams for the best matching video for "track" in
+    search results.
+
+    The `download_url` is expected as `None` since it's impossible
+    to predict its value before-hand.
+    """
+    return [
+        {"bitrate": 160, "content": None, "download_url": None, "encoding": "opus", "filesize": 3614184},
+        {"bitrate": 128, "content": None, "download_url": None, "encoding": "mp4a.40.2", "filesize": 3444850},
+        {"bitrate": 70, "content": None, "download_url": None, "encoding": "opus", "filesize": 1847626},
+        {"bitrate": 50, "content": None, "download_url": None, "encoding": "opus", "filesize": 1407962}
+    ]
+
+
+class TestYouTubeStreams:
+    @pytest.mark.network
+    def test_streams(self, content, expect_formatted_streams):
+        formatted_streams = YouTubeStreams(content.streams)
+        for index in range(len(formatted_streams.all)):
+            assert isinstance(formatted_streams.all[index]["download_url"], str)
+            assert formatted_streams.all[index]["connection"] is not None
+            # We `None` the `download_url` since it's impossible to
+            # predict its value before-hand.
+            formatted_streams.all[index]["download_url"] = None
+            formatted_streams.all[index]["connection"] = None
+
+        # assert formatted_streams.all == expect_formatted_streams
+        for f, e in zip(formatted_streams.all, expect_formatted_streams):
+            assert f["filesize"] == e["filesize"]
+
+    # @pytest.mark.mock
+    def test_mock_streams(self, mock_content, expect_formatted_streams, monkeypatch):
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_streams(mock_content, expect_formatted_streams)
+
+    @pytest.mark.network
+    def test_getbest(self, content):
+        formatted_streams = YouTubeStreams(content.streams)
+        best_stream = formatted_streams.getbest()
+        assert isinstance(best_stream["download_url"], str)
+        assert best_stream["connection"] is not None
+        # We `None` the `download_url` since it's impossible to
+        # predict its value before-hand.
+        best_stream["download_url"] = None
+        best_stream["connection"] = None
+        assert best_stream == {
+            "bitrate": 160,
+            "connection": None,
+            "download_url": None,
+            "encoding": "opus",
+            "filesize": 3614184
+        }
+
+    # @pytest.mark.mock
+    def test_mock_getbest(self, mock_content, monkeypatch):
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_getbest(mock_content)
+
+    @pytest.mark.network
+    def test_getworst(self, content):
+        formatted_streams = YouTubeStreams(content.streams)
+        worst_stream = formatted_streams.getworst()
+        assert isinstance(worst_stream["download_url"], str)
+        assert worst_stream["connection"] is not None
+        # We `None` the `download_url` since it's impossible to
+        # predict its value before-hand.
+        worst_stream["download_url"] = None
+        worst_stream["connection"] = None
+        assert worst_stream == {
+            "bitrate": 50,
+            "connection": None,
+            "download_url": None,
+            "encoding": 'opus',
+            "filesize": 1407962
+        }
+
+    # @pytest.mark.mock
+    def test_mock_getworst(self, mock_content, monkeypatch):
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_getworst(mock_content)
+
+
+class TestProviderYouTube:
+    @pytest.fixture(scope="module")
+    def youtube_provider(self):
+        return ProviderYouTube()
+
+    class MockYouTubeSearch:
+        watch_urls = []
+        def search(self, query):
+            return self.watch_urls
+
+    @pytest.mark.network
+    def test_from_query(self, track, youtube_provider):
+        metadata = youtube_provider.from_query(track)
+        assert isinstance(metadata["streams"], YouTubeStreams)
+        # We avoid testing each item for the `streams` key here
+        # again. It this has already been tested above.
+        metadata["streams"] = []
+        assert metadata == {
+          'album': {'artists': [{'name': None}],
+                    'images': [{'url': 'https://i.ytimg.com/vi/cH4E_t3m3xM/maxresdefault.jpg'}],
+                    'name': None},
+          'artists': [{'name': 'SelenaGomezVEVO'}],
+          'copyright': None,
+          'disc_number': 1,
+          'duration': 213,
+          'external_ids': {'isrc': None},
+          'external_urls': {'youtube': 'https://youtube.com/watch?v=cH4E_t3m3xM'},
+          'genre': None,
+          'lyrics': None,
+          'name': 'Selena Gomez, Marshmello - Wolves',
+          'provider': 'youtube',
+          'publisher': None,
+          'release_date': '2017-11-1',
+          'streams': [],
+          'total_tracks': 1,
+          'track_number': 1,
+          'type': 'track',
+          'year': '2017'
+       }
+
+
+
+    def test_mock_from_query(self, track, youtube_provider, expect_mock_search_results, monkeypatch):
+        self.MockYouTubeSearch.watch_urls = expect_mock_search_results
+        monkeypatch.setattr(youtube, "YouTubeSearch", self.MockYouTubeSearch)
+        monkeypatch.setattr(pytube, "YouTube", MockYouTube)
+        monkeypatch.setattr(urllib.request, "urlopen", MockHTTPResponse)
+        self.test_from_query(track, youtube_provider)
+
+    @pytest.mark.network
+    def test_error_exception_from_query(self, no_result_track, youtube_provider):
+        with pytest.raises(YouTubeMetadataNotFoundError):
+            youtube_provider.from_query(no_result_track)
+
+    def test_mock_error_exception_from_query(self, no_result_track, youtube_provider, monkeypatch):
+        self.MockYouTubeSearch.watch_urls = []
+        monkeypatch.setattr(youtube, "YouTubeSearch", self.MockYouTubeSearch)
+        monkeypatch.setattr(pytube, "YouTube", MockYouTube)
+        self.test_error_exception_from_query(no_result_track, youtube_provider)
+

spotdl/metadata/providers/youtube.py

@@ -0,0 +1,283 @@
+import pytube
+from bs4 import BeautifulSoup
+
+import urllib.request
+import threading
+from collections.abc import Sequence
+
+from spotdl.metadata import StreamsBase
+from spotdl.metadata import ProviderBase
+from spotdl.metadata.exceptions import YouTubeMetadataNotFoundError
+
+import spotdl.util
+
+import logging
+logger = logging.getLogger(__name__)
+
+BASE_SEARCH_URL = "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={}"
+HEADERS = [('Range', 'bytes=0-'),]
+
+
+class YouTubeVideos(Sequence):
+    def __init__(self, videos):
+        self.videos = videos
+        super().__init__()
+
+    def __repr__(self):
+        return "YouTubeVideos({})".format(self.videos)
+
+    def __len__(self):
+        return len(self.videos)
+
+    def __getitem__(self, index):
+        return self.videos[index]
+
+    def __eq__(self, instance):
+        return self.videos == instance.videos
+
+    def bestmatch(self):
+        video = self.videos[0]
+        logger.debug("Matched with: {title} ({url}) [{duration}]".format(
+            title=video["title"],
+            url=video["url"],
+            duration=video["duration"]
+        ))
+        return video
+
+
+class YouTubeSearch:
+    def __init__(self):
+        self.base_search_url = BASE_SEARCH_URL
+
+    def generate_search_url(self, query):
+        quoted_query = urllib.request.quote(query)
+        return self.base_search_url.format(quoted_query)
+
+    def _fetch_response_html(self, url, retries=5):
+        response = urllib.request.urlopen(url)
+        soup = BeautifulSoup(response.read(), "html.parser")
+        return soup
+
+    def _extract_video_details_from_result(self, html):
+        video_time = html.find("span", class_="video-time").get_text()
+        inner_html = html.find("div", class_="yt-lockup-content")
+        video_id = inner_html.find("a")["href"][-11:]
+        video_title = inner_html.find("a")["title"]
+        video_details = {
+            "url": "https://www.youtube.com/watch?v=" + video_id,
+            "title": video_title,
+            "duration": video_time,
+        }
+        return video_details
+
+    def _fetch_search_results(self, html, limit=10):
+        result_source = html.find_all(
+            "div", {"class": "yt-lockup-dismissable yt-uix-tile"}
+        )
+        videos = []
+
+        for result in result_source:
+            if not self._is_video(result):
+                continue
+
+            video = self._extract_video_details_from_result(result)
+            videos.append(video)
+
+            if len(videos) >= limit:
+                break
+
+        return videos
+
+    def _is_video(self, result):
+        # ensure result is not a channel
+        not_video = (
+            result.find("channel") is not None
+            or "yt-lockup-channel" in result.parent.attrs["class"]
+            or "yt-lockup-channel" in result.attrs["class"]
+        )
+
+        # ensure result is not a mix/playlist
+        not_video = not_video or "yt-lockup-playlist" in result.parent.attrs["class"]
+
+        # ensure video result is not an advertisement
+        not_video = not_video or result.find("googleads") is not None
+
+        video = not not_video
+        return video
+
+    def _is_server_side_invalid_response(self, videos, html):
+        if videos:
+            return False
+        search_message = html.find("div", {"class":"search-message"})
+        return search_message is None
+
+    def search(self, query, limit=10, retries=5):
+        """ Search and scrape YouTube to return a list of matching videos. """
+        search_url = self.generate_search_url(query)
+        logger.debug('Fetching YouTube results for "{}" at "{}".'.format(query, search_url))
+        html = self._fetch_response_html(search_url)
+        videos = self._fetch_search_results(html, limit=limit)
+        to_retry = retries > 0 and self._is_server_side_invalid_response(videos, html)
+        if to_retry:
+            retries -= 1
+            logger.debug(
+                "Retrying since YouTube returned invalid response for search "
+                "results. Retries left: {retries}.".format(retries=retries)
+            )
+            return self.search(query, limit=limit, retries=retries)
+        return YouTubeVideos(videos)
+
+
+class YouTubeStreams(StreamsBase):
+    def __init__(self, streams):
+        self.network_headers = HEADERS
+
+        audiostreams = streams.filter(only_audio=True).order_by("abr").desc()
+
+        thread_pool = []
+        self.all = []
+
+        for stream in audiostreams:
+            standard_stream = {
+                # Store only the integer part for bitrate. For example
+                # the given bitrate would be "192kbps", we store only
+                # the integer part (192) here and drop the rest.
+                "bitrate": int(stream.abr[:-4]),
+                "connection": None,
+                "download_url": stream.url,
+                "encoding": stream.audio_codec,
+                "filesize": None,
+            }
+            establish_connection = threading.Thread(
+               target=self._store_connection,
+               args=(standard_stream,),
+            )
+            thread_pool.append(establish_connection)
+            establish_connection.start()
+            self.all.append(standard_stream)
+
+        for thread in thread_pool:
+            thread.join()
+
+    def _store_connection(self, stream):
+        response = self._make_request(stream["download_url"])
+        stream["connection"] = response
+        stream["filesize"] = int(response.headers["Content-Length"])
+
+    def _make_request(self, url):
+        request = urllib.request.Request(url)
+        for header in self.network_headers:
+            request.add_header(*header)
+        return urllib.request.urlopen(request)
+
+    def get(self, quality="best", preftype="automatic"):
+        if quality == "best":
+            return self.getbest(preftype=preftype)
+        elif quality == "worst":
+            return self.getworst(preftype=preftype)
+        else:
+            return None
+
+    def getbest(self, preftype="automatic"):
+        selected_stream = None
+        if preftype == "automatic":
+            selected_stream = self.all[0]
+        else:
+            for stream in self.all:
+                if stream["encoding"] == preftype:
+                    selected_stream = stream
+                    break
+        logger.debug('Selected best quality stream for "{preftype}" format:\n{stream}'.format(
+            preftype=preftype,
+            stream=selected_stream,
+        ))
+        return selected_stream
+
+    def getworst(self, preftype="automatic"):
+        selected_stream = None
+        if preftype == "automatic":
+            selected_stream = self.all[-1]
+        else:
+            for stream in self.all[::-1]:
+                if stream["encoding"] == preftype:
+                    selected_stream = stream
+                    break
+        logger.debug('Selected worst quality stream for "{preftype}" format:\n{stream}'.format(
+            preftype=preftype,
+            stream=selected_stream,
+        ))
+        return selected_stream
+
+
+class ProviderYouTube(ProviderBase):
+    def from_query(self, query):
+        watch_urls = self.search(query)
+        if not watch_urls:
+            raise YouTubeMetadataNotFoundError(
+                'YouTube returned nothing for the given search '
+                'query ("{}")'.format(query)
+            )
+        return self.from_url(watch_urls[0])
+
+    def from_url(self, url, retries=5):
+        logger.debug('Fetching YouTube metadata for "{url}".'.format(url=url))
+        try:
+            content = pytube.YouTube(url)
+        except KeyError:
+            # Sometimes YouTube can return unexpected response, in such a case
+            # retry a few times before finally failing.
+            if retries > 0:
+                retries -= 1
+                logger.debug(
+                    "YouTube returned an unexpected response for "
+                    "`pytube.YouTube({url})`. Retries left: {retries}".format(
+                        url=url, retries=retries
+                    )
+                )
+                return self.from_url(url, retries=retries)
+            else:
+                raise
+        else:
+            return self.from_pytube_object(content)
+
+    def from_pytube_object(self, content):
+        return self.metadata_to_standard_form(content)
+
+    def search(self, query):
+        return YouTubeSearch().search(query)
+
+    def _fetch_publish_date(self, content):
+        # XXX: This needs to be supported in PyTube itself
+        # See https://github.com/nficano/pytube/issues/595
+        position = content.watch_html.find("publishDate")
+        publish_date = content.watch_html[position+16:position+25]
+        return publish_date
+
+    def metadata_to_standard_form(self, content):
+        """ Fetch a song's metadata from YouTube. """
+        publish_date = self._fetch_publish_date(content)
+        metadata = {
+            "name": content.title,
+            "artists": [{"name": content.author}],
+            "duration": content.length,
+            "external_urls": {"youtube": content.watch_url},
+            "album": {
+                "images": [{"url": content.thumbnail_url}],
+                "artists": [{"name": None}],
+                "name": None,
+            },
+            "year": publish_date.split("-")[0],
+            "release_date": publish_date,
+            "type": "track",
+            "disc_number": 1,
+            "track_number": 1,
+            "total_tracks": 1,
+            "publisher": None,
+            "external_ids": {"isrc": None},
+            "lyrics": None,
+            "copyright": None,
+            "genre": None,
+            "streams": YouTubeStreams(content.streams),
+            "provider": "youtube",
+        }
+        return metadata

spotdl/metadata/tests/test_embedder_base.py

@@ -0,0 +1,72 @@
+from spotdl.metadata import EmbedderBase
+
+import pytest
+
+class EmbedderKid(EmbedderBase):
+    def __init__(self):
+        super().__init__()
+
+
+class TestEmbedderBaseABC:
+    def test_error_base_class_embedderbase(self):
+        with pytest.raises(TypeError):
+            # This abstract base class must be inherited from
+            # for instantiation
+            EmbedderBase()
+
+    def test_inherit_abstract_base_class_streamsbase(self):
+        EmbedderKid()
+
+
+class TestMethods:
+    @pytest.fixture(scope="module")
+    def embedderkid(self):
+        return EmbedderKid()
+
+    def test_target_formats(self, embedderkid):
+        assert embedderkid.supported_formats == ()
+
+    @pytest.mark.parametrize("path, expect_encoding", (
+        ("/a/b/c/file.mp3", "mp3"),
+        ("music/pop/1.wav", "wav"),
+        ("/a path/with spaces/track.m4a", "m4a"),
+    ))
+    def test_get_encoding(self, embedderkid, path, expect_encoding):
+        assert embedderkid.get_encoding(path) == expect_encoding
+
+    def test_apply_metadata_with_explicit_encoding(self, embedderkid):
+        with pytest.raises(TypeError):
+            embedderkid.apply_metadata("/path/to/music.mp3", {}, cached_albumart="imagedata", encoding="mp3")
+
+    def test_apply_metadata_with_implicit_encoding(self, embedderkid):
+        with pytest.raises(TypeError):
+            embedderkid.apply_metadata("/path/to/music.wav", {}, cached_albumart="imagedata")
+
+    class MockHTTPResponse:
+        """
+        This mocks `urllib.request.urlopen` for custom response text.
+        """
+        response_file = ""
+
+        def __init__(self, url):
+            pass
+
+        def read(self):
+            pass
+
+    def test_apply_metadata_without_cached_image(self, embedderkid, monkeypatch):
+        monkeypatch.setattr("urllib.request.urlopen", self.MockHTTPResponse)
+        metadata = {"album": {"images": [{"url": "http://animageurl.com"},]}}
+        with pytest.raises(TypeError):
+            embedderkid.apply_metadata("/path/to/music.wav", metadata, cached_albumart=None)
+
+    @pytest.mark.parametrize("fmt_method_suffix", (
+        "as_mp3",
+        "as_m4a",
+        "as_flac",
+    ))
+    def test_embed_formats(self, fmt_method_suffix, embedderkid):
+        method = eval("embedderkid." + fmt_method_suffix)
+        with pytest.raises(NotImplementedError):
+            method("/a/random/path", {})
+

spotdl/metadata/tests/test_metadata_exceptions.py

@@ -0,0 +1,15 @@
+from spotdl.metadata.exceptions import MetadataNotFoundError
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+from spotdl.metadata.exceptions import YouTubeMetadataNotFoundError
+
+
+class TestMetadataNotFoundSubclass:
+    def test_metadata_not_found_subclass(self):
+        assert issubclass(MetadataNotFoundError, Exception)
+
+    def test_spotify_metadata_not_found(self):
+        assert issubclass(SpotifyMetadataNotFoundError, MetadataNotFoundError)
+
+    def test_youtube_metadata_not_found(self):
+        assert issubclass(YouTubeMetadataNotFoundError, MetadataNotFoundError)
+

spotdl/metadata/tests/test_provider_base.py

@@ -0,0 +1,60 @@
+from spotdl.metadata import ProviderBase
+from spotdl.metadata import StreamsBase
+
+import pytest
+
+class TestStreamsBaseABC:
+    def test_error_abstract_base_class_streamsbase(self):
+        with pytest.raises(TypeError):
+            # This abstract base class must be inherited from
+            # for instantiation
+            StreamsBase()
+
+    def test_inherit_abstract_base_class_streamsbase(self):
+        class StreamsKid(StreamsBase):
+            def __init__(self, streams):
+                super().__init__(streams)
+
+        streams = ("stream1", "stream2", "stream3")
+        kid = StreamsKid(streams)
+        assert kid.all == streams
+
+
+class TestMethods:
+    class StreamsKid(StreamsBase):
+        def __init__(self, streams):
+            super().__init__(streams)
+
+
+    @pytest.fixture(scope="module")
+    def streamskid(self):
+        streams = ("stream1", "stream2", "stream3")
+        streamskid = self.StreamsKid(streams)
+        return streamskid
+
+    def test_getbest(self, streamskid):
+        best_stream = streamskid.getbest()
+        assert best_stream == "stream1"
+
+    def test_getworst(self, streamskid):
+        worst_stream = streamskid.getworst()
+        assert worst_stream == "stream3"
+
+
+class TestProviderBaseABC:
+    def test_error_abstract_base_class_providerbase(self):
+        with pytest.raises(TypeError):
+            # This abstract base class must be inherited from
+            # for instantiation
+            ProviderBase()
+
+    def test_inherit_abstract_base_class_providerbase(self):
+        class ProviderKid(ProviderBase):
+            def from_url(self, query):
+                pass
+
+            def metadata_to_standard_form(self, metadata):
+                pass
+
+        ProviderKid()
+

spotdl/metadata_search.py

@@ -0,0 +1,260 @@
+from spotdl.metadata.providers import ProviderSpotify
+from spotdl.metadata.providers import ProviderYouTube
+from spotdl.lyrics.providers import Genius
+from spotdl.lyrics.exceptions import LyricsNotFoundError
+
+import spotdl.metadata
+import spotdl.util
+from spotdl.metadata.exceptions import SpotifyMetadataNotFoundError
+
+from spotdl.command_line.exceptions import NoYouTubeVideoFoundError
+from spotdl.command_line.exceptions import NoYouTubeVideoMatchError
+
+import sys
+import logging
+logger = logging.getLogger(__name__)
+
+PROVIDERS = {
+    "spotify": ProviderSpotify,
+    "youtube": ProviderYouTube,
+}
+
+
+def prompt_for_youtube_search_result(videos):
+    max_index_length = len(str(len(videos)))
+    max_title_length = max(len(v["title"]) for v in videos)
+    msg = "{index:>{max_index}}. Skip downloading this track".format(
+        index=0,
+        max_index=max_index_length,
+    )
+    print(msg, file=sys.stderr)
+    for index, video in enumerate(videos, 1):
+        vid_details = "{index:>{max_index}}. {title:<{max_title}}\n{new_line_gap}  {url} [{duration}]".format(
+            index=index,
+            max_index=max_index_length,
+            title=video["title"],
+            max_title=max_title_length,
+            new_line_gap=" " * max_index_length,
+            url=video["url"],
+            duration=video["duration"],
+        )
+        print(vid_details, file=sys.stderr)
+    print("", file=sys.stderr)
+
+    selection = spotdl.util.prompt_user_for_selection(range(1, len(videos)+1))
+
+    if selection is None:
+        return None
+    return videos[selection-1]
+
+
+class MetadataSearch:
+    def __init__(self, track, lyrics=False, yt_search_format="{artist} - {track-name}", yt_manual=False, providers=PROVIDERS):
+        self.track = track
+        self.track_type = spotdl.util.track_type(track)
+        self.lyrics = lyrics
+        self.yt_search_format = yt_search_format
+        self.yt_manual = yt_manual
+        self.providers = {}
+        for provider, parent in providers.items():
+            self.providers[provider] = parent()
+        self.lyric_provider = Genius()
+
+    def get_lyrics(self, query):
+        try:
+            lyrics = self.lyric_provider.from_query(query)
+        except LyricsNotFoundError as e:
+            logger.warning(e.args[0])
+            lyrics = None
+        return lyrics
+
+    def _make_lyric_search_query(self, metadata):
+        if self.track_type == "query":
+            lyric_query = self.track
+        else:
+            lyric_search_format = "{artist} - {track-name}"
+            lyric_query = spotdl.metadata.format_string(
+                lyric_search_format,
+                metadata
+            )
+        return lyric_query
+
+    def on_youtube_and_spotify(self):
+        track_type_mapper = {
+            "spotify": self._on_youtube_and_spotify_for_type_spotify,
+            "youtube": self._on_youtube_and_spotify_for_type_youtube,
+            "query": self._on_youtube_and_spotify_for_type_query,
+        }
+        caller = track_type_mapper[self.track_type]
+        metadata = caller()
+
+        if not self.lyrics:
+            return metadata
+
+        lyric_query = self._make_lyric_search_query(metadata)
+        metadata["lyrics"] = spotdl.util.ThreadWithReturnValue(
+            target=self.get_lyrics,
+            args=(lyric_query,),
+        )
+
+        return metadata
+
+    def on_youtube(self):
+        track_type_mapper = {
+            "spotify": self._on_youtube_for_type_spotify,
+            "youtube": self._on_youtube_for_type_youtube,
+            "query": self._on_youtube_for_type_query,
+        }
+        caller = track_type_mapper[self.track_type]
+        metadata = caller(self.track)
+
+        if not self.lyrics:
+            return metadata
+
+        lyric_query = self._make_lyric_search_query(metadata)
+        metadata["lyrics"] = spotdl.util.ThreadWithReturnValue(
+            target=self.get_lyrics,
+            arguments=(lyric_query,),
+        )
+
+        return metadata
+
+    def on_spotify(self):
+        track_type_mapper = {
+            "spotify": self._on_spotify_for_type_spotify,
+            "youtube": self._on_spotify_for_type_youtube,
+            "query": self._on_spotify_for_type_query,
+        }
+        caller = track_type_mapper[self.track_type]
+        metadata = caller(self.track)
+
+        if not self.lyrics:
+            return metadata
+
+        lyric_query = self._make_lyric_search_query(metadata)
+        metadata["lyrics"] = spotdl.util.ThreadWithReturnValue(
+            target=self.get_lyrics,
+            arguments=(lyric_query,),
+        )
+
+        return metadata
+
+    def best_on_youtube_search(self):
+        track_type_mapper = {
+            "spotify": self._best_on_youtube_search_for_type_spotify,
+            "youtube": self._best_on_youtube_search_for_type_youtube,
+            "query": self._best_on_youtube_search_for_type_query,
+        }
+        caller = track_type_mapper[self.track_type]
+        video = caller(self.track)
+        return video
+
+    def _best_on_youtube_search_for_type_query(self, query):
+        videos = self.providers["youtube"].search(query)
+        if not videos:
+            raise NoYouTubeVideoFoundError(
+                'YouTube returned no videos for the search query "{}".'.format(query)
+            )
+        if self.yt_manual:
+            video = prompt_for_youtube_search_result(videos)
+        else:
+            video = videos.bestmatch()
+
+        if video is None:
+            raise NoYouTubeVideoMatchError(
+                'No matching videos found on YouTube for the search query "{}".'.format(
+                    search_query
+                )
+            )
+        return video
+
+    def _best_on_youtube_search_for_type_youtube(self, url):
+        video = self._best_on_youtube_search_for_type_query(url)
+        return video
+
+    def _best_on_youtube_search_for_type_spotify(self, url):
+        spotify_metadata = self._on_spotify_for_type_spotify(self.track)
+        search_query = spotdl.metadata.format_string(self.yt_search_format, spotify_metadata)
+        video = self._best_on_youtube_search_for_type_query(search_query)
+        return video
+
+    def _on_youtube_and_spotify_for_type_spotify(self):
+        logger.debug("Extracting YouTube and Spotify metadata for input Spotify URI.")
+        spotify_metadata = self._on_spotify_for_type_spotify(self.track)
+        search_query = spotdl.metadata.format_string(self.yt_search_format, spotify_metadata)
+        youtube_video = self._best_on_youtube_search_for_type_spotify(search_query)
+        youtube_metadata = self.providers["youtube"].from_url(youtube_video["url"])
+        metadata = spotdl.util.merge_copy(
+            youtube_metadata,
+            spotify_metadata
+        )
+        return metadata
+
+    def _on_youtube_and_spotify_for_type_youtube(self):
+        logger.debug("Extracting YouTube and Spotify metadata for input YouTube URL.")
+        youtube_metadata = self._on_youtube_for_type_youtube(self.track)
+        search_query = spotdl.metadata.format_string("{track-name}", youtube_metadata)
+        spotify_metadata = self._on_spotify_for_type_query(search_query)
+        metadata = spotdl.util.merge_copy(
+            youtube_metadata,
+            spotify_metadata
+        )
+        return metadata
+
+    def _on_youtube_and_spotify_for_type_query(self):
+        logger.debug("Extracting YouTube and Spotify metadata for input track query.")
+        search_query = self.track
+        # Make use of threads here to search on both YouTube & Spotify
+        # at the same time.
+        spotify_metadata = spotdl.util.ThreadWithReturnValue(
+            target=self._on_spotify_for_type_query,
+            args=(search_query,)
+        )
+        spotify_metadata.start()
+        youtube_metadata = self._on_youtube_for_type_query(search_query)
+        metadata = spotdl.util.merge_copy(
+            youtube_metadata,
+            spotify_metadata.join()
+        )
+        return metadata
+
+    def _on_youtube_for_type_spotify(self):
+        logger.debug("Extracting YouTube metadata for input Spotify URI.")
+        spotify_metadata = self._on_spotify_for_type_spotify(self.track)
+        search_query = spotdl.metadata.format_string(self.yt_search_format, spotify_metadata)
+        youtube_video = self._best_on_youtube_search_for_type_spotify(search_query)
+        youtube_metadata = self.providers["youtube"].from_url(youtube_video["url"])
+        return youtube_metadata
+
+    def _on_youtube_for_type_youtube(self, url):
+        logger.debug("Extracting YouTube metadata for input YouTube URL.")
+        youtube_metadata = self.providers["youtube"].from_url(url)
+        return youtube_metadata
+
+    def _on_youtube_for_type_query(self, query):
+        logger.debug("Extracting YouTube metadata for input track query.")
+        youtube_video = self._best_on_youtube_search_for_type_query(query)
+        youtube_metadata = self.providers["youtube"].from_url(youtube_video["url"])
+        return youtube_metadata
+
+    def _on_spotify_for_type_youtube(self, url):
+        logger.debug("Extracting Spotify metadata for input YouTube URL.")
+        youtube_metadata = self.providers["youtube"].from_url(url)
+        search_query = spotdl.metadata.format_string("{track-name}", youtube_metadata)
+        spotify_metadata = self.providers["spotify"].from_query(search_query)
+        return spotify_metadata
+
+    def _on_spotify_for_type_spotify(self, url):
+        logger.debug("Extracting Spotify metadata for input Spotify URI.")
+        spotify_metadata = self.providers["spotify"].from_url(url)
+        return spotify_metadata
+
+    def _on_spotify_for_type_query(self, query):
+        logger.debug("Extracting Spotify metadata for input track query.")
+        try:
+            spotify_metadata = self.providers["spotify"].from_query(query)
+        except SpotifyMetadataNotFoundError as e:
+            logger.warn(e.args[0])
+            spotify_metadata = {}
+        return spotify_metadata
+

spotdl/patcher.py

@@ -1,64 +0,0 @@
-from pafy import backend_youtube_dl
-import pafy
-
-from spotdl import internals
-
-
-def _getbestthumb(self):
-    url = self._ydl_info["thumbnails"][0]["url"]
-    if url:
-        return url
-
-    part_url = "https://i.ytimg.com/vi/%s/" % self.videoid
-    # Thumbnail resolution sorted in descending order
-    thumbs = (
-        "maxresdefault.jpg",
-        "sddefault.jpg",
-        "hqdefault.jpg",
-        "mqdefault.jpg",
-        "default.jpg",
-    )
-    for thumb in thumbs:
-        url = part_url + thumb
-        if self._content_available(url):
-            return url
-
-
-def _process_streams(self):
-    for format_index in range(len(self._ydl_info["formats"])):
-        try:
-            self._ydl_info["formats"][format_index]["url"] = self._ydl_info["formats"][
-                format_index
-            ]["fragment_base_url"]
-        except KeyError:
-            pass
-    return backend_youtube_dl.YtdlPafy._old_process_streams(self)
-
-
-@classmethod
-def _content_available(cls, url):
-    return internals.content_available(url)
-
-
-class PatchPafy:
-    """
-    These patches have not been released by pafy on PyPI yet but
-    are useful to us.
-    """
-
-    def patch_getbestthumb(self):
-        # https://github.com/mps-youtube/pafy/pull/211
-        pafy.backend_shared.BasePafy._bestthumb = None
-        pafy.backend_shared.BasePafy._content_available = _content_available
-        pafy.backend_shared.BasePafy.getbestthumb = _getbestthumb
-
-    def patch_process_streams(self):
-        # https://github.com/mps-youtube/pafy/pull/230
-        backend_youtube_dl.YtdlPafy._old_process_streams = (
-            backend_youtube_dl.YtdlPafy._process_streams
-        )
-        backend_youtube_dl.YtdlPafy._process_streams = _process_streams
-
-    def patch_insecure_streams(self):
-        # https://github.com/mps-youtube/pafy/pull/235
-        pafy.g.def_ydl_opts["prefer_insecure"] = False

spotdl/spotdl.py

@@ -1,79 +0,0 @@
-#!/usr/bin/env python3
-
-import sys
-import platform
-import pprint
-import logzero
-from logzero import logger as log
-
-from spotdl import __version__
-from spotdl import const
-from spotdl import handle
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import downloader
-
-
-def debug_sys_info():
-    log.debug("Python version: {}".format(sys.version))
-    log.debug("Platform: {}".format(platform.platform()))
-    log.debug(pprint.pformat(const.args.__dict__))
-
-
-def match_args():
-    if const.args.song:
-        for track in const.args.song:
-            track_dl = downloader.Downloader(raw_song=track)
-            track_dl.download_single()
-    elif const.args.list:
-        if const.args.write_m3u:
-            youtube_tools.generate_m3u(
-                track_file=const.args.list
-            )
-        else:
-            list_dl = downloader.ListDownloader(
-                tracks_file=const.args.list,
-                skip_file=const.args.skip,
-                write_successful_file=const.args.write_successful,
-            )
-            list_dl.download_list()
-    elif const.args.playlist:
-        spotify_tools.write_playlist(
-            playlist_url=const.args.playlist, text_file=const.args.write_to
-        )
-    elif const.args.album:
-        spotify_tools.write_album(
-            album_url=const.args.album, text_file=const.args.write_to
-        )
-    elif const.args.all_albums:
-        spotify_tools.write_all_albums_from_artist(
-            artist_url=const.args.all_albums, text_file=const.args.write_to
-        )
-    elif const.args.username:
-        spotify_tools.write_user_playlist(
-            username=const.args.username, text_file=const.args.write_to
-        )
-
-
-def main():
-    const.args = handle.get_arguments()
-
-    internals.filter_path(const.args.folder)
-    youtube_tools.set_api_key()
-
-    logzero.setup_default_logger(formatter=const._formatter, level=const.args.log_level)
-
-    try:
-        match_args()
-        # actually we don't necessarily need this, but yeah...
-        # explicit is better than implicit!
-        sys.exit(0)
-
-    except KeyboardInterrupt as e:
-        log.exception(e)
-        sys.exit(3)
-
-
-if __name__ == "__main__":
-    main()

spotdl/spotify_tools.py

@@ -1,265 +0,0 @@
-import spotipy
-import spotipy.oauth2 as oauth2
-
-from slugify import slugify
-from titlecase import titlecase
-from logzero import logger as log
-import pprint
-import sys
-import os
-import functools
-
-from spotdl import const
-from spotdl import internals
-from spotdl.lyrics.providers import LyricClasses
-from spotdl.lyrics.exceptions import LyricsNotFound
-
-spotify = None
-
-
-def generate_token():
-    """ Generate the token. """
-    credentials = oauth2.SpotifyClientCredentials(
-        client_id=const.args.spotify_client_id,
-        client_secret=const.args.spotify_client_secret,
-    )
-    token = credentials.get_access_token()
-    return token
-
-
-def must_be_authorized(func, spotify=spotify):
-    def wrapper(*args, **kwargs):
-        global spotify
-        try:
-            assert spotify
-            return func(*args, **kwargs)
-        except (AssertionError, spotipy.client.SpotifyException):
-            token = generate_token()
-            spotify = spotipy.Spotify(auth=token)
-            return func(*args, **kwargs)
-
-    return wrapper
-
-
-@must_be_authorized
-def generate_metadata(raw_song):
-    """ Fetch a song's metadata from Spotify. """
-    if internals.is_spotify(raw_song):
-        # fetch track information directly if it is spotify link
-        log.debug("Fetching metadata for given track URL")
-        meta_tags = spotify.track(raw_song)
-    else:
-        # otherwise search on spotify and fetch information from first result
-        log.debug('Searching for "{}" on Spotify'.format(raw_song))
-        try:
-            meta_tags = spotify.search(raw_song, limit=1)["tracks"]["items"][0]
-        except IndexError:
-            return None
-    artist = spotify.artist(meta_tags["artists"][0]["id"])
-    album = spotify.album(meta_tags["album"]["id"])
-
-    try:
-        meta_tags[u"genre"] = titlecase(artist["genres"][0])
-    except IndexError:
-        meta_tags[u"genre"] = None
-    try:
-        meta_tags[u"copyright"] = album["copyrights"][0]["text"]
-    except IndexError:
-        meta_tags[u"copyright"] = None
-    try:
-        meta_tags[u"external_ids"][u"isrc"]
-    except KeyError:
-        meta_tags[u"external_ids"][u"isrc"] = None
-
-    meta_tags[u"release_date"] = album["release_date"]
-    meta_tags[u"publisher"] = album["label"]
-    meta_tags[u"total_tracks"] = album["tracks"]["total"]
-
-    log.debug("Fetching lyrics")
-    meta_tags["lyrics"] = None
-
-    for LyricClass in LyricClasses:
-        track = LyricClass(meta_tags["artists"][0]["name"], meta_tags["name"])
-        try:
-            meta_tags["lyrics"] = track.get_lyrics()
-        except LyricsNotFound:
-            continue
-        else:
-            break
-
-    # Some sugar
-    meta_tags["year"], *_ = meta_tags["release_date"].split("-")
-    meta_tags["duration"] = meta_tags["duration_ms"] / 1000.0
-    meta_tags["spotify_metadata"] = True
-    # Remove unwanted parameters
-    del meta_tags["duration_ms"]
-    del meta_tags["available_markets"]
-    del meta_tags["album"]["available_markets"]
-
-    log.debug(pprint.pformat(meta_tags))
-    return meta_tags
-
-
-@must_be_authorized
-def write_user_playlist(username, text_file=None):
-    """ Write user playlists to text_file """
-    links = get_playlists(username=username)
-    playlist = internals.input_link(links)
-    return write_playlist(playlist, text_file)
-
-
-@must_be_authorized
-def get_playlists(username):
-    """ Fetch user playlists when using the -u option. """
-    playlists = spotify.user_playlists(username)
-    links = []
-    check = 1
-
-    while True:
-        for playlist in playlists["items"]:
-            # in rare cases, playlists may not be found, so playlists['next']
-            # is None. Skip these. Also see Issue #91.
-            if playlist["name"] is not None:
-                log.info(
-                    u"{0:>5}. {1:<30}  ({2} tracks)".format(
-                        check, playlist["name"], playlist["tracks"]["total"]
-                    )
-                )
-                playlist_url = playlist["external_urls"]["spotify"]
-                log.debug(playlist_url)
-                links.append(playlist_url)
-                check += 1
-        if playlists["next"]:
-            playlists = spotify.next(playlists)
-        else:
-            break
-
-    return links
-
-
-@must_be_authorized
-def fetch_playlist(playlist):
-    try:
-        playlist_id = internals.extract_spotify_id(playlist)
-    except IndexError:
-        # Wrong format, in either case
-        log.error("The provided playlist URL is not in a recognized format!")
-        sys.exit(10)
-    try:
-        results = spotify.user_playlist(
-            user=None, playlist_id=playlist_id, fields="tracks,next,name"
-        )
-    except spotipy.client.SpotifyException:
-        log.error("Unable to find playlist")
-        log.info("Make sure the playlist is set to publicly visible and then try again")
-        sys.exit(11)
-
-    return results
-
-
-@must_be_authorized
-def write_playlist(playlist_url, text_file=None):
-    playlist = fetch_playlist(playlist_url)
-    tracks = playlist["tracks"]
-    if not text_file:
-        text_file = u"{0}.txt".format(slugify(playlist["name"], ok="-_()[]{}"))
-    return write_tracks(tracks, text_file)
-
-
-@must_be_authorized
-def fetch_album(album):
-    album_id = internals.extract_spotify_id(album)
-    album = spotify.album(album_id)
-    return album
-
-
-@must_be_authorized
-def fetch_albums_from_artist(artist_url, album_type=None):
-    """
-    This funcction returns all the albums from a give artist_url using the US
-    market
-    :param artist_url - spotify artist url
-    :param album_type - the type of album to fetch (ex: single) the default is
-                        all albums
-    :param return - the album from the artist
-    """
-
-    # fetching artist's albums limitting the results to the US to avoid duplicate
-    # albums from multiple markets
-    artist_id = internals.extract_spotify_id(artist_url)
-    results = spotify.artist_albums(artist_id, album_type=album_type, country="US")
-
-    albums = results["items"]
-
-    # indexing all pages of results
-    while results["next"]:
-        results = spotify.next(results)
-        albums.extend(results["items"])
-
-    return albums
-
-
-@must_be_authorized
-def write_all_albums_from_artist(artist_url, text_file=None):
-    """
-    This function gets all albums from an artist and writes it to a file in the
-    current working directory called [ARTIST].txt, where [ARTIST] is the artist
-    of the album
-    :param artist_url - spotify artist url
-    :param text_file - file to write albums to
-    """
-
-    album_base_url = "https://open.spotify.com/album/"
-
-    # fetching all default albums
-    albums = fetch_albums_from_artist(artist_url, album_type=None)
-
-    # if no file if given, the default save file is in the current working
-    # directory with the name of the artist
-    if text_file is None:
-        text_file = albums[0]["artists"][0]["name"] + ".txt"
-
-    for album in albums:
-        # logging album name
-        log.info("Fetching album: " + album["name"])
-        write_album(album_base_url + album["id"], text_file=text_file)
-
-
-@must_be_authorized
-def write_album(album_url, text_file=None):
-    album = fetch_album(album_url)
-    tracks = spotify.album_tracks(album["id"])
-    if not text_file:
-        text_file = u"{0}.txt".format(slugify(album["name"], ok="-_()[]{}"))
-    return write_tracks(tracks, text_file)
-
-
-@must_be_authorized
-def write_tracks(tracks, text_file):
-    log.info(u"Writing {0} tracks to {1}".format(tracks["total"], text_file))
-    track_urls = []
-    with open(text_file, "a") as file_out:
-        while True:
-            for item in tracks["items"]:
-                if "track" in item:
-                    track = item["track"]
-                else:
-                    track = item
-                try:
-                    track_url = track["external_urls"]["spotify"]
-                    log.debug(track_url)
-                    file_out.write(track_url + "\n")
-                    track_urls.append(track_url)
-                except KeyError:
-                    log.warning(
-                        u"Skipping track {0} by {1} (local only?)".format(
-                            track["name"], track["artists"][0]["name"]
-                        )
-                    )
-            # 1 page = 50 results
-            # check if there are more pages
-            if tracks["next"]:
-                tracks = spotify.next(tracks)
-            else:
-                break
-    return track_urls

spotdl/tests/test_config.py

@@ -0,0 +1,71 @@
+import spotdl.config
+
+import argparse
+import os
+import sys
+import yaml
+import pytest
+
+
+@pytest.mark.xfail
+@pytest.fixture(scope="module")
+def config_path(tmpdir_factory):
+    config_path = os.path.join(str(tmpdir_factory.mktemp("config")), "config.yml")
+    return config_path
+
+
+@pytest.mark.xfail
+@pytest.fixture(scope="module")
+def modified_config():
+    modified_config = dict(spotdl.config.DEFAULT_CONFIGURATION)
+    return modified_config
+
+
+def test_dump_n_read_config(config_path):
+    expect_config = spotdl.config.DEFAULT_CONFIGURATION
+    spotdl.config.dump_config(
+        config_path,
+        config=expect_config,
+    )
+    config = spotdl.config.read_config(config_path)
+    assert config == expect_config
+
+
+class TestDefaultConfigFile:
+    @pytest.mark.skipif(not sys.platform == "linux", reason="Linux only")
+    def test_linux_default_config_file(self):
+        expect_default_config_file = os.path.expanduser("~/.config/spotdl/config.yml")
+        assert spotdl.config.DEFAULT_CONFIG_FILE == expect_default_config_file
+
+    @pytest.mark.xfail
+    @pytest.mark.skipif(not sys.platform == "darwin" and not sys.platform == "win32",
+                        reason="Windows only")
+    def test_windows_default_config_file(self):
+        raise NotImplementedError
+
+    @pytest.mark.xfail
+    @pytest.mark.skipif(not sys.platform == "darwin",
+                        reason="OS X only")
+    def test_osx_default_config_file(self):
+        raise NotImplementedError
+
+
+class TestConfig:
+    @pytest.mark.xfail
+    def test_custom_config_path(self, config_path, modified_config):
+        parser = argparse.ArgumentParser()
+        with open(config_path, "w") as config_file:
+            yaml.dump(modified_config, config_file, default_flow_style=False)
+        overridden_config = spotdl.config.override_config(
+            config_path, parser, raw_args=""
+        )
+        modified_values = [
+            str(value)
+            for value in modified_config["spotify-downloader"].values()
+        ]
+        overridden_config.folder = os.path.realpath(overridden_config.folder)
+        overridden_values = [
+            str(value) for value in overridden_config.__dict__.values()
+        ]
+        assert sorted(overridden_values) == sorted(modified_values)
+

spotdl/tests/test_util.py

@@ -0,0 +1,60 @@
+import sys
+import os
+import subprocess
+
+import spotdl.util
+
+import pytest
+
+
+@pytest.fixture(scope="module")
+def directory_fixture(tmpdir_factory):
+    dir_path = os.path.join(str(tmpdir_factory.mktemp("tmpdir")), "filter_this_directory")
+    return dir_path
+
+
+@pytest.mark.parametrize("str_duration, sec_duration", [
+    ("0:23", 23),
+	("0:45", 45),
+	("2:19", 139),
+	("3:33", 213),
+	("7:38", 458),
+	("1:30:05", 5405),
+])
+def test_get_seconds_from_video_time(str_duration, sec_duration):
+    secs = spotdl.util.get_sec(str_duration)
+    assert secs == sec_duration
+
+
+@pytest.mark.parametrize("duplicates, expected", [
+    (("https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+      "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",),
+    ( "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",),),
+
+    (("https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+      "",
+      "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",),
+    ( "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+      "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",),),
+
+    (("ncs fade",
+      "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+      "",
+      "ncs fade",),
+    ("ncs fade",
+     "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),),
+
+    (("ncs spectre ",
+      "  https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+      ""),
+    ( "ncs spectre",
+      "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),),
+])
+def test_remove_duplicates(duplicates, expected):
+    uniques = spotdl.util.remove_duplicates(
+		duplicates,
+		condition=lambda x: x,
+		operation=str.strip,
+	)
+    assert tuple(uniques) == expected
+

spotdl/track.py

@@ -0,0 +1,111 @@
+import tqdm
+
+import urllib.request
+import subprocess
+import sys
+
+from spotdl.encode.encoders import EncoderFFmpeg
+from spotdl.metadata.embedders import EmbedderDefault
+
+import spotdl.util
+
+CHUNK_SIZE = 16 * 1024
+
+class Track:
+    def __init__(self, metadata, cache_albumart=False):
+        self.metadata = metadata
+        self._chunksize = CHUNK_SIZE
+
+        if cache_albumart:
+            self._albumart_thread = self._cache_albumart()
+
+        self._cache_albumart = cache_albumart
+
+    def _cache_albumart(self):
+        albumart_thread = spotdl.util.ThreadWithReturnValue(
+            target=lambda url: urllib.request.urlopen(url).read(),
+            args=(self.metadata["album"]["images"][0]["url"],)
+        )
+        albumart_thread.start()
+        return albumart_thread
+
+    def calculate_total_chunks(self, filesize):
+        return (filesize // self._chunksize) + 1
+
+    def make_progress_bar(self, total_chunks):
+        progress_bar = tqdm.trange(
+            total_chunks,
+            unit_scale=(self._chunksize // 1024),
+            unit="KiB",
+            dynamic_ncols=True,
+            bar_format='{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt}KiB '
+                '[{elapsed}<{remaining}, {rate_fmt}{postfix}]',
+        )
+        return progress_bar
+
+    def download_while_re_encoding(self, stream, target_path, target_encoding=None,
+                                   encoder=EncoderFFmpeg(), show_progress=True):
+        total_chunks = self.calculate_total_chunks(stream["filesize"])
+        process = encoder.re_encode_from_stdin(
+            stream["encoding"],
+            target_path,
+            target_encoding=target_encoding
+        )
+        response = stream["connection"]
+
+        progress_bar = self.make_progress_bar(total_chunks)
+        for _ in progress_bar:
+            chunk = response.read(self._chunksize)
+            process.stdin.write(chunk)
+
+        process.stdin.close()
+        process.wait()
+
+    def download(self, stream, target_path, show_progress=True):
+        total_chunks = self.calculate_total_chunks(stream["filesize"])
+        progress_bar = self.make_progress_bar(total_chunks)
+        response = stream["connection"]
+
+        def writer(response, progress_bar, file_io):
+            for _ in progress_bar:
+                chunk = response.read(self._chunksize)
+                file_io.write(chunk)
+
+        write_to_stdout = target_path == "-"
+        if write_to_stdout:
+            file_io = sys.stdout.buffer
+            writer(response, progress_bar, file_io)
+        else:
+            with open(target_path, "wb") as file_io:
+                writer(response, progress_bar, file_io)
+
+    def re_encode(self, input_path, target_path, target_encoding=None,
+                  encoder=EncoderFFmpeg(), show_progress=True):
+        stream = self.metadata["streams"].getbest()
+        total_chunks = self.calculate_total_chunks(stream["filesize"])
+        process = encoder.re_encode_from_stdin(
+            stream["encoding"],
+            target_path,
+            target_encoding=target_encoding
+        )
+        with open(input_path, "rb") as fin:
+            for _ in tqdm.trange(total_chunks):
+                chunk = fin.read(self._chunksize)
+                process.stdin.write(chunk)
+
+        process.stdin.close()
+        process.wait()
+
+    def apply_metadata(self, input_path, encoding=None, embedder=EmbedderDefault()):
+        if self._cache_albumart:
+            albumart = self._albumart_thread.join()
+        else:
+            albumart = None
+
+        embedder.apply_metadata(
+            input_path,
+            self.metadata,
+            cached_albumart=albumart,
+            encoding=encoding,
+        )
+

spotdl/util.py

@@ -0,0 +1,167 @@
+import os
+import sys
+import math
+import urllib.request
+import threading
+
+import logging
+logger = logging.getLogger(__name__)
+
+
+try:
+    import winreg
+except ImportError:
+    pass
+
+try:
+    from slugify import SLUG_OK, slugify
+except ImportError:
+    logger.error("Oops! `unicode-slugify` was not found.")
+    logger.info("Please remove any other slugify library and install `unicode-slugify`")
+    raise
+
+
+# This has been referred from
+# https://stackoverflow.com/a/6894023/6554943
+# It's because threaded functions do not return by default
+# Whereas this will return the value when `join` method
+# is called.
+class ThreadWithReturnValue(threading.Thread):
+    def __init__(self, target=lambda: None, args=()):
+        super().__init__(target=target, args=args)
+        self._return = None
+
+    def run(self):
+        if self._target is not None:
+            self._return = self._target(
+                *self._args,
+                **self._kwargs
+            )
+
+    def join(self, *args, **kwargs):
+        super().join(*args, **kwargs)
+        return self._return
+
+
+def merge_copy(base, overrider):
+    return merge(base.copy(), overrider)
+
+def merge(base, overrider):
+    """ Override base dict with an overrider dict, recursively. """
+    for key, value in overrider.items():
+        if isinstance(value, dict):
+            subitem = base.setdefault(key, {})
+            merge(subitem, value)
+        else:
+            base[key] = value
+
+    return base
+
+
+def prompt_user_for_selection(items):
+    """ Let the user input a choice. """
+    logger.info("Enter a number:")
+    while True:
+        try:
+            the_chosen_one = int(input("> "))
+            if 1 <= the_chosen_one <= len(items):
+                return items[the_chosen_one - 1]
+            elif the_chosen_one == 0:
+                return None
+            else:
+                logger.warning("Choose a valid number!")
+        except ValueError:
+            logger.warning("Choose a valid number!")
+
+
+def is_spotify(track):
+    """ Check if the input song is a Spotify link. """
+    status = len(track) == 22 and track.replace(" ", "%20") == track
+    status = status or track.find("spotify") > -1
+    return status
+
+
+def is_youtube(track):
+    """ Check if the input song is a YouTube link. """
+    status = len(track) == 11 and track.replace(" ", "%20") == track
+    status = status and not track.lower() == track
+    status = status or "youtube.com/watch?v=" in track
+    return status
+
+
+def track_type(track):
+    track_types = {
+        "spotify": is_spotify,
+        "youtube": is_youtube,
+    }
+    for provider, fn in track_types.items():
+        if fn(track):
+            return provider
+    return "query"
+
+
+def sanitize(string, ok="&-_()[]{}", spaces_to_underscores=False):
+    """ Generate filename of the song to be downloaded. """
+    if spaces_to_underscores:
+        string = string.replace(" ", "_")
+    # replace slashes with "-" to avoid directory creation errors
+    string = string.replace("/", "-").replace("\\", "-")
+    # slugify removes any special characters
+    string = slugify(string, ok=ok, lower=False, spaces=True)
+    return string
+
+
+def get_sec(time_str):
+    if ":" in time_str:
+        splitter = ":"
+    elif "." in time_str:
+        splitter = "."
+    else:
+        raise ValueError(
+            "No expected character found in {} to split" "time values.".format(time_str)
+        )
+    v = time_str.split(splitter, 3)
+    v.reverse()
+    sec = 0
+    if len(v) > 0:  # seconds
+        sec += int(v[0])
+    if len(v) > 1:  # minutes
+        sec += int(v[1]) * 60
+    if len(v) > 2:  # hours
+        sec += int(v[2]) * 3600
+    return sec
+
+
+def remove_duplicates(elements, condition=lambda _: True, operation=lambda x: x):
+    """
+    Removes duplicates from a list whilst preserving order.
+
+    We could directly call `set()` on the list but it changes
+    the order of elements.
+    """
+
+    local_set = set()
+    local_set_add = local_set.add
+    filtered_list = []
+    for x in elements:
+        if condition(x) and not (x in local_set or local_set_add(x)):
+            operated = operation(x)
+            filtered_list.append(operated)
+            local_set_add(operated)
+    return filtered_list
+
+
+def titlecase(string):
+    return " ".join(word.capitalize() for word in string.split())
+
+
+def readlines_from_nonbinary_file(path):
+    with open(path, "r") as fin:
+        lines = fin.read().splitlines()
+    return lines
+
+
+def writelines_to_nonbinary_file(path, lines):
+    with open(path, "w") as fout:
+        fout.writelines(map(lambda x: x + "\n", lines))
+

spotdl/version.py

@@ -0,0 +1,2 @@
+__version__ = "2.0.0"
+

spotdl/youtube_tools.py

@@ -1,412 +0,0 @@
-from bs4 import BeautifulSoup
-import urllib
-import pafy
-
-from slugify import slugify
-from logzero import logger as log
-import os
-
-from spotdl import spotify_tools
-from spotdl import internals
-from spotdl import const
-
-# Fix download speed throttle on short duration tracks
-# Read more on mps-youtube/pafy#199
-pafy.g.opener.addheaders.append(("Range", "bytes=0-"))
-
-# Implement unreleased methods on Pafy object
-# More info: https://github.com/mps-youtube/pafy/pull/211
-if pafy.__version__ <= "0.5.5":
-    from spotdl import patcher
-
-    pafy_patcher = patcher.PatchPafy()
-    pafy_patcher.patch_getbestthumb()
-    pafy_patcher.patch_process_streams()
-    pafy_patcher.patch_insecure_streams()
-
-
-def set_api_key():
-    if const.args.youtube_api_key:
-        key = const.args.youtube_api_key
-    else:
-        # Please respect this YouTube token :)
-        key = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
-    pafy.set_api_key(key)
-
-
-def go_pafy(raw_song, meta_tags=None):
-    """ Parse track from YouTube. """
-    if internals.is_youtube(raw_song):
-        track_info = pafy.new(raw_song)
-    else:
-        track_url = generate_youtube_url(raw_song, meta_tags)
-
-        if track_url:
-            track_info = pafy.new(track_url)
-        else:
-            track_info = None
-
-    return track_info
-
-
-def match_video_and_metadata(track):
-    """ Get and match track data from YouTube and Spotify. """
-    meta_tags = None
-
-    def fallback_metadata(meta_tags):
-        fallback_metadata_info = (
-            "Track not found on Spotify, falling back on YouTube metadata"
-        )
-        skip_fallback_metadata_warning = (
-            "Fallback condition not met, shall not embed metadata"
-        )
-        if meta_tags is None:
-            if const.args.no_fallback_metadata:
-                log.warning(skip_fallback_metadata_warning)
-            else:
-                log.info(fallback_metadata_info)
-                meta_tags = generate_metadata(content)
-        return meta_tags
-
-    if internals.is_youtube(track):
-        log.debug("Input song is a YouTube URL")
-        content = go_pafy(track, meta_tags=None)
-        track = slugify(content.title).replace("-", " ")
-        if not const.args.no_metadata:
-            meta_tags = spotify_tools.generate_metadata(track)
-            meta_tags = fallback_metadata(meta_tags)
-
-    elif internals.is_spotify(track):
-        log.debug("Input song is a Spotify URL")
-        # Let it generate metadata, YouTube doesn't know Spotify slang
-        meta_tags = spotify_tools.generate_metadata(track)
-        content = go_pafy(track, meta_tags)
-        if const.args.no_metadata:
-            meta_tags = None
-
-    else:
-        log.debug("Input song is plain text based")
-        if const.args.no_metadata:
-            content = go_pafy(track, meta_tags=None)
-        else:
-            meta_tags = spotify_tools.generate_metadata(track)
-            content = go_pafy(track, meta_tags=meta_tags)
-            meta_tags = fallback_metadata(meta_tags)
-
-    return content, meta_tags
-
-
-def generate_metadata(content):
-    """ Fetch a song's metadata from YouTube. """
-    meta_tags = {
-        "spotify_metadata": False,
-        "name": content.title,
-        "artists": [{"name": content.author}],
-        "duration": content.length,
-        "external_urls": {"youtube": content.watchv_url},
-        "album": {
-            "images": [{"url": content.getbestthumb()}],
-            "artists": [{"name": None}],
-            "name": None,
-        },
-        "year": None,
-        "release_date": None,
-        "type": "track",
-        "disc_number": 1,
-        "track_number": 1,
-        "total_tracks": 1,
-        "publisher": None,
-        "external_ids": {"isrc": None},
-        "lyrics": None,
-        "copyright": None,
-        "genre": None,
-    }
-
-    # Workaround for
-    # https://github.com/ritiek/spotify-downloader/issues/671
-    try:
-        meta_tags["year"] = content.published.split("-")[0]
-        meta_tags["release_date"] = content.published.split(" ")[0]
-    except pafy.util.GdataError:
-        pass
-
-    return meta_tags
-
-
-def get_youtube_title(content, number=None):
-    """ Get the YouTube video's title. """
-    title = content.title
-    if number:
-        return "{0}. {1}".format(number, title)
-    else:
-        return title
-
-
-def generate_m3u(track_file):
-    tracks = internals.get_unique_tracks(track_file)
-    target_file = "{}.m3u".format(track_file.split(".")[0])
-    total_tracks = len(tracks)
-    log.info("Generating {0} from {1} YouTube URLs".format(target_file, total_tracks))
-    with open(target_file, "w") as output_file:
-        output_file.write("#EXTM3U\n\n")
-
-    videos = []
-    for n, track in enumerate(tracks, 1):
-        content, _ = match_video_and_metadata(track)
-        if content is None:
-            log.warning("Skipping {}".format(track))
-        else:
-            log.info(
-                "Matched track {0}/{1} ({2})".format(
-                    n, total_tracks, content.watchv_url
-                )
-            )
-            log.debug(track)
-            m3u_key = "#EXTINF:{duration},{title}\n{youtube_url}\n".format(
-                duration=internals.get_sec(content.duration),
-                title=content.title,
-                youtube_url=content.watchv_url,
-            )
-            log.debug(m3u_key)
-            with open(target_file, "a") as output_file:
-                output_file.write(m3u_key)
-            videos.append(content.watchv_url)
-
-    return videos
-
-
-def download_song(file_name, content):
-    """ Download the audio file from YouTube. """
-    _, extension = os.path.splitext(file_name)
-    if extension in (".webm", ".m4a"):
-        link = content.getbestaudio(preftype=extension[1:])
-    else:
-        log.debug("No audio streams available for {} type".format(extension))
-        return False
-
-    if link:
-        log.debug("Downloading from URL: " + link.url)
-        filepath = os.path.join(const.args.folder, file_name)
-        log.debug("Saving to: " + filepath)
-        link.download(filepath=filepath)
-        return True
-    else:
-        log.debug("No audio streams available")
-        return False
-
-
-def generate_search_url(query):
-    """ Generate YouTube search URL for the given song. """
-    # urllib.request.quote() encodes string with special characters
-    quoted_query = urllib.request.quote(query)
-    # Special YouTube URL filter to search only for videos
-    url = "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={0}".format(
-        quoted_query
-    )
-    return url
-
-
-def is_video(result):
-    # ensure result is not a channel
-    not_video = (
-        result.find("channel") is not None
-        or "yt-lockup-channel" in result.parent.attrs["class"]
-        or "yt-lockup-channel" in result.attrs["class"]
-    )
-
-    # ensure result is not a mix/playlist
-    not_video = not_video or "yt-lockup-playlist" in result.parent.attrs["class"]
-
-    # ensure video result is not an advertisement
-    not_video = not_video or result.find("googleads") is not None
-
-    video = not not_video
-    return video
-
-
-def generate_youtube_url(raw_song, meta_tags):
-    url_fetch = GenerateYouTubeURL(raw_song, meta_tags)
-    if const.args.youtube_api_key:
-        url = url_fetch.api()
-    else:
-        url = url_fetch.scrape()
-    return url
-
-
-class GenerateYouTubeURL:
-    def __init__(self, raw_song, meta_tags):
-        self.raw_song = raw_song
-        self.meta_tags = meta_tags
-
-        if meta_tags is None:
-            self.search_query = raw_song
-        else:
-            self.search_query = internals.format_string(
-                const.args.search_format, meta_tags, force_spaces=True
-            )
-
-    def _best_match(self, videos):
-        if not videos:
-            log.error("No videos found on YouTube for a given search")
-            return None
-
-        """ Select the best matching video from a list of videos. """
-        if const.args.manual:
-            log.info(self.raw_song)
-            log.info("0. Skip downloading this song.\n")
-            # fetch all video links on first page on YouTube
-            for i, v in enumerate(videos):
-                log.info(
-                    u"{0}. {1} {2} {3}".format(
-                        i + 1,
-                        v["title"],
-                        v["videotime"],
-                        "http://youtube.com/watch?v=" + v["link"],
-                    )
-                )
-            # let user select the song to download
-            result = internals.input_link(videos)
-            if result is None:
-                return None
-        else:
-            if not self.meta_tags:
-                # if the metadata could not be acquired, take the first result
-                # from Youtube because the proper song length is unknown
-                result = videos[0]
-                log.debug(
-                    "Since no metadata found on Spotify, going with the first result"
-                )
-            else:
-                # filter out videos that do not have a similar length to the Spotify song
-                duration_tolerance = 10
-                max_duration_tolerance = 20
-                possible_videos_by_duration = []
-
-                # start with a reasonable duration_tolerance, and increment duration_tolerance
-                # until one of the Youtube results falls within the correct duration or
-                # the duration_tolerance has reached the max_duration_tolerance
-                while len(possible_videos_by_duration) == 0:
-                    possible_videos_by_duration = list(
-                        filter(
-                            lambda x: abs(x["seconds"] - self.meta_tags["duration"])
-                            <= duration_tolerance,
-                            videos,
-                        )
-                    )
-                    duration_tolerance += 1
-                    if duration_tolerance > max_duration_tolerance:
-                        log.error(
-                            "{0} by {1} was not found.".format(
-                                self.meta_tags["name"],
-                                self.meta_tags["artists"][0]["name"],
-                            )
-                        )
-                        return None
-
-                result = possible_videos_by_duration[0]
-
-        if result:
-            url = "http://youtube.com/watch?v={0}".format(result["link"])
-        else:
-            url = None
-
-        return url
-
-    def scrape(self, bestmatch=True, tries_remaining=5):
-        """ Search and scrape YouTube to return a list of matching videos. """
-
-        # prevents an infinite loop but allows for a few retries
-        if tries_remaining == 0:
-            log.debug("No tries left. I quit.")
-            return
-
-        search_url = generate_search_url(self.search_query)
-        log.debug("Opening URL: {0}".format(search_url))
-
-        item = self._fetch_response(search_url).read()
-        items_parse = BeautifulSoup(item, "html.parser")
-
-        videos = []
-        for x in items_parse.find_all(
-            "div", {"class": "yt-lockup-dismissable yt-uix-tile"}
-        ):
-
-            if not is_video(x):
-                continue
-
-            y = x.find("div", class_="yt-lockup-content")
-            link = y.find("a")["href"][-11:]
-            title = y.find("a")["title"]
-
-            try:
-                videotime = x.find("span", class_="video-time").get_text()
-            except AttributeError:
-                log.debug("Could not find video duration on YouTube, retrying..")
-                return self.scrape(
-                    bestmatch=bestmatch, tries_remaining=tries_remaining - 1
-                )
-
-            youtubedetails = {
-                "link": link,
-                "title": title,
-                "videotime": videotime,
-                "seconds": internals.get_sec(videotime),
-            }
-            videos.append(youtubedetails)
-
-        if bestmatch:
-            return self._best_match(videos)
-
-        return videos
-
-    def api(self, bestmatch=True):
-        """ Use YouTube API to search and return a list of matching videos. """
-
-        query = {"part": "snippet", "maxResults": 50, "type": "video"}
-
-        if const.args.music_videos_only:
-            query["videoCategoryId"] = "10"
-
-        if not self.meta_tags:
-            song = self.raw_song
-            query["q"] = song
-        else:
-            query["q"] = self.search_query
-        log.debug("query: {0}".format(query))
-
-        data = pafy.call_gdata("search", query)
-        data["items"] = list(
-            filter(lambda x: x["id"].get("videoId") is not None, data["items"])
-        )
-        query_results = {
-            "part": "contentDetails,snippet,statistics",
-            "maxResults": 50,
-            "id": ",".join(i["id"]["videoId"] for i in data["items"]),
-        }
-        log.debug("query_results: {0}".format(query_results))
-
-        vdata = pafy.call_gdata("videos", query_results)
-
-        videos = []
-        for x in vdata["items"]:
-            duration_s = pafy.playlist.parseISO8591(x["contentDetails"]["duration"])
-            youtubedetails = {
-                "link": x["id"],
-                "title": x["snippet"]["title"],
-                "videotime": internals.videotime_from_seconds(duration_s),
-                "seconds": duration_s,
-            }
-            videos.append(youtubedetails)
-
-        if bestmatch:
-            return self._best_match(videos)
-
-        return videos
-
-    @staticmethod
-    def _fetch_response(url):
-        # XXX: This method exists only because it helps us indirectly
-        # monkey patch `urllib.request.open`, directly monkey patching
-        # `urllib.request.open` causes us to end up in an infinite recursion
-        # during the test since `urllib.request.open` would monkeypatch itself.
-        return urllib.request.urlopen(url)

test/loader.py

@@ -1,26 +0,0 @@
-from spotdl import const
-from spotdl import handle
-from spotdl import spotdl
-import urllib
-
-import pytest
-
-
-def load_defaults():
-    const.args = handle.get_arguments(raw_args="", to_group=False, to_merge=False)
-    const.args.overwrite = "skip"
-
-    spotdl.args = const.args
-    spotdl.log = const.logzero.setup_logger(
-        formatter=const._formatter, level=const.args.log_level
-    )
-
-
-# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
-# so that we get same results even if YouTube changes the list/order of videos on their page.
-GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
-
-
-def monkeypatch_youtube_search_page(*args, **kwargs):
-    fake_urlopen = urllib.request.urlopen(GIST_URL)
-    return fake_urlopen

test/test_download_with_metadata.py

@@ -1,244 +0,0 @@
-import subprocess
-import os
-
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import convert
-from spotdl import metadata
-from spotdl import downloader
-
-import pytest
-import loader
-
-loader.load_defaults()
-
-SPOTIFY_TRACK_URL = "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD"
-EXPECTED_YOUTUBE_TITLE = "Janji - Heroes Tonight (feat. Johnning) [NCS Release]"
-EXPECTED_SPOTIFY_TITLE = "Janji - Heroes Tonight"
-EXPECTED_YOUTUBE_URL = "http://youtube.com/watch?v=3nQNiWdeH2Q"
-# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
-# so that we get same results even if YouTube changes the list/order of videos on their page.
-GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
-
-
-def pytest_namespace():
-    # XXX: We override the value of `content_fixture` later in the tests.
-    # We do not use an acutal @pytest.fixture because it does not accept
-    # the monkeypatch parameter and we need to monkeypatch the network
-    # request before creating the Pafy object.
-    return {"content_fixture": None}
-
-
-@pytest.fixture(scope="module")
-def metadata_fixture():
-    meta_tags = spotify_tools.generate_metadata(SPOTIFY_TRACK_URL)
-    return meta_tags
-
-
-def test_metadata(metadata_fixture):
-    expect_number = 24
-    assert len(metadata_fixture) == expect_number
-
-
-class TestFileFormat:
-    def test_with_spaces(self, metadata_fixture):
-        title = internals.format_string(const.args.file_format, metadata_fixture)
-        assert title == EXPECTED_SPOTIFY_TITLE
-
-    def test_without_spaces(self, metadata_fixture):
-        const.args.no_spaces = True
-        title = internals.format_string(const.args.file_format, metadata_fixture)
-        assert title == EXPECTED_SPOTIFY_TITLE.replace(" ", "_")
-
-
-def test_youtube_url(metadata_fixture, monkeypatch):
-    monkeypatch.setattr(
-        youtube_tools.GenerateYouTubeURL,
-        "_fetch_response",
-        loader.monkeypatch_youtube_search_page,
-    )
-    url = youtube_tools.generate_youtube_url(SPOTIFY_TRACK_URL, metadata_fixture)
-    assert url == EXPECTED_YOUTUBE_URL
-
-
-def test_youtube_title(metadata_fixture, monkeypatch):
-    monkeypatch.setattr(
-        youtube_tools.GenerateYouTubeURL,
-        "_fetch_response",
-        loader.monkeypatch_youtube_search_page,
-    )
-    content = youtube_tools.go_pafy(SPOTIFY_TRACK_URL, metadata_fixture)
-    pytest.content_fixture = content
-    title = youtube_tools.get_youtube_title(content)
-    assert title == EXPECTED_YOUTUBE_TITLE
-
-
-@pytest.fixture(scope="module")
-def filename_fixture(metadata_fixture):
-    songname = internals.format_string(const.args.file_format, metadata_fixture)
-    filename = internals.sanitize_title(songname)
-    return filename
-
-
-def test_check_track_exists_before_download(tmpdir, metadata_fixture, filename_fixture):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
-    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
-    assert check == expect_check
-
-
-class TestDownload:
-    def blank_audio_generator(self, filepath):
-        if filepath.endswith(".m4a"):
-            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a aac {}".format(filepath)
-        elif filepath.endswith(".webm"):
-            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a libopus {}".format(filepath)
-        subprocess.call(cmd.split(" "))
-
-    def test_m4a(self, monkeypatch, filename_fixture):
-        expect_download = True
-        monkeypatch.setattr(
-            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
-        )
-        monkeypatch.setattr(
-            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
-        )
-        download = youtube_tools.download_song(
-            filename_fixture + ".m4a", pytest.content_fixture
-        )
-        assert download == expect_download
-
-    def test_webm(self, monkeypatch, filename_fixture):
-        expect_download = True
-        monkeypatch.setattr(
-            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
-        )
-        monkeypatch.setattr(
-            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
-        )
-        download = youtube_tools.download_song(
-            filename_fixture + ".webm", pytest.content_fixture
-        )
-        assert download == expect_download
-
-
-class TestFFmpeg:
-    def test_convert_from_webm_to_mp3(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        monkeypatch.setattr("os.remove", lambda x: None)
-        _, command = convert.song(
-            filename_fixture + ".webm", filename_fixture + ".mp3", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-    def test_convert_from_webm_to_m4a(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -cutoff 20000 -codec:a aac -ar 44100 -b:a 192k -vn {0}.m4a".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        monkeypatch.setattr("os.remove", lambda x: None)
-        _, command = convert.song(
-            filename_fixture + ".webm", filename_fixture + ".m4a", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:v copy -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        monkeypatch.setattr("os.remove", lambda x: None)
-        _, command = convert.song(
-            filename_fixture + ".m4a", filename_fixture + ".mp3", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-    def test_convert_from_m4a_to_webm(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a libopus -vbr on -b:a 192k -vn {0}.webm".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        monkeypatch.setattr("os.remove", lambda x: None)
-        _, command = convert.song(
-            filename_fixture + ".m4a", filename_fixture + ".webm", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-    def test_convert_from_m4a_to_flac(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a flac -ar 44100 -b:a 192k -vn {0}.flac".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        monkeypatch.setattr("os.remove", lambda x: None)
-        _, command = convert.song(
-            filename_fixture + ".m4a", filename_fixture + ".flac", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-    def test_correct_container_for_m4a(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a.temp -acodec copy -b:a 192k -vn {0}.m4a".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        _, command = convert.song(
-            filename_fixture + ".m4a", filename_fixture + ".m4a", const.args.folder
-        )
-        assert " ".join(command) == expect_command
-
-
-class TestAvconv:
-    @pytest.mark.skip(reason="avconv is no longer provided with FFmpeg")
-    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
-        monkeypatch.setattr("os.remove", lambda x: None)
-        expect_command = "avconv -loglevel 0 -i {0}.m4a -ab 192k {0}.mp3 -y".format(
-            os.path.join(const.args.folder, filename_fixture)
-        )
-        _, command = convert.song(
-            filename_fixture + ".m4a",
-            filename_fixture + ".mp3",
-            const.args.folder,
-            avconv=True,
-        )
-        assert " ".join(command) == expect_command
-
-
-@pytest.fixture(scope="module")
-def trackpath_fixture(filename_fixture):
-    trackpath = os.path.join(const.args.folder, filename_fixture)
-    return trackpath
-
-
-class TestEmbedMetadata:
-    def test_embed_in_mp3(self, metadata_fixture, trackpath_fixture):
-        expect_embed = True
-        embed = metadata.embed(trackpath_fixture + ".mp3", metadata_fixture)
-        assert embed == expect_embed
-
-    def test_embed_in_m4a(self, metadata_fixture, trackpath_fixture):
-        expect_embed = True
-        embed = metadata.embed(trackpath_fixture + ".m4a", metadata_fixture)
-        os.remove(trackpath_fixture + ".m4a")
-        assert embed == expect_embed
-
-    def test_embed_in_webm(self, metadata_fixture, trackpath_fixture):
-        expect_embed = False
-        embed = metadata.embed(trackpath_fixture + ".webm", metadata_fixture)
-        os.remove(trackpath_fixture + ".webm")
-        assert embed == expect_embed
-
-    def test_embed_in_flac(self, metadata_fixture, trackpath_fixture):
-        expect_embed = True
-        embed = metadata.embed(trackpath_fixture + ".flac", metadata_fixture)
-        os.remove(trackpath_fixture + ".flac")
-        assert embed == expect_embed
-
-
-def test_check_track_exists_after_download(
-    metadata_fixture, filename_fixture, trackpath_fixture
-):
-    expect_check = True
-    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
-    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
-    os.remove(trackpath_fixture + ".mp3")
-    assert check == expect_check

test/test_dry_run.py

@@ -1,21 +0,0 @@
-import os
-
-from spotdl import const
-from spotdl import downloader
-
-import loader
-
-loader.load_defaults()
-
-TRACK_URL = "http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU"
-
-
-def test_dry_download_list(tmpdir):
-    const.args.folder = str(tmpdir)
-    const.args.dry_run = True
-    file_path = os.path.join(const.args.folder, "test_list.txt")
-    with open(file_path, "w") as f:
-        f.write(TRACK_URL)
-    list_dl = downloader.ListDownloader(file_path)
-    downloaded_song, *_ = list_dl.download_list()
-    assert downloaded_song == TRACK_URL

test/test_handle.py

@@ -1,70 +0,0 @@
-import os
-import sys
-import argparse
-
-from spotdl import handle
-
-import pytest
-import yaml
-
-
-def test_error_m3u_without_list():
-    with pytest.raises(SystemExit):
-        handle.get_arguments(raw_args=("-s cool song", "--write-m3u"), to_group=True)
-
-
-def test_m3u_with_list():
-    handle.get_arguments(raw_args=("-l cool_list.txt", "--write-m3u"), to_group=True)
-
-
-def test_log_str_to_int():
-    expect_levels = [20, 30, 40, 10]
-    levels = [handle.log_leveller(level) for level in handle._LOG_LEVELS_STR]
-    assert levels == expect_levels
-
-
-@pytest.fixture(scope="module")
-def config_path_fixture(tmpdir_factory):
-    config_path = os.path.join(str(tmpdir_factory.mktemp("config")), "config.yml")
-    return config_path
-
-
-@pytest.fixture(scope="module")
-def modified_config_fixture():
-    modified_config = dict(handle.default_conf)
-    return modified_config
-
-
-class TestConfig:
-    def test_default_config(self, config_path_fixture):
-        expect_config = handle.default_conf["spotify-downloader"]
-        config = handle.get_config(config_path_fixture)
-        assert config == expect_config
-
-    def test_modified_config(self, modified_config_fixture):
-        modified_config_fixture["spotify-downloader"]["file-format"] = "just_a_test"
-        merged_config = handle.merge(handle.default_conf, modified_config_fixture)
-        assert merged_config == modified_config_fixture
-
-    def test_custom_config_path(self, config_path_fixture, modified_config_fixture):
-        parser = argparse.ArgumentParser()
-        with open(config_path_fixture, "w") as config_file:
-            yaml.dump(modified_config_fixture, config_file, default_flow_style=False)
-        overridden_config = handle.override_config(
-            config_path_fixture, parser, raw_args=""
-        )
-        modified_values = [
-            str(value)
-            for value in modified_config_fixture["spotify-downloader"].values()
-        ]
-        overridden_config.folder = os.path.realpath(overridden_config.folder)
-        overridden_values = [
-            str(value) for value in overridden_config.__dict__.values()
-        ]
-        assert sorted(overridden_values) == sorted(modified_values)
-
-
-def test_grouped_arguments(tmpdir):
-    sys.path[0] = str(tmpdir)
-    with pytest.raises(SystemExit):
-        handle.get_arguments(to_group=True, to_merge=True)

test/test_internals.py

@@ -1,180 +0,0 @@
-import sys
-import os
-import subprocess
-
-from spotdl import internals
-
-import pytest
-
-
-DUPLICATE_TRACKS_TEST_TABLE = [
-    (
-        (
-            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
-            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
-        ),
-        ("https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",),
-    ),
-    (
-        (
-            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
-            "",
-            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
-        ),
-        (
-            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
-            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
-        ),
-    ),
-    (
-        (
-            "ncs fade",
-            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
-            "",
-            "ncs fade",
-        ),
-        ("ncs fade", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
-    ),
-    (
-        ("ncs spectre ", "  https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ", ""),
-        ("ncs spectre", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
-    ),
-]
-
-STRING_IDS_TEST_TABLE = [
-    (
-        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8?si=_cVm-FBRQmi7VWML7E49Ig",
-        "1feoGrmmD8QmNqtK2Gdwy8",
-    ),
-    (
-        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8",
-        "1feoGrmmD8QmNqtK2Gdwy8",
-    ),
-    ("spotify:artist:1feoGrmmD8QmNqtK2Gdwy8", "1feoGrmmD8QmNqtK2Gdwy8"),
-    (
-        "https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp?si=LkVQLJGGT--Lh8BWM4MGvg",
-        "1d1l3UkeAjtM7kVTDyR8yp",
-    ),
-    ("https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
-    ("spotify:album:1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
-    (
-        "https://open.spotify.com/user/5kkyy50uu8btnagp30pobxz2f/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
-        "3SFKRjUXm0IMQJMkEgPHeY",
-    ),
-    (
-        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
-        "3SFKRjUXm0IMQJMkEgPHeY",
-    ),
-    (
-        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY",
-        "3SFKRjUXm0IMQJMkEgPHeY",
-    ),
-    (
-        "spotify:user:5kkyy50uu8btnagp30pobxz2f:playlist:3SFKRjUXm0IMQJMkEgPHeY",
-        "3SFKRjUXm0IMQJMkEgPHeY",
-    ),
-    (
-        "https://open.spotify.com/user/uqlakumu7wslkoen46s5bulq0",
-        "uqlakumu7wslkoen46s5bulq0",
-    ),
-]
-
-
-FROM_SECONDS_TEST_TABLE = [
-    (35, "35"),
-    (23, "23"),
-    (158, "2:38"),
-    (263, "4:23"),
-    (4562, "1:16:02"),
-    (26762, "7:26:02"),
-]
-
-
-TO_SECONDS_TEST_TABLE = [
-    ("0:23", 23),
-    ("0:45", 45),
-    ("2:19", 139),
-    ("3:33", 213),
-    ("7:38", 458),
-    ("1:30:05", 5405),
-]
-
-
-def test_default_music_directory():
-    if sys.platform.startswith("linux"):
-        output = subprocess.check_output(["xdg-user-dir", "MUSIC"])
-        expect_directory = output.decode("utf-8").rstrip()
-    else:
-        home = os.path.expanduser("~")
-        expect_directory = os.path.join(home, "Music")
-
-    directory = internals.get_music_dir()
-    assert directory == expect_directory
-
-
-@pytest.fixture(scope="module")
-def directory_fixture(tmpdir_factory):
-    dir_path = os.path.join(str(tmpdir_factory.mktemp("tmpdir")), "filter_this_folder")
-    return dir_path
-
-
-class TestPathFilterer:
-    def test_create_directory(self, directory_fixture):
-        expect_path = True
-        internals.filter_path(directory_fixture)
-        is_path = os.path.isdir(directory_fixture)
-        assert is_path == expect_path
-
-    def test_remove_temp_files(self, directory_fixture):
-        expect_file = False
-        file_path = os.path.join(directory_fixture, "pesky_file.temp")
-        open(file_path, "a")
-        internals.filter_path(directory_fixture)
-        is_file = os.path.isfile(file_path)
-        assert is_file == expect_file
-
-
-@pytest.mark.parametrize("sec_duration, str_duration", FROM_SECONDS_TEST_TABLE)
-def test_video_time_from_seconds(sec_duration, str_duration):
-    duration = internals.videotime_from_seconds(sec_duration)
-    assert duration == str_duration
-
-
-@pytest.mark.parametrize("str_duration, sec_duration", TO_SECONDS_TEST_TABLE)
-def test_get_seconds_from_video_time(str_duration, sec_duration):
-    secs = internals.get_sec(str_duration)
-    assert secs == sec_duration
-
-
-@pytest.mark.parametrize("duplicates, expected", DUPLICATE_TRACKS_TEST_TABLE)
-def test_get_unique_tracks(tmpdir, duplicates, expected):
-    file_path = os.path.join(str(tmpdir), "test_duplicates.txt")
-    with open(file_path, "w") as f:
-        f.write("\n".join(duplicates))
-
-    unique_tracks = internals.get_unique_tracks(file_path)
-    assert tuple(unique_tracks) == expected
-
-
-@pytest.mark.parametrize("input_str, expected_spotify_id", STRING_IDS_TEST_TABLE)
-def test_extract_spotify_id(input_str, expected_spotify_id):
-    spotify_id = internals.extract_spotify_id(input_str)
-    assert spotify_id == expected_spotify_id
-
-
-def test_trim(tmpdir):
-    text_file = os.path.join(str(tmpdir), "test_trim.txt")
-    with open(text_file, "w") as track_file:
-        track_file.write("ncs - spectre\nncs - heroes\nncs - hope")
-
-    with open(text_file, "r") as track_file:
-        tracks = track_file.readlines()
-
-    expect_number = len(tracks) - 1
-    expect_track = tracks[0]
-    track = internals.trim_song(text_file)
-
-    with open(text_file, "r") as track_file:
-        number = len(track_file.readlines())
-
-    assert expect_number == number and expect_track == track

test/test_patcher.py

@@ -1,36 +0,0 @@
-from spotdl import patcher
-import pafy
-
-import pytest
-
-pafy_patcher = patcher.PatchPafy()
-pafy_patcher.patch_getbestthumb()
-
-
-class TestPafyContentAvailable:
-    pass
-
-
-class TestMethodAssignment:
-    def test_pafy_getbestthumb(self):
-        pafy.backend_shared.BasePafy.getbestthumb == patcher._getbestthumb
-
-
-class TestMethodCalls:
-    @pytest.fixture(scope="module")
-    def content_fixture(self):
-        content = pafy.new("http://youtube.com/watch?v=3nQNiWdeH2Q")
-        return content
-
-    def test_pafy_getbestthumb(self, content_fixture):
-        thumbnail = patcher._getbestthumb(content_fixture)
-        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/hqdefault.jpg"
-
-    def test_pafy_getbestthumb_without_ytdl(self, content_fixture):
-        content_fixture._ydl_info["thumbnails"][0]["url"] = None
-        thumbnail = patcher._getbestthumb(content_fixture)
-        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/sddefault.jpg"
-
-    def test_pafy_content_available(self):
-        TestPafyContentAvailable._content_available = patcher._content_available
-        assert TestPafyContentAvailable()._content_available("https://youtube.com/")

test/test_spotify_tools.py

@@ -1,178 +0,0 @@
-from spotdl import spotify_tools
-from spotdl import const
-
-import spotipy
-
-import os
-import pytest
-import loader
-
-loader.load_defaults()
-
-
-def test_generate_token():
-    token = spotify_tools.generate_token()
-    assert len(token) == 83
-
-
-class TestMustBeAuthorizedDecorator:
-    def test_spotify_instance_is_unset(self):
-        spotify_tools.spotify = None
-
-        @spotify_tools.must_be_authorized
-        def sample_func():
-            return True
-
-        assert sample_func()
-
-    def test_spotify_instance_forces_assertion_error(self):
-        @spotify_tools.must_be_authorized
-        def sample_func():
-            raise AssertionError
-
-        with pytest.raises(AssertionError):
-            sample_func()
-
-    def test_fake_token_generator(self, monkeypatch):
-        spotify_tools.spotify = None
-        monkeypatch.setattr(spotify_tools, "generate_token", lambda: 123123)
-
-        with pytest.raises(spotipy.client.SpotifyException):
-            spotify_tools.generate_metadata("ncs - spectre")
-
-    def test_correct_token(self):
-        assert spotify_tools.generate_metadata("ncs - spectre")
-
-
-class TestGenerateMetadata:
-    @pytest.fixture(scope="module")
-    def metadata_fixture(self):
-        metadata = spotify_tools.generate_metadata("ncs - spectre")
-        return metadata
-
-    def test_len(self, metadata_fixture):
-        assert len(metadata_fixture) == 24
-
-    def test_trackname(self, metadata_fixture):
-        assert metadata_fixture["name"] == "Spectre"
-
-    def test_artist(self, metadata_fixture):
-        assert metadata_fixture["artists"][0]["name"] == "Alan Walker"
-
-    def test_duration(self, metadata_fixture):
-        assert metadata_fixture["duration"] == 230.634
-
-
-def test_get_playlists():
-    expect_playlist_ids = [
-        "34gWCK8gVeYDPKcctB6BQJ",
-        "04wTU2c2WNQG9XE5oSLYfj",
-        "0fWBMhGh38y0wsYWwmM9Kt",
-    ]
-
-    expect_playlists = [
-        "https://open.spotify.com/playlist/" + playlist_id
-        for playlist_id in expect_playlist_ids
-    ]
-
-    playlists = spotify_tools.get_playlists("uqlakumu7wslkoen46s5bulq0")
-    assert playlists == expect_playlists
-
-
-def test_write_user_playlist(tmpdir, monkeypatch):
-    expect_tracks = 17
-    text_file = os.path.join(str(tmpdir), "test_us.txt")
-    monkeypatch.setattr("builtins.input", lambda x: 1)
-    spotify_tools.write_user_playlist("uqlakumu7wslkoen46s5bulq0", text_file)
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-class TestFetchPlaylist:
-    @pytest.fixture(scope="module")
-    def playlist_fixture(self):
-        playlist = spotify_tools.fetch_playlist(
-            "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt"
-        )
-        return playlist
-
-    def test_name(self, playlist_fixture):
-        assert playlist_fixture["name"] == "special_test_playlist"
-
-    def test_tracks(self, playlist_fixture):
-        assert playlist_fixture["tracks"]["total"] == 14
-
-
-def test_write_playlist(tmpdir):
-    expect_tracks = 14
-    text_file = os.path.join(str(tmpdir), "test_pl.txt")
-    spotify_tools.write_playlist(
-        "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt", text_file
-    )
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-# XXX: Monkeypatch these tests if they fail in future
-class TestFetchAlbum:
-    @pytest.fixture(scope="module")
-    def album_fixture(self):
-        album = spotify_tools.fetch_album(
-            "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg"
-        )
-        return album
-
-    def test_name(self, album_fixture):
-        assert album_fixture["name"] == "NCS: Infinity"
-
-    def test_tracks(self, album_fixture):
-        assert album_fixture["tracks"]["total"] == 15
-
-
-# XXX: Monkeypatch these tests if they fail in future
-class TestFetchAlbumsFromArtist:
-    @pytest.fixture(scope="module")
-    def albums_from_artist_fixture(self):
-        albums = spotify_tools.fetch_albums_from_artist(
-            "https://open.spotify.com/artist/7oPftvlwr6VrsViSDV7fJY"
-        )
-        return albums
-
-    def test_len(self, albums_from_artist_fixture):
-        assert len(albums_from_artist_fixture) == 54
-
-    def test_zeroth_album_name(self, albums_from_artist_fixture):
-        assert albums_from_artist_fixture[0]["name"] == "Revolution Radio"
-
-    def test_zeroth_album_tracks(self, albums_from_artist_fixture):
-        assert albums_from_artist_fixture[0]["total_tracks"] == 12
-
-    def test_fist_album_name(self, albums_from_artist_fixture):
-        assert albums_from_artist_fixture[1]["name"] == "Demolicious"
-
-    def test_first_album_tracks(self, albums_from_artist_fixture):
-        assert albums_from_artist_fixture[0]["total_tracks"] == 12
-
-
-def test_write_all_albums_from_artist(tmpdir):
-    expect_tracks = 282
-    text_file = os.path.join(str(tmpdir), "test_ab.txt")
-    spotify_tools.write_all_albums_from_artist(
-        "https://open.spotify.com/artist/4dpARuHxo51G3z768sgnrY", text_file
-    )
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-def test_write_album(tmpdir):
-    expect_tracks = 15
-    text_file = os.path.join(str(tmpdir), "test_al.txt")
-    spotify_tools.write_album(
-        "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg", text_file
-    )
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks

test/test_youtube_tools.py

@@ -1,243 +0,0 @@
-import os
-import builtins
-
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import downloader
-
-import loader
-import pytest
-
-loader.load_defaults()
-
-YT_API_KEY = "AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90"
-
-TRACK_SEARCH = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
-EXPECTED_TITLE = TRACK_SEARCH
-EXPECTED_YT_URL = "http://youtube.com/watch?v=qOOcy2-tmbk"
-
-RESULT_COUNT_SEARCH = "she is still sleeping SAO"
-
-EXPECTED_YT_API_KEY = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
-EXPECTED_YT_API_KEY_CUSTOM = "some_api_key"
-
-
-class TestYouTubeAPIKeys:
-    def test_custom(self):
-        const.args.youtube_api_key = EXPECTED_YT_API_KEY_CUSTOM
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == EXPECTED_YT_API_KEY_CUSTOM
-
-    def test_default(self):
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == EXPECTED_YT_API_KEY
-
-
-@pytest.fixture(scope="module")
-def metadata_fixture():
-    metadata = spotify_tools.generate_metadata(TRACK_SEARCH)
-    return metadata
-
-
-def test_metadata(metadata_fixture):
-    expect_metadata = None
-    assert metadata_fixture == expect_metadata
-
-
-class TestArgsManualResultCount:
-    # Regresson test for issue #264
-    def test_scrape(self):
-        const.args.manual = True
-        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-        video_ids = url.scrape(bestmatch=False)
-        # Web scraping gives us all videos on the 1st page
-        assert len(video_ids) == 20
-
-    def test_api(self):
-        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-        video_ids = url.api(bestmatch=False)
-        const.args.manual = False
-        # API gives us 50 videos (or as requested)
-        assert len(video_ids) == 50
-
-
-class TestYouTubeURL:
-    def test_only_music_category(self, metadata_fixture):
-        const.args.music_videos_only = True
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
-        assert url == EXPECTED_YT_URL
-
-    def test_all_categories(self, metadata_fixture):
-        const.args.music_videos_only = False
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
-        assert url == EXPECTED_YT_URL
-
-    def test_args_manual(self, metadata_fixture, monkeypatch):
-        const.args.manual = True
-        monkeypatch.setattr("builtins.input", lambda x: "1")
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
-        assert url == EXPECTED_YT_URL
-
-    def test_args_manual_none(self, metadata_fixture, monkeypatch):
-        expect_url = None
-        monkeypatch.setattr("builtins.input", lambda x: "0")
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
-        const.args.manual = False
-        assert url == expect_url
-
-
-@pytest.fixture(scope="module")
-def content_fixture(metadata_fixture):
-    content = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
-    return content
-
-
-# True = Metadata must be fetched from Spotify
-# False = Metadata must be fetched from YouTube
-# None = Metadata must be `None`
-
-MATCH_METADATA_NO_FALLBACK_TEST_TABLE = [
-    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
-    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
-    ("Linux Talk | Working with Drives and Filesystems", None),
-]
-
-MATCH_METADATA_FALLBACK_TEST_TABLE = [
-    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
-    ("http://youtube.com/watch?v=3nQNiWdeH2Q", False),
-    ("Linux Talk | Working with Drives and Filesystems", False),
-]
-
-MATCH_METADATA_NO_METADATA_TEST_TABLE = [
-    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", None),
-    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
-    ("Linux Talk | Working with Drives and Filesystems", None),
-]
-
-
-class TestMetadataOrigin:
-    def match_metadata(self, track, metadata_type):
-        _, metadata = youtube_tools.match_video_and_metadata(track)
-        if metadata_type is None:
-            assert metadata == metadata_type
-        else:
-            assert metadata["spotify_metadata"] == metadata_type
-
-    @pytest.mark.parametrize(
-        "track, metadata_type", MATCH_METADATA_NO_FALLBACK_TEST_TABLE
-    )
-    def test_match_metadata_with_no_fallback(
-        self, track, metadata_type, content_fixture, monkeypatch
-    ):
-        monkeypatch.setattr(
-            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
-        )
-        const.args.no_fallback_metadata = True
-        self.match_metadata(track, metadata_type)
-
-    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_FALLBACK_TEST_TABLE)
-    def test_match_metadata_with_fallback(
-        self, track, metadata_type, content_fixture, monkeypatch
-    ):
-        monkeypatch.setattr(
-            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
-        )
-        const.args.no_fallback_metadata = False
-        self.match_metadata(track, metadata_type)
-
-    @pytest.mark.parametrize(
-        "track, metadata_type", MATCH_METADATA_NO_METADATA_TEST_TABLE
-    )
-    def test_match_metadata_with_no_metadata(
-        self, track, metadata_type, content_fixture, monkeypatch
-    ):
-        monkeypatch.setattr(
-            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
-        )
-        const.args.no_metadata = True
-        self.match_metadata(track, metadata_type)
-
-
-@pytest.fixture(scope="module")
-def title_fixture(content_fixture):
-    title = youtube_tools.get_youtube_title(content_fixture)
-    return title
-
-
-class TestYouTubeTitle:
-    def test_single_download_with_youtube_api(self, title_fixture):
-        const.args.youtube_api_key = YT_API_KEY
-        youtube_tools.set_api_key()
-        assert title_fixture == EXPECTED_TITLE
-
-    def test_download_from_list_without_youtube_api(
-        self, metadata_fixture, content_fixture
-    ):
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        content_fixture = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
-        title = youtube_tools.get_youtube_title(content_fixture, 1)
-        assert title == "1. {0}".format(EXPECTED_TITLE)
-
-
-@pytest.fixture(scope="module")
-def filename_fixture(title_fixture):
-    filename = internals.sanitize_title(title_fixture)
-    return filename
-
-
-def test_check_exists(metadata_fixture, filename_fixture, tmpdir):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
-    check = track_existence.already_exists(TRACK_SEARCH)
-    assert check == expect_check
-
-
-def test_generate_m3u(tmpdir, monkeypatch):
-    monkeypatch.setattr(
-        youtube_tools.GenerateYouTubeURL,
-        "_fetch_response",
-        loader.monkeypatch_youtube_search_page,
-    )
-    expect_m3u = (
-        "#EXTM3U\n\n"
-        "#EXTINF:208,Janji - Heroes Tonight (feat. Johnning) [NCS Release]\n"
-        "http://www.youtube.com/watch?v=3nQNiWdeH2Q\n"
-        "#EXTINF:226,Alan Walker - Spectre [NCS Release]\n"
-        "http://www.youtube.com/watch?v=AOeY-nDp7hI\n"
-    )
-    m3u_track_file = os.path.join(str(tmpdir), "m3u_test.txt")
-    with open(m3u_track_file, "w") as track_file:
-        track_file.write("\nhttps://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD")
-        track_file.write("\nhttp://www.youtube.com/watch?v=AOeY-nDp7hI")
-    youtube_tools.generate_m3u(m3u_track_file)
-    m3u_file = "{}.m3u".format(m3u_track_file.split(".")[0])
-    with open(m3u_file, "r") as m3u_in:
-        m3u = m3u_in.readlines()
-    assert "".join(m3u) == expect_m3u
-
-
-class TestDownload:
-    def test_webm(self, content_fixture, filename_fixture, monkeypatch):
-        # content_fixture does not have any .webm audiostream
-        expect_download = False
-        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
-        download = youtube_tools.download_song(
-            filename_fixture + ".webm", content_fixture
-        )
-        assert download == expect_download
-
-    def test_other(self, content_fixture, filename_fixture, monkeypatch):
-        expect_download = False
-        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
-        download = youtube_tools.download_song(
-            filename_fixture + ".fake_extension", content_fixture
-        )
-        assert download == expect_download