Merge pull request #558 from ritiek/pafy-prefer-secure-by-default

Pafy prefer secure HTTPS by default

.gitignore

@@ -1,8 +1,10 @@
+# Spotdl generated files
+*.m4a
+*.mp3
 config.yml
 Music/
 *.txt
 *.m3u
-upload.sh
 .pytest_cache/
 
 

.travis.yml

@@ -1,9 +1,11 @@
-dist: trusty
+dist: xenial
 language: python
+sudo: required
 python:
   - "3.4"
   - "3.5"
   - "3.6"
+  - "3.7"
 before_install:
   - pip install tinydownload
   - pip install pytest-cov

CHANGES.md

@@ -5,8 +5,66 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+-
 
-*Please keep this section up to date when creating a PR!*
+### Fixed
+-
+
+### Changed
+-
+
+## [1.2.2] - 2019-06-03
+### Fixed
+- Patch bug in Pafy to prefer secure HTTPS ([@ritiek](https://github.com/ritiek)) (#558)
+
+## [1.2.1] - 2019-04-28
+### Fixed
+- Patch bug in Pafy when fetching audiostreams with latest youtube-dl ([@ritiek](https://github.com/ritiek)) (#539)
+
+### Changed
+- Removed duplicate debug log entry from `internals.trim_song` ([@ritiek](https://github.com/ritiek)) (#519)
+- Fix YAMLLoadWarning ([@cyberboysumanjay](https://github.com/cyberboysumanjay)) (#517)
+
+## [1.2.0] - 2019-03-01
+### Added
+- `--write-to` parameter for setting custom file to write Spotify track URLs to ([@ritiek](https://github.com/ritiek)) (#507)
+- Set custom Spotify Client ID and Client Secret via config.yml ([@ManveerBasra](https://github.com/ManveerBasra)) (#502)
+- Use YouTube as fallback metadata if track not found on Spotify. Also added `--no-fallback-metadata`
+  to preserve old behaviour ([@ritiek](https://github.com/ritiek)) (#457)
+
+### Fixed
+- Fix already downloaded prompt when using "/" in `--file-format` to create sub-directories ([@ritiek](https://github.com/ritiek)) (#503)
+- Fix writing playlist tracks to file ([@ritiek](https://github.com/ritiek)) (#506)
+
+## [1.1.2] - 2019-02-10
+### Changed
+- Fetch all artist albums by default instead of only fetching the "album" type ([@ritiek](https://github.com/ritiek)) (#493)
+- Option `-f` (`--folder`) is used when exporting text files using `-p` (`--playlist`) for playlists or `-b` (`--album`) for albums ([@Silverfeelin](https://github.com/Silverfeelin)) (#476)
+- Use first artist from album object for album artist ([@tillhainbach](https://github.com/tillhainbach))
+
+### Fixed
+- Fix renaming files when encoder is not found ([@ritiek](https://github.com/ritiek)) (#475)
+- Add missing `import time` ([@ifduyue](https://github.com/ifduyue)) (#465)
+
+## [1.1.1] - 2019-01-03
+### Added
+- Output informative message in case of no result found in YouTube search ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Ability to pass multiple tracks with `-s` option ([@ritiek](https://github.com/ritiek)) (#442)
+
+### Changed
+- Allowed to fetch metadata from Spotify upon searching Spotify-URL and  `--no-metadata` to gather YouTube custom-search fields ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Change FFmpeg to use the built-in encoder `aac` instead of 3rd party `libfdk-aac` which does not
+  ship with the apt package ([@ritiek](https://github.com/ritiek)) (#448)
+- Monkeypatch ever-changing network-relying tests ([@ritiek](https://github.com/ritiek)) (#448)
+- Correct `.m4a` container before writing metadata so metadata fields shows up properly in
+  media players (especially iTunes) ([@ritiek](https://github.com/ritiek) with thanks to [@Amit-L](https://github.com/Amit-L)!) (#453)
+- Refactored core downloading module ([@ritiek](https://github.com/ritiek)) (#410)
+
+### Fixed
+- Workaround conversion conflicts when input and output filename are same ([@ritiek](https://github.com/ritiek)) (#459)
+- Applied a check on result in case of search using Spotify-URL  `--no-metadata` option ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Included a missing `import spotipy` in downloader.py ([@ritiek](https://github.com/ritiek)) (#440)
 
 ## [1.1.0] - 2018-11-13
 ### Added

ISSUE_TEMPLATE.md

@@ -1,13 +1,15 @@
 <!--
 Please follow the guide below
 
-- You will be asked some questions and requested to provide some information, please read them CAREFULLY and answer honestly
-- Put an `x` into all the boxes [ ] relevant to your *issue* (like that [x])
-- Use *Preview* tab to see how your issue will actually look like
+- Before opening your ticket, make sure you either installed the latest release from PyPI
+  or installed directly from the master branch and have searched through existing issues
+  including closed ones.
 -->
 
-- [ ] Using latest version as provided on the [master branch](https://github.com/ritiek/spotify-downloader/tree/master)
-- [ ] [Searched](https://github.com/ritiek/spotify-downloader/issues?utf8=%E2%9C%93&q=is%3Aissue) for similar issues including closed ones
+<!--
+- Put an `x` into the box [ ] below (like [x]) depending on the purpose of your issue
+- Use *Preview* tab to see how your issue will actually look like
+-->
 
 #### What is the purpose of your *issue*?
 - [ ] Bug

README.md

@@ -21,7 +21,7 @@ Below is how your music library will look!
 
 ## Installation
 
-:heavy_exclamation_mark: **This tool works only with Python 3.**
+❗️ **This tool works only with Python 3.**
 
 Python 2 compatibility was dropped because of the way it deals with unicode (2020 is coming soon too).
 If you still need to use Python 2 - check out the (outdated)
@@ -30,7 +30,7 @@ If you still need to use Python 2 - check out the (outdated)
 spotify-downloader works with all major distributions and even on low-powered devices such as a Raspberry Pi.
 
 spotify-downloader can be installed via pip with:
-```
+```console
 $ pip3 install spotdl
 ```
 
@@ -41,7 +41,7 @@ page for detailed OS-specific instructions to get it and other dependencies it r
 
 For the most basic usage, downloading tracks is as easy as
 
-```
+```console
 $ spotdl --song https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ
 $ spotdl --song "ncs - spectre"
 ```
@@ -49,7 +49,7 @@ $ spotdl --song "ncs - spectre"
 For downloading playlist and albums, you need to first load all the tracks into text file and then pass
 this text file to `--list` argument. Here is how you would do it for a playlist
 
-```
+```console
 $ spotdl --playlist https://open.spotify.com/user/nocopyrightsounds/playlist/7sZbq8QGyMnhKPcLJvCUFD
 INFO: Writing 62 tracks to ncs-releases.txt
 $ spotdl --list ncs-releases.txt
@@ -65,11 +65,7 @@ contains detailed information about different available ways to download tracks.
 
 ## FAQ
 
-- [
-How to specify a custom folder where tracks should be downloaded?](https://github.com/ritiek/spotify-downloader/wiki/FAQ#how-to-specify-a-custom-folder-where-tracks-should-be-downloaded)
-
-Check out our [FAQ wiki page](https://github.com/ritiek/spotify-downloader/wiki/FAQ)
-for more info.
+All FAQs will be mentioned in our [FAQ wiki page](https://github.com/ritiek/spotify-downloader/wiki/FAQ).
 
 ## Contributing
 
@@ -77,7 +73,7 @@ Check out [CONTRIBUTING.md](CONTRIBUTING.md) for more info.
 
 ## Running Tests
 
-```
+```console
 $ python3 -m pytest test
 ```
 

setup.py

@@ -1,6 +1,6 @@
 from setuptools import setup
 
-with open("README.md", "r") as f:
+with open("README.md", "r", encoding="utf-8") as f:
     long_description = f.read()
 
 import spotdl

spotdl/__init__.py

@@ -1 +1 @@
-__version__ = "1.1.0"
+__version__ = "1.2.2"

spotdl/const.py

@@ -2,13 +2,16 @@ import logzero
 
 _log_format = "%(color)s%(levelname)s:%(end_color)s %(message)s"
 _formatter = logzero.LogFormatter(fmt=_log_format)
+_log_level = 0
 
 # Set up a temporary logger with default log level so that
 # it can be used before log level argument is determined
-logzero.setup_default_logger(formatter=_formatter)
+logzero.setup_default_logger(formatter=_formatter, level=_log_level)
 
-# options
-args = None
+# Options
+# Initialize an empty object which can be assigned attributes
+# (useful when using spotdl as a library)
+args = type("", (), {})()
 
 # Apple has specific tags - see mutagen docs -
 # http://mutagen.readthedocs.io/en/latest/api/mp4.html

spotdl/convert.py

@@ -3,7 +3,8 @@ import os
 from logzero import logger as log
 
 
-"""What are the differences and similarities between ffmpeg, libav, and avconv?
+"""
+What are the differences and similarities between ffmpeg, libav, and avconv?
 https://stackoverflow.com/questions/9477115
 
 ffmeg encoders high to lower quality
@@ -17,22 +18,48 @@ https://trac.ffmpeg.org/wiki/Encode/AAC
 
 def song(input_song, output_song, folder, avconv=False, trim_silence=False):
     """ Do the audio format conversion. """
-    if input_song == output_song:
-        return 0
-    convert = Converter(input_song, output_song, folder, trim_silence)
-    log.info("Converting {0} to {1}".format(input_song, output_song.split(".")[-1]))
-    if avconv:
-        exit_code = convert.with_avconv()
+    if avconv and trim_silence:
+        raise ValueError("avconv does not support trim_silence")
+
+    if not input_song == output_song:
+        log.info("Converting {0} to {1}".format(input_song, output_song.split(".")[-1]))
+    elif input_song.endswith(".m4a"):
+        log.info('Correcting container in "{}"'.format(input_song))
     else:
-        exit_code = convert.with_ffmpeg()
-    return exit_code
+        return 0
+
+    convert = Converter(input_song, output_song, folder, delete_original=True)
+    if avconv:
+        exit_code, command = convert.with_avconv()
+    else:
+        exit_code, command = convert.with_ffmpeg(trim_silence=trim_silence)
+    return exit_code, command
 
 
 class Converter:
-    def __init__(self, input_song, output_song, folder, trim_silence=False):
-        self.input_file = os.path.join(folder, input_song)
+    def __init__(self, input_song, output_song, folder, delete_original):
+        _, self.input_ext = os.path.splitext(input_song)
+        _, self.output_ext = os.path.splitext(output_song)
+
         self.output_file = os.path.join(folder, output_song)
-        self.trim_silence = trim_silence
+        rename_to_temp = False
+
+        same_file = os.path.abspath(input_song) == os.path.abspath(output_song)
+        if same_file:
+            # FFmpeg/avconv cannot have the same file for both input and output
+            # This would happen when the extensions are same, so rename
+            # the input track to append ".temp"
+            log.debug(
+                'Input file and output file are going will be same during encoding, will append ".temp" to input file just before starting encoding to avoid conflict'
+            )
+            input_song = output_song + ".temp"
+            rename_to_temp = True
+            delete_original = True
+
+        self.input_file = os.path.join(folder, input_song)
+
+        self.rename_to_temp = rename_to_temp
+        self.delete_original = delete_original
 
     def with_avconv(self):
         if log.level == 10:
@@ -52,43 +79,53 @@ class Converter:
             "-y",
         ]
 
-        if self.trim_silence:
-            log.warning("--trim-silence not supported with avconv")
+        if self.rename_to_temp:
+            os.rename(self.output_file, self.input_file)
 
         log.debug(command)
-        return subprocess.call(command)
+        try:
+            code = subprocess.call(command)
+        except FileNotFoundError:
+            if self.rename_to_temp:
+                os.rename(self.input_file, self.output_file)
+            raise
 
-    def with_ffmpeg(self):
+        if self.delete_original:
+            log.debug('Removing original file: "{}"'.format(self.input_file))
+            os.remove(self.input_file)
+
+        return code, command
+
+    def with_ffmpeg(self, trim_silence=False):
         ffmpeg_pre = "ffmpeg -y "
 
         if not log.level == 10:
             ffmpeg_pre += "-hide_banner -nostats -v panic "
 
-        _, input_ext = os.path.splitext(self.input_file)
-        _, output_ext = os.path.splitext(self.output_file)
-
         ffmpeg_params = ""
 
-        if input_ext == ".m4a":
-            if output_ext == ".mp3":
+        if self.input_ext == ".m4a":
+            if self.output_ext == ".mp3":
                 ffmpeg_params = "-codec:v copy -codec:a libmp3lame -ar 44100 "
-            elif output_ext == ".webm":
+            elif self.output_ext == ".webm":
                 ffmpeg_params = "-codec:a libopus -vbr on "
+            elif self.output_ext == ".m4a":
+                ffmpeg_params = "-acodec copy "
 
-        elif input_ext == ".webm":
-            if output_ext == ".mp3":
+        elif self.input_ext == ".webm":
+            if self.output_ext == ".mp3":
                 ffmpeg_params = "-codec:a libmp3lame -ar 44100 "
-            elif output_ext == ".m4a":
-                ffmpeg_params = "-cutoff 20000 -codec:a libfdk_aac -ar 44100 "
+            elif self.output_ext == ".m4a":
+                ffmpeg_params = "-cutoff 20000 -codec:a aac -ar 44100 "
 
-        if output_ext == ".flac":
+        if self.output_ext == ".flac":
             ffmpeg_params = "-codec:a flac -ar 44100 "
 
         # add common params for any of the above combination
         ffmpeg_params += "-b:a 192k -vn "
-        ffmpeg_pre += " -i"
+        ffmpeg_pre += "-i "
 
-        if self.trim_silence:
+        if trim_silence:
             ffmpeg_params += "-af silenceremove=start_periods=1 "
 
         command = (
@@ -98,5 +135,19 @@ class Converter:
             + [self.output_file]
         )
 
+        if self.rename_to_temp:
+            os.rename(self.output_file, self.input_file)
+
         log.debug(command)
-        return subprocess.call(command)
+        try:
+            code = subprocess.call(command)
+        except FileNotFoundError:
+            if self.rename_to_temp:
+                os.rename(self.input_file, self.output_file)
+            raise
+
+        if self.delete_original:
+            log.debug('Removing original file: "{}"'.format(self.input_file))
+            os.remove(self.input_file)
+
+        return code, command

spotdl/downloader.py

@@ -0,0 +1,251 @@
+import spotipy
+import urllib
+import os
+import time
+from logzero import logger as log
+
+from spotdl import const
+from spotdl import metadata
+from spotdl import convert
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+
+
+class CheckExists:
+    def __init__(self, music_file, meta_tags=None):
+        self.meta_tags = meta_tags
+        basepath, filename = os.path.split(music_file)
+        filepath = os.path.join(const.args.folder, basepath)
+        os.makedirs(filepath, exist_ok=True)
+        self.filepath = filepath
+        self.filename = filename
+
+    def already_exists(self, raw_song):
+        """ Check if the input song already exists in the given folder. """
+        log.debug(
+            "Cleaning any temp files and checking "
+            'if "{}" already exists'.format(self.filename)
+        )
+        songs = os.listdir(self.filepath)
+        self._remove_temp_files(songs)
+
+        for song in songs:
+            # check if a song with the same name is already present in the given folder
+            if self._match_filenames(song):
+                if internals.is_spotify(raw_song) and not self._has_metadata(song):
+                    return False
+
+                log.warning('"{}" already exists'.format(song))
+                if const.args.overwrite == "prompt":
+                    return self._prompt_song(song)
+                elif const.args.overwrite == "force":
+                    return self._force_overwrite_song(song)
+                elif const.args.overwrite == "skip":
+                    return self._skip_song(song)
+
+        return False
+
+    def _remove_temp_files(self, songs):
+        for song in songs:
+            if song.endswith(".temp"):
+                os.remove(os.path.join(self.filepath, song))
+
+    def _has_metadata(self, song):
+        # check if the already downloaded song has correct metadata
+        # if not, remove it and download again without prompt
+        already_tagged = metadata.compare(
+            os.path.join(self.filepath, song), self.meta_tags
+        )
+        log.debug("Checking if it is already tagged correctly? {}", already_tagged)
+        if not already_tagged:
+            os.remove(os.path.join(self.filepath, song))
+            return False
+
+        return True
+
+    def _prompt_song(self, song):
+        log.info(
+            '"{}" has already been downloaded. ' "Re-download? (y/N): ".format(song)
+        )
+        prompt = input("> ")
+        if prompt.lower() == "y":
+            return self._force_overwrite_song(song)
+        else:
+            return self._skip_song(song)
+
+    def _force_overwrite_song(self, song):
+        os.remove(os.path.join(const.args.folder, song))
+        log.info('Overwriting "{}"'.format(song))
+        return False
+
+    def _skip_song(self, song):
+        log.info('Skipping "{}"'.format(song))
+        return True
+
+    def _match_filenames(self, song):
+        if os.path.splitext(song)[0] == self.filename:
+            log.debug('Found an already existing song: "{}"'.format(song))
+            return True
+
+        return False
+
+
+class Downloader:
+    def __init__(self, raw_song, number=None):
+        self.raw_song = raw_song
+        self.number = number
+        self.content, self.meta_tags = youtube_tools.match_video_and_metadata(raw_song)
+
+    def download_single(self):
+        """ Logic behind downloading a song. """
+
+        if self._to_skip():
+            return
+
+        # "[number]. [artist] - [song]" if downloading from list
+        # otherwise "[artist] - [song]"
+        youtube_title = youtube_tools.get_youtube_title(self.content, self.number)
+        log.info("{} ({})".format(youtube_title, self.content.watchv_url))
+
+        # generate file name of the song to download
+        songname = self.refine_songname(self.content.title)
+
+        if const.args.dry_run:
+            return
+
+        song_existence = CheckExists(songname, self.meta_tags)
+        if not song_existence.already_exists(self.raw_song):
+            return self._download_single(songname)
+
+    def _download_single(self, songname):
+        # deal with file formats containing slashes to non-existent directories
+        songpath = os.path.join(const.args.folder, os.path.dirname(songname))
+        os.makedirs(songpath, exist_ok=True)
+        input_song = songname + const.args.input_ext
+        output_song = songname + const.args.output_ext
+        if youtube_tools.download_song(input_song, self.content):
+            print("")
+            try:
+                convert.song(
+                    input_song,
+                    output_song,
+                    const.args.folder,
+                    avconv=const.args.avconv,
+                    trim_silence=const.args.trim_silence,
+                )
+            except FileNotFoundError:
+                encoder = "avconv" if const.args.avconv else "ffmpeg"
+                log.warning("Could not find {0}, skip encoding".format(encoder))
+                output_song = self.unconverted_filename(songname)
+
+            if not const.args.no_metadata and self.meta_tags is not None:
+                metadata.embed(
+                    os.path.join(const.args.folder, output_song), self.meta_tags
+                )
+            return True
+
+    def _to_skip(self):
+        if self.content is None:
+            log.debug("Found no matching video")
+            return True
+
+        if const.args.download_only_metadata and self.meta_tags is None:
+            log.info("Found no metadata. Skipping the download")
+            return True
+
+    def refine_songname(self, songname):
+        if self.meta_tags is not None:
+            refined_songname = internals.format_string(
+                const.args.file_format, self.meta_tags, slugification=True
+            )
+            log.debug(
+                'Refining songname from "{0}" to "{1}"'.format(
+                    songname, refined_songname
+                )
+            )
+            if not refined_songname == " - ":
+                songname = refined_songname
+        else:
+            songname = internals.sanitize_title(songname)
+
+        return songname
+
+    @staticmethod
+    def unconverted_filename(songname):
+        const.args.output_ext = const.args.input_ext
+        output_song = songname + const.args.output_ext
+        return output_song
+
+
+class ListDownloader:
+    def __init__(self, tracks_file, skip_file=None, write_successful_file=None):
+        self.tracks_file = tracks_file
+        self.skip_file = skip_file
+        self.write_successful_file = write_successful_file
+        self.tracks = internals.get_unique_tracks(self.tracks_file)
+
+    def download_list(self):
+        """ Download all songs from the list. """
+        # override file with unique tracks
+        log.info("Overriding {} with unique tracks".format(self.tracks_file))
+        self._override_file()
+
+        # Remove tracks to skip from tracks list
+        if self.skip_file is not None:
+            self.tracks = self._filter_tracks_against_skip_file()
+
+        log.info(u"Preparing to download {} songs".format(len(self.tracks)))
+        return self._download_list()
+
+    def _download_list(self):
+        downloaded_songs = []
+
+        for number, raw_song in enumerate(self.tracks, 1):
+            print("")
+            try:
+                track_dl = Downloader(raw_song, number=number)
+                track_dl.download_single()
+            except (urllib.request.URLError, TypeError, IOError) as e:
+                # detect network problems
+                self._cleanup(raw_song, e)
+                # TODO: remove this sleep once #397 is fixed
+                # wait 0.5 sec to avoid infinite looping
+                time.sleep(0.5)
+                continue
+
+            downloaded_songs.append(raw_song)
+            # Add track to file of successful downloads
+            if self.write_successful_file is not None:
+                self._write_successful(raw_song)
+
+            log.debug("Removing downloaded song from tracks file")
+            internals.trim_song(self.tracks_file)
+
+        return downloaded_songs
+
+    def _override_file(self):
+        with open(self.tracks_file, "w") as f:
+            f.write("\n".join(self.tracks))
+
+    def _write_successful(self, raw_song):
+        log.debug("Adding downloaded song to write successful file")
+        with open(self.write_successful_file, "a") as f:
+            f.write("\n" + raw_song)
+
+    def _cleanup(self, raw_song, exception):
+        self.tracks.append(raw_song)
+        # remove the downloaded song from file
+        internals.trim_song(self.tracks_file)
+        # and append it at the end of file
+        with open(self.tracks_file, "a") as f:
+            f.write("\n" + raw_song)
+        log.exception(exception)
+        log.warning("Failed to download song. Will retry after other songs\n")
+
+    def _filter_tracks_against_skip_file(self):
+        skip_tracks = internals.get_unique_tracks(self.skip_file)
+        len_before = len(self.tracks)
+        tracks = [track for track in self.tracks if track not in skip_tracks]
+        log.info("Skipping {} tracks".format(len_before - len(tracks)))
+        return tracks

spotdl/handle.py

@@ -1,14 +1,14 @@
-import appdirs
-from spotdl import internals, const
 from logzero import logger as log
+import appdirs
 
 import logging
 import yaml
 import argparse
 import mimetypes
-
 import os
-import sys
+
+import spotdl
+from spotdl import internals
 
 
 _LOG_LEVELS_STR = ["INFO", "WARNING", "ERROR", "DEBUG"]
@@ -17,11 +17,13 @@ default_conf = {
     "spotify-downloader": {
         "manual": False,
         "no-metadata": False,
+        "no-fallback-metadata": False,
         "avconv": False,
         "folder": internals.get_music_dir(),
         "overwrite": "prompt",
         "input-ext": ".m4a",
         "output-ext": ".mp3",
+        "write-to": None,
         "trim-silence": False,
         "download-only-metadata": False,
         "dry-run": False,
@@ -33,6 +35,8 @@ default_conf = {
         "skip": None,
         "write-successful": None,
         "log-level": "INFO",
+        "spotify_client_id": "4fe3fecfe5334023a1472516cc99d805",
+        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c"
     }
 }
 
@@ -54,7 +58,7 @@ def merge(default, config):
 def get_config(config_file):
     try:
         with open(config_file, "r") as ymlfile:
-            cfg = yaml.load(ymlfile)
+            cfg = yaml.safe_load(ymlfile)
     except FileNotFoundError:
         log.info("Writing default configuration to {0}:".format(config_file))
         with open(config_file, "w") as ymlfile:
@@ -100,7 +104,7 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         group = parser.add_mutually_exclusive_group(required=True)
 
         group.add_argument(
-            "-s", "--song", help="download track by spotify link or name"
+            "-s", "--song", nargs="+", help="download track by spotify link or name"
         )
         group.add_argument("-l", "--list", help="download tracks from a file")
         group.add_argument(
@@ -114,28 +118,25 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         group.add_argument(
             "-ab",
             "--all-albums",
-            help="load all tracks from artist URL into <artist_name>.txt"
+            help="load all tracks from artist URL into <artist_name>.txt",
         )
         group.add_argument(
             "-u",
             "--username",
             help="load tracks from user's playlist into <playlist_name>.txt",
         )
-        group.add_argument(
-            "-V", "--version", help="show version and exit", action="store_true"
-        )
 
     parser.add_argument(
-        '--write-m3u',
+        "--write-m3u",
         help="generate an .m3u playlist file with youtube links given "
-             "a text file containing tracks",
-        action='store_true'
+        "a text file containing tracks",
+        action="store_true",
     )
     parser.add_argument(
         "-m",
         "--manual",
         default=config["manual"],
-        help="choose the track to download manually from a list " "of matching tracks",
+        help="choose the track to download manually from a list of matching tracks",
         action="store_true",
     )
     parser.add_argument(
@@ -145,6 +146,13 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         help="do not embed metadata in tracks",
         action="store_true",
     )
+    parser.add_argument(
+        "-nf",
+        "--no-fallback-metadata",
+        default=config["no-fallback-metadata"],
+        help="do not use YouTube as fallback for metadata if track not found on Spotify",
+        action="store_true",
+    )
     parser.add_argument(
         "-a",
         "--avconv",
@@ -177,6 +185,11 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         default=config["output-ext"],
         help="preferred output format .mp3, .m4a (AAC), .flac, etc.",
     )
+    parser.add_argument(
+        "--write-to",
+        default=config["write-to"],
+        help="write tracks from Spotify playlist, album, etc. to this file",
+    )
     parser.add_argument(
         "-ff",
         "--file-format",
@@ -255,21 +268,55 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         default=config["write-successful"],
         help="path to file to write successful tracks to",
     )
+    parser.add_argument(
+        "-sci",
+        "--spotify-client-id",
+        default=config["spotify_client_id"],
+        help=argparse.SUPPRESS
+    )
+    parser.add_argument(
+        "-scs",
+        "--spotify-client-secret",
+        default=config["spotify_client_secret"],
+        help=argparse.SUPPRESS
+    )
     parser.add_argument(
         "-c", "--config", default=None, help="path to custom config.yml file"
     )
+    parser.add_argument(
+        "-V",
+        "--version",
+        action="version",
+        version="%(prog)s {}".format(spotdl.__version__),
+    )
 
     parsed = parser.parse_args(raw_args)
 
     if parsed.config is not None and to_merge:
         parsed = override_config(parsed.config, parser)
 
-    if to_group and parsed.list and \
-            not mimetypes.MimeTypes().guess_type(parsed.list)[0] == "text/plain":
-        parser.error("{0} is not of a valid argument to --list, argument must be plain text file".format(parsed.list))
+    if (
+        to_group
+        and parsed.list
+        and not mimetypes.MimeTypes().guess_type(parsed.list)[0] == "text/plain"
+    ):
+        parser.error(
+            "{0} is not of a valid argument to --list, argument must be plain text file".format(
+                parsed.list
+            )
+        )
 
     if parsed.write_m3u and not parsed.list:
-        parser.error('--write-m3u can only be used with --list')
+        parser.error("--write-m3u can only be used with --list")
+
+    if parsed.avconv and parsed.trim_silence:
+        parser.error("--trim-silence can only be used with FFmpeg")
+
+    if parsed.write_to and not (parsed.playlist \
+            or parsed.album \
+            or parsed.all_albums \
+            or parsed.username):
+        parser.error("--write-to can only be used with --playlist, --album, --all-albums, or --username")
 
     parsed.log_level = log_leveller(parsed.log_level)
 

spotdl/internals.py

@@ -1,6 +1,7 @@
+from logzero import logger as log
 import os
 import sys
-from logzero import logger as log
+import urllib.request
 
 from spotdl import const
 
@@ -48,11 +49,11 @@ def input_link(links):
             log.warning("Choose a valid number!")
 
 
-def trim_song(text_file):
+def trim_song(tracks_file):
     """ Remove the first song from file. """
-    with open(text_file, "r") as file_in:
+    with open(tracks_file, "r") as file_in:
         data = file_in.read().splitlines(True)
-    with open(text_file, "w") as file_out:
+    with open(tracks_file, "w") as file_out:
         file_out.writelines(data[1:])
     return data[0]
 
@@ -181,20 +182,23 @@ def extract_spotify_id(raw_string):
     return spotify_id
 
 
-def get_unique_tracks(text_file):
+def get_unique_tracks(tracks_file):
     """
     Returns a list of unique tracks given a path to a
     file containing tracks.
     """
 
-    with open(text_file, "r") as listed:
+    log.info(
+        "Checking and removing any duplicate tracks "
+        "in reading {}".format(tracks_file)
+    )
+    with open(tracks_file, "r") as tracks_in:
         # Read tracks into a list and remove any duplicates
-        lines = listed.read().splitlines()
+        lines = tracks_in.read().splitlines()
 
     # Remove blank and strip whitespaces from lines (if any)
     lines = [line.strip() for line in lines if line.strip()]
     lines = remove_duplicates(lines)
-
     return lines
 
 
@@ -249,3 +253,12 @@ def remove_duplicates(tracks):
     local_set = set()
     local_set_add = local_set.add
     return [x for x in tracks if not (x in local_set or local_set_add(x))]
+
+
+def content_available(url):
+    try:
+        response = urllib.request.urlopen(url)
+    except HTTPError:
+        return False
+    else:
+        return response.getcode() < 300

spotdl/metadata.py

@@ -2,10 +2,11 @@ from mutagen.easyid3 import EasyID3
 from mutagen.id3 import ID3, TORY, TYER, TPUB, APIC, USLT, COMM
 from mutagen.mp4 import MP4, MP4Cover
 from mutagen.flac import Picture, FLAC
-from logzero import logger as log
-from spotdl.const import TAG_PRESET, M4A_TAG_PRESET
 
 import urllib.request
+from logzero import logger as log
+
+from spotdl.const import TAG_PRESET, M4A_TAG_PRESET
 
 
 def compare(music_file, metadata):
@@ -45,6 +46,8 @@ class EmbedMetadata:
     def __init__(self, music_file, meta_tags):
         self.music_file = music_file
         self.meta_tags = meta_tags
+        self.spotify_metadata = meta_tags["spotify_metadata"]
+        self.provider = "spotify" if meta_tags["spotify_metadata"] else "youtube"
 
     def as_mp3(self):
         """ Embed metadata to MP3 files. """
@@ -61,7 +64,7 @@ class EmbedMetadata:
         audiofile["lyricist"] = meta_tags["artists"][0]["name"]
         audiofile["arranger"] = meta_tags["artists"][0]["name"]
         audiofile["performer"] = meta_tags["artists"][0]["name"]
-        audiofile["website"] = meta_tags["external_urls"]["spotify"]
+        audiofile["website"] = meta_tags["external_urls"][self.provider]
         audiofile["length"] = str(meta_tags["duration"])
         if meta_tags["publisher"]:
             audiofile["encodedby"] = meta_tags["publisher"]
@@ -77,7 +80,7 @@ class EmbedMetadata:
         audiofile["TYER"] = TYER(encoding=3, text=meta_tags["year"])
         if meta_tags["publisher"]:
             audiofile["TPUB"] = TPUB(encoding=3, text=meta_tags["publisher"])
-        audiofile["COMM"] = COMM(encoding=3, text=meta_tags["external_urls"]["spotify"])
+        audiofile["COMM"] = COMM(encoding=3, text=meta_tags["external_urls"][self.provider])
         if meta_tags["lyrics"]:
             audiofile["USLT"] = USLT(
                 encoding=3, desc=u"Lyrics", text=meta_tags["lyrics"]
@@ -105,7 +108,7 @@ class EmbedMetadata:
         audiofile = MP4(music_file)
         self._embed_basic_metadata(audiofile, preset=M4A_TAG_PRESET)
         audiofile[M4A_TAG_PRESET["year"]] = meta_tags["year"]
-        audiofile[M4A_TAG_PRESET["comment"]] = meta_tags["external_urls"]["spotify"]
+        audiofile[M4A_TAG_PRESET["comment"]] = meta_tags["external_urls"][self.provider]
         if meta_tags["lyrics"]:
             audiofile[M4A_TAG_PRESET["lyrics"]] = meta_tags["lyrics"]
         try:
@@ -126,7 +129,7 @@ class EmbedMetadata:
         audiofile = FLAC(music_file)
         self._embed_basic_metadata(audiofile)
         audiofile["year"] = meta_tags["year"]
-        audiofile["comment"] = meta_tags["external_urls"]["spotify"]
+        audiofile["comment"] = meta_tags["external_urls"][self.provider]
         if meta_tags["lyrics"]:
             audiofile["lyrics"] = meta_tags["lyrics"]
 
@@ -145,8 +148,10 @@ class EmbedMetadata:
     def _embed_basic_metadata(self, audiofile, preset=TAG_PRESET):
         meta_tags = self.meta_tags
         audiofile[preset["artist"]] = meta_tags["artists"][0]["name"]
-        audiofile[preset["albumartist"]] = meta_tags["artists"][0]["name"]
-        audiofile[preset["album"]] = meta_tags["album"]["name"]
+        if meta_tags["album"]["artists"][0]["name"]:
+            audiofile[preset["albumartist"]] = meta_tags["album"]["artists"][0]["name"]
+        if meta_tags["album"]["name"]:
+            audiofile[preset["album"]] = meta_tags["album"]["name"]
         audiofile[preset["title"]] = meta_tags["name"]
         audiofile[preset["date"]] = meta_tags["release_date"]
         audiofile[preset["originaldate"]] = meta_tags["release_date"]

spotdl/patcher.py

@@ -0,0 +1,55 @@
+from pafy import backend_youtube_dl
+import pafy
+
+from spotdl import internals
+
+
+def _getbestthumb(self):
+    url = self._ydl_info["thumbnails"][0]["url"]
+    if url:
+        return url
+
+    part_url = "https://i.ytimg.com/vi/%s/" % self.videoid
+    # Thumbnail resolution sorted in descending order
+    thumbs = ("maxresdefault.jpg",
+              "sddefault.jpg",
+              "hqdefault.jpg",
+              "mqdefault.jpg",
+              "default.jpg")
+    for thumb in thumbs:
+        url = part_url + thumb
+        if self._content_available(url):
+            return url
+
+def _process_streams(self):
+    for format_index in range(len(self._ydl_info['formats'])):
+        try:
+            self._ydl_info['formats'][format_index]['url'] = self._ydl_info['formats'][format_index]['fragment_base_url']
+        except KeyError:
+            pass
+    return backend_youtube_dl.YtdlPafy._old_process_streams(self)
+
+@classmethod
+def _content_available(cls, url):
+    return internals.content_available(url)
+
+
+class PatchPafy:
+    """
+    These patches have not been released by pafy on PyPI yet but
+    are useful to us.
+    """
+    def patch_getbestthumb(self):
+        # https://github.com/mps-youtube/pafy/pull/211
+        pafy.backend_shared.BasePafy._bestthumb = None
+        pafy.backend_shared.BasePafy._content_available = _content_available
+        pafy.backend_shared.BasePafy.getbestthumb = _getbestthumb
+
+    def patch_process_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/230
+        backend_youtube_dl.YtdlPafy._old_process_streams = backend_youtube_dl.YtdlPafy._process_streams
+        backend_youtube_dl.YtdlPafy._process_streams = _process_streams
+
+    def patch_insecure_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/235
+        pafy.g.def_ydl_opts["prefer_insecure"] = False

spotdl/spotdl.py

@@ -1,233 +1,66 @@
 #!/usr/bin/env python3
-# -*- coding: UTF-8 -*-
+
+import sys
+import platform
+import pprint
+import logzero
+from logzero import logger as log
+
 from spotdl import __version__
 from spotdl import const
 from spotdl import handle
-from spotdl import metadata
-from spotdl import convert
 from spotdl import internals
 from spotdl import spotify_tools
 from spotdl import youtube_tools
-from logzero import logger as log
-from slugify import slugify
-import spotipy
-import urllib.request
-import logzero
-import os
-import sys
-import time
-import platform
-import pprint
+from spotdl import downloader
 
 
-def check_exists(music_file, raw_song, meta_tags):
-    """ Check if the input song already exists in the given folder. """
-    log.debug(
-        "Cleaning any temp files and checking "
-        'if "{}" already exists'.format(music_file)
-    )
-    songs = os.listdir(const.args.folder)
-    for song in songs:
-        if song.endswith(".temp"):
-            os.remove(os.path.join(const.args.folder, song))
-            continue
-        # check if a song with the same name is already present in the given folder
-        if os.path.splitext(song)[0] == music_file:
-            log.debug('Found an already existing song: "{}"'.format(song))
-            if internals.is_spotify(raw_song):
-                # check if the already downloaded song has correct metadata
-                # if not, remove it and download again without prompt
-                already_tagged = metadata.compare(
-                    os.path.join(const.args.folder, song), meta_tags
-                )
-                log.debug(
-                    "Checking if it is already tagged correctly? {}", already_tagged
-                )
-                if not already_tagged:
-                    os.remove(os.path.join(const.args.folder, song))
-                    return False
-
-            log.warning('"{}" already exists'.format(song))
-            if const.args.overwrite == "prompt":
-                log.info(
-                    '"{}" has already been downloaded. '
-                    "Re-download? (y/N): ".format(song)
-                )
-                prompt = input("> ")
-                if prompt.lower() == "y":
-                    os.remove(os.path.join(const.args.folder, song))
-                    return False
-                else:
-                    return True
-            elif const.args.overwrite == "force":
-                os.remove(os.path.join(const.args.folder, song))
-                log.info('Overwriting "{}"'.format(song))
-                return False
-            elif const.args.overwrite == "skip":
-                log.info('Skipping "{}"'.format(song))
-                return True
-    return False
+def debug_sys_info():
+    log.debug("Python version: {}".format(sys.version))
+    log.debug("Platform: {}".format(platform.platform()))
+    log.debug(pprint.pformat(const.args.__dict__))
 
 
-def download_list(tracks_file, skip_file=None, write_successful_file=None):
-    """ Download all songs from the list. """
-
-    log.info("Checking and removing any duplicate tracks")
-    tracks = internals.get_unique_tracks(tracks_file)
-
-    # override file with unique tracks
-    with open(tracks_file, "w") as f:
-        f.write("\n".join(tracks))
-
-    # Remove tracks to skip from tracks list
-    if skip_file is not None:
-        skip_tracks = internals.get_unique_tracks(skip_file)
-        len_before = len(tracks)
-        tracks = [track for track in tracks if track not in skip_tracks]
-        log.info("Skipping {} tracks".format(len_before - len(tracks)))
-
-    log.info(u"Preparing to download {} songs".format(len(tracks)))
-    downloaded_songs = []
-
-    for number, raw_song in enumerate(tracks, 1):
-        print("")
-        try:
-            download_single(raw_song, number=number)
-        # token expires after 1 hour
-        except spotipy.client.SpotifyException:
-            # refresh token when it expires
-            log.debug("Token expired, generating new one and authorizing")
-            spotify_tools.refresh_token()
-            download_single(raw_song, number=number)
-        # detect network problems
-        except (urllib.request.URLError, TypeError, IOError) as e:
-            tracks.append(raw_song)
-            # remove the downloaded song from file
-            internals.trim_song(tracks_file)
-            # and append it at the end of file
-            with open(tracks_file, "a") as f:
-                f.write("\n" + raw_song)
-            log.exception(e)
-            log.warning("Failed to download song. Will retry after other songs\n")
-            # wait 0.5 sec to avoid infinite looping
-            time.sleep(0.5)
-            continue
-
-        downloaded_songs.append(raw_song)
-        # Add track to file of successful downloads
-        log.debug("Adding downloaded song to write successful file")
-        if write_successful_file is not None:
-            with open(write_successful_file, "a") as f:
-                f.write("\n" + raw_song)
-        log.debug("Removing downloaded song from tracks file")
-        internals.trim_song(tracks_file)
-
-    return downloaded_songs
-
-
-def download_single(raw_song, number=None):
-    """ Logic behind downloading a song. """
-    content, meta_tags = youtube_tools.match_video_and_metadata(raw_song)
-
-    if content is None:
-        log.debug("Found no matching video")
-        return
-
-    if const.args.download_only_metadata and meta_tags is None:
-        log.info("Found no metadata. Skipping the download")
-        return
-
-    # "[number]. [artist] - [song]" if downloading from list
-    # otherwise "[artist] - [song]"
-    youtube_title = youtube_tools.get_youtube_title(content, number)
-    log.info("{} ({})".format(youtube_title, content.watchv_url))
-
-    # generate file name of the song to download
-    songname = content.title
-
-    if meta_tags is not None:
-        refined_songname = internals.format_string(
-            const.args.file_format, meta_tags, slugification=True
-        )
-        log.debug(
-            'Refining songname from "{0}" to "{1}"'.format(songname, refined_songname)
-        )
-        if not refined_songname == " - ":
-            songname = refined_songname
-    else:
-        if not const.args.no_metadata:
-            log.warning("Could not find metadata")
-        songname = internals.sanitize_title(songname)
-
-    if const.args.dry_run:
-        return
-
-    if not check_exists(songname, raw_song, meta_tags):
-        # deal with file formats containing slashes to non-existent directories
-        songpath = os.path.join(const.args.folder, os.path.dirname(songname))
-        os.makedirs(songpath, exist_ok=True)
-        input_song = songname + const.args.input_ext
-        output_song = songname + const.args.output_ext
-        if youtube_tools.download_song(input_song, content):
-            print("")
-            try:
-                convert.song(
-                    input_song,
-                    output_song,
-                    const.args.folder,
-                    avconv=const.args.avconv,
-                    trim_silence=const.args.trim_silence,
-                )
-            except FileNotFoundError:
-                encoder = "avconv" if const.args.avconv else "ffmpeg"
-                log.warning("Could not find {0}, skipping conversion".format(encoder))
-                const.args.output_ext = const.args.input_ext
-                output_song = songname + const.args.output_ext
-
-            if not const.args.input_ext == const.args.output_ext:
-                os.remove(os.path.join(const.args.folder, input_song))
-            if not const.args.no_metadata and meta_tags is not None:
-                metadata.embed(os.path.join(const.args.folder, output_song), meta_tags)
-            return True
+def match_args():
+    if const.args.song:
+        for track in const.args.song:
+            track_dl = downloader.Downloader(raw_song=track)
+            track_dl.download_single()
+    elif const.args.list:
+        if const.args.write_m3u:
+            youtube_tools.generate_m3u(track_file=const.args.list,
+                                       text_file=const.args.write_to)
+        else:
+            list_dl = downloader.ListDownloader(
+                tracks_file=const.args.list,
+                skip_file=const.args.skip,
+                write_successful_file=const.args.write_successful,
+            )
+            list_dl.download_list()
+    elif const.args.playlist:
+        spotify_tools.write_playlist(playlist_url=const.args.playlist,
+                                     text_file=const.args.write_to)
+    elif const.args.album:
+        spotify_tools.write_album(album_url=const.args.album,
+                                  text_file=const.args.write_to)
+    elif const.args.all_albums:
+        spotify_tools.write_all_albums_from_artist(artist_url=const.args.all_albums,
+                                                   text_file=const.args.write_to)
+    elif const.args.username:
+        spotify_tools.write_user_playlist(username=const.args.username,
+                                          text_file=const.args.write_to)
 
 
 def main():
     const.args = handle.get_arguments()
 
-    if const.args.version:
-        print("spotdl {version}".format(version=__version__))
-        sys.exit()
-
     internals.filter_path(const.args.folder)
     youtube_tools.set_api_key()
 
     logzero.setup_default_logger(formatter=const._formatter, level=const.args.log_level)
 
-    log.debug("Python version: {}".format(sys.version))
-    log.debug("Platform: {}".format(platform.platform()))
-    log.debug(pprint.pformat(const.args.__dict__))
-
     try:
-        if const.args.song:
-            download_single(raw_song=const.args.song)
-        elif const.args.list:
-            if const.args.write_m3u:
-                youtube_tools.generate_m3u(track_file=const.args.list)
-            else:
-                download_list(
-                    tracks_file=const.args.list,
-                    skip_file=const.args.skip,
-                    write_successful_file=const.args.write_successful,
-                )
-        elif const.args.playlist:
-            spotify_tools.write_playlist(playlist_url=const.args.playlist)
-        elif const.args.album:
-            spotify_tools.write_album(album_url=const.args.album)
-        elif const.args.all_albums:
-            spotify_tools.write_all_albums_from_artist(artist_url=const.args.all_albums)
-        elif const.args.username:
-            spotify_tools.write_user_playlist(username=const.args.username)
-
+        match_args()
         # actually we don't necessarily need this, but yeah...
         # explicit is better than implicit!
         sys.exit(0)

spotdl/spotify_tools.py

@@ -1,38 +1,45 @@
 import spotipy
 import spotipy.oauth2 as oauth2
 import lyricwikia
-from logzero import logger as log
-
-from spotdl import internals
 
 from slugify import slugify
 from titlecase import titlecase
+from logzero import logger as log
 import pprint
 import sys
+import os
+import functools
+
+from spotdl import const
+from spotdl import internals
+
+spotify = None
 
 
 def generate_token():
-    """ Generate the token. Please respect these credentials :) """
+    """ Generate the token. """
     credentials = oauth2.SpotifyClientCredentials(
-        client_id="4fe3fecfe5334023a1472516cc99d805",
-        client_secret="0f02b7c483c04257984695007a4a8d5c",
+        client_id=const.args.spotify_client_id,
+        client_secret=const.args.spotify_client_secret,
     )
     token = credentials.get_access_token()
     return token
 
 
-def refresh_token():
-    """ Refresh expired token"""
-    global spotify
-    new_token = generate_token()
-    spotify = spotipy.Spotify(auth=new_token)
-
-# token is mandatory when using Spotify's API
-# https://developer.spotify.com/news-stories/2017/01/27/removing-unauthenticated-calls-to-the-web-api/
-token = generate_token()
-spotify = spotipy.Spotify(auth=token)
+def must_be_authorized(func, spotify=spotify):
+    def wrapper(*args, **kwargs):
+        global spotify
+        try:
+            assert spotify
+            return func(*args, **kwargs)
+        except (AssertionError, spotipy.client.SpotifyException):
+            token = generate_token()
+            spotify = spotipy.Spotify(auth=token)
+            return func(*args, **kwargs)
+    return wrapper
 
 
+@must_be_authorized
 def generate_metadata(raw_song):
     """ Fetch a song's metadata from Spotify. """
     if internals.is_spotify(raw_song):
@@ -78,6 +85,7 @@ def generate_metadata(raw_song):
     # Some sugar
     meta_tags["year"], *_ = meta_tags["release_date"].split("-")
     meta_tags["duration"] = meta_tags["duration_ms"] / 1000.0
+    meta_tags["spotify_metadata"] = True
     # Remove unwanted parameters
     del meta_tags["duration_ms"]
     del meta_tags["available_markets"]
@@ -87,12 +95,15 @@ def generate_metadata(raw_song):
     return meta_tags
 
 
+@must_be_authorized
 def write_user_playlist(username, text_file=None):
+    """ Write user playlists to text_file """
     links = get_playlists(username=username)
     playlist = internals.input_link(links)
     return write_playlist(playlist, text_file)
 
 
+@must_be_authorized
 def get_playlists(username):
     """ Fetch user playlists when using the -u option. """
     playlists = spotify.user_playlists(username)
@@ -121,6 +132,7 @@ def get_playlists(username):
     return links
 
 
+@must_be_authorized
 def fetch_playlist(playlist):
     try:
         playlist_id = internals.extract_spotify_id(playlist)
@@ -140,6 +152,7 @@ def fetch_playlist(playlist):
     return results
 
 
+@must_be_authorized
 def write_playlist(playlist_url, text_file=None):
     playlist = fetch_playlist(playlist_url)
     tracks = playlist["tracks"]
@@ -148,19 +161,21 @@ def write_playlist(playlist_url, text_file=None):
     return write_tracks(tracks, text_file)
 
 
+@must_be_authorized
 def fetch_album(album):
     album_id = internals.extract_spotify_id(album)
     album = spotify.album(album_id)
     return album
 
 
-def fetch_album_from_artist(artist_url, album_type="album"):
+@must_be_authorized
+def fetch_albums_from_artist(artist_url, album_type=None):
     """
     This funcction returns all the albums from a give artist_url using the US
     market
     :param artist_url - spotify artist url
     :param album_type - the type of album to fetch (ex: single) the default is
-                        a standard album
+                        all albums
     :param return - the album from the artist
     """
 
@@ -179,6 +194,7 @@ def fetch_album_from_artist(artist_url, album_type="album"):
     return albums
 
 
+@must_be_authorized
 def write_all_albums_from_artist(artist_url, text_file=None):
     """
     This function gets all albums from an artist and writes it to a file in the
@@ -191,7 +207,7 @@ def write_all_albums_from_artist(artist_url, text_file=None):
     album_base_url = "https://open.spotify.com/album/"
 
     # fetching all default albums
-    albums = fetch_album_from_artist(artist_url)
+    albums = fetch_albums_from_artist(artist_url, album_type=None)
 
     # if no file if given, the default save file is in the current working
     # directory with the name of the artist
@@ -203,14 +219,8 @@ def write_all_albums_from_artist(artist_url, text_file=None):
         log.info("Fetching album: " + album["name"])
         write_album(album_base_url + album["id"], text_file=text_file)
 
-    # fetching all single albums
-    singles = fetch_album_from_artist(artist_url, album_type="single")
-
-    for single in singles:
-        log.info("Fetching single: " + single["name"])
-        write_album(album_base_url + single["id"], text_file=text_file)
-
 
+@must_be_authorized
 def write_album(album_url, text_file=None):
     album = fetch_album(album_url)
     tracks = spotify.album_tracks(album["id"])
@@ -219,6 +229,7 @@ def write_album(album_url, text_file=None):
     return write_tracks(tracks, text_file)
 
 
+@must_be_authorized
 def write_tracks(tracks, text_file):
     log.info(u"Writing {0} tracks to {1}".format(tracks["total"], text_file))
     track_urls = []

spotdl/youtube_tools.py

@@ -1,20 +1,28 @@
 from bs4 import BeautifulSoup
 import urllib
 import pafy
+
 from slugify import slugify
 from logzero import logger as log
+import os
 
 from spotdl import spotify_tools
 from spotdl import internals
 from spotdl import const
 
-import os
-import pprint
-
 # Fix download speed throttle on short duration tracks
 # Read more on mps-youtube/pafy#199
 pafy.g.opener.addheaders.append(("Range", "bytes=0-"))
 
+# Implement unreleased methods on Pafy object
+# More info: https://github.com/mps-youtube/pafy/pull/211
+if pafy.__version__ <= "0.5.4":
+    from spotdl import patcher
+    pafy_patcher = patcher.PatchPafy()
+    pafy_patcher.patch_getbestthumb()
+    pafy_patcher.patch_process_streams()
+    pafy_patcher.patch_insecure_streams()
+
 
 def set_api_key():
     if const.args.youtube_api_key:
@@ -40,26 +48,76 @@ def go_pafy(raw_song, meta_tags=None):
     return track_info
 
 
-def match_video_and_metadata(track, force_pafy=True):
+def match_video_and_metadata(track):
     """ Get and match track data from YouTube and Spotify. """
     meta_tags = None
 
+
+    def fallback_metadata(meta_tags):
+        fallback_metadata_info = "Track not found on Spotify, falling back on YouTube metadata"
+        skip_fallback_metadata_warning = "Fallback condition not met, shall not embed metadata"
+        if meta_tags is None:
+            if const.args.no_fallback_metadata:
+                log.warning(skip_fallback_metadata_warning)
+            else:
+                log.info(fallback_metadata_info)
+                meta_tags = generate_metadata(content)
+        return meta_tags
+
+
     if internals.is_youtube(track):
         log.debug("Input song is a YouTube URL")
         content = go_pafy(track, meta_tags=None)
         track = slugify(content.title).replace("-", " ")
         if not const.args.no_metadata:
             meta_tags = spotify_tools.generate_metadata(track)
+            meta_tags = fallback_metadata(meta_tags)
+
+    elif internals.is_spotify(track):
+        log.debug("Input song is a Spotify URL")
+        # Let it generate metadata, YouTube doesn't know Spotify slang
+        meta_tags = spotify_tools.generate_metadata(track)
+        content = go_pafy(track, meta_tags)
+        if const.args.no_metadata:
+            meta_tags = None
+
     else:
-        if not const.args.no_metadata:
-            meta_tags = spotify_tools.generate_metadata(track)
-        if force_pafy:
-            content = go_pafy(track, meta_tags)
+        log.debug("Input song is plain text based")
+        if const.args.no_metadata:
+            content = go_pafy(track, meta_tags=None)
         else:
-            content = None
+            meta_tags = spotify_tools.generate_metadata(track)
+            content = go_pafy(track, meta_tags=meta_tags)
+            meta_tags = fallback_metadata(meta_tags)
+
     return content, meta_tags
 
 
+def generate_metadata(content):
+    """ Fetch a song's metadata from YouTube. """
+    meta_tags = {"spotify_metadata": False,
+                 "name": content.title,
+                 "artists": [{"name": content.author}],
+                 "duration": content.length,
+                 "external_urls": {"youtube": content.watchv_url},
+                 "album": {"images" : [{"url": content.getbestthumb()}],
+                           "artists": [{"name": None}],"name": None},
+                 "year": content.published.split("-")[0],
+                 "release_date": content.published.split(" ")[0],
+                 "type": "track",
+                 "disc_number": 1,
+                 "track_number": 1,
+                 "total_tracks": 1,
+                 "publisher": None,
+                 "external_ids": {"isrc": None},
+                 "lyrics": None,
+                 "copyright": None,
+                 "genre": None,
+                 }
+
+    return meta_tags
+
+
 def get_youtube_title(content, number=None):
     """ Get the YouTube video's title. """
     title = content.title
@@ -76,6 +134,8 @@ def generate_m3u(track_file):
     log.info("Generating {0} from {1} YouTube URLs".format(target_file, total_tracks))
     with open(target_file, "w") as output_file:
         output_file.write("#EXTM3U\n\n")
+
+    videos = []
     for n, track in enumerate(tracks, 1):
         content, _ = match_video_and_metadata(track)
         if content is None:
@@ -95,6 +155,9 @@ def generate_m3u(track_file):
             log.debug(m3u_key)
             with open(target_file, "a") as output_file:
                 output_file.write(m3u_key)
+            videos.append(content.watchv_url)
+
+    return videos
 
 
 def download_song(file_name, content):
@@ -168,6 +231,10 @@ class GenerateYouTubeURL:
             )
 
     def _best_match(self, videos):
+        if not videos:
+            log.error("No videos found on YouTube for a given search")
+            return None
+
         """ Select the best matching video from a list of videos. """
         if const.args.manual:
             log.info(self.raw_song)
@@ -241,7 +308,7 @@ class GenerateYouTubeURL:
         search_url = generate_search_url(self.search_query)
         log.debug("Opening URL: {0}".format(search_url))
 
-        item = urllib.request.urlopen(search_url).read()
+        item = self._fetch_response(search_url).read()
         items_parse = BeautifulSoup(item, "html.parser")
 
         videos = []
@@ -320,3 +387,11 @@ class GenerateYouTubeURL:
             return self._best_match(videos)
 
         return videos
+
+    @staticmethod
+    def _fetch_response(url):
+        # XXX: This method exists only because it helps us indirectly
+        # monkey patch `urllib.request.open`, directly monkey patching
+        # `urllib.request.open` causes us to end up in an infinite recursion
+        # during the test since `urllib.request.open` would monkeypatch itself.
+        return urllib.request.urlopen(url)

test/loader.py

@@ -1,6 +1,7 @@
 from spotdl import const
 from spotdl import handle
 from spotdl import spotdl
+import urllib
 
 import pytest
 
@@ -8,9 +9,18 @@ import pytest
 def load_defaults():
     const.args = handle.get_arguments(raw_args="", to_group=False, to_merge=False)
     const.args.overwrite = "skip"
-    const.args.log_level = 10
 
     spotdl.args = const.args
     spotdl.log = const.logzero.setup_logger(
         formatter=const._formatter, level=const.args.log_level
     )
+
+
+# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
+# so that we get same results even if YouTube changes the list/order of videos on their page.
+GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
+
+def monkeypatch_youtube_search_page(*args, **kwargs):
+    fake_urlopen = urllib.request.urlopen(GIST_URL)
+    return fake_urlopen
+

test/test_download_with_metadata.py

@@ -0,0 +1,232 @@
+import subprocess
+import os
+
+from spotdl import const
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+from spotdl import convert
+from spotdl import metadata
+from spotdl import downloader
+
+import pytest
+import loader
+
+loader.load_defaults()
+
+SPOTIFY_TRACK_URL = "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD"
+EXPECTED_YOUTUBE_TITLE = "Janji - Heroes Tonight (feat. Johnning) [NCS Release]"
+EXPECTED_SPOTIFY_TITLE = "Janji - Heroes Tonight"
+EXPECTED_YOUTUBE_URL = "http://youtube.com/watch?v=3nQNiWdeH2Q"
+# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
+# so that we get same results even if YouTube changes the list/order of videos on their page.
+GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
+
+
+def pytest_namespace():
+    # XXX: We override the value of `content_fixture` later in the tests.
+    # We do not use an acutal @pytest.fixture because it does not accept
+    # the monkeypatch parameter and we need to monkeypatch the network
+    # request before creating the Pafy object.
+    return {"content_fixture": None}
+
+
+@pytest.fixture(scope="module")
+def metadata_fixture():
+    meta_tags = spotify_tools.generate_metadata(SPOTIFY_TRACK_URL)
+    return meta_tags
+
+
+def test_metadata(metadata_fixture):
+    expect_number = 24
+    assert len(metadata_fixture) == expect_number
+
+
+class TestFileFormat:
+    def test_with_spaces(self, metadata_fixture):
+        title = internals.format_string(const.args.file_format, metadata_fixture)
+        assert title == EXPECTED_SPOTIFY_TITLE
+
+    def test_without_spaces(self, metadata_fixture):
+        const.args.no_spaces = True
+        title = internals.format_string(const.args.file_format, metadata_fixture)
+        assert title == EXPECTED_SPOTIFY_TITLE.replace(" ", "_")
+
+
+def test_youtube_url(metadata_fixture, monkeypatch):
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
+    url = youtube_tools.generate_youtube_url(SPOTIFY_TRACK_URL, metadata_fixture)
+    assert url == EXPECTED_YOUTUBE_URL
+
+
+def test_youtube_title(metadata_fixture, monkeypatch):
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
+    content = youtube_tools.go_pafy(SPOTIFY_TRACK_URL, metadata_fixture)
+    pytest.content_fixture = content
+    title = youtube_tools.get_youtube_title(content)
+    assert title == EXPECTED_YOUTUBE_TITLE
+
+
+@pytest.fixture(scope="module")
+def filename_fixture(metadata_fixture):
+    songname = internals.format_string(const.args.file_format, metadata_fixture)
+    filename = internals.sanitize_title(songname)
+    return filename
+
+
+def test_check_track_exists_before_download(tmpdir, metadata_fixture, filename_fixture):
+    expect_check = False
+    const.args.folder = str(tmpdir)
+    # prerequisites for determining filename
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
+    assert check == expect_check
+
+
+class TestDownload:
+    def blank_audio_generator(self, filepath):
+        if filepath.endswith(".m4a"):
+            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a aac {}".format(filepath)
+        elif filepath.endswith(".webm"):
+            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a libopus {}".format(filepath)
+        subprocess.call(cmd.split(" "))
+
+    def test_m4a(self, monkeypatch, filename_fixture):
+        expect_download = True
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", self.blank_audio_generator)
+        monkeypatch.setattr("pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator)
+        download = youtube_tools.download_song(filename_fixture + ".m4a", pytest.content_fixture)
+        assert download == expect_download
+
+    def test_webm(self, monkeypatch, filename_fixture):
+        expect_download = True
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", self.blank_audio_generator)
+        monkeypatch.setattr("pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator)
+        download = youtube_tools.download_song(filename_fixture + ".webm", pytest.content_fixture)
+        assert download == expect_download
+
+
+class TestFFmpeg:
+    def test_convert_from_webm_to_mp3(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.webm -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".webm", filename_fixture + ".mp3", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_webm_to_m4a(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.webm -cutoff 20000 -codec:a aac -ar 44100 -b:a 192k -vn {0}.m4a".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".webm", filename_fixture + ".m4a", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:v copy -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".mp3", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_webm(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:a libopus -vbr on -b:a 192k -vn {0}.webm".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".webm", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_flac(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:a flac -ar 44100 -b:a 192k -vn {0}.flac".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".flac", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_correct_container_for_m4a(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a.temp -acodec copy -b:a 192k -vn {0}.m4a".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".m4a", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+
+class TestAvconv:
+    @pytest.mark.skip(reason="avconv is no longer provided with FFmpeg")
+    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
+        monkeypatch.setattr("os.remove", lambda x: None)
+        expect_command = "avconv -loglevel 0 -i {0}.m4a -ab 192k {0}.mp3 -y".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        _, command = convert.song(
+            filename_fixture + ".m4a",
+            filename_fixture + ".mp3",
+            const.args.folder,
+            avconv=True,
+        )
+        assert " ".join(command) == expect_command
+
+
+@pytest.fixture(scope="module")
+def trackpath_fixture(filename_fixture):
+    trackpath = os.path.join(const.args.folder, filename_fixture)
+    return trackpath
+
+
+class TestEmbedMetadata:
+    def test_embed_in_mp3(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".mp3", metadata_fixture)
+        assert embed == expect_embed
+
+    def test_embed_in_m4a(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".m4a", metadata_fixture)
+        os.remove(trackpath_fixture + ".m4a")
+        assert embed == expect_embed
+
+    def test_embed_in_webm(self, metadata_fixture, trackpath_fixture):
+        expect_embed = False
+        embed = metadata.embed(trackpath_fixture + ".webm", metadata_fixture)
+        os.remove(trackpath_fixture + ".webm")
+        assert embed == expect_embed
+
+    def test_embed_in_flac(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".flac", metadata_fixture)
+        os.remove(trackpath_fixture + ".flac")
+        assert embed == expect_embed
+
+
+def test_check_track_exists_after_download(
+    metadata_fixture, filename_fixture, trackpath_fixture
+):
+    expect_check = True
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
+    os.remove(trackpath_fixture + ".mp3")
+    assert check == expect_check

test/test_dry_run.py

@@ -1,7 +1,7 @@
 import os
 
 from spotdl import const
-from spotdl import spotdl
+from spotdl import downloader
 
 import loader
 
@@ -16,5 +16,6 @@ def test_dry_download_list(tmpdir):
     file_path = os.path.join(const.args.folder, "test_list.txt")
     with open(file_path, "w") as f:
         f.write(TRACK_URL)
-    downloaded_song, *_ = spotdl.download_list(file_path)
+    list_dl = downloader.ListDownloader(file_path)
+    downloaded_song, *_ = list_dl.download_list()
     assert downloaded_song == TRACK_URL

test/test_handle.py

@@ -10,13 +10,11 @@ import yaml
 
 def test_error_m3u_without_list():
     with pytest.raises(SystemExit):
-        handle.get_arguments(raw_args=('-s cool song', '--write-m3u',),
-                             to_group=True)
+        handle.get_arguments(raw_args=("-s cool song", "--write-m3u"), to_group=True)
 
 
 def test_m3u_with_list():
-    handle.get_arguments(raw_args=('-l cool_list.txt', '--write-m3u',),
-                         to_group=True)
+    handle.get_arguments(raw_args=("-l cool_list.txt", "--write-m3u"), to_group=True)
 
 
 def test_log_str_to_int():
@@ -25,28 +23,39 @@ def test_log_str_to_int():
     assert levels == expect_levels
 
 
+@pytest.fixture(scope="module")
+def config_path_fixture(tmpdir_factory):
+    config_path = os.path.join(str(tmpdir_factory.mktemp("config")), "config.yml")
+    return config_path
+
+
+@pytest.fixture(scope="module")
+def modified_config_fixture():
+    modified_config = dict(handle.default_conf)
+    return modified_config
+
+
 class TestConfig:
-    def test_default_config(self, tmpdir):
+    def test_default_config(self, config_path_fixture):
         expect_config = handle.default_conf["spotify-downloader"]
-        global config_path
-        config_path = os.path.join(str(tmpdir), "config.yml")
-        config = handle.get_config(config_path)
+        config = handle.get_config(config_path_fixture)
         assert config == expect_config
 
-    def test_modified_config(self):
-        global modified_config
-        modified_config = dict(handle.default_conf)
-        modified_config["spotify-downloader"]["file-format"] = "just_a_test"
-        merged_config = handle.merge(handle.default_conf, modified_config)
-        assert merged_config == modified_config
+    def test_modified_config(self, modified_config_fixture):
+        modified_config_fixture["spotify-downloader"]["file-format"] = "just_a_test"
+        merged_config = handle.merge(handle.default_conf, modified_config_fixture)
+        assert merged_config == modified_config_fixture
 
-    def test_custom_config_path(self, tmpdir):
+    def test_custom_config_path(self, config_path_fixture, modified_config_fixture):
         parser = argparse.ArgumentParser()
-        with open(config_path, "w") as config_file:
-            yaml.dump(modified_config, config_file, default_flow_style=False)
-        overridden_config = handle.override_config(config_path, parser, raw_args="")
+        with open(config_path_fixture, "w") as config_file:
+            yaml.dump(modified_config_fixture, config_file, default_flow_style=False)
+        overridden_config = handle.override_config(
+            config_path_fixture, parser, raw_args=""
+        )
         modified_values = [
-            str(value) for value in modified_config["spotify-downloader"].values()
+            str(value)
+            for value in modified_config_fixture["spotify-downloader"].values()
         ]
         overridden_config.folder = os.path.realpath(overridden_config.folder)
         overridden_values = [

test/test_internals.py

@@ -80,6 +80,26 @@ STRING_IDS_TEST_TABLE = [
 ]
 
 
+FROM_SECONDS_TEST_TABLE = [
+    (35, "35"),
+    (23, "23"),
+    (158, "2:38"),
+    (263, "4:23"),
+    (4562, "1:16:02"),
+    (26762, "7:26:02"),
+]
+
+
+TO_SECONDS_TEST_TABLE = [
+    ("0:23", 23),
+    ("0:45", 45),
+    ("2:19", 139),
+    ("3:33", 213),
+    ("7:38", 458),
+    ("1:30:05", 5405),
+]
+
+
 def test_default_music_directory():
     if sys.platform.startswith("linux"):
         output = subprocess.check_output(["xdg-user-dir", "MUSIC"])
@@ -92,68 +112,38 @@ def test_default_music_directory():
     assert directory == expect_directory
 
 
+@pytest.fixture(scope="module")
+def directory_fixture(tmpdir_factory):
+    dir_path = os.path.join(str(tmpdir_factory.mktemp("tmpdir")), "filter_this_folder")
+    return dir_path
+
+
 class TestPathFilterer:
-    def test_create_directory(self, tmpdir):
+    def test_create_directory(self, directory_fixture):
         expect_path = True
-        global folder_path
-        folder_path = os.path.join(str(tmpdir), "filter_this_folder")
-        internals.filter_path(folder_path)
-        is_path = os.path.isdir(folder_path)
+        internals.filter_path(directory_fixture)
+        is_path = os.path.isdir(directory_fixture)
         assert is_path == expect_path
 
-    def test_remove_temp_files(self, tmpdir):
+    def test_remove_temp_files(self, directory_fixture):
         expect_file = False
-        file_path = os.path.join(folder_path, "pesky_file.temp")
+        file_path = os.path.join(directory_fixture, "pesky_file.temp")
         open(file_path, "a")
-        internals.filter_path(folder_path)
+        internals.filter_path(directory_fixture)
         is_file = os.path.isfile(file_path)
         assert is_file == expect_file
 
 
-class TestVideoTimeFromSeconds:
-    def test_from_seconds(self):
-        expect_duration = "35"
-        duration = internals.videotime_from_seconds(35)
-        assert duration == expect_duration
-
-    def test_from_minutes(self):
-        expect_duration = "2:38"
-        duration = internals.videotime_from_seconds(158)
-        assert duration == expect_duration
-
-    def test_from_hours(self):
-        expect_duration = "1:16:02"
-        duration = internals.videotime_from_seconds(4562)
-        assert duration == expect_duration
+@pytest.mark.parametrize("sec_duration, str_duration", FROM_SECONDS_TEST_TABLE)
+def test_video_time_from_seconds(sec_duration, str_duration):
+    duration = internals.videotime_from_seconds(sec_duration)
+    assert duration == str_duration
 
 
-class TestGetSeconds:
-    def test_from_seconds(self):
-        expect_secs = 45
-        secs = internals.get_sec("0:45")
-        assert secs == expect_secs
-        secs = internals.get_sec("0.45")
-        assert secs == expect_secs
-
-    def test_from_minutes(self):
-        expect_secs = 213
-        secs = internals.get_sec("3.33")
-        assert secs == expect_secs
-        secs = internals.get_sec("3:33")
-        assert secs == expect_secs
-
-    def test_from_hours(self):
-        expect_secs = 5405
-        secs = internals.get_sec("1.30.05")
-        assert secs == expect_secs
-        secs = internals.get_sec("1:30:05")
-        assert secs == expect_secs
-
-    def test_raise_error(self):
-        with pytest.raises(ValueError):
-            internals.get_sec("10*05")
-        with pytest.raises(ValueError):
-            internals.get_sec("02,28,46")
+@pytest.mark.parametrize("str_duration, sec_duration", TO_SECONDS_TEST_TABLE)
+def test_get_seconds_from_video_time(str_duration, sec_duration):
+    secs = internals.get_sec(str_duration)
+    assert secs == sec_duration
 
 
 @pytest.mark.parametrize("duplicates, expected", DUPLICATE_TRACKS_TEST_TABLE)
@@ -170,3 +160,21 @@ def test_get_unique_tracks(tmpdir, duplicates, expected):
 def test_extract_spotify_id(input_str, expected_spotify_id):
     spotify_id = internals.extract_spotify_id(input_str)
     assert spotify_id == expected_spotify_id
+
+
+def test_trim(tmpdir):
+    text_file = os.path.join(str(tmpdir), "test_trim.txt")
+    with open(text_file, "w") as track_file:
+        track_file.write("ncs - spectre\nncs - heroes\nncs - hope")
+
+    with open(text_file, "r") as track_file:
+        tracks = track_file.readlines()
+
+    expect_number = len(tracks) - 1
+    expect_track = tracks[0]
+    track = internals.trim_song(text_file)
+
+    with open(text_file, "r") as track_file:
+        number = len(track_file.readlines())
+
+    assert expect_number == number and expect_track == track

test/test_list.py

@@ -1,90 +0,0 @@
-import builtins
-import os
-
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import const
-from spotdl import spotdl
-
-import loader
-
-
-USERNAME = "uqlakumu7wslkoen46s5bulq0"
-PLAYLIST_URL = "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt"
-ALBUM_URL = "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg"
-ARTIST_URL = "https://open.spotify.com/artist/4dpARuHxo51G3z768sgnrY"
-
-loader.load_defaults()
-
-
-def test_user_playlists(tmpdir, monkeypatch):
-    expect_tracks = 17
-    text_file = os.path.join(str(tmpdir), "test_us.txt")
-    monkeypatch.setattr("builtins.input", lambda x: 1)
-    spotify_tools.write_user_playlist(USERNAME, text_file)
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-def test_playlist(tmpdir):
-    expect_tracks = 14
-    text_file = os.path.join(str(tmpdir), "test_pl.txt")
-    spotify_tools.write_playlist(PLAYLIST_URL, text_file)
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-def test_album(tmpdir):
-    expect_tracks = 15
-    text_file = os.path.join(str(tmpdir), "test_al.txt")
-    spotify_tools.write_album(ALBUM_URL, text_file)
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-def test_m3u(tmpdir):
-    expect_m3u = (
-        "#EXTM3U\n\n"
-        "#EXTINF:31,Eminem - 01 - Eminem - Curtains Up (Skit)\n"
-        "http://www.youtube.com/watch?v=qk13SFlwG9A\n"
-        "#EXTINF:226,Alan Walker - Spectre [NCS Release]\n"
-        "http://www.youtube.com/watch?v=AOeY-nDp7hI\n"
-    )
-    m3u_track_file = os.path.join(str(tmpdir), "m3u_test.txt")
-    with open(m3u_track_file, "w") as track_file:
-        track_file.write("\nhttps://open.spotify.com/track/2nT5m433s95hvYJH4S7ont")
-        track_file.write("\nhttp://www.youtube.com/watch?v=AOeY-nDp7hI")
-    youtube_tools.generate_m3u(m3u_track_file)
-    m3u_file = "{}.m3u".format(m3u_track_file.split(".")[0])
-    with open(m3u_file, "r") as m3u_in:
-        m3u = m3u_in.readlines()
-    assert "".join(m3u) == expect_m3u
-
-
-def test_all_albums(tmpdir):
-    # current number of tracks on spotify since as of 10/10/2018
-    # in US market only
-    expect_tracks = 49
-    global text_file
-    text_file = os.path.join(str(tmpdir), "test_ab.txt")
-    spotify_tools.write_all_albums_from_artist(ARTIST_URL, text_file)
-    with open(text_file, "r") as f:
-        tracks = len(f.readlines())
-    assert tracks == expect_tracks
-
-
-def test_trim():
-    with open(text_file, "r") as track_file:
-        tracks = track_file.readlines()
-
-    expect_number = len(tracks) - 1
-    expect_track = tracks[0]
-    track = spotdl.internals.trim_song(text_file)
-
-    with open(text_file, "r") as track_file:
-        number = len(track_file.readlines())
-
-    assert expect_number == number and expect_track == track

test/test_patcher.py

@@ -0,0 +1,36 @@
+from spotdl import patcher
+import pafy
+
+import pytest
+
+pafy_patcher = patcher.PatchPafy()
+pafy_patcher.patch_getbestthumb()
+
+class TestPafyContentAvailable:
+    pass
+
+
+class TestMethodAssignment:
+    def test_pafy_getbestthumb(self):
+        pafy.backend_shared.BasePafy.getbestthumb == patcher._getbestthumb
+
+
+class TestMethodCalls:
+    @pytest.fixture(scope="module")
+    def content_fixture(self):
+        content = pafy.new("http://youtube.com/watch?v=3nQNiWdeH2Q")
+        return content
+
+    def test_pafy_getbestthumb(self, content_fixture):
+        thumbnail = patcher._getbestthumb(content_fixture)
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/maxresdefault.jpg"
+
+    def test_pafy_getbestthumb_without_ytdl(self, content_fixture):
+        content_fixture._ydl_info["thumbnails"][0]["url"] = None
+        thumbnail = patcher._getbestthumb(content_fixture)
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/maxresdefault.jpg"
+
+
+    def test_pafy_content_available(self):
+        TestPafyContentAvailable._content_available = patcher._content_available
+        assert TestPafyContentAvailable()._content_available("https://youtube.com/")

test/test_spotify_tools.py

@@ -0,0 +1,178 @@
+from spotdl import spotify_tools
+from spotdl import const
+
+import spotipy
+
+import os
+import pytest
+import loader
+
+loader.load_defaults()
+
+
+def test_generate_token():
+    token = spotify_tools.generate_token()
+    assert len(token) == 83
+
+
+class TestMustBeAuthorizedDecorator:
+    def test_spotify_instance_is_unset(self):
+        spotify_tools.spotify = None
+
+        @spotify_tools.must_be_authorized
+        def sample_func():
+            return True
+
+        assert sample_func()
+
+    def test_spotify_instance_forces_assertion_error(self):
+        @spotify_tools.must_be_authorized
+        def sample_func():
+            raise AssertionError
+
+        with pytest.raises(AssertionError):
+            sample_func()
+
+    def test_fake_token_generator(self, monkeypatch):
+        spotify_tools.spotify = None
+        monkeypatch.setattr(spotify_tools, "generate_token", lambda: 123123)
+
+        with pytest.raises(spotipy.client.SpotifyException):
+            spotify_tools.generate_metadata("ncs - spectre")
+
+    def test_correct_token(self):
+        assert spotify_tools.generate_metadata("ncs - spectre")
+
+
+class TestGenerateMetadata:
+    @pytest.fixture(scope="module")
+    def metadata_fixture(self):
+        metadata = spotify_tools.generate_metadata("ncs - spectre")
+        return metadata
+
+    def test_len(self, metadata_fixture):
+        assert len(metadata_fixture) == 24
+
+    def test_trackname(self, metadata_fixture):
+        assert metadata_fixture["name"] == "Spectre"
+
+    def test_artist(self, metadata_fixture):
+        assert metadata_fixture["artists"][0]["name"] == "Alan Walker"
+
+    def test_duration(self, metadata_fixture):
+        assert metadata_fixture["duration"] == 230.634
+
+
+def test_get_playlists():
+    expect_playlist_ids = [
+        "34gWCK8gVeYDPKcctB6BQJ",
+        "04wTU2c2WNQG9XE5oSLYfj",
+        "0fWBMhGh38y0wsYWwmM9Kt",
+    ]
+
+    expect_playlists = [
+        "https://open.spotify.com/playlist/" + playlist_id
+        for playlist_id in expect_playlist_ids
+    ]
+
+    playlists = spotify_tools.get_playlists("uqlakumu7wslkoen46s5bulq0")
+    assert playlists == expect_playlists
+
+
+def test_write_user_playlist(tmpdir, monkeypatch):
+    expect_tracks = 17
+    text_file = os.path.join(str(tmpdir), "test_us.txt")
+    monkeypatch.setattr("builtins.input", lambda x: 1)
+    spotify_tools.write_user_playlist("uqlakumu7wslkoen46s5bulq0", text_file)
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+class TestFetchPlaylist:
+    @pytest.fixture(scope="module")
+    def playlist_fixture(self):
+        playlist = spotify_tools.fetch_playlist(
+            "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt"
+        )
+        return playlist
+
+    def test_name(self, playlist_fixture):
+        assert playlist_fixture["name"] == "special_test_playlist"
+
+    def test_tracks(self, playlist_fixture):
+        assert playlist_fixture["tracks"]["total"] == 14
+
+
+def test_write_playlist(tmpdir):
+    expect_tracks = 14
+    text_file = os.path.join(str(tmpdir), "test_pl.txt")
+    spotify_tools.write_playlist(
+        "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+# XXX: Monkeypatch these tests if they fail in future
+class TestFetchAlbum:
+    @pytest.fixture(scope="module")
+    def album_fixture(self):
+        album = spotify_tools.fetch_album(
+            "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg"
+        )
+        return album
+
+    def test_name(self, album_fixture):
+        assert album_fixture["name"] == "NCS: Infinity"
+
+    def test_tracks(self, album_fixture):
+        assert album_fixture["tracks"]["total"] == 15
+
+
+# XXX: Monkeypatch these tests if they fail in future
+class TestFetchAlbumsFromArtist:
+    @pytest.fixture(scope="module")
+    def albums_from_artist_fixture(self):
+        albums = spotify_tools.fetch_albums_from_artist(
+            "https://open.spotify.com/artist/7oPftvlwr6VrsViSDV7fJY"
+        )
+        return albums
+
+    def test_len(self, albums_from_artist_fixture):
+        assert len(albums_from_artist_fixture) == 53
+
+    def test_zeroth_album_name(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["name"] == "Revolution Radio"
+
+    def test_zeroth_album_tracks(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["total_tracks"] == 12
+
+    def test_fist_album_name(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[1]["name"] == "Demolicious"
+
+    def test_first_album_tracks(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["total_tracks"] == 12
+
+
+def test_write_all_albums_from_artist(tmpdir):
+    expect_tracks = 282
+    text_file = os.path.join(str(tmpdir), "test_ab.txt")
+    spotify_tools.write_all_albums_from_artist(
+        "https://open.spotify.com/artist/4dpARuHxo51G3z768sgnrY", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+def test_write_album(tmpdir):
+    expect_tracks = 15
+    text_file = os.path.join(str(tmpdir), "test_al.txt")
+    spotify_tools.write_album(
+        "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks

test/test_with_metadata.py

@@ -1,151 +0,0 @@
-import os
-
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import convert
-from spotdl import metadata
-from spotdl import spotdl
-
-import loader
-
-loader.load_defaults()
-
-TRACK_URL = "https://open.spotify.com/track/2nT5m433s95hvYJH4S7ont"
-EXPECTED_TITLE = "Eminem - Curtains Up"
-EXPECTED_YT_TITLE = "Eminem - 01 - Eminem - Curtains Up (Skit)"
-EXPECTED_YT_URL = "http://youtube.com/watch?v=qk13SFlwG9A"
-
-
-def test_metadata():
-    expect_number = 23
-    global meta_tags
-    meta_tags = spotify_tools.generate_metadata(TRACK_URL)
-    assert len(meta_tags) == expect_number
-
-
-class TestFileFormat:
-    def test_with_spaces(self):
-        title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == EXPECTED_TITLE
-
-    def test_without_spaces(self):
-        const.args.no_spaces = True
-        title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == EXPECTED_TITLE.replace(" ", "_")
-
-
-def test_youtube_url():
-    url = youtube_tools.generate_youtube_url(TRACK_URL, meta_tags)
-    assert url == EXPECTED_YT_URL
-
-
-def test_youtube_title():
-    global content
-    content = youtube_tools.go_pafy(TRACK_URL, meta_tags)
-    title = youtube_tools.get_youtube_title(content)
-    assert title == EXPECTED_YT_TITLE
-
-
-def test_check_track_exists_before_download(tmpdir):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    songname = internals.format_string(const.args.file_format, meta_tags)
-    global file_name
-    file_name = internals.sanitize_title(songname)
-    check = spotdl.check_exists(file_name, TRACK_URL, meta_tags)
-    assert check == expect_check
-
-
-class TestDownload:
-    def test_m4a(self):
-        expect_download = True
-        download = youtube_tools.download_song(file_name + ".m4a", content)
-        assert download == expect_download
-
-    def test_webm(self):
-        expect_download = True
-        download = youtube_tools.download_song(file_name + ".webm", content)
-        assert download == expect_download
-
-
-class TestFFmpeg:
-    def test_convert_from_webm_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".webm", file_name + ".mp3", const.args.folder
-        )
-        assert return_code == expect_return_code
-
-    def test_convert_from_webm_to_m4a(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".webm", file_name + ".m4a", const.args.folder
-        )
-        assert return_code == expect_return_code
-
-    def test_convert_from_m4a_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".m4a", file_name + ".mp3", const.args.folder
-        )
-        assert return_code == expect_return_code
-
-    def test_convert_from_m4a_to_webm(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".m4a", file_name + ".webm", const.args.folder
-        )
-        assert return_code == expect_return_code
-
-    def test_convert_from_m4a_to_flac(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".m4a", file_name + ".flac", const.args.folder
-        )
-        assert return_code == expect_return_code
-
-
-class TestAvconv:
-    def test_convert_from_m4a_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(
-            file_name + ".m4a", file_name + ".mp3", const.args.folder, avconv=True
-        )
-        assert return_code == expect_return_code
-
-
-class TestEmbedMetadata:
-    def test_embed_in_mp3(self):
-        expect_embed = True
-        global track_path
-        track_path = os.path.join(const.args.folder, file_name)
-        embed = metadata.embed(track_path + ".mp3", meta_tags)
-        assert embed == expect_embed
-
-    def test_embed_in_m4a(self):
-        expect_embed = True
-        embed = metadata.embed(track_path + ".m4a", meta_tags)
-        os.remove(track_path + ".m4a")
-        assert embed == expect_embed
-
-    def test_embed_in_webm(self):
-        expect_embed = False
-        embed = metadata.embed(track_path + ".webm", meta_tags)
-        os.remove(track_path + ".webm")
-        assert embed == expect_embed
-
-    def test_embed_in_flac(self):
-        expect_embed = True
-        embed = metadata.embed(track_path + ".flac", meta_tags)
-        os.remove(track_path + ".flac")
-        assert embed == expect_embed
-
-
-def test_check_track_exists_after_download():
-    expect_check = True
-    check = spotdl.check_exists(file_name, TRACK_URL, meta_tags)
-    os.remove(track_path + ".mp3")
-    assert check == expect_check

test/test_without_metadata.py

@@ -1,129 +0,0 @@
-import os
-import builtins
-
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import spotdl
-
-import loader
-
-loader.load_defaults()
-
-YT_API_KEY = "AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90"
-
-TRACK_SEARCH = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
-EXPECTED_TITLE = TRACK_SEARCH
-EXPECTED_YT_URL = "http://youtube.com/watch?v=qOOcy2-tmbk"
-EXPECTED_YT_URLS = (EXPECTED_YT_URL, "http://youtube.com/watch?v=5USR1Omo7f0")
-
-RESULT_COUNT_SEARCH = "she is still sleeping SAO"
-
-EXPECTED_YT_API_KEY = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
-EXPECTED_YT_API_KEY_CUSTOM = "some_api_key"
-
-
-class TestYouTubeAPIKeys:
-    def test_custom(self):
-        const.args.youtube_api_key = EXPECTED_YT_API_KEY_CUSTOM
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == EXPECTED_YT_API_KEY_CUSTOM
-
-    def test_default(self):
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == EXPECTED_YT_API_KEY
-
-
-def test_metadata():
-    expect_metadata = None
-    global metadata
-    metadata = spotify_tools.generate_metadata(TRACK_SEARCH)
-    assert metadata == expect_metadata
-
-
-class TestArgsManualResultCount:
-    # Regresson test for issue #264
-    def test_scrape(self):
-        const.args.manual = True
-        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-        video_ids = url.scrape(bestmatch=False)
-        # Web scraping gives us all videos on the 1st page
-        assert len(video_ids) == 20
-
-    def test_api(self):
-        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-        video_ids = url.api(bestmatch=False)
-        const.args.manual = False
-        # API gives us 50 videos (or as requested)
-        assert len(video_ids) == 50
-
-
-class TestYouTubeURL:
-    def test_only_music_category(self):
-        const.args.music_videos_only = True
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        # YouTube keeps changing its results
-        assert url in EXPECTED_YT_URLS
-
-    def test_all_categories(self):
-        const.args.music_videos_only = False
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        assert url == EXPECTED_YT_URL
-
-    def test_args_manual(self, monkeypatch):
-        const.args.manual = True
-        monkeypatch.setattr("builtins.input", lambda x: "1")
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        assert url == EXPECTED_YT_URL
-
-    def test_args_manual_none(self, monkeypatch):
-        expect_url = None
-        monkeypatch.setattr("builtins.input", lambda x: "0")
-        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        const.args.manual = False
-        assert url == expect_url
-
-
-class TestYouTubeTitle:
-    def test_single_download_with_youtube_api(self):
-        global content
-        global title
-        const.args.youtube_api_key = YT_API_KEY
-        youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(TRACK_SEARCH, metadata)
-        title = youtube_tools.get_youtube_title(content)
-        assert title == EXPECTED_TITLE
-
-    def test_download_from_list_without_youtube_api(self):
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(TRACK_SEARCH, metadata)
-        title = youtube_tools.get_youtube_title(content, 1)
-        assert title == "1. {0}".format(EXPECTED_TITLE)
-
-
-def test_check_exists(tmpdir):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    global file_name
-    file_name = internals.sanitize_title(title)
-    check = spotdl.check_exists(file_name, TRACK_SEARCH, metadata)
-    assert check == expect_check
-
-
-class TestDownload:
-    def test_webm(self):
-        # content does not have any .webm audiostream
-        expect_download = False
-        download = youtube_tools.download_song(file_name + ".webm", content)
-        assert download == expect_download
-
-    def test_other(self):
-        expect_download = False
-        download = youtube_tools.download_song(file_name + ".fake_extension", content)
-        assert download == expect_download

test/test_youtube_tools.py

@@ -0,0 +1,223 @@
+import os
+import builtins
+
+from spotdl import const
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+from spotdl import downloader
+
+import loader
+import pytest
+
+loader.load_defaults()
+
+YT_API_KEY = "AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90"
+
+TRACK_SEARCH = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
+EXPECTED_TITLE = TRACK_SEARCH
+EXPECTED_YT_URL = "http://youtube.com/watch?v=qOOcy2-tmbk"
+
+RESULT_COUNT_SEARCH = "she is still sleeping SAO"
+
+EXPECTED_YT_API_KEY = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
+EXPECTED_YT_API_KEY_CUSTOM = "some_api_key"
+
+
+class TestYouTubeAPIKeys:
+    def test_custom(self):
+        const.args.youtube_api_key = EXPECTED_YT_API_KEY_CUSTOM
+        youtube_tools.set_api_key()
+        key = youtube_tools.pafy.g.api_key
+        assert key == EXPECTED_YT_API_KEY_CUSTOM
+
+    def test_default(self):
+        const.args.youtube_api_key = None
+        youtube_tools.set_api_key()
+        key = youtube_tools.pafy.g.api_key
+        assert key == EXPECTED_YT_API_KEY
+
+
+@pytest.fixture(scope="module")
+def metadata_fixture():
+    metadata = spotify_tools.generate_metadata(TRACK_SEARCH)
+    return metadata
+
+
+def test_metadata(metadata_fixture):
+    expect_metadata = None
+    assert metadata_fixture == expect_metadata
+
+
+class TestArgsManualResultCount:
+    # Regresson test for issue #264
+    def test_scrape(self):
+        const.args.manual = True
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
+        video_ids = url.scrape(bestmatch=False)
+        # Web scraping gives us all videos on the 1st page
+        assert len(video_ids) == 20
+
+    def test_api(self):
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
+        video_ids = url.api(bestmatch=False)
+        const.args.manual = False
+        # API gives us 50 videos (or as requested)
+        assert len(video_ids) == 50
+
+
+class TestYouTubeURL:
+    def test_only_music_category(self, metadata_fixture):
+        const.args.music_videos_only = True
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_all_categories(self, metadata_fixture):
+        const.args.music_videos_only = False
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_args_manual(self, metadata_fixture, monkeypatch):
+        const.args.manual = True
+        monkeypatch.setattr("builtins.input", lambda x: "1")
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_args_manual_none(self, metadata_fixture, monkeypatch):
+        expect_url = None
+        monkeypatch.setattr("builtins.input", lambda x: "0")
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        const.args.manual = False
+        assert url == expect_url
+
+
+@pytest.fixture(scope="module")
+def content_fixture(metadata_fixture):
+    content = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
+    return content
+
+
+# True = Metadata must be fetched from Spotify
+# False = Metadata must be fetched from YouTube
+# None = Metadata must be `None`
+
+MATCH_METADATA_NO_FALLBACK_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
+    ("Linux Talk | Working with Drives and Filesystems", None)
+]
+
+MATCH_METADATA_FALLBACK_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", False),
+    ("Linux Talk | Working with Drives and Filesystems", False)
+]
+
+MATCH_METADATA_NO_METADATA_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", None),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
+    ("Linux Talk | Working with Drives and Filesystems", None)
+]
+
+
+class TestMetadataOrigin:
+    def match_metadata(self, track, metadata_type):
+        _, metadata = youtube_tools.match_video_and_metadata(track)
+        if metadata_type is None:
+            assert metadata == metadata_type
+        else:
+            assert metadata["spotify_metadata"] == metadata_type
+
+    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_NO_FALLBACK_TEST_TABLE)
+    def test_match_metadata_with_no_fallback(self, track, metadata_type, content_fixture, monkeypatch):
+        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+        const.args.no_fallback_metadata = True
+        self.match_metadata(track, metadata_type)
+
+    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_FALLBACK_TEST_TABLE)
+    def test_match_metadata_with_fallback(self, track, metadata_type, content_fixture, monkeypatch):
+        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+        const.args.no_fallback_metadata = False
+        self.match_metadata(track, metadata_type)
+
+    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_NO_METADATA_TEST_TABLE)
+    def test_match_metadata_with_no_metadata(self, track, metadata_type, content_fixture, monkeypatch):
+        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+        const.args.no_metadata = True
+        self.match_metadata(track, metadata_type)
+
+
+@pytest.fixture(scope="module")
+def title_fixture(content_fixture):
+    title = youtube_tools.get_youtube_title(content_fixture)
+    return title
+
+
+class TestYouTubeTitle:
+    def test_single_download_with_youtube_api(self, title_fixture):
+        const.args.youtube_api_key = YT_API_KEY
+        youtube_tools.set_api_key()
+        assert title_fixture == EXPECTED_TITLE
+
+    def test_download_from_list_without_youtube_api(
+        self, metadata_fixture, content_fixture
+    ):
+        const.args.youtube_api_key = None
+        youtube_tools.set_api_key()
+        content_fixture = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
+        title = youtube_tools.get_youtube_title(content_fixture, 1)
+        assert title == "1. {0}".format(EXPECTED_TITLE)
+
+
+@pytest.fixture(scope="module")
+def filename_fixture(title_fixture):
+    filename = internals.sanitize_title(title_fixture)
+    return filename
+
+
+def test_check_exists(metadata_fixture, filename_fixture, tmpdir):
+    expect_check = False
+    const.args.folder = str(tmpdir)
+    # prerequisites for determining filename
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(TRACK_SEARCH)
+    assert check == expect_check
+
+
+def test_generate_m3u(tmpdir, monkeypatch):
+    monkeypatch.setattr(youtube_tools.GenerateYouTubeURL, "_fetch_response", loader.monkeypatch_youtube_search_page)
+    expect_m3u = (
+        "#EXTM3U\n\n"
+        "#EXTINF:208,Janji - Heroes Tonight (feat. Johnning) [NCS Release]\n"
+        "http://www.youtube.com/watch?v=3nQNiWdeH2Q\n"
+        "#EXTINF:226,Alan Walker - Spectre [NCS Release]\n"
+        "http://www.youtube.com/watch?v=AOeY-nDp7hI\n"
+    )
+    m3u_track_file = os.path.join(str(tmpdir), "m3u_test.txt")
+    with open(m3u_track_file, "w") as track_file:
+        track_file.write("\nhttps://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD")
+        track_file.write("\nhttp://www.youtube.com/watch?v=AOeY-nDp7hI")
+    youtube_tools.generate_m3u(m3u_track_file)
+    m3u_file = "{}.m3u".format(m3u_track_file.split(".")[0])
+    with open(m3u_file, "r") as m3u_in:
+        m3u = m3u_in.readlines()
+    assert "".join(m3u) == expect_m3u
+
+
+class TestDownload:
+    def test_webm(self, content_fixture, filename_fixture, monkeypatch):
+        # content_fixture does not have any .webm audiostream
+        expect_download = False
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
+        download = youtube_tools.download_song(
+            filename_fixture + ".webm", content_fixture
+        )
+        assert download == expect_download
+
+    def test_other(self, content_fixture, filename_fixture, monkeypatch):
+        expect_download = False
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
+        download = youtube_tools.download_song(
+            filename_fixture + ".fake_extension", content_fixture
+        )
+        assert download == expect_download