Merge pull request #641 from ritiek/release-v1.2.3

Bump version for v1.2.3 release

.travis.yml

@@ -1,6 +1,4 @@
-dist: xenial
 language: python
-sudo: required
 python:
   - "3.4"
   - "3.5"
@@ -40,7 +38,7 @@ install:
   - tinydownload 07426048687547254773 -o ~/bin/ffmpeg
   - chmod 755 ~/bin/ffmpeg
   - xdg-user-dirs-update
-script: python -m pytest test --cov=.
+script: travis_retry pytest --cov=.
 after_success:
   - pip install codecov
   - codecov

CHANGES.md

@@ -5,14 +5,31 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [1.2.3] - 2019-12-20
 ### Added
--
+- Added `--no-remove-original-file` ([@NightMachinary](https://github.com/NightMachinary)) (#580)
+- Added leading Zeros in `track_number` for correct sorting ([@Dsujan](https://github.com/Dsujan)) (#592)
+- Added `track_id` key for `--file-format` parameter ([@kadaliao](https://github.com/kadaliao)) (#568)
 
 ### Fixed
--
+- Some tracks randomly fail to download with Pafy v0.5.5 ([@ritiek](https://github.com/ritiek)) (#638)
+- Generate list error --write-m3u ([@arthurlutz](https://github.com/arthurlutz)) (#559)
 
 ### Changed
--
+- Fetch lyrics from Genius and fallback to LyricWikia if not found ([@ritiek](https://github.com/ritiek)) (#585)
+
+## [1.2.2] - 2019-06-03
+### Fixed
+- Patch bug in Pafy to prefer secure HTTPS ([@ritiek](https://github.com/ritiek)) (#558)
+
+## [1.2.1] - 2019-04-28
+### Fixed
+- Patch bug in Pafy when fetching audiostreams with latest youtube-dl ([@ritiek](https://github.com/ritiek)) (#539)
+
+### Changed
+- Removed duplicate debug log entry from `internals.trim_song` ([@ritiek](https://github.com/ritiek)) (#519)
+- Fix YAMLLoadWarning ([@cyberboysumanjay](https://github.com/cyberboysumanjay)) (#517)
 
 ## [1.2.0] - 2019-03-01
 ### Added

CONTRIBUTING.md

@@ -24,7 +24,7 @@ don't feel bad. Open an issue any way!
 unless mentioned otherwise.
 - Code should be formatted using [black](https://github.com/ambv/black). Don't worry if you forgot or don't know how to do this, the codebase will be black-formatted with each release.
 - All tests are placed in the [test directory](https://github.com/ritiek/spotify-downloader/tree/master/test). We use [pytest](https://github.com/pytest-dev/pytest)
-to run the test suite: `$ python3 -m pytest test`.
+to run the test suite: `$ pytest`.
 If you don't have pytest, you can install it with `$ pip3 install pytest`.
 - Add a note about the changes, your GitHub username and a reference to the PR to the `Unreleased` section of the [`CHANGES.md`](CHANGES.md) file (see existing releases for examples), add the appropriate section ("Added", "Changed", "Fixed" etc.) if necessary. You don't have to increment version numbers. See https://keepachangelog.com/en/1.0.0/ for more information.
 - If you are planning to work on something big, let us know through an issue. So we can discuss more about it.

README.md

@@ -11,7 +11,7 @@
 - Can also download a song by entering its artist and song name (in case if you don't have the Spotify's HTTP link for some song).
 - Automatically applies metadata to the downloaded song which includes:
 
-  - `Title`, `Artist`, `Album`, `Album art`, `Lyrics` (if found on [lyrics wikia](http://lyrics.wikia.com)), `Album artist`, `Genre`, `Track number`, `Disc number`, `Release date`, and more...
+  - `Title`, `Artist`, `Album`, `Album art`, `Lyrics` (if found either on [Genius](https://genius.com/) or [LyricsWikia](http://lyrics.wikia.com)), `Album artist`, `Genre`, `Track number`, `Disc number`, `Release date`, and more...
 
 - Works straight out of the box and does not require you to generate or mess with your API keys (already included).
 
@@ -30,7 +30,7 @@ If you still need to use Python 2 - check out the (outdated)
 spotify-downloader works with all major distributions and even on low-powered devices such as a Raspberry Pi.
 
 spotify-downloader can be installed via pip with:
-```
+```console
 $ pip3 install spotdl
 ```
 
@@ -41,7 +41,7 @@ page for detailed OS-specific instructions to get it and other dependencies it r
 
 For the most basic usage, downloading tracks is as easy as
 
-```
+```console
 $ spotdl --song https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ
 $ spotdl --song "ncs - spectre"
 ```
@@ -49,7 +49,7 @@ $ spotdl --song "ncs - spectre"
 For downloading playlist and albums, you need to first load all the tracks into text file and then pass
 this text file to `--list` argument. Here is how you would do it for a playlist
 
-```
+```console
 $ spotdl --playlist https://open.spotify.com/user/nocopyrightsounds/playlist/7sZbq8QGyMnhKPcLJvCUFD
 INFO: Writing 62 tracks to ncs-releases.txt
 $ spotdl --list ncs-releases.txt
@@ -73,8 +73,8 @@ Check out [CONTRIBUTING.md](CONTRIBUTING.md) for more info.
 
 ## Running Tests
 
-```
+```console
-$ python3 -m pytest test
+$ pytest
 ```
 
 Obviously this requires the `pytest` module to be installed.

setup.py

@@ -10,7 +10,7 @@ setup(
     name="spotdl",
     # Tests are included automatically:
     # https://docs.python.org/3.6/distutils/sourcedist.html#specifying-the-files-to-distribute
-    packages=["spotdl"],
+    packages=["spotdl", "spotdl.lyrics", "spotdl.lyrics.providers"],
     version=spotdl.__version__,
     install_requires=[
         "pathlib >= 1.0.1",
@@ -46,7 +46,6 @@ setup(
         "metadata",
     ],
     classifiers=[
-        "Development Status :: 4 - Beta",
         "Intended Audience :: End Users/Desktop",
         "License :: OSI Approved :: MIT License",
         "Programming Language :: Python",

spotdl/__init__.py

@@ -1 +1 @@
-__version__ = "1.2.0"
+__version__ = "1.2.3"

spotdl/convert.py

@@ -16,7 +16,14 @@ https://trac.ffmpeg.org/wiki/Encode/AAC
 """
 
 
-def song(input_song, output_song, folder, avconv=False, trim_silence=False):
+def song(
+    input_song,
+    output_song,
+    folder,
+    avconv=False,
+    trim_silence=False,
+    delete_original=True,
+):
     """ Do the audio format conversion. """
     if avconv and trim_silence:
         raise ValueError("avconv does not support trim_silence")
@@ -28,7 +35,9 @@ def song(input_song, output_song, folder, avconv=False, trim_silence=False):
     else:
         return 0
 
-    convert = Converter(input_song, output_song, folder, delete_original=True)
+    convert = Converter(
+        input_song, output_song, folder, delete_original=delete_original
+    )
     if avconv:
         exit_code, command = convert.with_avconv()
     else:
@@ -97,7 +106,9 @@ class Converter:
         return code, command
 
     def with_ffmpeg(self, trim_silence=False):
-        ffmpeg_pre = "ffmpeg -y "
+        ffmpeg_pre = (
+            "ffmpeg -y -nostdin "
+        )  # -nostdin is necessary for spotdl to be able to run in the backgroung.
 
         if not log.level == 10:
             ffmpeg_pre += "-hide_banner -nostats -v panic "

spotdl/downloader.py

@@ -96,6 +96,7 @@ class Downloader:
         self.raw_song = raw_song
         self.number = number
         self.content, self.meta_tags = youtube_tools.match_video_and_metadata(raw_song)
+        self.total_songs = int(self.meta_tags["total_tracks"])
 
     def download_single(self):
         """ Logic behind downloading a song. """
@@ -133,6 +134,7 @@ class Downloader:
                     const.args.folder,
                     avconv=const.args.avconv,
                     trim_silence=const.args.trim_silence,
+                    delete_original=not const.args.no_remove_original,
                 )
             except FileNotFoundError:
                 encoder = "avconv" if const.args.avconv else "ffmpeg"
@@ -157,7 +159,10 @@ class Downloader:
     def refine_songname(self, songname):
         if self.meta_tags is not None:
             refined_songname = internals.format_string(
-                const.args.file_format, self.meta_tags, slugification=True
+                const.args.file_format,
+                self.meta_tags,
+                slugification=True,
+                total_songs=self.total_songs,
             )
             log.debug(
                 'Refining songname from "{0}" to "{1}"'.format(

spotdl/handle.py

@@ -15,6 +15,7 @@ _LOG_LEVELS_STR = ["INFO", "WARNING", "ERROR", "DEBUG"]
 
 default_conf = {
     "spotify-downloader": {
+        "no-remove-original": False,
         "manual": False,
         "no-metadata": False,
         "no-fallback-metadata": False,
@@ -36,7 +37,7 @@ default_conf = {
         "write-successful": None,
         "log-level": "INFO",
         "spotify_client_id": "4fe3fecfe5334023a1472516cc99d805",
-        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c"
+        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c",
     }
 }
 
@@ -58,7 +59,7 @@ def merge(default, config):
 def get_config(config_file):
     try:
         with open(config_file, "r") as ymlfile:
-            cfg = yaml.load(ymlfile)
+            cfg = yaml.safe_load(ymlfile)
     except FileNotFoundError:
         log.info("Writing default configuration to {0}:".format(config_file))
         with open(config_file, "w") as ymlfile:
@@ -139,6 +140,13 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         help="choose the track to download manually from a list of matching tracks",
         action="store_true",
     )
+    parser.add_argument(
+        "-nr",
+        "--no-remove-original",
+        default=config["no-remove-original"],
+        help="do not remove the original file after conversion",
+        action="store_true",
+    )
     parser.add_argument(
         "-nm",
         "--no-metadata",
@@ -272,13 +280,13 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
         "-sci",
         "--spotify-client-id",
         default=config["spotify_client_id"],
-        help=argparse.SUPPRESS
+        help=argparse.SUPPRESS,
     )
     parser.add_argument(
         "-scs",
         "--spotify-client-secret",
         default=config["spotify_client_secret"],
-        help=argparse.SUPPRESS
+        help=argparse.SUPPRESS,
     )
     parser.add_argument(
         "-c", "--config", default=None, help="path to custom config.yml file"
@@ -312,11 +320,12 @@ def get_arguments(raw_args=None, to_group=True, to_merge=True):
     if parsed.avconv and parsed.trim_silence:
         parser.error("--trim-silence can only be used with FFmpeg")
 
-    if parsed.write_to and not (parsed.playlist \
+    if parsed.write_to and not (
-            or parsed.album \
+        parsed.playlist or parsed.album or parsed.all_albums or parsed.username
-            or parsed.all_albums \
+    ):
-            or parsed.username):
+        parser.error(
-        parser.error("--write-to can only be used with --playlist, --album, --all-albums, or --username")
+            "--write-to can only be used with --playlist, --album, --all-albums, or --username"
+        )
 
     parsed.log_level = log_leveller(parsed.log_level)
 

spotdl/internals.py

@@ -1,8 +1,10 @@
 from logzero import logger as log
 import os
 import sys
+import math
 import urllib.request
 
+
 from spotdl import const
 
 try:
@@ -30,6 +32,7 @@ formats = {
     9: "track_number",
     10: "total_tracks",
     11: "isrc",
+    12: "track_id",
 }
 
 
@@ -51,7 +54,6 @@ def input_link(links):
 
 def trim_song(tracks_file):
     """ Remove the first song from file. """
-    log.debug("Removing downloaded song from tracks file")
     with open(tracks_file, "r") as file_in:
         data = file_in.read().splitlines(True)
     with open(tracks_file, "w") as file_out:
@@ -74,7 +76,9 @@ def is_youtube(raw_song):
     return status
 
 
-def format_string(string_format, tags, slugification=False, force_spaces=False):
+def format_string(
+    string_format, tags, slugification=False, force_spaces=False, total_songs=0
+):
     """ Generate a string of the format '[artist] - [song]' for the given spotify song. """
     format_tags = dict(formats)
     format_tags[0] = tags["name"]
@@ -89,14 +93,22 @@ def format_string(string_format, tags, slugification=False, force_spaces=False):
     format_tags[9] = tags["track_number"]
     format_tags[10] = tags["total_tracks"]
     format_tags[11] = tags["external_ids"]["isrc"]
+    format_tags[12] = tags["id"]
 
     format_tags_sanitized = {
         k: sanitize_title(str(v), ok="'-_()[]{}") if slugification else str(v)
         for k, v in format_tags.items()
     }
+    # calculating total digits presnet in total_songs to prepare a zfill.
+    total_digits = 0 if total_songs == 0 else int(math.log10(total_songs)) + 1
 
     for x in formats:
         format_tag = "{" + formats[x] + "}"
+        # Making consistent track number by prepending zero
+        # on it according to number of digits in total songs
+        if format_tag == "{track_number}":
+            format_tags_sanitized[x] = format_tags_sanitized[x].zfill(total_digits)
+
         string_format = string_format.replace(format_tag, format_tags_sanitized[x])
 
     if const.args.no_spaces and not force_spaces:
@@ -259,7 +271,7 @@ def remove_duplicates(tracks):
 def content_available(url):
     try:
         response = urllib.request.urlopen(url)
-    except HTTPError:
+    except urllib.request.HTTPError:
         return False
     else:
         return response.getcode() < 300

spotdl/lyrics/__init__.py

@@ -0,0 +1 @@
+from spotdl.lyrics.lyric_base import LyricBase

spotdl/lyrics/exceptions.py

@@ -0,0 +1,5 @@
+class LyricsNotFound(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super(LyricsNotFound, self).__init__(message)

spotdl/lyrics/lyric_base.py

@@ -0,0 +1,14 @@
+import lyricwikia
+
+from abc import ABC
+from abc import abstractmethod
+
+
+class LyricBase(ABC):
+    @abstractmethod
+    def __init__(self, artist, song):
+        pass
+
+    @abstractmethod
+    def get_lyrics(self, linesep="\n", timeout=None):
+        pass

spotdl/lyrics/providers/__init__.py

@@ -0,0 +1,4 @@
+from spotdl.lyrics.providers.genius import Genius
+from spotdl.lyrics.providers.lyricwikia_wrapper import LyricWikia
+
+LyricClasses = (Genius, LyricWikia)

spotdl/lyrics/providers/genius.py

@@ -0,0 +1,47 @@
+from bs4 import BeautifulSoup
+import urllib.request
+
+from spotdl.lyrics.lyric_base import LyricBase
+from spotdl.lyrics.exceptions import LyricsNotFound
+
+BASE_URL = "https://genius.com"
+
+
+class Genius(LyricBase):
+    def __init__(self, artist, song):
+        self.artist = artist
+        self.song = song
+        self.base_url = BASE_URL
+
+    def _guess_lyric_url(self):
+        query = "/{} {} lyrics".format(self.artist, self.song)
+        query = query.replace(" ", "-")
+        encoded_query = urllib.request.quote(query)
+        lyric_url = self.base_url + encoded_query
+        return lyric_url
+
+    def _fetch_page(self, url, timeout=None):
+        request = urllib.request.Request(url)
+        request.add_header("User-Agent", "urllib")
+        try:
+            response = urllib.request.urlopen(request, timeout=timeout)
+        except urllib.request.HTTPError:
+            raise LyricsNotFound(
+                "Could not find lyrics for {} - {} at URL: {}".format(
+                    self.artist, self.song, url
+                )
+            )
+        else:
+            return response.read()
+
+    def _get_lyrics_text(self, html):
+        soup = BeautifulSoup(html, "html.parser")
+        lyrics_paragraph = soup.find("p")
+        lyrics = lyrics_paragraph.get_text()
+        return lyrics
+
+    def get_lyrics(self, linesep="\n", timeout=None):
+        url = self._guess_lyric_url()
+        html_page = self._fetch_page(url, timeout=timeout)
+        lyrics = self._get_lyrics_text(html_page)
+        return lyrics.replace("\n", linesep)

spotdl/lyrics/providers/lyricwikia_wrapper.py

@@ -0,0 +1,18 @@
+import lyricwikia
+
+from spotdl.lyrics.lyric_base import LyricBase
+from spotdl.lyrics.exceptions import LyricsNotFound
+
+
+class LyricWikia(LyricBase):
+    def __init__(self, artist, song):
+        self.artist = artist
+        self.song = song
+
+    def get_lyrics(self, linesep="\n", timeout=None):
+        try:
+            lyrics = lyricwikia.get_lyrics(self.artist, self.song, linesep, timeout)
+        except lyricwikia.LyricsNotFound as e:
+            raise LyricsNotFound(e.args[0])
+        else:
+            return lyrics

spotdl/lyrics/providers/tests/test_genius.py

@@ -0,0 +1,37 @@
+from spotdl.lyrics import LyricBase
+from spotdl.lyrics import exceptions
+from spotdl.lyrics.providers import Genius
+
+import urllib.request
+import pytest
+
+
+class TestGenius:
+    def test_subclass(self):
+        assert issubclass(Genius, LyricBase)
+
+    @pytest.fixture(scope="module")
+    def track(self):
+        return Genius("artist", "song")
+
+    def test_base_url(self, track):
+        assert track.base_url == "https://genius.com"
+
+    def test_get_lyrics(self, track, monkeypatch):
+        def mocked_urlopen(url, timeout=None):
+            class DummyHTTPResponse:
+                def read(self):
+                    return "<p>amazing lyrics!</p>"
+
+            return DummyHTTPResponse()
+
+        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
+        assert track.get_lyrics() == "amazing lyrics!"
+
+    def test_lyrics_not_found_error(self, track, monkeypatch):
+        def mocked_urlopen(url, timeout=None):
+            raise urllib.request.HTTPError("", "", "", "", "")
+
+        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
+        with pytest.raises(exceptions.LyricsNotFound):
+            track.get_lyrics()

spotdl/lyrics/providers/tests/test_lyricwikia_wrapper.py

@@ -0,0 +1,35 @@
+import lyricwikia
+
+from spotdl.lyrics import LyricBase
+from spotdl.lyrics import exceptions
+from spotdl.lyrics.providers import LyricWikia
+
+import pytest
+
+
+class TestLyricWikia:
+    def test_subclass(self):
+        assert issubclass(LyricWikia, LyricBase)
+
+    def test_get_lyrics(self, monkeypatch):
+        # `LyricWikia` class uses the 3rd party method `lyricwikia.get_lyrics`
+        # internally and there is no need to test a 3rd party library as they
+        # have their own implementation of tests.
+        monkeypatch.setattr(
+            "lyricwikia.get_lyrics", lambda a, b, c, d: "awesome lyrics!"
+        )
+        track = LyricWikia("Lyricwikia", "Lyricwikia")
+        assert track.get_lyrics() == "awesome lyrics!"
+
+    def test_lyrics_not_found_error(self, monkeypatch):
+        def lyricwikia_lyrics_not_found(msg):
+            raise lyricwikia.LyricsNotFound(msg)
+
+        # Wrap `lyricwikia.LyricsNotFound` with `exceptions.LyricsNotFound` error.
+        monkeypatch.setattr(
+            "lyricwikia.get_lyrics",
+            lambda a, b, c, d: lyricwikia_lyrics_not_found("Nope, no lyrics."),
+        )
+        track = LyricWikia("Lyricwikia", "Lyricwikia")
+        with pytest.raises(exceptions.LyricsNotFound):
+            track.get_lyrics()

spotdl/metadata.py

@@ -80,7 +80,9 @@ class EmbedMetadata:
         audiofile["TYER"] = TYER(encoding=3, text=meta_tags["year"])
         if meta_tags["publisher"]:
             audiofile["TPUB"] = TPUB(encoding=3, text=meta_tags["publisher"])
-        audiofile["COMM"] = COMM(encoding=3, text=meta_tags["external_urls"][self.provider])
+        audiofile["COMM"] = COMM(
+            encoding=3, text=meta_tags["external_urls"][self.provider]
+        )
         if meta_tags["lyrics"]:
             audiofile["USLT"] = USLT(
                 encoding=3, desc=u"Lyrics", text=meta_tags["lyrics"]

spotdl/patcher.py

@@ -1,3 +1,4 @@
+from pafy import backend_youtube_dl
 import pafy
 
 from spotdl import internals
@@ -10,22 +11,54 @@ def _getbestthumb(self):
 
     part_url = "https://i.ytimg.com/vi/%s/" % self.videoid
     # Thumbnail resolution sorted in descending order
-    thumbs = ("maxresdefault.jpg",
+    thumbs = (
-              "sddefault.jpg",
+        "maxresdefault.jpg",
-              "hqdefault.jpg",
+        "sddefault.jpg",
-              "mqdefault.jpg",
+        "hqdefault.jpg",
-              "default.jpg")
+        "mqdefault.jpg",
+        "default.jpg",
+    )
     for thumb in thumbs:
         url = part_url + thumb
         if self._content_available(url):
             return url
 
 
+def _process_streams(self):
+    for format_index in range(len(self._ydl_info["formats"])):
+        try:
+            self._ydl_info["formats"][format_index]["url"] = self._ydl_info["formats"][
+                format_index
+            ]["fragment_base_url"]
+        except KeyError:
+            pass
+    return backend_youtube_dl.YtdlPafy._old_process_streams(self)
+
+
 @classmethod
 def _content_available(cls, url):
     return internals.content_available(url)
 
-def patch_pafy():
+
-    pafy.backend_shared.BasePafy._bestthumb = None
+class PatchPafy:
-    pafy.backend_shared.BasePafy._content_available = _content_available
+    """
-    pafy.backend_shared.BasePafy.getbestthumb = _getbestthumb
+    These patches have not been released by pafy on PyPI yet but
+    are useful to us.
+    """
+
+    def patch_getbestthumb(self):
+        # https://github.com/mps-youtube/pafy/pull/211
+        pafy.backend_shared.BasePafy._bestthumb = None
+        pafy.backend_shared.BasePafy._content_available = _content_available
+        pafy.backend_shared.BasePafy.getbestthumb = _getbestthumb
+
+    def patch_process_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/230
+        backend_youtube_dl.YtdlPafy._old_process_streams = (
+            backend_youtube_dl.YtdlPafy._process_streams
+        )
+        backend_youtube_dl.YtdlPafy._process_streams = _process_streams
+
+    def patch_insecure_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/235
+        pafy.g.def_ydl_opts["prefer_insecure"] = False

spotdl/spotdl.py

@@ -28,8 +28,9 @@ def match_args():
             track_dl.download_single()
     elif const.args.list:
         if const.args.write_m3u:
-            youtube_tools.generate_m3u(track_file=const.args.list,
+            youtube_tools.generate_m3u(
-                                       text_file=const.args.write_to)
+                track_file=const.args.list
+            )
         else:
             list_dl = downloader.ListDownloader(
                 tracks_file=const.args.list,
@@ -38,17 +39,21 @@ def match_args():
             )
             list_dl.download_list()
     elif const.args.playlist:
-        spotify_tools.write_playlist(playlist_url=const.args.playlist,
+        spotify_tools.write_playlist(
-                                     text_file=const.args.write_to)
+            playlist_url=const.args.playlist, text_file=const.args.write_to
+        )
     elif const.args.album:
-        spotify_tools.write_album(album_url=const.args.album,
+        spotify_tools.write_album(
-                                  text_file=const.args.write_to)
+            album_url=const.args.album, text_file=const.args.write_to
+        )
     elif const.args.all_albums:
-        spotify_tools.write_all_albums_from_artist(artist_url=const.args.all_albums,
+        spotify_tools.write_all_albums_from_artist(
-                                                   text_file=const.args.write_to)
+            artist_url=const.args.all_albums, text_file=const.args.write_to
+        )
     elif const.args.username:
-        spotify_tools.write_user_playlist(username=const.args.username,
+        spotify_tools.write_user_playlist(
-                                          text_file=const.args.write_to)
+            username=const.args.username, text_file=const.args.write_to
+        )
 
 
 def main():

spotdl/spotify_tools.py

@@ -1,6 +1,5 @@
 import spotipy
 import spotipy.oauth2 as oauth2
-import lyricwikia
 
 from slugify import slugify
 from titlecase import titlecase
@@ -12,6 +11,8 @@ import functools
 
 from spotdl import const
 from spotdl import internals
+from spotdl.lyrics.providers import LyricClasses
+from spotdl.lyrics.exceptions import LyricsNotFound
 
 spotify = None
 
@@ -36,6 +37,7 @@ def must_be_authorized(func, spotify=spotify):
             token = generate_token()
             spotify = spotipy.Spotify(auth=token)
             return func(*args, **kwargs)
+
     return wrapper
 
 
@@ -74,13 +76,16 @@ def generate_metadata(raw_song):
     meta_tags[u"total_tracks"] = album["tracks"]["total"]
 
     log.debug("Fetching lyrics")
+    meta_tags["lyrics"] = None
 
-    try:
+    for LyricClass in LyricClasses:
-        meta_tags["lyrics"] = lyricwikia.get_lyrics(
+        track = LyricClass(meta_tags["artists"][0]["name"], meta_tags["name"])
-            meta_tags["artists"][0]["name"], meta_tags["name"]
+        try:
-        )
+            meta_tags["lyrics"] = track.get_lyrics()
-    except lyricwikia.LyricsNotFound:
+        except LyricsNotFound:
-        meta_tags["lyrics"] = None
+            continue
+        else:
+            break
 
     # Some sugar
     meta_tags["year"], *_ = meta_tags["release_date"].split("-")

spotdl/youtube_tools.py

@@ -16,9 +16,13 @@ pafy.g.opener.addheaders.append(("Range", "bytes=0-"))
 
 # Implement unreleased methods on Pafy object
 # More info: https://github.com/mps-youtube/pafy/pull/211
-if pafy.__version__ <= "0.5.4":
+if pafy.__version__ <= "0.5.5":
     from spotdl import patcher
-    patcher.patch_pafy()
+
+    pafy_patcher = patcher.PatchPafy()
+    pafy_patcher.patch_getbestthumb()
+    pafy_patcher.patch_process_streams()
+    pafy_patcher.patch_insecure_streams()
 
 
 def set_api_key():
@@ -49,10 +53,13 @@ def match_video_and_metadata(track):
     """ Get and match track data from YouTube and Spotify. """
     meta_tags = None
 
-
     def fallback_metadata(meta_tags):
-        fallback_metadata_info = "Track not found on Spotify, falling back on YouTube metadata"
+        fallback_metadata_info = (
-        skip_fallback_metadata_warning = "Fallback condition not met, shall not embed metadata"
+            "Track not found on Spotify, falling back on YouTube metadata"
+        )
+        skip_fallback_metadata_warning = (
+            "Fallback condition not met, shall not embed metadata"
+        )
         if meta_tags is None:
             if const.args.no_fallback_metadata:
                 log.warning(skip_fallback_metadata_warning)
@@ -61,7 +68,6 @@ def match_video_and_metadata(track):
                 meta_tags = generate_metadata(content)
         return meta_tags
 
-
     if internals.is_youtube(track):
         log.debug("Input song is a YouTube URL")
         content = go_pafy(track, meta_tags=None)
@@ -92,25 +98,29 @@ def match_video_and_metadata(track):
 
 def generate_metadata(content):
     """ Fetch a song's metadata from YouTube. """
-    meta_tags = {"spotify_metadata": False,
+    meta_tags = {
-                 "name": content.title,
+        "spotify_metadata": False,
-                 "artists": [{"name": content.author}],
+        "name": content.title,
-                 "duration": content.length,
+        "artists": [{"name": content.author}],
-                 "external_urls": {"youtube": content.watchv_url},
+        "duration": content.length,
-                 "album": {"images" : [{"url": content.getbestthumb()}],
+        "external_urls": {"youtube": content.watchv_url},
-                           "artists": [{"name": None}],"name": None},
+        "album": {
-                 "year": content.published.split("-")[0],
+            "images": [{"url": content.getbestthumb()}],
-                 "release_date": content.published.split(" ")[0],
+            "artists": [{"name": None}],
-                 "type": "track",
+            "name": None,
-                 "disc_number": 1,
+        },
-                 "track_number": 1,
+        "year": content.published.split("-")[0],
-                 "total_tracks": 1,
+        "release_date": content.published.split(" ")[0],
-                 "publisher": None,
+        "type": "track",
-                 "external_ids": {"isrc": None},
+        "disc_number": 1,
-                 "lyrics": None,
+        "track_number": 1,
-                 "copyright": None,
+        "total_tracks": 1,
-                 "genre": None,
+        "publisher": None,
-                 }
+        "external_ids": {"isrc": None},
+        "lyrics": None,
+        "copyright": None,
+        "genre": None,
+    }
 
     return meta_tags
 

test/loader.py

@@ -20,7 +20,7 @@ def load_defaults():
 # so that we get same results even if YouTube changes the list/order of videos on their page.
 GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
 
+
 def monkeypatch_youtube_search_page(*args, **kwargs):
     fake_urlopen = urllib.request.urlopen(GIST_URL)
     return fake_urlopen
-

test/test_download_with_metadata.py

@@ -101,22 +101,34 @@ class TestDownload:
 
     def test_m4a(self, monkeypatch, filename_fixture):
         expect_download = True
-        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", self.blank_audio_generator)
+        monkeypatch.setattr(
-        monkeypatch.setattr("pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator)
+            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
-        download = youtube_tools.download_song(filename_fixture + ".m4a", pytest.content_fixture)
+        )
+        monkeypatch.setattr(
+            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
+        )
+        download = youtube_tools.download_song(
+            filename_fixture + ".m4a", pytest.content_fixture
+        )
         assert download == expect_download
 
     def test_webm(self, monkeypatch, filename_fixture):
         expect_download = True
-        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", self.blank_audio_generator)
+        monkeypatch.setattr(
-        monkeypatch.setattr("pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator)
+            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
-        download = youtube_tools.download_song(filename_fixture + ".webm", pytest.content_fixture)
+        )
+        monkeypatch.setattr(
+            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
+        )
+        download = youtube_tools.download_song(
+            filename_fixture + ".webm", pytest.content_fixture
+        )
         assert download == expect_download
 
 
 class TestFFmpeg:
     def test_convert_from_webm_to_mp3(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.webm -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         monkeypatch.setattr("os.remove", lambda x: None)
@@ -126,7 +138,7 @@ class TestFFmpeg:
         assert " ".join(command) == expect_command
 
     def test_convert_from_webm_to_m4a(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.webm -cutoff 20000 -codec:a aac -ar 44100 -b:a 192k -vn {0}.m4a".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -cutoff 20000 -codec:a aac -ar 44100 -b:a 192k -vn {0}.m4a".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         monkeypatch.setattr("os.remove", lambda x: None)
@@ -136,7 +148,7 @@ class TestFFmpeg:
         assert " ".join(command) == expect_command
 
     def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:v copy -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:v copy -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         monkeypatch.setattr("os.remove", lambda x: None)
@@ -146,7 +158,7 @@ class TestFFmpeg:
         assert " ".join(command) == expect_command
 
     def test_convert_from_m4a_to_webm(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:a libopus -vbr on -b:a 192k -vn {0}.webm".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a libopus -vbr on -b:a 192k -vn {0}.webm".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         monkeypatch.setattr("os.remove", lambda x: None)
@@ -156,7 +168,7 @@ class TestFFmpeg:
         assert " ".join(command) == expect_command
 
     def test_convert_from_m4a_to_flac(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a -codec:a flac -ar 44100 -b:a 192k -vn {0}.flac".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a flac -ar 44100 -b:a 192k -vn {0}.flac".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         monkeypatch.setattr("os.remove", lambda x: None)
@@ -166,7 +178,7 @@ class TestFFmpeg:
         assert " ".join(command) == expect_command
 
     def test_correct_container_for_m4a(self, filename_fixture, monkeypatch):
-        expect_command = "ffmpeg -y -hide_banner -nostats -v panic -i {0}.m4a.temp -acodec copy -b:a 192k -vn {0}.m4a".format(
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a.temp -acodec copy -b:a 192k -vn {0}.m4a".format(
             os.path.join(const.args.folder, filename_fixture)
         )
         _, command = convert.song(
@@ -176,6 +188,7 @@ class TestFFmpeg:
 
 
 class TestAvconv:
+    @pytest.mark.skip(reason="avconv is no longer provided with FFmpeg")
     def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
         monkeypatch.setattr("os.remove", lambda x: None)
         expect_command = "avconv -loglevel 0 -i {0}.m4a -ab 192k {0}.mp3 -y".format(

test/test_patcher.py

@@ -3,7 +3,9 @@ import pafy
 
 import pytest
 
-patcher.patch_pafy()
+pafy_patcher = patcher.PatchPafy()
+pafy_patcher.patch_getbestthumb()
+
 
 class TestPafyContentAvailable:
     pass
@@ -22,13 +24,12 @@ class TestMethodCalls:
 
     def test_pafy_getbestthumb(self, content_fixture):
         thumbnail = patcher._getbestthumb(content_fixture)
-        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/maxresdefault.jpg"
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/hqdefault.jpg"
 
     def test_pafy_getbestthumb_without_ytdl(self, content_fixture):
         content_fixture._ydl_info["thumbnails"][0]["url"] = None
         thumbnail = patcher._getbestthumb(content_fixture)
-        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/maxresdefault.jpg"
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/sddefault.jpg"
-
 
     def test_pafy_content_available(self):
         TestPafyContentAvailable._content_available = patcher._content_available

test/test_spotify_tools.py

@@ -115,7 +115,7 @@ def test_write_playlist(tmpdir):
     assert tracks == expect_tracks
 
 
-# XXX: Mock this test off if it fails in future
+# XXX: Monkeypatch these tests if they fail in future
 class TestFetchAlbum:
     @pytest.fixture(scope="module")
     def album_fixture(self):
@@ -131,7 +131,7 @@ class TestFetchAlbum:
         assert album_fixture["tracks"]["total"] == 15
 
 
-# XXX: Mock this test off if it fails in future
+# XXX: Monkeypatch these tests if they fail in future
 class TestFetchAlbumsFromArtist:
     @pytest.fixture(scope="module")
     def albums_from_artist_fixture(self):
@@ -141,7 +141,7 @@ class TestFetchAlbumsFromArtist:
         return albums
 
     def test_len(self, albums_from_artist_fixture):
-        assert len(albums_from_artist_fixture) == 52
+        assert len(albums_from_artist_fixture) == 54
 
     def test_zeroth_album_name(self, albums_from_artist_fixture):
         assert albums_from_artist_fixture[0]["name"] == "Revolution Radio"

test/test_youtube_tools.py

@@ -104,19 +104,19 @@ def content_fixture(metadata_fixture):
 MATCH_METADATA_NO_FALLBACK_TEST_TABLE = [
     ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
     ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
-    ("Linux Talk | Working with Drives and Filesystems", None)
+    ("Linux Talk | Working with Drives and Filesystems", None),
 ]
 
 MATCH_METADATA_FALLBACK_TEST_TABLE = [
     ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
     ("http://youtube.com/watch?v=3nQNiWdeH2Q", False),
-    ("Linux Talk | Working with Drives and Filesystems", False)
+    ("Linux Talk | Working with Drives and Filesystems", False),
 ]
 
 MATCH_METADATA_NO_METADATA_TEST_TABLE = [
     ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", None),
     ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
-    ("Linux Talk | Working with Drives and Filesystems", None)
+    ("Linux Talk | Working with Drives and Filesystems", None),
 ]
 
 
@@ -128,21 +128,37 @@ class TestMetadataOrigin:
         else:
             assert metadata["spotify_metadata"] == metadata_type
 
-    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_NO_FALLBACK_TEST_TABLE)
+    @pytest.mark.parametrize(
-    def test_match_metadata_with_no_fallback(self, track, metadata_type, content_fixture, monkeypatch):
+        "track, metadata_type", MATCH_METADATA_NO_FALLBACK_TEST_TABLE
-        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+    )
+    def test_match_metadata_with_no_fallback(
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
         const.args.no_fallback_metadata = True
         self.match_metadata(track, metadata_type)
 
     @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_FALLBACK_TEST_TABLE)
-    def test_match_metadata_with_fallback(self, track, metadata_type, content_fixture, monkeypatch):
+    def test_match_metadata_with_fallback(
-        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
         const.args.no_fallback_metadata = False
         self.match_metadata(track, metadata_type)
 
-    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_NO_METADATA_TEST_TABLE)
+    @pytest.mark.parametrize(
-    def test_match_metadata_with_no_metadata(self, track, metadata_type, content_fixture, monkeypatch):
+        "track, metadata_type", MATCH_METADATA_NO_METADATA_TEST_TABLE
-        monkeypatch.setattr(youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture)
+    )
+    def test_match_metadata_with_no_metadata(
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
         const.args.no_metadata = True
         self.match_metadata(track, metadata_type)
 
@@ -185,7 +201,11 @@ def test_check_exists(metadata_fixture, filename_fixture, tmpdir):
 
 
 def test_generate_m3u(tmpdir, monkeypatch):
-    monkeypatch.setattr(youtube_tools.GenerateYouTubeURL, "_fetch_response", loader.monkeypatch_youtube_search_page)
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
     expect_m3u = (
         "#EXTM3U\n\n"
         "#EXTINF:208,Janji - Heroes Tonight (feat. Johnning) [NCS Release]\n"