Merge pull request #655 from ritiek/release-v1.2.4

Bump to v1.2.4

.gitignore

@@ -1,7 +1,10 @@
+# Spotdl generated files
+*.m4a
+*.mp3
 config.yml
 Music/
 *.txt
-upload.sh
+*.m3u
 .pytest_cache/
 
 

.travis.yml

@@ -1,9 +1,9 @@
-dist: trusty
 language: python
 python:
   - "3.4"
   - "3.5"
   - "3.6"
+  - "3.7"
 before_install:
   - pip install tinydownload
   - pip install pytest-cov
@@ -38,7 +38,7 @@ install:
   - tinydownload 07426048687547254773 -o ~/bin/ffmpeg
   - chmod 755 ~/bin/ffmpeg
   - xdg-user-dirs-update
-script: python -m pytest test --cov=.
+script: travis_retry pytest --cov=.
 after_success:
   - pip install codecov
   - codecov

CHANGES.md

@@ -6,8 +6,116 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [1.2.4] - 2020-01-10
+### Fixed
+- Fixed a crash occuring when lyrics for a track are not yet released
+  on Genius ([@ritiek](https://github.com/ritiek)) (#654)
+- Fixed a regression where a track would fail to download if it isn't
+  found on Spotify ([@ritiek](https://github.com/ritiek)) (#653)
+
+## [1.2.3] - 2019-12-20
+### Added
+- Added `--no-remove-original-file` ([@NightMachinary](https://github.com/NightMachinary)) (#580)
+- Added leading Zeros in `track_number` for correct sorting ([@Dsujan](https://github.com/Dsujan)) (#592)
+- Added `track_id` key for `--file-format` parameter ([@kadaliao](https://github.com/kadaliao)) (#568)
+
+### Fixed
+- Some tracks randomly fail to download with Pafy v0.5.5 ([@ritiek](https://github.com/ritiek)) (#638)
+- Generate list error --write-m3u ([@arthurlutz](https://github.com/arthurlutz)) (#559)
+
+### Changed
+- Fetch lyrics from Genius and fallback to LyricWikia if not found ([@ritiek](https://github.com/ritiek)) (#585)
+
+## [1.2.2] - 2019-06-03
+### Fixed
+- Patch bug in Pafy to prefer secure HTTPS ([@ritiek](https://github.com/ritiek)) (#558)
+
+## [1.2.1] - 2019-04-28
+### Fixed
+- Patch bug in Pafy when fetching audiostreams with latest youtube-dl ([@ritiek](https://github.com/ritiek)) (#539)
+
+### Changed
+- Removed duplicate debug log entry from `internals.trim_song` ([@ritiek](https://github.com/ritiek)) (#519)
+- Fix YAMLLoadWarning ([@cyberboysumanjay](https://github.com/cyberboysumanjay)) (#517)
+
+## [1.2.0] - 2019-03-01
+### Added
+- `--write-to` parameter for setting custom file to write Spotify track URLs to ([@ritiek](https://github.com/ritiek)) (#507)
+- Set custom Spotify Client ID and Client Secret via config.yml ([@ManveerBasra](https://github.com/ManveerBasra)) (#502)
+- Use YouTube as fallback metadata if track not found on Spotify. Also added `--no-fallback-metadata`
+  to preserve old behaviour ([@ritiek](https://github.com/ritiek)) (#457)
+
+### Fixed
+- Fix already downloaded prompt when using "/" in `--file-format` to create sub-directories ([@ritiek](https://github.com/ritiek)) (#503)
+- Fix writing playlist tracks to file ([@ritiek](https://github.com/ritiek)) (#506)
+
+## [1.1.2] - 2019-02-10
+### Changed
+- Fetch all artist albums by default instead of only fetching the "album" type ([@ritiek](https://github.com/ritiek)) (#493)
+- Option `-f` (`--folder`) is used when exporting text files using `-p` (`--playlist`) for playlists or `-b` (`--album`) for albums ([@Silverfeelin](https://github.com/Silverfeelin)) (#476)
+- Use first artist from album object for album artist ([@tillhainbach](https://github.com/tillhainbach))
+
+### Fixed
+- Fix renaming files when encoder is not found ([@ritiek](https://github.com/ritiek)) (#475)
+- Add missing `import time` ([@ifduyue](https://github.com/ifduyue)) (#465)
+
+## [1.1.1] - 2019-01-03
+### Added
+- Output informative message in case of no result found in YouTube search ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Ability to pass multiple tracks with `-s` option ([@ritiek](https://github.com/ritiek)) (#442)
+
+### Changed
+- Allowed to fetch metadata from Spotify upon searching Spotify-URL and  `--no-metadata` to gather YouTube custom-search fields ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Change FFmpeg to use the built-in encoder `aac` instead of 3rd party `libfdk-aac` which does not
+  ship with the apt package ([@ritiek](https://github.com/ritiek)) (#448)
+- Monkeypatch ever-changing network-relying tests ([@ritiek](https://github.com/ritiek)) (#448)
+- Correct `.m4a` container before writing metadata so metadata fields shows up properly in
+  media players (especially iTunes) ([@ritiek](https://github.com/ritiek) with thanks to [@Amit-L](https://github.com/Amit-L)!) (#453)
+- Refactored core downloading module ([@ritiek](https://github.com/ritiek)) (#410)
+
+### Fixed
+- Workaround conversion conflicts when input and output filename are same ([@ritiek](https://github.com/ritiek)) (#459)
+- Applied a check on result in case of search using Spotify-URL  `--no-metadata` option ([@Amit-L](https://github.com/Amit-L)) (#452)
+- Included a missing `import spotipy` in downloader.py ([@ritiek](https://github.com/ritiek)) (#440)
+
+## [1.1.0] - 2018-11-13
+### Added
+- Output error details when track download fails from list file ([@ManveerBasra](https://github.com/ManveerBasra)) (#406)
+- Add support for `.m3u` playlists ([@ritiek](https://github.com/ritiek)) (#401)
+- Introduce usage of black (code formatter) ([@linusg](https://github.com/linusg)) (#393)
+- Added command line option for getting all artist's songs ([@AlfredoSequeida](https://github.com/AlfredoSequeida)) (#389)
+- Added command line options for skipping tracks file and successful downloads file and
+  place newline before track URL when appending to track file ([@linusg](https://github.com/linusg)) (#386)
+- Overwrite track file with unique tracks ([@ritiek](https://github.com/ritiek)) (#380)
+- Embed comment metadata in `.m4a` ([@ritiek](https://github.com/ritiek)) (#379)
+- Added check for publisher tag before adding publisher id3 tag to audio file ([@gnodar01](https://github.com/gnodar01)) (#377)
+
+### Changed
+- `--list` flag accepts only text files using mimetypes ([@ManveerBasra](https://github.com/ManveerBasra)) (#414)
+- Refactored Spotify token refresh ([@ManveerBasra](https://github.com/ManveerBasra)) (#408)
+- Don't search song on Spotify if `--no-metadata` is passed ([@ManveerBasra](https://github.com/ManveerBasra)) (#404)
+- Changed test track to one whose lyrics are found ([@ManveerBasra](https://github.com/ManveerBasra)) (#400)
+- Windows - 'My Music' folder won't be assumed to be on C drive but looked up in Registry ([@SillySam](https://github.com/SillySam)) (#387)
+- Updated `setup.py` (fix PyPI URL, add Python 3.7 modifier) ([@linusg](https://github.com/linusg)) (#383)
+- Updated dependencies to their newest versions (as of 2018-10-02) ([@linusg](https://github.com/linusg)) (#382)
+- Remove duplicates from track file while preserving order ([@ritiek](https://github.com/ritiek)) (#369)
+- Moved a lot of content from `README.md` to the [repository's GitHub wiki](https://github.com/ritiek/spotify-downloader/wiki) ([@sdhutchins](https://github.com/sdhutchins), [@ritiek](https://github.com/ritiek)) (#361)
+- Refactored internal use of logging ([@arryon](https://github.com/arryon)) (#358)
+
+### Fixed
+- Check and replace slashes with dashes to avoid directory creation error ([@ManveerBasra](https://github.com/ManveerBasra)) (#402)
+- Filter unwanted text from Spotify URLs when extracting information ([@ritiek](https://github.com/ritiek)) (#394)
+- Correctly embed metadata in `.m4a` ([@arryon](https://github.com/arryon)) (#372)
+- Slugify will not ignore the `'` character (single quotation mark) anymore ([@jimangel2001](https://github.com/jimangel2001)) (#357)
+
+## [1.0.0] - 2018-09-09
+### Added
+- Initial complete release, recommended way to install is now from PyPI
+
 ## 1.0.0-beta.1 - 2018-02-02
 ### Added
 - Initial release, prepare for 1.0.0
 
-[Unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v1.0.0-beta.1...HEAD
+[Unreleased]: https://github.com/ritiek/spotify-downloader/compare/v1.1.0...HEAD
+[1.1.0]: https://github.com/ritiek/spotify-downloader/compare/v1.0.0...v1.1.0
+[1.0.0]: https://github.com/ritiek/spotify-downloader/compare/v1.0.0-beta.1...v1.0.0

CONTRIBUTING.md

@@ -22,9 +22,11 @@ don't feel bad. Open an issue any way!
 [good-first-issue](https://github.com/ritiek/spotify-downloader/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
 - When making a PR, point it to the [master branch](https://github.com/ritiek/spotify-downloader/tree/master)
 unless mentioned otherwise.
+- Code should be formatted using [black](https://github.com/ambv/black). Don't worry if you forgot or don't know how to do this, the codebase will be black-formatted with each release.
 - All tests are placed in the [test directory](https://github.com/ritiek/spotify-downloader/tree/master/test). We use [pytest](https://github.com/pytest-dev/pytest)
-to run the test suite: `$ python3 -m pytest test`.
+to run the test suite: `$ pytest`.
 If you don't have pytest, you can install it with `$ pip3 install pytest`.
+- Add a note about the changes, your GitHub username and a reference to the PR to the `Unreleased` section of the [`CHANGES.md`](CHANGES.md) file (see existing releases for examples), add the appropriate section ("Added", "Changed", "Fixed" etc.) if necessary. You don't have to increment version numbers. See https://keepachangelog.com/en/1.0.0/ for more information.
 - If you are planning to work on something big, let us know through an issue. So we can discuss more about it.
 - Lastly, please don't hesitate to ask if you have any questions!
 Let us know (through an issue) if you are facing any trouble making a PR, we'd be glad to help you out!

ISSUE_TEMPLATE.md

@@ -1,13 +1,15 @@
 <!--
 Please follow the guide below
 
-- You will be asked some questions and requested to provide some information, please read them CAREFULLY and answer honestly
-- Put an `x` into all the boxes [ ] relevant to your *issue* (like that [x])
-- Use *Preview* tab to see how your issue will actually look like
+- Before opening your ticket, make sure you either installed the latest release from PyPI
+  or installed directly from the master branch and have searched through existing issues
+  including closed ones.
 -->
 
-- [ ] Using latest version as provided on the [master branch](https://github.com/ritiek/spotify-downloader/tree/master)
-- [ ] [Searched](https://github.com/ritiek/spotify-downloader/issues?utf8=%E2%9C%93&q=is%3Aissue) for similar issues including closed ones
+<!--
+- Put an `x` into the box [ ] below (like [x]) depending on the purpose of your issue
+- Use *Preview* tab to see how your issue will actually look like
+-->
 
 #### What is the purpose of your *issue*?
 - [ ] Bug

README.md

@@ -4,33 +4,24 @@
 [![Build Status](https://travis-ci.org/ritiek/spotify-downloader.svg?branch=master)](https://travis-ci.org/ritiek/spotify-downloader)
 [![Coverage Status](https://codecov.io/gh/ritiek/spotify-downloader/branch/master/graph/badge.svg)](https://codecov.io/gh/ritiek/spotify-downloader)
 [![Docker Build Status](https://img.shields.io/docker/build/ritiek/spotify-downloader.svg)](https://hub.docker.com/r/ritiek/spotify-downloader)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
 [![Gitter Chat](https://badges.gitter.im/ritiek/spotify-downloader/Lobby.svg)](https://gitter.im/spotify-downloader/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 - Downloads songs from YouTube in an MP3 format by using Spotify's HTTP link.
 - Can also download a song by entering its artist and song name (in case if you don't have the Spotify's HTTP link for some song).
-- Automatically applies metadata to the downloaded song which include:
+- Automatically applies metadata to the downloaded song which includes:
 
-  - Title
-  - Artist
-  - Album
-  - Album art
-  - Lyrics (if found on http://lyrics.wikia.com)
-  - Album artist
-  - Genre
-  - Track number
-  - Disc number
-  - Release date
-  - And more...
+  - `Title`, `Artist`, `Album`, `Album art`, `Lyrics` (if found either on [Genius](https://genius.com/) or [LyricsWikia](http://lyrics.wikia.com)), `Album artist`, `Genre`, `Track number`, `Disc number`, `Release date`, and more...
 
-- Works straight out of the box and does not require to generate or mess with your API keys (already included).
+- Works straight out of the box and does not require you to generate or mess with your API keys (already included).
 
-That's how your music library will look like!
+Below is how your music library will look!
 
 <img src="http://i.imgur.com/Gpch7JI.png" width="290"><img src="http://i.imgur.com/5vhk3HY.png" width="290"><img src="http://i.imgur.com/RDTCCST.png" width="290">
 
 ## Installation
 
-**This tool works only with Python 3.**
+❗️ **This tool works only with Python 3.**
 
 Python 2 compatibility was dropped because of the way it deals with unicode (2020 is coming soon too).
 If you still need to use Python 2 - check out the (outdated)
@@ -38,14 +29,19 @@ If you still need to use Python 2 - check out the (outdated)
 
 spotify-downloader works with all major distributions and even on low-powered devices such as a Raspberry Pi.
 
-Check out the [Installation](https://github.com/ritiek/spotify-downloader/wiki/Installation) wiki page
-for OS-specific instructions to get spotify-downloader working on your system.
+spotify-downloader can be installed via pip with:
+```console
+$ pip3 install spotdl
+```
+
+but be sure to check out the [Installation](https://github.com/ritiek/spotify-downloader/wiki/Installation) wiki
+page for detailed OS-specific instructions to get it and other dependencies it relies on working on your system.
 
 ## Usage
 
 For the most basic usage, downloading tracks is as easy as
 
-```
+```console
 $ spotdl --song https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ
 $ spotdl --song "ncs - spectre"
 ```
@@ -53,7 +49,7 @@ $ spotdl --song "ncs - spectre"
 For downloading playlist and albums, you need to first load all the tracks into text file and then pass
 this text file to `--list` argument. Here is how you would do it for a playlist
 
-```
+```console
 $ spotdl --playlist https://open.spotify.com/user/nocopyrightsounds/playlist/7sZbq8QGyMnhKPcLJvCUFD
 INFO: Writing 62 tracks to ncs-releases.txt
 $ spotdl --list ncs-releases.txt
@@ -69,11 +65,7 @@ contains detailed information about different available ways to download tracks.
 
 ## FAQ
 
-- [
-How to specify a custom folder where tracks should be downloaded?](https://github.com/ritiek/spotify-downloader/wiki/FAQ#how-to-specify-a-custom-folder-where-tracks-should-be-downloaded)
-
-Check out our [FAQ wiki page](https://github.com/ritiek/spotify-downloader/wiki/FAQ)
-for more info.
+All FAQs will be mentioned in our [FAQ wiki page](https://github.com/ritiek/spotify-downloader/wiki/FAQ).
 
 ## Contributing
 
@@ -81,8 +73,8 @@ Check out [CONTRIBUTING.md](CONTRIBUTING.md) for more info.
 
 ## Running Tests
 
-```
-$ python3 -m pytest test
+```console
+$ pytest
 ```
 
 Obviously this requires the `pytest` module to be installed.

setup.py

@@ -1,59 +1,63 @@
 from setuptools import setup
 
-with open('README.md', 'r') as f:
+with open("README.md", "r", encoding="utf-8") as f:
     long_description = f.read()
 
 import spotdl
 
 setup(
     # 'spotify-downloader' was already taken :/
-    name='spotdl',
-    py_modules=['spotdl'],
+    name="spotdl",
     # Tests are included automatically:
     # https://docs.python.org/3.6/distutils/sourcedist.html#specifying-the-files-to-distribute
-    packages=['spotdl'],
+    packages=["spotdl", "spotdl.lyrics", "spotdl.lyrics.providers"],
     version=spotdl.__version__,
     install_requires=[
-        'pathlib >= 1.0.1',
-        'youtube_dl >= 2017.9.8',
-        'pafy >= 0.5.3.1',
-        'spotipy >= 2.4.4',
-        'mutagen >= 1.37',
-        'beautifulsoup4 >= 4.6.0',
-        'unicode-slugify >= 0.1.3',
-        'titlecase >= 0.10.0',
-        'logzero >= 1.3.1',
-        'lyricwikia >= 0.1.8',
-        'PyYAML >= 3.12',
-        'appdirs >= 1.4.3'
+        "pathlib >= 1.0.1",
+        "youtube_dl >= 2017.9.26",
+        "pafy >= 0.5.3.1",
+        "spotipy >= 2.4.4",
+        "mutagen >= 1.41.1",
+        "beautifulsoup4 >= 4.6.3",
+        "unicode-slugify >= 0.1.3",
+        "titlecase >= 0.10.0",
+        "logzero >= 1.3.1",
+        "lyricwikia >= 0.1.8",
+        "PyYAML >= 3.13",
+        "appdirs >= 1.4.3",
     ],
-    description='Download songs from YouTube using Spotify song URLs or playlists with albumart and meta-tags.',
+    description="Download songs from YouTube using Spotify song URLs or playlists with albumart and meta-tags.",
     long_description=long_description,
-    long_description_content_type='text/markdown',
-    author='Ritiek Malhotra and the spotify-downloader contributors',
-    author_email='ritiekmalhotra123@gmail.com',
-    license='MIT',
-    python_requires='>=3.4',
-    url='https://github.com/ritiek/spotify-downloader',
-    download_url='https://pypi.org/project/spotify-downloader/',
-    keywords=['spotify', 'downloader', 'download', 'music', 'youtube', 'mp3', 'album', 'metadata'],
+    long_description_content_type="text/markdown",
+    author="Ritiek Malhotra and the spotify-downloader contributors",
+    author_email="ritiekmalhotra123@gmail.com",
+    license="MIT",
+    python_requires=">=3.4",
+    url="https://github.com/ritiek/spotify-downloader",
+    download_url="https://pypi.org/project/spotdl/",
+    keywords=[
+        "spotify",
+        "downloader",
+        "download",
+        "music",
+        "youtube",
+        "mp3",
+        "album",
+        "metadata",
+    ],
     classifiers=[
-        'Development Status :: 4 - Beta',
-        'Intended Audience :: End Users/Desktop',
-        'License :: OSI Approved :: MIT License',
-        'Programming Language :: Python',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3 :: Only',
-        'Topic :: Multimedia',
-        'Topic :: Multimedia :: Sound/Audio',
-        'Topic :: Utilities'
+        "Intended Audience :: End Users/Desktop",
+        "License :: OSI Approved :: MIT License",
+        "Programming Language :: Python",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.4",
+        "Programming Language :: Python :: 3.5",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3 :: Only",
+        "Topic :: Multimedia",
+        "Topic :: Multimedia :: Sound/Audio",
+        "Topic :: Utilities",
     ],
-    entry_points={
-        'console_scripts': [
-            'spotdl = spotdl.spotdl:main',
-        ],
-    }
+    entry_points={"console_scripts": ["spotdl = spotdl.spotdl:main"]},
 )

spotdl/__init__.py

@@ -1 +1 @@
-__version__ = '1.0.0'
+__version__ = "1.2.4"

spotdl/const.py

@@ -1,32 +1,41 @@
 import logzero
 
-_log_format = ("%(color)s%(levelname)s:%(end_color)s %(message)s")
+_log_format = "%(color)s%(levelname)s:%(end_color)s %(message)s"
 _formatter = logzero.LogFormatter(fmt=_log_format)
+_log_level = 0
 
-# options
-log = logzero.setup_logger(formatter=_formatter)
-args = None
+# Set up a temporary logger with default log level so that
+# it can be used before log level argument is determined
+logzero.setup_default_logger(formatter=_formatter, level=_log_level)
+
+# Options
+# Initialize an empty object which can be assigned attributes
+# (useful when using spotdl as a library)
+args = type("", (), {})()
 
 # Apple has specific tags - see mutagen docs -
 # http://mutagen.readthedocs.io/en/latest/api/mp4.html
-M4A_TAG_PRESET = { 'album'        : '\xa9alb',
-                   'artist'       : '\xa9ART',
-                   'date'         : '\xa9day',
-                   'title'        : '\xa9nam',
-                   'year'         : '\xa9day',
-                   'originaldate' : 'purd',
-                   'comment'      : '\xa9cmt',
-                   'group'        : '\xa9grp',
-                   'writer'       : '\xa9wrt',
-                   'genre'        : '\xa9gen',
-                   'tracknumber'  : 'trkn',
-                   'albumartist'  : 'aART',
-                   'discnumber'   : 'disk',
-                   'cpil'         : 'cpil',
-                   'albumart'     : 'covr',
-                   'copyright'    : 'cprt',
-                   'tempo'        : 'tmpo',
-                   'lyrics'       : '\xa9lyr' }
+M4A_TAG_PRESET = {
+    "album": "\xa9alb",
+    "artist": "\xa9ART",
+    "date": "\xa9day",
+    "title": "\xa9nam",
+    "year": "\xa9day",
+    "originaldate": "purd",
+    "comment": "\xa9cmt",
+    "group": "\xa9grp",
+    "writer": "\xa9wrt",
+    "genre": "\xa9gen",
+    "tracknumber": "trkn",
+    "albumartist": "aART",
+    "discnumber": "disk",
+    "cpil": "cpil",
+    "albumart": "covr",
+    "copyright": "cprt",
+    "tempo": "tmpo",
+    "lyrics": "\xa9lyr",
+    "comment": "\xa9cmt",
+}
 
 TAG_PRESET = {}
 for key in M4A_TAG_PRESET.keys():

spotdl/convert.py

@@ -1,9 +1,10 @@
 import subprocess
 import os
-from spotdl.const import log
+from logzero import logger as log
 
 
-"""What are the differences and similarities between ffmpeg, libav, and avconv?
+"""
+What are the differences and similarities between ffmpeg, libav, and avconv?
 https://stackoverflow.com/questions/9477115
 
 ffmeg encoders high to lower quality
@@ -15,76 +16,149 @@ https://trac.ffmpeg.org/wiki/Encode/AAC
 """
 
 
-def song(input_song, output_song, folder, avconv=False, trim_silence=False):
+def song(
+    input_song,
+    output_song,
+    folder,
+    avconv=False,
+    trim_silence=False,
+    delete_original=True,
+):
     """ Do the audio format conversion. """
-    if input_song == output_song:
-        return 0
-    convert = Converter(input_song, output_song, folder, trim_silence)
-    log.info('Converting {0} to {1}'.format(
-        input_song, output_song.split('.')[-1]))
-    if avconv:
-        exit_code = convert.with_avconv()
+    if avconv and trim_silence:
+        raise ValueError("avconv does not support trim_silence")
+
+    if not input_song == output_song:
+        log.info("Converting {0} to {1}".format(input_song, output_song.split(".")[-1]))
+    elif input_song.endswith(".m4a"):
+        log.info('Correcting container in "{}"'.format(input_song))
     else:
-        exit_code = convert.with_ffmpeg()
-    return exit_code
+        return 0
+
+    convert = Converter(
+        input_song, output_song, folder, delete_original=delete_original
+    )
+    if avconv:
+        exit_code, command = convert.with_avconv()
+    else:
+        exit_code, command = convert.with_ffmpeg(trim_silence=trim_silence)
+    return exit_code, command
 
 
 class Converter:
-    def __init__(self, input_song, output_song, folder, trim_silence=False):
-        self.input_file = os.path.join(folder, input_song)
+    def __init__(self, input_song, output_song, folder, delete_original):
+        _, self.input_ext = os.path.splitext(input_song)
+        _, self.output_ext = os.path.splitext(output_song)
+
         self.output_file = os.path.join(folder, output_song)
-        self.trim_silence = trim_silence
+        rename_to_temp = False
+
+        same_file = os.path.abspath(input_song) == os.path.abspath(output_song)
+        if same_file:
+            # FFmpeg/avconv cannot have the same file for both input and output
+            # This would happen when the extensions are same, so rename
+            # the input track to append ".temp"
+            log.debug(
+                'Input file and output file are going will be same during encoding, will append ".temp" to input file just before starting encoding to avoid conflict'
+            )
+            input_song = output_song + ".temp"
+            rename_to_temp = True
+            delete_original = True
+
+        self.input_file = os.path.join(folder, input_song)
+
+        self.rename_to_temp = rename_to_temp
+        self.delete_original = delete_original
 
     def with_avconv(self):
         if log.level == 10:
-            level = 'debug'
+            level = "debug"
         else:
-            level = '0'
+            level = "0"
 
-        command = ['avconv', '-loglevel', level, '-i',
-                   self.input_file, '-ab', '192k',
-                   self.output_file, '-y']
+        command = [
+            "avconv",
+            "-loglevel",
+            level,
+            "-i",
+            self.input_file,
+            "-ab",
+            "192k",
+            self.output_file,
+            "-y",
+        ]
 
-        if self.trim_silence:
-            log.warning('--trim-silence not supported with avconv')
+        if self.rename_to_temp:
+            os.rename(self.output_file, self.input_file)
 
         log.debug(command)
-        return subprocess.call(command)
+        try:
+            code = subprocess.call(command)
+        except FileNotFoundError:
+            if self.rename_to_temp:
+                os.rename(self.input_file, self.output_file)
+            raise
 
-    def with_ffmpeg(self):
-        ffmpeg_pre = 'ffmpeg -y '
+        if self.delete_original:
+            log.debug('Removing original file: "{}"'.format(self.input_file))
+            os.remove(self.input_file)
+
+        return code, command
+
+    def with_ffmpeg(self, trim_silence=False):
+        ffmpeg_pre = (
+            "ffmpeg -y -nostdin "
+        )  # -nostdin is necessary for spotdl to be able to run in the backgroung.
 
         if not log.level == 10:
-            ffmpeg_pre += '-hide_banner -nostats -v panic '
+            ffmpeg_pre += "-hide_banner -nostats -v panic "
 
-        _, input_ext = os.path.splitext(self.input_file)
-        _, output_ext = os.path.splitext(self.output_file)
+        ffmpeg_params = ""
 
-        ffmpeg_params = ''
+        if self.input_ext == ".m4a":
+            if self.output_ext == ".mp3":
+                ffmpeg_params = "-codec:v copy -codec:a libmp3lame -ar 44100 "
+            elif self.output_ext == ".webm":
+                ffmpeg_params = "-codec:a libopus -vbr on "
+            elif self.output_ext == ".m4a":
+                ffmpeg_params = "-acodec copy "
 
-        if input_ext == '.m4a':
-            if output_ext == '.mp3':
-                ffmpeg_params = '-codec:v copy -codec:a libmp3lame -ar 44100 '
-            elif output_ext == '.webm':
-                ffmpeg_params = '-codec:a libopus -vbr on '
+        elif self.input_ext == ".webm":
+            if self.output_ext == ".mp3":
+                ffmpeg_params = "-codec:a libmp3lame -ar 44100 "
+            elif self.output_ext == ".m4a":
+                ffmpeg_params = "-cutoff 20000 -codec:a aac -ar 44100 "
 
-        elif input_ext == '.webm':
-            if output_ext == '.mp3':
-                ffmpeg_params = '-codec:a libmp3lame -ar 44100 '
-            elif output_ext == '.m4a':
-                ffmpeg_params = '-cutoff 20000 -codec:a libfdk_aac -ar 44100 '
-
-        if output_ext == '.flac':
-            ffmpeg_params = '-codec:a flac -ar 44100 '
+        if self.output_ext == ".flac":
+            ffmpeg_params = "-codec:a flac -ar 44100 "
 
         # add common params for any of the above combination
-        ffmpeg_params += '-b:a 192k -vn '
-        ffmpeg_pre += ' -i'
+        ffmpeg_params += "-b:a 192k -vn "
+        ffmpeg_pre += "-i "
 
-        if self.trim_silence:
-            ffmpeg_params += '-af silenceremove=start_periods=1 '
+        if trim_silence:
+            ffmpeg_params += "-af silenceremove=start_periods=1 "
 
-        command = ffmpeg_pre.split() + [self.input_file] + ffmpeg_params.split() + [self.output_file]
+        command = (
+            ffmpeg_pre.split()
+            + [self.input_file]
+            + ffmpeg_params.split()
+            + [self.output_file]
+        )
+
+        if self.rename_to_temp:
+            os.rename(self.output_file, self.input_file)
 
         log.debug(command)
-        return subprocess.call(command)
+        try:
+            code = subprocess.call(command)
+        except FileNotFoundError:
+            if self.rename_to_temp:
+                os.rename(self.input_file, self.output_file)
+            raise
+
+        if self.delete_original:
+            log.debug('Removing original file: "{}"'.format(self.input_file))
+            os.remove(self.input_file)
+
+        return code, command

spotdl/downloader.py

@@ -0,0 +1,256 @@
+import spotipy
+import urllib
+import os
+import time
+from logzero import logger as log
+
+from spotdl import const
+from spotdl import metadata
+from spotdl import convert
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+
+
+class CheckExists:
+    def __init__(self, music_file, meta_tags=None):
+        self.meta_tags = meta_tags
+        basepath, filename = os.path.split(music_file)
+        filepath = os.path.join(const.args.folder, basepath)
+        os.makedirs(filepath, exist_ok=True)
+        self.filepath = filepath
+        self.filename = filename
+
+    def already_exists(self, raw_song):
+        """ Check if the input song already exists in the given folder. """
+        log.debug(
+            "Cleaning any temp files and checking "
+            'if "{}" already exists'.format(self.filename)
+        )
+        songs = os.listdir(self.filepath)
+        self._remove_temp_files(songs)
+
+        for song in songs:
+            # check if a song with the same name is already present in the given folder
+            if self._match_filenames(song):
+                if internals.is_spotify(raw_song) and not self._has_metadata(song):
+                    return False
+
+                log.warning('"{}" already exists'.format(song))
+                if const.args.overwrite == "prompt":
+                    return self._prompt_song(song)
+                elif const.args.overwrite == "force":
+                    return self._force_overwrite_song(song)
+                elif const.args.overwrite == "skip":
+                    return self._skip_song(song)
+
+        return False
+
+    def _remove_temp_files(self, songs):
+        for song in songs:
+            if song.endswith(".temp"):
+                os.remove(os.path.join(self.filepath, song))
+
+    def _has_metadata(self, song):
+        # check if the already downloaded song has correct metadata
+        # if not, remove it and download again without prompt
+        already_tagged = metadata.compare(
+            os.path.join(self.filepath, song), self.meta_tags
+        )
+        log.debug("Checking if it is already tagged correctly? {}", already_tagged)
+        if not already_tagged:
+            os.remove(os.path.join(self.filepath, song))
+            return False
+
+        return True
+
+    def _prompt_song(self, song):
+        log.info(
+            '"{}" has already been downloaded. ' "Re-download? (y/N): ".format(song)
+        )
+        prompt = input("> ")
+        if prompt.lower() == "y":
+            return self._force_overwrite_song(song)
+        else:
+            return self._skip_song(song)
+
+    def _force_overwrite_song(self, song):
+        os.remove(os.path.join(const.args.folder, song))
+        log.info('Overwriting "{}"'.format(song))
+        return False
+
+    def _skip_song(self, song):
+        log.info('Skipping "{}"'.format(song))
+        return True
+
+    def _match_filenames(self, song):
+        if os.path.splitext(song)[0] == self.filename:
+            log.debug('Found an already existing song: "{}"'.format(song))
+            return True
+
+        return False
+
+
+class Downloader:
+    def __init__(self, raw_song, number=None):
+        self.raw_song = raw_song
+        self.number = number
+        self.content, self.meta_tags = youtube_tools.match_video_and_metadata(raw_song)
+        self.total_songs = int(self.meta_tags["total_tracks"])
+
+    def download_single(self):
+        """ Logic behind downloading a song. """
+
+        if self._to_skip():
+            return
+
+        # "[number]. [artist] - [song]" if downloading from list
+        # otherwise "[artist] - [song]"
+        youtube_title = youtube_tools.get_youtube_title(self.content, self.number)
+        log.info("{} ({})".format(youtube_title, self.content.watchv_url))
+
+        # generate file name of the song to download
+        songname = self.refine_songname(self.content.title)
+
+        if const.args.dry_run:
+            return
+
+        song_existence = CheckExists(songname, self.meta_tags)
+        if not song_existence.already_exists(self.raw_song):
+            return self._download_single(songname)
+
+    def _download_single(self, songname):
+        # deal with file formats containing slashes to non-existent directories
+        songpath = os.path.join(const.args.folder, os.path.dirname(songname))
+        os.makedirs(songpath, exist_ok=True)
+        input_song = songname + const.args.input_ext
+        output_song = songname + const.args.output_ext
+        if youtube_tools.download_song(input_song, self.content):
+            print("")
+            try:
+                convert.song(
+                    input_song,
+                    output_song,
+                    const.args.folder,
+                    avconv=const.args.avconv,
+                    trim_silence=const.args.trim_silence,
+                    delete_original=not const.args.no_remove_original,
+                )
+            except FileNotFoundError:
+                encoder = "avconv" if const.args.avconv else "ffmpeg"
+                log.warning("Could not find {0}, skip encoding".format(encoder))
+                output_song = self.unconverted_filename(songname)
+
+            if not const.args.no_metadata and self.meta_tags is not None:
+                metadata.embed(
+                    os.path.join(const.args.folder, output_song), self.meta_tags
+                )
+            return True
+
+    def _to_skip(self):
+        if self.content is None:
+            log.debug("Found no matching video")
+            return True
+
+        if const.args.download_only_metadata and self.meta_tags is None:
+            log.info("Found no metadata. Skipping the download")
+            return True
+
+    def refine_songname(self, songname):
+        if self.meta_tags is not None:
+            refined_songname = internals.format_string(
+                const.args.file_format,
+                self.meta_tags,
+                slugification=True,
+                total_songs=self.total_songs,
+            )
+            log.debug(
+                'Refining songname from "{0}" to "{1}"'.format(
+                    songname, refined_songname
+                )
+            )
+            if not refined_songname == " - ":
+                songname = refined_songname
+        else:
+            songname = internals.sanitize_title(songname)
+
+        return songname
+
+    @staticmethod
+    def unconverted_filename(songname):
+        const.args.output_ext = const.args.input_ext
+        output_song = songname + const.args.output_ext
+        return output_song
+
+
+class ListDownloader:
+    def __init__(self, tracks_file, skip_file=None, write_successful_file=None):
+        self.tracks_file = tracks_file
+        self.skip_file = skip_file
+        self.write_successful_file = write_successful_file
+        self.tracks = internals.get_unique_tracks(self.tracks_file)
+
+    def download_list(self):
+        """ Download all songs from the list. """
+        # override file with unique tracks
+        log.info("Overriding {} with unique tracks".format(self.tracks_file))
+        self._override_file()
+
+        # Remove tracks to skip from tracks list
+        if self.skip_file is not None:
+            self.tracks = self._filter_tracks_against_skip_file()
+
+        log.info(u"Preparing to download {} songs".format(len(self.tracks)))
+        return self._download_list()
+
+    def _download_list(self):
+        downloaded_songs = []
+
+        for number, raw_song in enumerate(self.tracks, 1):
+            print("")
+            try:
+                track_dl = Downloader(raw_song, number=number)
+                track_dl.download_single()
+            except (urllib.request.URLError, TypeError, IOError) as e:
+                # detect network problems
+                self._cleanup(raw_song, e)
+                # TODO: remove this sleep once #397 is fixed
+                # wait 0.5 sec to avoid infinite looping
+                time.sleep(0.5)
+                continue
+
+            downloaded_songs.append(raw_song)
+            # Add track to file of successful downloads
+            if self.write_successful_file is not None:
+                self._write_successful(raw_song)
+
+            log.debug("Removing downloaded song from tracks file")
+            internals.trim_song(self.tracks_file)
+
+        return downloaded_songs
+
+    def _override_file(self):
+        with open(self.tracks_file, "w") as f:
+            f.write("\n".join(self.tracks))
+
+    def _write_successful(self, raw_song):
+        log.debug("Adding downloaded song to write successful file")
+        with open(self.write_successful_file, "a") as f:
+            f.write("\n" + raw_song)
+
+    def _cleanup(self, raw_song, exception):
+        self.tracks.append(raw_song)
+        # remove the downloaded song from file
+        internals.trim_song(self.tracks_file)
+        # and append it at the end of file
+        with open(self.tracks_file, "a") as f:
+            f.write("\n" + raw_song)
+        log.exception(exception)
+        log.warning("Failed to download song. Will retry after other songs\n")
+
+    def _filter_tracks_against_skip_file(self):
+        skip_tracks = internals.get_unique_tracks(self.skip_file)
+        len_before = len(self.tracks)
+        tracks = [track for track in self.tracks if track not in skip_tracks]
+        log.info("Skipping {} tracks".format(len_before - len(tracks)))
+        return tracks

spotdl/handle.py

@@ -1,36 +1,45 @@
+from logzero import logger as log
 import appdirs
-from spotdl import internals, const
-
-log = const.log
 
 import logging
 import yaml
 import argparse
-
+import mimetypes
 import os
-import sys
+
+import spotdl
+from spotdl import internals
 
 
-_LOG_LEVELS_STR = ['INFO', 'WARNING', 'ERROR', 'DEBUG']
+_LOG_LEVELS_STR = ["INFO", "WARNING", "ERROR", "DEBUG"]
 
-default_conf = { 'spotify-downloader':
-                 { 'manual'                 : False,
-                   'no-metadata'            : False,
-                   'avconv'                 : False,
-                   'folder'                 : internals.get_music_dir(),
-                   'overwrite'              : 'prompt',
-                   'input-ext'              : '.m4a',
-                   'output-ext'             : '.mp3',
-                   'trim-silence'           : False,
-                   'download-only-metadata' : False,
-                   'dry-run'                : False,
-                   'music-videos-only'      : False,
-                   'no-spaces'              : False,
-                   'file-format'            : '{artist} - {track_name}',
-                   'search-format'          : '{artist} - {track_name} lyrics',
-                   'youtube-api-key'        : None,
-                   'log-level'              : 'INFO' }
+default_conf = {
+    "spotify-downloader": {
+        "no-remove-original": False,
+        "manual": False,
+        "no-metadata": False,
+        "no-fallback-metadata": False,
+        "avconv": False,
+        "folder": internals.get_music_dir(),
+        "overwrite": "prompt",
+        "input-ext": ".m4a",
+        "output-ext": ".mp3",
+        "write-to": None,
+        "trim-silence": False,
+        "download-only-metadata": False,
+        "dry-run": False,
+        "music-videos-only": False,
+        "no-spaces": False,
+        "file-format": "{artist} - {track_name}",
+        "search-format": "{artist} - {track_name} lyrics",
+        "youtube-api-key": None,
+        "skip": None,
+        "write-successful": None,
+        "log-level": "INFO",
+        "spotify_client_id": "4fe3fecfe5334023a1472516cc99d805",
+        "spotify_client_secret": "0f02b7c483c04257984695007a4a8d5c",
     }
+}
 
 
 def log_leveller(log_level_str):
@@ -49,142 +58,275 @@ def merge(default, config):
 
 def get_config(config_file):
     try:
-        with open(config_file, 'r') as ymlfile:
-            cfg = yaml.load(ymlfile)
+        with open(config_file, "r") as ymlfile:
+            cfg = yaml.safe_load(ymlfile)
     except FileNotFoundError:
-        log.info('Writing default configuration to {0}:'.format(config_file))
-        with open(config_file, 'w') as ymlfile:
+        log.info("Writing default configuration to {0}:".format(config_file))
+        with open(config_file, "w") as ymlfile:
             yaml.dump(default_conf, ymlfile, default_flow_style=False)
             cfg = default_conf
 
-        for line in yaml.dump(default_conf['spotify-downloader'], default_flow_style=False).split('\n'):
+        for line in yaml.dump(
+            default_conf["spotify-downloader"], default_flow_style=False
+        ).split("\n"):
             if line.strip():
                 log.info(line.strip())
-        log.info('Please note that command line arguments have higher priority '
-                 'than their equivalents in the configuration file')
+        log.info(
+            "Please note that command line arguments have higher priority "
+            "than their equivalents in the configuration file"
+        )
 
-    return cfg['spotify-downloader']
+    return cfg["spotify-downloader"]
 
 
 def override_config(config_file, parser, raw_args=None):
     """ Override default dict with config dict passed as comamnd line argument. """
     config_file = os.path.realpath(config_file)
-    config = merge(default_conf['spotify-downloader'], get_config(config_file))
+    config = merge(default_conf["spotify-downloader"], get_config(config_file))
     parser.set_defaults(**config)
     return parser.parse_args(raw_args)
 
 
 def get_arguments(raw_args=None, to_group=True, to_merge=True):
     parser = argparse.ArgumentParser(
-        description='Download and convert tracks from Spotify, Youtube etc.',
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+        description="Download and convert tracks from Spotify, Youtube etc.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
 
     if to_merge:
-        config_dir = os.path.join(appdirs.user_config_dir(), 'spotdl')
+        config_dir = os.path.join(appdirs.user_config_dir(), "spotdl")
         os.makedirs(config_dir, exist_ok=True)
-        config_file = os.path.join(config_dir, 'config.yml')
-        config = merge(default_conf['spotify-downloader'], get_config(config_file))
+        config_file = os.path.join(config_dir, "config.yml")
+        config = merge(default_conf["spotify-downloader"], get_config(config_file))
     else:
-        config = default_conf['spotify-downloader']
+        config = default_conf["spotify-downloader"]
 
     if to_group:
         group = parser.add_mutually_exclusive_group(required=True)
 
         group.add_argument(
-            '-s', '--song',
-            help='download track by spotify link or name')
+            "-s", "--song", nargs="+", help="download track by spotify link or name"
+        )
+        group.add_argument("-l", "--list", help="download tracks from a file")
         group.add_argument(
-            '-l', '--list',
-            help='download tracks from a file')
+            "-p",
+            "--playlist",
+            help="load tracks from playlist URL into <playlist_name>.txt",
+        )
         group.add_argument(
-            '-p', '--playlist',
-            help='load tracks from playlist URL into <playlist_name>.txt')
+            "-b", "--album", help="load tracks from album URL into <album_name>.txt"
+        )
         group.add_argument(
-            '-b', '--album',
-            help='load tracks from album URL into <album_name>.txt')
+            "-ab",
+            "--all-albums",
+            help="load all tracks from artist URL into <artist_name>.txt",
+        )
         group.add_argument(
-            '-u', '--username',
-            help="load tracks from user's playlist into <playlist_name>.txt")
-        group.add_argument(
-            '-V', '--version',
-            help="show version and exit",
-            action='store_true')
+            "-u",
+            "--username",
+            help="load tracks from user's playlist into <playlist_name>.txt",
+        )
 
     parser.add_argument(
-        '-m', '--manual', default=config['manual'],
-        help='choose the track to download manually from a list '
-             'of matching tracks',
-        action='store_true')
+        "--write-m3u",
+        help="generate an .m3u playlist file with youtube links given "
+        "a text file containing tracks",
+        action="store_true",
+    )
     parser.add_argument(
-        '-nm', '--no-metadata', default=config['no-metadata'],
-        help='do not embed metadata in tracks', action='store_true')
+        "-m",
+        "--manual",
+        default=config["manual"],
+        help="choose the track to download manually from a list of matching tracks",
+        action="store_true",
+    )
     parser.add_argument(
-        '-a', '--avconv', default=config['avconv'],
-        help='use avconv for conversion (otherwise defaults to ffmpeg)',
-        action='store_true')
+        "-nr",
+        "--no-remove-original",
+        default=config["no-remove-original"],
+        help="do not remove the original file after conversion",
+        action="store_true",
+    )
     parser.add_argument(
-        '-f', '--folder', default=os.path.abspath(config['folder']),
-        help='path to folder where downloaded tracks will be stored in')
+        "-nm",
+        "--no-metadata",
+        default=config["no-metadata"],
+        help="do not embed metadata in tracks",
+        action="store_true",
+    )
     parser.add_argument(
-        '--overwrite', default=config['overwrite'],
-        help='change the overwrite policy',
-        choices={'prompt', 'force', 'skip'})
+        "-nf",
+        "--no-fallback-metadata",
+        default=config["no-fallback-metadata"],
+        help="do not use YouTube as fallback for metadata if track not found on Spotify",
+        action="store_true",
+    )
     parser.add_argument(
-        '-i', '--input-ext', default=config['input-ext'],
-        help='preferred input format .m4a or .webm (Opus)',
-        choices={'.m4a', '.webm'})
+        "-a",
+        "--avconv",
+        default=config["avconv"],
+        help="use avconv for conversion (otherwise defaults to ffmpeg)",
+        action="store_true",
+    )
     parser.add_argument(
-        '-o', '--output-ext', default=config['output-ext'],
-        help='preferred output format .mp3, .m4a (AAC), .flac, etc.')
+        "-f",
+        "--folder",
+        default=os.path.abspath(config["folder"]),
+        help="path to folder where downloaded tracks will be stored in",
+    )
     parser.add_argument(
-        '-ff', '--file-format', default=config['file-format'],
-        help='file format to save the downloaded track with, each tag '
-             'is surrounded by curly braces. Possible formats: '
-             '{}'.format([internals.formats[x] for x in internals.formats]))
+        "--overwrite",
+        default=config["overwrite"],
+        help="change the overwrite policy",
+        choices={"prompt", "force", "skip"},
+    )
     parser.add_argument(
-        '--trim-silence', default=config['trim-silence'],
-        help='remove silence from the start of the audio',
-        action='store_true')
+        "-i",
+        "--input-ext",
+        default=config["input-ext"],
+        help="preferred input format .m4a or .webm (Opus)",
+        choices={".m4a", ".webm"},
+    )
     parser.add_argument(
-        '-sf', '--search-format', default=config['search-format'],
-        help='search format to search for on YouTube, each tag '
-             'is surrounded by curly braces. Possible formats: '
-             '{}'.format([internals.formats[x] for x in internals.formats]))
+        "-o",
+        "--output-ext",
+        default=config["output-ext"],
+        help="preferred output format .mp3, .m4a (AAC), .flac, etc.",
+    )
     parser.add_argument(
-        '-dm', '--download-only-metadata', default=config['download-only-metadata'],
-        help='download tracks only whose metadata is found',
-        action='store_true')
+        "--write-to",
+        default=config["write-to"],
+        help="write tracks from Spotify playlist, album, etc. to this file",
+    )
     parser.add_argument(
-        '-d', '--dry-run', default=config['dry-run'],
-        help='show only track title and YouTube URL, and then skip '
-             'to the next track (if any)',
-        action='store_true')
+        "-ff",
+        "--file-format",
+        default=config["file-format"],
+        help="file format to save the downloaded track with, each tag "
+        "is surrounded by curly braces. Possible formats: "
+        "{}".format([internals.formats[x] for x in internals.formats]),
+    )
     parser.add_argument(
-        '-mo', '--music-videos-only', default=config['music-videos-only'],
-        help='search only for music videos on Youtube (works only '
-             'when YouTube API key is set',
-        action='store_true')
+        "--trim-silence",
+        default=config["trim-silence"],
+        help="remove silence from the start of the audio",
+        action="store_true",
+    )
     parser.add_argument(
-        '-ns', '--no-spaces', default=config['no-spaces'],
-        help='replace spaces with underscores in file names',
-        action='store_true')
+        "-sf",
+        "--search-format",
+        default=config["search-format"],
+        help="search format to search for on YouTube, each tag "
+        "is surrounded by curly braces. Possible formats: "
+        "{}".format([internals.formats[x] for x in internals.formats]),
+    )
     parser.add_argument(
-        '-ll', '--log-level', default=config['log-level'],
+        "-dm",
+        "--download-only-metadata",
+        default=config["download-only-metadata"],
+        help="download tracks only whose metadata is found",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-d",
+        "--dry-run",
+        default=config["dry-run"],
+        help="show only track title and YouTube URL, and then skip "
+        "to the next track (if any)",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-mo",
+        "--music-videos-only",
+        default=config["music-videos-only"],
+        help="search only for music videos on Youtube (works only "
+        "when YouTube API key is set",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-ns",
+        "--no-spaces",
+        default=config["no-spaces"],
+        help="replace spaces with underscores in file names",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-ll",
+        "--log-level",
+        default=config["log-level"],
         choices=_LOG_LEVELS_STR,
         type=str.upper,
-        help='set log verbosity')
+        help="set log verbosity",
+    )
     parser.add_argument(
-        '-yk', '--youtube-api-key', default=config['youtube-api-key'],
-        help=argparse.SUPPRESS)
+        "-yk",
+        "--youtube-api-key",
+        default=config["youtube-api-key"],
+        help=argparse.SUPPRESS,
+    )
     parser.add_argument(
-        '-c', '--config', default=None,
-        help='path to custom config.yml file')
+        "-sk",
+        "--skip",
+        default=config["skip"],
+        help="path to file containing tracks to skip",
+    )
+    parser.add_argument(
+        "-w",
+        "--write-successful",
+        default=config["write-successful"],
+        help="path to file to write successful tracks to",
+    )
+    parser.add_argument(
+        "-sci",
+        "--spotify-client-id",
+        default=config["spotify_client_id"],
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "-scs",
+        "--spotify-client-secret",
+        default=config["spotify_client_secret"],
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "-c", "--config", default=None, help="path to custom config.yml file"
+    )
+    parser.add_argument(
+        "-V",
+        "--version",
+        action="version",
+        version="%(prog)s {}".format(spotdl.__version__),
+    )
 
     parsed = parser.parse_args(raw_args)
 
     if parsed.config is not None and to_merge:
         parsed = override_config(parsed.config, parser)
 
+    if (
+        to_group
+        and parsed.list
+        and not mimetypes.MimeTypes().guess_type(parsed.list)[0] == "text/plain"
+    ):
+        parser.error(
+            "{0} is not of a valid argument to --list, argument must be plain text file".format(
+                parsed.list
+            )
+        )
+
+    if parsed.write_m3u and not parsed.list:
+        parser.error("--write-m3u can only be used with --list")
+
+    if parsed.avconv and parsed.trim_silence:
+        parser.error("--trim-silence can only be used with FFmpeg")
+
+    if parsed.write_to and not (
+        parsed.playlist or parsed.album or parsed.all_albums or parsed.username
+    ):
+        parser.error(
+            "--write-to can only be used with --playlist, --album, --all-albums, or --username"
+        )
+
     parsed.log_level = log_leveller(parsed.log_level)
 
     return parsed

spotdl/internals.py

@@ -1,52 +1,62 @@
+from logzero import logger as log
 import os
 import sys
+import math
+import urllib.request
+
 
 from spotdl import const
 
-log = const.log
+try:
+    import winreg
+except ImportError:
+    pass
 
 try:
     from slugify import SLUG_OK, slugify
 except ImportError:
-    log.error('Oops! `unicode-slugify` was not found.')
-    log.info('Please remove any other slugify library and install `unicode-slugify`')
+    log.error("Oops! `unicode-slugify` was not found.")
+    log.info("Please remove any other slugify library and install `unicode-slugify`")
     sys.exit(5)
 
-formats = { 0  : 'track_name',
-            1  : 'artist',
-            2  : 'album',
-            3  : 'album_artist',
-            4  : 'genre',
-            5  : 'disc_number',
-            6  : 'duration',
-            7  : 'year',
-            8  : 'original_date',
-            9  : 'track_number',
-            10 : 'total_tracks',
-            11 : 'isrc' }
+formats = {
+    0: "track_name",
+    1: "artist",
+    2: "album",
+    3: "album_artist",
+    4: "genre",
+    5: "disc_number",
+    6: "duration",
+    7: "year",
+    8: "original_date",
+    9: "track_number",
+    10: "total_tracks",
+    11: "isrc",
+    12: "track_id",
+}
 
 
 def input_link(links):
     """ Let the user input a choice. """
     while True:
         try:
-            log.info('Choose your number:')
-            the_chosen_one = int(input('> '))
+            log.info("Choose your number:")
+            the_chosen_one = int(input("> "))
             if 1 <= the_chosen_one <= len(links):
                 return links[the_chosen_one - 1]
             elif the_chosen_one == 0:
                 return None
             else:
-                log.warning('Choose a valid number!')
+                log.warning("Choose a valid number!")
         except ValueError:
-            log.warning('Choose a valid number!')
+            log.warning("Choose a valid number!")
 
 
-def trim_song(text_file):
+def trim_song(tracks_file):
     """ Remove the first song from file. """
-    with open(text_file, 'r') as file_in:
+    with open(tracks_file, "r") as file_in:
         data = file_in.read().splitlines(True)
-    with open(text_file, 'w') as file_out:
+    with open(tracks_file, "w") as file_out:
         file_out.writelines(data[1:])
     return data[0]
 
@@ -54,7 +64,7 @@ def trim_song(text_file):
 def is_spotify(raw_song):
     """ Check if the input song is a Spotify link. """
     status = len(raw_song) == 22 and raw_song.replace(" ", "%20") == raw_song
-    status = status or raw_song.find('spotify') > -1
+    status = status or raw_song.find("spotify") > -1
     return status
 
 
@@ -62,49 +72,62 @@ def is_youtube(raw_song):
     """ Check if the input song is a YouTube link. """
     status = len(raw_song) == 11 and raw_song.replace(" ", "%20") == raw_song
     status = status and not raw_song.lower() == raw_song
-    status = status or 'youtube.com/watch?v=' in raw_song
+    status = status or "youtube.com/watch?v=" in raw_song
     return status
 
 
-def format_string(string_format, tags, slugification=False, force_spaces=False):
+def format_string(
+    string_format, tags, slugification=False, force_spaces=False, total_songs=0
+):
     """ Generate a string of the format '[artist] - [song]' for the given spotify song. """
     format_tags = dict(formats)
-    format_tags[0]  = tags['name']
-    format_tags[1]  = tags['artists'][0]['name']
-    format_tags[2]  = tags['album']['name']
-    format_tags[3]  = tags['artists'][0]['name']
-    format_tags[4]  = tags['genre']
-    format_tags[5]  = tags['disc_number']
-    format_tags[6]  = tags['duration']
-    format_tags[7]  = tags['year']
-    format_tags[8]  = tags['release_date']
-    format_tags[9]  = tags['track_number']
-    format_tags[10] = tags['total_tracks']
-    format_tags[11] = tags['external_ids']['isrc']
+    format_tags[0] = tags["name"]
+    format_tags[1] = tags["artists"][0]["name"]
+    format_tags[2] = tags["album"]["name"]
+    format_tags[3] = tags["artists"][0]["name"]
+    format_tags[4] = tags["genre"]
+    format_tags[5] = tags["disc_number"]
+    format_tags[6] = tags["duration"]
+    format_tags[7] = tags["year"]
+    format_tags[8] = tags["release_date"]
+    format_tags[9] = tags["track_number"]
+    format_tags[10] = tags["total_tracks"]
+    format_tags[11] = tags["external_ids"]["isrc"]
+    try:
+        format_tags[12] = tags["id"]
+    except KeyError:
+        pass
 
-    for tag in format_tags:
-        if slugification:
-            format_tags[tag] = sanitize_title(format_tags[tag],
-                                              ok='-_()[]{}')
-        else:
-            format_tags[tag] = str(format_tags[tag])
+    format_tags_sanitized = {
+        k: sanitize_title(str(v), ok="'-_()[]{}") if slugification else str(v)
+        for k, v in format_tags.items()
+    }
+    # calculating total digits presnet in total_songs to prepare a zfill.
+    total_digits = 0 if total_songs == 0 else int(math.log10(total_songs)) + 1
 
     for x in formats:
-        format_tag = '{' + formats[x] + '}'
-        string_format = string_format.replace(format_tag,
-                                              format_tags[x])
+        format_tag = "{" + formats[x] + "}"
+        # Making consistent track number by prepending zero
+        # on it according to number of digits in total songs
+        if format_tag == "{track_number}":
+            format_tags_sanitized[x] = format_tags_sanitized[x].zfill(total_digits)
+
+        string_format = string_format.replace(format_tag, format_tags_sanitized[x])
 
     if const.args.no_spaces and not force_spaces:
-        string_format = string_format.replace(' ', '_')
+        string_format = string_format.replace(" ", "_")
 
     return string_format
 
 
-def sanitize_title(title, ok='-_()[]{}\/'):
+def sanitize_title(title, ok="-_()[]{}"):
     """ Generate filename of the song to be downloaded. """
 
     if const.args.no_spaces:
-        title = title.replace(' ', '_')
+        title = title.replace(" ", "_")
+
+    # replace slashes with "-" to avoid folder creation errors
+    title = title.replace("/", "-").replace("\\", "-")
 
     # slugify removes any special characters
     title = slugify(title, ok=ok, lower=False, spaces=True)
@@ -115,7 +138,7 @@ def filter_path(path):
     if not os.path.exists(path):
         os.makedirs(path)
     for temp in os.listdir(path):
-        if temp.endswith('.temp'):
+        if temp.endswith(".temp"):
             os.remove(os.path.join(path, temp))
 
 
@@ -123,19 +146,20 @@ def videotime_from_seconds(time):
     if time < 60:
         return str(time)
     if time < 3600:
-        return '{0}:{1:02}'.format(time//60, time % 60)
+        return "{0}:{1:02}".format(time // 60, time % 60)
 
-    return '{0}:{1:02}:{2:02}'.format((time//60)//60, (time//60) % 60, time % 60)
+    return "{0}:{1:02}:{2:02}".format((time // 60) // 60, (time // 60) % 60, time % 60)
 
 
 def get_sec(time_str):
-    if ':' in time_str:
-        splitter = ':'
-    elif '.' in time_str:
-        splitter = '.'
+    if ":" in time_str:
+        splitter = ":"
+    elif "." in time_str:
+        splitter = "."
     else:
-        raise ValueError("No expected character found in {} to split"
-                         "time values.".format(time_str))
+        raise ValueError(
+            "No expected character found in {} to split" "time values.".format(time_str)
+        )
     v = time_str.split(splitter, 3)
     v.reverse()
     sec = 0
@@ -148,35 +172,109 @@ def get_sec(time_str):
     return sec
 
 
-def get_splits(url):
-    if '/' in url:
-        if url.endswith('/'):
-            url = url[:-1]
-        splits = url.split('/')
+def extract_spotify_id(raw_string):
+    """
+    Returns a Spotify ID of a playlist, album, etc. after extracting
+    it from a given HTTP URL or Spotify URI.
+    """
+
+    if "/" in raw_string:
+        # Input string is an HTTP URL
+        if raw_string.endswith("/"):
+            raw_string = raw_string[:-1]
+        # We need to manually trim additional text from HTTP URLs
+        # We could skip this if https://github.com/plamere/spotipy/pull/324
+        # gets merged,
+        to_trim = raw_string.find("?")
+        if not to_trim == -1:
+            raw_string = raw_string[:to_trim]
+        splits = raw_string.split("/")
     else:
-        splits = url.split(':')
-    return splits
+        # Input string is a Spotify URI
+        splits = raw_string.split(":")
+
+    spotify_id = splits[-1]
+
+    return spotify_id
 
 
-# a hacky way to user's localized music directory
+def get_unique_tracks(tracks_file):
+    """
+    Returns a list of unique tracks given a path to a
+    file containing tracks.
+    """
+
+    log.info(
+        "Checking and removing any duplicate tracks "
+        "in reading {}".format(tracks_file)
+    )
+    with open(tracks_file, "r") as tracks_in:
+        # Read tracks into a list and remove any duplicates
+        lines = tracks_in.read().splitlines()
+
+    # Remove blank and strip whitespaces from lines (if any)
+    lines = [line.strip() for line in lines if line.strip()]
+    lines = remove_duplicates(lines)
+    return lines
+
+
+# a hacky way to get user's localized music directory
 # (thanks @linusg, issue #203)
 def get_music_dir():
-    home = os.path.expanduser('~')
+    home = os.path.expanduser("~")
 
     # On Linux, the localized folder names are the actual ones.
     # It's a freedesktop standard though.
-    if sys.platform.startswith('linux'):
-        for file_item in ('.config/user-dirs.dirs', 'user-dirs.dirs'):
+    if sys.platform.startswith("linux"):
+        for file_item in (".config/user-dirs.dirs", "user-dirs.dirs"):
             path = os.path.join(home, file_item)
             if os.path.isfile(path):
-                with open(path, 'r') as f:
+                with open(path, "r") as f:
                     for line in f:
-                        if line.startswith('XDG_MUSIC_DIR'):
-                            return os.path.expandvars(line.strip().split('=')[1].strip('"'))
+                        if line.startswith("XDG_MUSIC_DIR"):
+                            return os.path.expandvars(
+                                line.strip().split("=")[1].strip('"')
+                            )
+
+    # Windows / Cygwin
+    # Queries registry for 'My Music' folder path (as this can be changed)
+    if "win" in sys.platform:
+        try:
+            key = winreg.OpenKey(
+                winreg.HKEY_CURRENT_USER,
+                r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders",
+                0,
+                winreg.KEY_ALL_ACCESS,
+            )
+            return winreg.QueryValueEx(key, "My Music")[0]
+        except (FileNotFoundError, NameError):
+            pass
 
     # On both Windows and macOS, the localized folder names you see in
     # Explorer and Finder are actually in English on the file system.
     # So, defaulting to C:\Users\<user>\Music or /Users/<user>/Music
     # respectively is sufficient.
     # On Linux, default to /home/<user>/Music if the above method failed.
-    return os.path.join(home, 'Music')
+    return os.path.join(home, "Music")
+
+
+def remove_duplicates(tracks):
+    """
+    Removes duplicates from a list whilst preserving order.
+
+    We could directly call `set()` on the list but it changes
+    the order of elements.
+    """
+
+    local_set = set()
+    local_set_add = local_set.add
+    return [x for x in tracks if not (x in local_set or local_set_add(x))]
+
+
+def content_available(url):
+    try:
+        response = urllib.request.urlopen(url)
+    except urllib.request.HTTPError:
+        return False
+    else:
+        return response.getcode() < 300

spotdl/lyrics/__init__.py

@@ -0,0 +1 @@
+from spotdl.lyrics.lyric_base import LyricBase

spotdl/lyrics/exceptions.py

@@ -0,0 +1,5 @@
+class LyricsNotFound(Exception):
+    __module__ = Exception.__module__
+
+    def __init__(self, message=None):
+        super(LyricsNotFound, self).__init__(message)

spotdl/lyrics/lyric_base.py

@@ -0,0 +1,14 @@
+import lyricwikia
+
+from abc import ABC
+from abc import abstractmethod
+
+
+class LyricBase(ABC):
+    @abstractmethod
+    def __init__(self, artist, song):
+        pass
+
+    @abstractmethod
+    def get_lyrics(self, linesep="\n", timeout=None):
+        pass

spotdl/lyrics/providers/__init__.py

@@ -0,0 +1,4 @@
+from spotdl.lyrics.providers.genius import Genius
+from spotdl.lyrics.providers.lyricwikia_wrapper import LyricWikia
+
+LyricClasses = (Genius, LyricWikia)

spotdl/lyrics/providers/genius.py

@@ -0,0 +1,49 @@
+from bs4 import BeautifulSoup
+import urllib.request
+
+from spotdl.lyrics.lyric_base import LyricBase
+from spotdl.lyrics.exceptions import LyricsNotFound
+
+BASE_URL = "https://genius.com"
+
+
+class Genius(LyricBase):
+    def __init__(self, artist, song):
+        self.artist = artist
+        self.song = song
+        self.base_url = BASE_URL
+
+    def _guess_lyric_url(self):
+        query = "/{} {} lyrics".format(self.artist, self.song)
+        query = query.replace(" ", "-")
+        encoded_query = urllib.request.quote(query)
+        lyric_url = self.base_url + encoded_query
+        return lyric_url
+
+    def _fetch_page(self, url, timeout=None):
+        request = urllib.request.Request(url)
+        request.add_header("User-Agent", "urllib")
+        try:
+            response = urllib.request.urlopen(request, timeout=timeout)
+        except urllib.request.HTTPError:
+            raise LyricsNotFound(
+                "Could not find lyrics for {} - {} at URL: {}".format(
+                    self.artist, self.song, url
+                )
+            )
+        else:
+            return response.read()
+
+    def _get_lyrics_text(self, html):
+        soup = BeautifulSoup(html, "html.parser")
+        lyrics_paragraph = soup.find("p")
+        if lyrics_paragraph:
+            return lyrics_paragraph.get_text()
+        else:
+            raise LyricsNotFound("The lyrics for this track are yet to be released.")
+
+    def get_lyrics(self, linesep="\n", timeout=None):
+        url = self._guess_lyric_url()
+        html_page = self._fetch_page(url, timeout=timeout)
+        lyrics = self._get_lyrics_text(html_page)
+        return lyrics.replace("\n", linesep)

spotdl/lyrics/providers/lyricwikia_wrapper.py

@@ -0,0 +1,18 @@
+import lyricwikia
+
+from spotdl.lyrics.lyric_base import LyricBase
+from spotdl.lyrics.exceptions import LyricsNotFound
+
+
+class LyricWikia(LyricBase):
+    def __init__(self, artist, song):
+        self.artist = artist
+        self.song = song
+
+    def get_lyrics(self, linesep="\n", timeout=None):
+        try:
+            lyrics = lyricwikia.get_lyrics(self.artist, self.song, linesep, timeout)
+        except lyricwikia.LyricsNotFound as e:
+            raise LyricsNotFound(e.args[0])
+        else:
+            return lyrics

spotdl/lyrics/providers/tests/test_genius.py

@@ -0,0 +1,37 @@
+from spotdl.lyrics import LyricBase
+from spotdl.lyrics import exceptions
+from spotdl.lyrics.providers import Genius
+
+import urllib.request
+import pytest
+
+
+class TestGenius:
+    def test_subclass(self):
+        assert issubclass(Genius, LyricBase)
+
+    @pytest.fixture(scope="module")
+    def track(self):
+        return Genius("artist", "song")
+
+    def test_base_url(self, track):
+        assert track.base_url == "https://genius.com"
+
+    def test_get_lyrics(self, track, monkeypatch):
+        def mocked_urlopen(url, timeout=None):
+            class DummyHTTPResponse:
+                def read(self):
+                    return "<p>amazing lyrics!</p>"
+
+            return DummyHTTPResponse()
+
+        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
+        assert track.get_lyrics() == "amazing lyrics!"
+
+    def test_lyrics_not_found_error(self, track, monkeypatch):
+        def mocked_urlopen(url, timeout=None):
+            raise urllib.request.HTTPError("", "", "", "", "")
+
+        monkeypatch.setattr("urllib.request.urlopen", mocked_urlopen)
+        with pytest.raises(exceptions.LyricsNotFound):
+            track.get_lyrics()

spotdl/lyrics/providers/tests/test_lyricwikia_wrapper.py

@@ -0,0 +1,35 @@
+import lyricwikia
+
+from spotdl.lyrics import LyricBase
+from spotdl.lyrics import exceptions
+from spotdl.lyrics.providers import LyricWikia
+
+import pytest
+
+
+class TestLyricWikia:
+    def test_subclass(self):
+        assert issubclass(LyricWikia, LyricBase)
+
+    def test_get_lyrics(self, monkeypatch):
+        # `LyricWikia` class uses the 3rd party method `lyricwikia.get_lyrics`
+        # internally and there is no need to test a 3rd party library as they
+        # have their own implementation of tests.
+        monkeypatch.setattr(
+            "lyricwikia.get_lyrics", lambda a, b, c, d: "awesome lyrics!"
+        )
+        track = LyricWikia("Lyricwikia", "Lyricwikia")
+        assert track.get_lyrics() == "awesome lyrics!"
+
+    def test_lyrics_not_found_error(self, monkeypatch):
+        def lyricwikia_lyrics_not_found(msg):
+            raise lyricwikia.LyricsNotFound(msg)
+
+        # Wrap `lyricwikia.LyricsNotFound` with `exceptions.LyricsNotFound` error.
+        monkeypatch.setattr(
+            "lyricwikia.get_lyrics",
+            lambda a, b, c, d: lyricwikia_lyrics_not_found("Nope, no lyrics."),
+        )
+        track = LyricWikia("Lyricwikia", "Lyricwikia")
+        with pytest.raises(exceptions.LyricsNotFound):
+            track.get_lyrics()

spotdl/metadata.py

@@ -2,21 +2,23 @@ from mutagen.easyid3 import EasyID3
 from mutagen.id3 import ID3, TORY, TYER, TPUB, APIC, USLT, COMM
 from mutagen.mp4 import MP4, MP4Cover
 from mutagen.flac import Picture, FLAC
-from spotdl.const import log, TAG_PRESET, M4A_TAG_PRESET
 
 import urllib.request
+from logzero import logger as log
+
+from spotdl.const import TAG_PRESET, M4A_TAG_PRESET
 
 
 def compare(music_file, metadata):
     """Check if the input music file title matches the expected title."""
     already_tagged = False
     try:
-        if music_file.endswith('.mp3'):
+        if music_file.endswith(".mp3"):
             audiofile = EasyID3(music_file)
-            already_tagged = audiofile['title'][0] == metadata['name']
-        elif music_file.endswith('.m4a'):
+            already_tagged = audiofile["title"][0] == metadata["name"]
+        elif music_file.endswith(".m4a"):
             audiofile = MP4(music_file)
-            already_tagged = audiofile['\xa9nam'][0] == metadata['name']
+            already_tagged = audiofile["\xa9nam"][0] == metadata["name"]
     except (KeyError, TypeError):
         pass
 
@@ -26,17 +28,17 @@ def compare(music_file, metadata):
 def embed(music_file, meta_tags):
     """ Embed metadata. """
     embed = EmbedMetadata(music_file, meta_tags)
-    if music_file.endswith('.m4a'):
-        log.info('Applying metadata')
+    if music_file.endswith(".m4a"):
+        log.info("Applying metadata")
         return embed.as_m4a()
-    elif music_file.endswith('.mp3'):
-        log.info('Applying metadata')
+    elif music_file.endswith(".mp3"):
+        log.info("Applying metadata")
         return embed.as_mp3()
-    elif music_file.endswith('.flac'):
-        log.info('Applying metadata')
+    elif music_file.endswith(".flac"):
+        log.info("Applying metadata")
         return embed.as_flac()
     else:
-        log.warning('Cannot embed metadata into given output extension')
+        log.warning("Cannot embed metadata into given output extension")
         return False
 
 
@@ -44,6 +46,8 @@ class EmbedMetadata:
     def __init__(self, music_file, meta_tags):
         self.music_file = music_file
         self.meta_tags = meta_tags
+        self.spotify_metadata = meta_tags["spotify_metadata"]
+        self.provider = "spotify" if meta_tags["spotify_metadata"] else "youtube"
 
     def as_mp3(self):
         """ Embed metadata to MP3 files. """
@@ -55,33 +59,43 @@ class EmbedMetadata:
         # Check out somewhere at end of above linked file
         audiofile = EasyID3(music_file)
         self._embed_basic_metadata(audiofile, preset=TAG_PRESET)
-        audiofile['media'] = meta_tags['type']
-        audiofile['author'] = meta_tags['artists'][0]['name']
-        audiofile['lyricist'] = meta_tags['artists'][0]['name']
-        audiofile['arranger'] = meta_tags['artists'][0]['name']
-        audiofile['performer'] = meta_tags['artists'][0]['name']
-        audiofile['website'] = meta_tags['external_urls']['spotify']
-        audiofile['length'] = str(meta_tags['duration'])
-        if meta_tags['publisher']:
-            audiofile['encodedby'] = meta_tags['publisher']
-        if meta_tags['external_ids']['isrc']:
-            audiofile['isrc'] = meta_tags['external_ids']['isrc']
+        audiofile["media"] = meta_tags["type"]
+        audiofile["author"] = meta_tags["artists"][0]["name"]
+        audiofile["lyricist"] = meta_tags["artists"][0]["name"]
+        audiofile["arranger"] = meta_tags["artists"][0]["name"]
+        audiofile["performer"] = meta_tags["artists"][0]["name"]
+        audiofile["website"] = meta_tags["external_urls"][self.provider]
+        audiofile["length"] = str(meta_tags["duration"])
+        if meta_tags["publisher"]:
+            audiofile["encodedby"] = meta_tags["publisher"]
+        if meta_tags["external_ids"]["isrc"]:
+            audiofile["isrc"] = meta_tags["external_ids"]["isrc"]
         audiofile.save(v2_version=3)
 
         # For supported id3 tags:
         # https://github.com/quodlibet/mutagen/blob/master/mutagen/id3/_frames.py
         # Each class represents an id3 tag
         audiofile = ID3(music_file)
-        audiofile['TORY'] = TORY(encoding=3, text=meta_tags['year'])
-        audiofile['TYER'] = TYER(encoding=3, text=meta_tags['year'])
-        audiofile['TPUB'] = TPUB(encoding=3, text=meta_tags['publisher'])
-        audiofile['COMM'] = COMM(encoding=3, text=meta_tags['external_urls']['spotify'])
-        if meta_tags['lyrics']:
-            audiofile['USLT'] = USLT(encoding=3, desc=u'Lyrics', text=meta_tags['lyrics'])
+        audiofile["TORY"] = TORY(encoding=3, text=meta_tags["year"])
+        audiofile["TYER"] = TYER(encoding=3, text=meta_tags["year"])
+        if meta_tags["publisher"]:
+            audiofile["TPUB"] = TPUB(encoding=3, text=meta_tags["publisher"])
+        audiofile["COMM"] = COMM(
+            encoding=3, text=meta_tags["external_urls"][self.provider]
+        )
+        if meta_tags["lyrics"]:
+            audiofile["USLT"] = USLT(
+                encoding=3, desc=u"Lyrics", text=meta_tags["lyrics"]
+            )
         try:
-            albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
-            audiofile['APIC'] = APIC(encoding=3, mime='image/jpeg', type=3,
-                                     desc=u'Cover', data=albumart.read())
+            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
+            audiofile["APIC"] = APIC(
+                encoding=3,
+                mime="image/jpeg",
+                type=3,
+                desc=u"Cover",
+                data=albumart.read(),
+            )
             albumart.close()
         except IndexError:
             pass
@@ -95,13 +109,15 @@ class EmbedMetadata:
         meta_tags = self.meta_tags
         audiofile = MP4(music_file)
         self._embed_basic_metadata(audiofile, preset=M4A_TAG_PRESET)
-        audiofile[M4A_TAG_PRESET['year']] = meta_tags['year']
-        if meta_tags['lyrics']:
-            audiofile['lyrics'] = meta_tags['lyrics']
+        audiofile[M4A_TAG_PRESET["year"]] = meta_tags["year"]
+        audiofile[M4A_TAG_PRESET["comment"]] = meta_tags["external_urls"][self.provider]
+        if meta_tags["lyrics"]:
+            audiofile[M4A_TAG_PRESET["lyrics"]] = meta_tags["lyrics"]
         try:
-            albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
-            audiofile[M4A_TAG_PRESET['albumart']] = [MP4Cover(
-                albumart.read(), imageformat=MP4Cover.FORMAT_JPEG)]
+            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
+            audiofile[M4A_TAG_PRESET["albumart"]] = [
+                MP4Cover(albumart.read(), imageformat=MP4Cover.FORMAT_JPEG)
+            ]
             albumart.close()
         except IndexError:
             pass
@@ -114,16 +130,16 @@ class EmbedMetadata:
         meta_tags = self.meta_tags
         audiofile = FLAC(music_file)
         self._embed_basic_metadata(audiofile)
-        audiofile['year'] = meta_tags['year']
-        audiofile['comment'] = meta_tags['external_urls']['spotify']
-        if meta_tags['lyrics']:
-            audiofile['lyrics'] = meta_tags['lyrics']
+        audiofile["year"] = meta_tags["year"]
+        audiofile["comment"] = meta_tags["external_urls"][self.provider]
+        if meta_tags["lyrics"]:
+            audiofile["lyrics"] = meta_tags["lyrics"]
 
         image = Picture()
         image.type = 3
-        image.desc = 'Cover'
-        image.mime = 'image/jpeg'
-        albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
+        image.desc = "Cover"
+        image.mime = "image/jpeg"
+        albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
         image.data = albumart.read()
         albumart.close()
         audiofile.add_picture(image)
@@ -133,27 +149,30 @@ class EmbedMetadata:
 
     def _embed_basic_metadata(self, audiofile, preset=TAG_PRESET):
         meta_tags = self.meta_tags
-        audiofile[preset['artist']] = meta_tags['artists'][0]['name']
-        audiofile[preset['albumartist']] = meta_tags['artists'][0]['name']
-        audiofile[preset['album']] = meta_tags['album']['name']
-        audiofile[preset['title']] = meta_tags['name']
-        audiofile[preset['date']] = meta_tags['release_date']
-        audiofile[preset['originaldate']] = meta_tags['release_date']
-        if meta_tags['genre']:
-            audiofile[preset['genre']] = meta_tags['genre']
-        if meta_tags['copyright']:
-            audiofile[preset['copyright']] = meta_tags['copyright']
-        if self.music_file.endswith('.flac'):
-            audiofile[preset['discnumber']] = str(meta_tags['disc_number'])
+        audiofile[preset["artist"]] = meta_tags["artists"][0]["name"]
+        if meta_tags["album"]["artists"][0]["name"]:
+            audiofile[preset["albumartist"]] = meta_tags["album"]["artists"][0]["name"]
+        if meta_tags["album"]["name"]:
+            audiofile[preset["album"]] = meta_tags["album"]["name"]
+        audiofile[preset["title"]] = meta_tags["name"]
+        audiofile[preset["date"]] = meta_tags["release_date"]
+        audiofile[preset["originaldate"]] = meta_tags["release_date"]
+        if meta_tags["genre"]:
+            audiofile[preset["genre"]] = meta_tags["genre"]
+        if meta_tags["copyright"]:
+            audiofile[preset["copyright"]] = meta_tags["copyright"]
+        if self.music_file.endswith(".flac"):
+            audiofile[preset["discnumber"]] = str(meta_tags["disc_number"])
         else:
-            audiofile[preset['discnumber']] = [(meta_tags['disc_number'], 0)]
-        if self.music_file.endswith('.flac'):
-            audiofile[preset['tracknumber']] = str(meta_tags['track_number'])
+            audiofile[preset["discnumber"]] = [(meta_tags["disc_number"], 0)]
+        if self.music_file.endswith(".flac"):
+            audiofile[preset["tracknumber"]] = str(meta_tags["track_number"])
         else:
-            if preset['tracknumber'] == TAG_PRESET['tracknumber']:
-                audiofile[preset['tracknumber']] = '{}/{}'.format(meta_tags['track_number'],
-                                                                  meta_tags['total_tracks'])
+            if preset["tracknumber"] == TAG_PRESET["tracknumber"]:
+                audiofile[preset["tracknumber"]] = "{}/{}".format(
+                    meta_tags["track_number"], meta_tags["total_tracks"]
+                )
             else:
-                audiofile[preset['tracknumber']] = [
-                    (meta_tags['track_number'], meta_tags['total_tracks'])
+                audiofile[preset["tracknumber"]] = [
+                    (meta_tags["track_number"], meta_tags["total_tracks"])
                 ]

spotdl/patcher.py

@@ -0,0 +1,64 @@
+from pafy import backend_youtube_dl
+import pafy
+
+from spotdl import internals
+
+
+def _getbestthumb(self):
+    url = self._ydl_info["thumbnails"][0]["url"]
+    if url:
+        return url
+
+    part_url = "https://i.ytimg.com/vi/%s/" % self.videoid
+    # Thumbnail resolution sorted in descending order
+    thumbs = (
+        "maxresdefault.jpg",
+        "sddefault.jpg",
+        "hqdefault.jpg",
+        "mqdefault.jpg",
+        "default.jpg",
+    )
+    for thumb in thumbs:
+        url = part_url + thumb
+        if self._content_available(url):
+            return url
+
+
+def _process_streams(self):
+    for format_index in range(len(self._ydl_info["formats"])):
+        try:
+            self._ydl_info["formats"][format_index]["url"] = self._ydl_info["formats"][
+                format_index
+            ]["fragment_base_url"]
+        except KeyError:
+            pass
+    return backend_youtube_dl.YtdlPafy._old_process_streams(self)
+
+
+@classmethod
+def _content_available(cls, url):
+    return internals.content_available(url)
+
+
+class PatchPafy:
+    """
+    These patches have not been released by pafy on PyPI yet but
+    are useful to us.
+    """
+
+    def patch_getbestthumb(self):
+        # https://github.com/mps-youtube/pafy/pull/211
+        pafy.backend_shared.BasePafy._bestthumb = None
+        pafy.backend_shared.BasePafy._content_available = _content_available
+        pafy.backend_shared.BasePafy.getbestthumb = _getbestthumb
+
+    def patch_process_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/230
+        backend_youtube_dl.YtdlPafy._old_process_streams = (
+            backend_youtube_dl.YtdlPafy._process_streams
+        )
+        backend_youtube_dl.YtdlPafy._process_streams = _process_streams
+
+    def patch_insecure_streams(self):
+        # https://github.com/mps-youtube/pafy/pull/235
+        pafy.g.def_ydl_opts["prefer_insecure"] = False

spotdl/spotdl.py

@@ -1,207 +1,71 @@
 #!/usr/bin/env python3
-# -*- coding: UTF-8 -*-
+
+import sys
+import platform
+import pprint
+import logzero
+from logzero import logger as log
+
 from spotdl import __version__
 from spotdl import const
 from spotdl import handle
-from spotdl import metadata
-from spotdl import convert
 from spotdl import internals
 from spotdl import spotify_tools
 from spotdl import youtube_tools
-from slugify import slugify
-import spotipy
-import urllib.request
-import os
-import sys
-import time
-import platform
-import pprint
+from spotdl import downloader
 
 
-def check_exists(music_file, raw_song, meta_tags):
-    """ Check if the input song already exists in the given folder. """
-    log.debug('Cleaning any temp files and checking '
-              'if "{}" already exists'.format(music_file))
-    songs = os.listdir(const.args.folder)
-    for song in songs:
-        if song.endswith('.temp'):
-            os.remove(os.path.join(const.args.folder, song))
-            continue
-        # check if a song with the same name is already present in the given folder
-        if os.path.splitext(song)[0] == music_file:
-            log.debug('Found an already existing song: "{}"'.format(song))
-            if internals.is_spotify(raw_song):
-                # check if the already downloaded song has correct metadata
-                # if not, remove it and download again without prompt
-                already_tagged = metadata.compare(os.path.join(const.args.folder, song),
-                                                  meta_tags)
-                log.debug('Checking if it is already tagged correctly? {}',
-                                                            already_tagged)
-                if not already_tagged:
-                    os.remove(os.path.join(const.args.folder, song))
-                    return False
+def debug_sys_info():
+    log.debug("Python version: {}".format(sys.version))
+    log.debug("Platform: {}".format(platform.platform()))
+    log.debug(pprint.pformat(const.args.__dict__))
 
-            log.warning('"{}" already exists'.format(song))
-            if const.args.overwrite == 'prompt':
-                log.info('"{}" has already been downloaded. '
-                         'Re-download? (y/N): '.format(song))
-                prompt = input('> ')
-                if prompt.lower() == 'y':
-                    os.remove(os.path.join(const.args.folder, song))
-                    return False
+
+def match_args():
+    if const.args.song:
+        for track in const.args.song:
+            track_dl = downloader.Downloader(raw_song=track)
+            track_dl.download_single()
+    elif const.args.list:
+        if const.args.write_m3u:
+            youtube_tools.generate_m3u(
+                track_file=const.args.list
+            )
         else:
-                    return True
-            elif const.args.overwrite == 'force':
-                os.remove(os.path.join(const.args.folder, song))
-                log.info('Overwriting "{}"'.format(song))
-                return False
-            elif const.args.overwrite == 'skip':
-                log.info('Skipping "{}"'.format(song))
-                return True
-    return False
-
-
-def download_list(text_file):
-    """ Download all songs from the list. """
-    with open(text_file, 'r') as listed:
-        # read tracks into a list and remove any duplicates
-        lines = listed.read().splitlines()
-        lines = list(set(lines))
-    # ignore blank lines in text_file (if any)
-    try:
-        lines.remove('')
-    except ValueError:
-        pass
-
-    log.info(u'Preparing to download {} songs'.format(len(lines)))
-    downloaded_songs = []
-
-    for number, raw_song in enumerate(lines, 1):
-        print('')
-        try:
-            download_single(raw_song, number=number)
-        # token expires after 1 hour
-        except spotipy.client.SpotifyException:
-            # refresh token when it expires
-            log.debug('Token expired, generating new one and authorizing')
-            new_token = spotify_tools.generate_token()
-            spotify_tools.spotify = spotipy.Spotify(auth=new_token)
-            download_single(raw_song, number=number)
-        # detect network problems
-        except (urllib.request.URLError, TypeError, IOError):
-            lines.append(raw_song)
-            # remove the downloaded song from file
-            internals.trim_song(text_file)
-            # and append it at the end of file
-            with open(text_file, 'a') as myfile:
-                myfile.write(raw_song + '\n')
-            log.warning('Failed to download song. Will retry after other songs\n')
-            # wait 0.5 sec to avoid infinite looping
-            time.sleep(0.5)
-            continue
-
-        downloaded_songs.append(raw_song)
-        log.debug('Removing downloaded song from text file')
-        internals.trim_song(text_file)
-
-    return downloaded_songs
-
-
-def download_single(raw_song, number=None):
-    """ Logic behind downloading a song. """
-    if internals.is_youtube(raw_song):
-        log.debug('Input song is a YouTube URL')
-        content = youtube_tools.go_pafy(raw_song, meta_tags=None)
-        raw_song = slugify(content.title).replace('-', ' ')
-        meta_tags = spotify_tools.generate_metadata(raw_song)
-    else:
-        meta_tags = spotify_tools.generate_metadata(raw_song)
-        content = youtube_tools.go_pafy(raw_song, meta_tags)
-
-    if content is None:
-        log.debug('Found no matching video')
-        return
-
-    if const.args.download_only_metadata and meta_tags is None:
-        log.info('Found no metadata. Skipping the download')
-        return
-
-    # "[number]. [artist] - [song]" if downloading from list
-    # otherwise "[artist] - [song]"
-    youtube_title = youtube_tools.get_youtube_title(content, number)
-    log.info('{} ({})'.format(youtube_title, content.watchv_url))
-
-    # generate file name of the song to download
-    songname = content.title
-
-    if meta_tags is not None:
-        refined_songname = internals.format_string(const.args.file_format,
-                                                   meta_tags,
-                                                   slugification=True)
-        log.debug('Refining songname from "{0}" to "{1}"'.format(songname, refined_songname))
-        if not refined_songname == ' - ':
-            songname = refined_songname
-    else:
-        log.warning('Could not find metadata')
-        songname = internals.sanitize_title(songname)
-
-    if const.args.dry_run:
-        return
-
-    if not check_exists(songname, raw_song, meta_tags):
-        # deal with file formats containing slashes to non-existent directories
-        songpath = os.path.join(const.args.folder, os.path.dirname(songname))
-        os.makedirs(songpath, exist_ok=True)
-        input_song = songname + const.args.input_ext
-        output_song = songname + const.args.output_ext
-        if youtube_tools.download_song(input_song, content):
-            print('')
-            try:
-                convert.song(input_song, output_song, const.args.folder,
-                             avconv=const.args.avconv, trim_silence=const.args.trim_silence)
-            except FileNotFoundError:
-                encoder = 'avconv' if const.args.avconv else 'ffmpeg'
-                log.warning('Could not find {0}, skipping conversion'.format(encoder))
-                const.args.output_ext = const.args.input_ext
-                output_song = songname + const.args.output_ext
-
-            if not const.args.input_ext == const.args.output_ext:
-                os.remove(os.path.join(const.args.folder, input_song))
-            if not const.args.no_metadata and meta_tags is not None:
-                metadata.embed(os.path.join(const.args.folder, output_song), meta_tags)
-            return True
+            list_dl = downloader.ListDownloader(
+                tracks_file=const.args.list,
+                skip_file=const.args.skip,
+                write_successful_file=const.args.write_successful,
+            )
+            list_dl.download_list()
+    elif const.args.playlist:
+        spotify_tools.write_playlist(
+            playlist_url=const.args.playlist, text_file=const.args.write_to
+        )
+    elif const.args.album:
+        spotify_tools.write_album(
+            album_url=const.args.album, text_file=const.args.write_to
+        )
+    elif const.args.all_albums:
+        spotify_tools.write_all_albums_from_artist(
+            artist_url=const.args.all_albums, text_file=const.args.write_to
+        )
+    elif const.args.username:
+        spotify_tools.write_user_playlist(
+            username=const.args.username, text_file=const.args.write_to
+        )
 
 
 def main():
     const.args = handle.get_arguments()
 
-    if const.args.version:
-        print('spotdl {version}'.format(version=__version__))
-        sys.exit()
-
     internals.filter_path(const.args.folder)
     youtube_tools.set_api_key()
 
-    const.log = const.logzero.setup_logger(formatter=const._formatter,
-                                      level=const.args.log_level)
-    global log
-    log = const.log
-    log.debug('Python version: {}'.format(sys.version))
-    log.debug('Platform: {}'.format(platform.platform()))
-    log.debug(pprint.pformat(const.args.__dict__))
+    logzero.setup_default_logger(formatter=const._formatter, level=const.args.log_level)
 
     try:
-        if const.args.song:
-            download_single(raw_song=const.args.song)
-        elif const.args.list:
-            download_list(text_file=const.args.list)
-        elif const.args.playlist:
-            spotify_tools.write_playlist(playlist_url=const.args.playlist)
-        elif const.args.album:
-            spotify_tools.write_album(album_url=const.args.album)
-        elif const.args.username:
-            spotify_tools.write_user_playlist(username=const.args.username)
-
+        match_args()
         # actually we don't necessarily need this, but yeah...
         # explicit is better than implicit!
         sys.exit(0)
@@ -211,5 +75,5 @@ def main():
         sys.exit(3)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()

spotdl/spotify_tools.py

@@ -1,90 +1,114 @@
 import spotipy
 import spotipy.oauth2 as oauth2
-import lyricwikia
-
-from spotdl import internals
-from spotdl.const import log
 
 from slugify import slugify
 from titlecase import titlecase
+from logzero import logger as log
 import pprint
 import sys
+import os
+import functools
+
+from spotdl import const
+from spotdl import internals
+from spotdl.lyrics.providers import LyricClasses
+from spotdl.lyrics.exceptions import LyricsNotFound
+
+spotify = None
 
 
 def generate_token():
-    """ Generate the token. Please respect these credentials :) """
+    """ Generate the token. """
     credentials = oauth2.SpotifyClientCredentials(
-        client_id='4fe3fecfe5334023a1472516cc99d805',
-        client_secret='0f02b7c483c04257984695007a4a8d5c')
+        client_id=const.args.spotify_client_id,
+        client_secret=const.args.spotify_client_secret,
+    )
     token = credentials.get_access_token()
     return token
 
-# token is mandatory when using Spotify's API
-# https://developer.spotify.com/news-stories/2017/01/27/removing-unauthenticated-calls-to-the-web-api/
-token = generate_token()
-spotify = spotipy.Spotify(auth=token)
+
+def must_be_authorized(func, spotify=spotify):
+    def wrapper(*args, **kwargs):
+        global spotify
+        try:
+            assert spotify
+            return func(*args, **kwargs)
+        except (AssertionError, spotipy.client.SpotifyException):
+            token = generate_token()
+            spotify = spotipy.Spotify(auth=token)
+            return func(*args, **kwargs)
+
+    return wrapper
 
 
+@must_be_authorized
 def generate_metadata(raw_song):
     """ Fetch a song's metadata from Spotify. """
     if internals.is_spotify(raw_song):
         # fetch track information directly if it is spotify link
-        log.debug('Fetching metadata for given track URL')
+        log.debug("Fetching metadata for given track URL")
         meta_tags = spotify.track(raw_song)
     else:
         # otherwise search on spotify and fetch information from first result
         log.debug('Searching for "{}" on Spotify'.format(raw_song))
         try:
-            meta_tags = spotify.search(raw_song, limit=1)['tracks']['items'][0]
+            meta_tags = spotify.search(raw_song, limit=1)["tracks"]["items"][0]
         except IndexError:
             return None
-    artist = spotify.artist(meta_tags['artists'][0]['id'])
-    album = spotify.album(meta_tags['album']['id'])
+    artist = spotify.artist(meta_tags["artists"][0]["id"])
+    album = spotify.album(meta_tags["album"]["id"])
 
     try:
-        meta_tags[u'genre'] = titlecase(artist['genres'][0])
+        meta_tags[u"genre"] = titlecase(artist["genres"][0])
     except IndexError:
-        meta_tags[u'genre'] = None
+        meta_tags[u"genre"] = None
     try:
-        meta_tags[u'copyright'] = album['copyrights'][0]['text']
+        meta_tags[u"copyright"] = album["copyrights"][0]["text"]
     except IndexError:
-        meta_tags[u'copyright'] = None
+        meta_tags[u"copyright"] = None
     try:
-        meta_tags[u'external_ids'][u'isrc']
+        meta_tags[u"external_ids"][u"isrc"]
     except KeyError:
-        meta_tags[u'external_ids'][u'isrc'] = None
+        meta_tags[u"external_ids"][u"isrc"] = None
 
-    meta_tags[u'release_date'] = album['release_date']
-    meta_tags[u'publisher'] = album['label']
-    meta_tags[u'total_tracks'] = album['tracks']['total']
+    meta_tags[u"release_date"] = album["release_date"]
+    meta_tags[u"publisher"] = album["label"]
+    meta_tags[u"total_tracks"] = album["tracks"]["total"]
 
-    log.debug('Fetching lyrics')
+    log.debug("Fetching lyrics")
+    meta_tags["lyrics"] = None
 
+    for LyricClass in LyricClasses:
+        track = LyricClass(meta_tags["artists"][0]["name"], meta_tags["name"])
         try:
-        meta_tags['lyrics'] = lyricwikia.get_lyrics(
-                        meta_tags['artists'][0]['name'],
-                        meta_tags['name'])
-    except lyricwikia.LyricsNotFound:
-        meta_tags['lyrics'] = None
+            meta_tags["lyrics"] = track.get_lyrics()
+        except LyricsNotFound:
+            continue
+        else:
+            break
 
     # Some sugar
-    meta_tags['year'], *_ = meta_tags['release_date'].split('-')
-    meta_tags['duration'] = meta_tags['duration_ms'] / 1000.0
+    meta_tags["year"], *_ = meta_tags["release_date"].split("-")
+    meta_tags["duration"] = meta_tags["duration_ms"] / 1000.0
+    meta_tags["spotify_metadata"] = True
     # Remove unwanted parameters
-    del meta_tags['duration_ms']
-    del meta_tags['available_markets']
-    del meta_tags['album']['available_markets']
+    del meta_tags["duration_ms"]
+    del meta_tags["available_markets"]
+    del meta_tags["album"]["available_markets"]
 
     log.debug(pprint.pformat(meta_tags))
     return meta_tags
 
 
+@must_be_authorized
 def write_user_playlist(username, text_file=None):
+    """ Write user playlists to text_file """
     links = get_playlists(username=username)
     playlist = internals.input_link(links)
     return write_playlist(playlist, text_file)
 
 
+@must_be_authorized
 def get_playlists(username):
     """ Fetch user playlists when using the -u option. """
     playlists = spotify.user_playlists(username)
@@ -92,18 +116,20 @@ def get_playlists(username):
     check = 1
 
     while True:
-        for playlist in playlists['items']:
+        for playlist in playlists["items"]:
             # in rare cases, playlists may not be found, so playlists['next']
             # is None. Skip these. Also see Issue #91.
-            if playlist['name'] is not None:
-                log.info(u'{0:>5}. {1:<30}  ({2} tracks)'.format(
-                    check, playlist['name'],
-                    playlist['tracks']['total']))
-                playlist_url = playlist['external_urls']['spotify']
+            if playlist["name"] is not None:
+                log.info(
+                    u"{0:>5}. {1:<30}  ({2} tracks)".format(
+                        check, playlist["name"], playlist["tracks"]["total"]
+                    )
+                )
+                playlist_url = playlist["external_urls"]["spotify"]
                 log.debug(playlist_url)
                 links.append(playlist_url)
                 check += 1
-        if playlists['next']:
+        if playlists["next"]:
             playlists = spotify.next(playlists)
         else:
             break
@@ -111,71 +137,128 @@ def get_playlists(username):
     return links
 
 
+@must_be_authorized
 def fetch_playlist(playlist):
-    splits = internals.get_splits(playlist)
     try:
-        username = splits[-3]
+        playlist_id = internals.extract_spotify_id(playlist)
     except IndexError:
         # Wrong format, in either case
-        log.error('The provided playlist URL is not in a recognized format!')
+        log.error("The provided playlist URL is not in a recognized format!")
         sys.exit(10)
-    playlist_id = splits[-1]
     try:
-        results = spotify.user_playlist(username, playlist_id,
-                                        fields='tracks,next,name')
+        results = spotify.user_playlist(
+            user=None, playlist_id=playlist_id, fields="tracks,next,name"
+        )
     except spotipy.client.SpotifyException:
-        log.error('Unable to find playlist')
-        log.info('Make sure the playlist is set to publicly visible and then try again')
+        log.error("Unable to find playlist")
+        log.info("Make sure the playlist is set to publicly visible and then try again")
         sys.exit(11)
 
     return results
 
 
+@must_be_authorized
 def write_playlist(playlist_url, text_file=None):
     playlist = fetch_playlist(playlist_url)
-    tracks = playlist['tracks']
+    tracks = playlist["tracks"]
     if not text_file:
-        text_file = u'{0}.txt'.format(slugify(playlist['name'], ok='-_()[]{}'))
+        text_file = u"{0}.txt".format(slugify(playlist["name"], ok="-_()[]{}"))
     return write_tracks(tracks, text_file)
 
 
+@must_be_authorized
 def fetch_album(album):
-    splits = internals.get_splits(album)
-    album_id = splits[-1]
+    album_id = internals.extract_spotify_id(album)
     album = spotify.album(album_id)
     return album
 
 
+@must_be_authorized
+def fetch_albums_from_artist(artist_url, album_type=None):
+    """
+    This funcction returns all the albums from a give artist_url using the US
+    market
+    :param artist_url - spotify artist url
+    :param album_type - the type of album to fetch (ex: single) the default is
+                        all albums
+    :param return - the album from the artist
+    """
+
+    # fetching artist's albums limitting the results to the US to avoid duplicate
+    # albums from multiple markets
+    artist_id = internals.extract_spotify_id(artist_url)
+    results = spotify.artist_albums(artist_id, album_type=album_type, country="US")
+
+    albums = results["items"]
+
+    # indexing all pages of results
+    while results["next"]:
+        results = spotify.next(results)
+        albums.extend(results["items"])
+
+    return albums
+
+
+@must_be_authorized
+def write_all_albums_from_artist(artist_url, text_file=None):
+    """
+    This function gets all albums from an artist and writes it to a file in the
+    current working directory called [ARTIST].txt, where [ARTIST] is the artist
+    of the album
+    :param artist_url - spotify artist url
+    :param text_file - file to write albums to
+    """
+
+    album_base_url = "https://open.spotify.com/album/"
+
+    # fetching all default albums
+    albums = fetch_albums_from_artist(artist_url, album_type=None)
+
+    # if no file if given, the default save file is in the current working
+    # directory with the name of the artist
+    if text_file is None:
+        text_file = albums[0]["artists"][0]["name"] + ".txt"
+
+    for album in albums:
+        # logging album name
+        log.info("Fetching album: " + album["name"])
+        write_album(album_base_url + album["id"], text_file=text_file)
+
+
+@must_be_authorized
 def write_album(album_url, text_file=None):
     album = fetch_album(album_url)
-    tracks = spotify.album_tracks(album['id'])
+    tracks = spotify.album_tracks(album["id"])
     if not text_file:
-        text_file = u'{0}.txt'.format(slugify(album['name'], ok='-_()[]{}'))
+        text_file = u"{0}.txt".format(slugify(album["name"], ok="-_()[]{}"))
     return write_tracks(tracks, text_file)
 
 
+@must_be_authorized
 def write_tracks(tracks, text_file):
-    log.info(u'Writing {0} tracks to {1}'.format(
-               tracks['total'], text_file))
+    log.info(u"Writing {0} tracks to {1}".format(tracks["total"], text_file))
     track_urls = []
-    with open(text_file, 'a') as file_out:
+    with open(text_file, "a") as file_out:
         while True:
-            for item in tracks['items']:
-                if 'track' in item:
-                    track = item['track']
+            for item in tracks["items"]:
+                if "track" in item:
+                    track = item["track"]
                 else:
                     track = item
                 try:
-                    track_url = track['external_urls']['spotify']
+                    track_url = track["external_urls"]["spotify"]
                     log.debug(track_url)
-                    file_out.write(track_url + '\n')
+                    file_out.write(track_url + "\n")
                     track_urls.append(track_url)
                 except KeyError:
-                    log.warning(u'Skipping track {0} by {1} (local only?)'.format(
-                        track['name'], track['artists'][0]['name']))
+                    log.warning(
+                        u"Skipping track {0} by {1} (local only?)".format(
+                            track["name"], track["artists"][0]["name"]
+                        )
+                    )
             # 1 page = 50 results
             # check if there are more pages
-            if tracks['next']:
+            if tracks["next"]:
                 tracks = spotify.next(tracks)
             else:
                 break

spotdl/youtube_tools.py

@@ -2,17 +2,27 @@ from bs4 import BeautifulSoup
 import urllib
 import pafy
 
+from slugify import slugify
+from logzero import logger as log
+import os
+
+from spotdl import spotify_tools
 from spotdl import internals
 from spotdl import const
 
-import os
-import pprint
-
-log = const.log
-
 # Fix download speed throttle on short duration tracks
 # Read more on mps-youtube/pafy#199
-pafy.g.opener.addheaders.append(('Range', 'bytes=0-'))
+pafy.g.opener.addheaders.append(("Range", "bytes=0-"))
+
+# Implement unreleased methods on Pafy object
+# More info: https://github.com/mps-youtube/pafy/pull/211
+if pafy.__version__ <= "0.5.5":
+    from spotdl import patcher
+
+    pafy_patcher = patcher.PatchPafy()
+    pafy_patcher.patch_getbestthumb()
+    pafy_patcher.patch_process_streams()
+    pafy_patcher.patch_insecure_streams()
 
 
 def set_api_key():
@@ -20,7 +30,7 @@ def set_api_key():
         key = const.args.youtube_api_key
     else:
         # Please respect this YouTube token :)
-        key = 'AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0'
+        key = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
     pafy.set_api_key(key)
 
 
@@ -39,32 +49,141 @@ def go_pafy(raw_song, meta_tags=None):
     return track_info
 
 
+def match_video_and_metadata(track):
+    """ Get and match track data from YouTube and Spotify. """
+    meta_tags = None
+
+    def fallback_metadata(meta_tags):
+        fallback_metadata_info = (
+            "Track not found on Spotify, falling back on YouTube metadata"
+        )
+        skip_fallback_metadata_warning = (
+            "Fallback condition not met, shall not embed metadata"
+        )
+        if meta_tags is None:
+            if const.args.no_fallback_metadata:
+                log.warning(skip_fallback_metadata_warning)
+            else:
+                log.info(fallback_metadata_info)
+                meta_tags = generate_metadata(content)
+        return meta_tags
+
+    if internals.is_youtube(track):
+        log.debug("Input song is a YouTube URL")
+        content = go_pafy(track, meta_tags=None)
+        track = slugify(content.title).replace("-", " ")
+        if not const.args.no_metadata:
+            meta_tags = spotify_tools.generate_metadata(track)
+            meta_tags = fallback_metadata(meta_tags)
+
+    elif internals.is_spotify(track):
+        log.debug("Input song is a Spotify URL")
+        # Let it generate metadata, YouTube doesn't know Spotify slang
+        meta_tags = spotify_tools.generate_metadata(track)
+        content = go_pafy(track, meta_tags)
+        if const.args.no_metadata:
+            meta_tags = None
+
+    else:
+        log.debug("Input song is plain text based")
+        if const.args.no_metadata:
+            content = go_pafy(track, meta_tags=None)
+        else:
+            meta_tags = spotify_tools.generate_metadata(track)
+            content = go_pafy(track, meta_tags=meta_tags)
+            meta_tags = fallback_metadata(meta_tags)
+
+    return content, meta_tags
+
+
+def generate_metadata(content):
+    """ Fetch a song's metadata from YouTube. """
+    meta_tags = {
+        "spotify_metadata": False,
+        "name": content.title,
+        "artists": [{"name": content.author}],
+        "duration": content.length,
+        "external_urls": {"youtube": content.watchv_url},
+        "album": {
+            "images": [{"url": content.getbestthumb()}],
+            "artists": [{"name": None}],
+            "name": None,
+        },
+        "year": content.published.split("-")[0],
+        "release_date": content.published.split(" ")[0],
+        "type": "track",
+        "disc_number": 1,
+        "track_number": 1,
+        "total_tracks": 1,
+        "publisher": None,
+        "external_ids": {"isrc": None},
+        "lyrics": None,
+        "copyright": None,
+        "genre": None,
+    }
+
+    return meta_tags
+
+
 def get_youtube_title(content, number=None):
     """ Get the YouTube video's title. """
     title = content.title
     if number:
-        return '{0}. {1}'.format(number, title)
+        return "{0}. {1}".format(number, title)
     else:
         return title
 
 
+def generate_m3u(track_file):
+    tracks = internals.get_unique_tracks(track_file)
+    target_file = "{}.m3u".format(track_file.split(".")[0])
+    total_tracks = len(tracks)
+    log.info("Generating {0} from {1} YouTube URLs".format(target_file, total_tracks))
+    with open(target_file, "w") as output_file:
+        output_file.write("#EXTM3U\n\n")
+
+    videos = []
+    for n, track in enumerate(tracks, 1):
+        content, _ = match_video_and_metadata(track)
+        if content is None:
+            log.warning("Skipping {}".format(track))
+        else:
+            log.info(
+                "Matched track {0}/{1} ({2})".format(
+                    n, total_tracks, content.watchv_url
+                )
+            )
+            log.debug(track)
+            m3u_key = "#EXTINF:{duration},{title}\n{youtube_url}\n".format(
+                duration=internals.get_sec(content.duration),
+                title=content.title,
+                youtube_url=content.watchv_url,
+            )
+            log.debug(m3u_key)
+            with open(target_file, "a") as output_file:
+                output_file.write(m3u_key)
+            videos.append(content.watchv_url)
+
+    return videos
+
+
 def download_song(file_name, content):
     """ Download the audio file from YouTube. """
     _, extension = os.path.splitext(file_name)
-    if extension in ('.webm', '.m4a'):
+    if extension in (".webm", ".m4a"):
         link = content.getbestaudio(preftype=extension[1:])
     else:
-        log.debug('No audio streams available for {} type'.format(extension))
+        log.debug("No audio streams available for {} type".format(extension))
         return False
 
     if link:
-        log.debug('Downloading from URL: ' + link.url)
+        log.debug("Downloading from URL: " + link.url)
         filepath = os.path.join(const.args.folder, file_name)
-        log.debug('Saving to: ' + filepath)
+        log.debug("Saving to: " + filepath)
         link.download(filepath=filepath)
         return True
     else:
-        log.debug('No audio streams available')
+        log.debug("No audio streams available")
         return False
 
 
@@ -73,23 +192,25 @@ def generate_search_url(query):
     # urllib.request.quote() encodes string with special characters
     quoted_query = urllib.request.quote(query)
     # Special YouTube URL filter to search only for videos
-    url = 'https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={0}'.format(quoted_query)
+    url = "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={0}".format(
+        quoted_query
+    )
     return url
 
 
 def is_video(result):
     # ensure result is not a channel
-    not_video = result.find('channel') is not None or \
-                'yt-lockup-channel' in result.parent.attrs['class'] or \
-                'yt-lockup-channel' in result.attrs['class']
+    not_video = (
+        result.find("channel") is not None
+        or "yt-lockup-channel" in result.parent.attrs["class"]
+        or "yt-lockup-channel" in result.attrs["class"]
+    )
 
     # ensure result is not a mix/playlist
-    not_video = not_video or \
-               'yt-lockup-playlist' in result.parent.attrs['class']
+    not_video = not_video or "yt-lockup-playlist" in result.parent.attrs["class"]
 
     # ensure video result is not an advertisement
-    not_video = not_video or \
-                result.find('googleads') is not None
+    not_video = not_video or result.find("googleads") is not None
 
     video = not not_video
     return video
@@ -112,18 +233,29 @@ class GenerateYouTubeURL:
         if meta_tags is None:
             self.search_query = raw_song
         else:
-            self.search_query = internals.format_string(const.args.search_format,
-                                                        meta_tags, force_spaces=True)
+            self.search_query = internals.format_string(
+                const.args.search_format, meta_tags, force_spaces=True
+            )
 
     def _best_match(self, videos):
+        if not videos:
+            log.error("No videos found on YouTube for a given search")
+            return None
+
         """ Select the best matching video from a list of videos. """
         if const.args.manual:
             log.info(self.raw_song)
-            log.info('0. Skip downloading this song.\n')
+            log.info("0. Skip downloading this song.\n")
             # fetch all video links on first page on YouTube
             for i, v in enumerate(videos):
-                log.info(u'{0}. {1} {2} {3}'.format(i+1, v['title'], v['videotime'],
-                      "http://youtube.com/watch?v="+v['link']))
+                log.info(
+                    u"{0}. {1} {2} {3}".format(
+                        i + 1,
+                        v["title"],
+                        v["videotime"],
+                        "http://youtube.com/watch?v=" + v["link"],
+                    )
+                )
             # let user select the song to download
             result = internals.input_link(videos)
             if result is None:
@@ -133,7 +265,9 @@ class GenerateYouTubeURL:
                 # if the metadata could not be acquired, take the first result
                 # from Youtube because the proper song length is unknown
                 result = videos[0]
-                log.debug('Since no metadata found on Spotify, going with the first result')
+                log.debug(
+                    "Since no metadata found on Spotify, going with the first result"
+                )
             else:
                 # filter out videos that do not have a similar length to the Spotify song
                 duration_tolerance = 10
@@ -144,16 +278,27 @@ class GenerateYouTubeURL:
                 # until one of the Youtube results falls within the correct duration or
                 # the duration_tolerance has reached the max_duration_tolerance
                 while len(possible_videos_by_duration) == 0:
-                    possible_videos_by_duration = list(filter(lambda x: abs(x['seconds'] - self.meta_tags['duration']) <= duration_tolerance, videos))
+                    possible_videos_by_duration = list(
+                        filter(
+                            lambda x: abs(x["seconds"] - self.meta_tags["duration"])
+                            <= duration_tolerance,
+                            videos,
+                        )
+                    )
                     duration_tolerance += 1
                     if duration_tolerance > max_duration_tolerance:
-                        log.error("{0} by {1} was not found.\n".format(self.meta_tags['name'], self.meta_tags['artists'][0]['name']))
+                        log.error(
+                            "{0} by {1} was not found.".format(
+                                self.meta_tags["name"],
+                                self.meta_tags["artists"][0]["name"],
+                            )
+                        )
                         return None
 
                 result = possible_videos_by_duration[0]
 
         if result:
-            url = "http://youtube.com/watch?v={0}".format(result['link'])
+            url = "http://youtube.com/watch?v={0}".format(result["link"])
         else:
             url = None
 
@@ -164,33 +309,41 @@ class GenerateYouTubeURL:
 
         # prevents an infinite loop but allows for a few retries
         if tries_remaining == 0:
-            log.debug('No tries left. I quit.')
+            log.debug("No tries left. I quit.")
             return
 
         search_url = generate_search_url(self.search_query)
-        log.debug('Opening URL: {0}'.format(search_url))
+        log.debug("Opening URL: {0}".format(search_url))
 
-        item = urllib.request.urlopen(search_url).read()
+        item = self._fetch_response(search_url).read()
         items_parse = BeautifulSoup(item, "html.parser")
 
         videos = []
-        for x in items_parse.find_all('div', {'class': 'yt-lockup-dismissable yt-uix-tile'}):
+        for x in items_parse.find_all(
+            "div", {"class": "yt-lockup-dismissable yt-uix-tile"}
+        ):
 
             if not is_video(x):
                 continue
 
-            y = x.find('div', class_='yt-lockup-content')
-            link = y.find('a')['href'][-11:]
-            title = y.find('a')['title']
+            y = x.find("div", class_="yt-lockup-content")
+            link = y.find("a")["href"][-11:]
+            title = y.find("a")["title"]
 
             try:
-                videotime = x.find('span', class_="video-time").get_text()
+                videotime = x.find("span", class_="video-time").get_text()
             except AttributeError:
-                log.debug('Could not find video duration on YouTube, retrying..')
-                return self.scrape(bestmatch=bestmatch, tries_remaining=tries_remaining-1)
+                log.debug("Could not find video duration on YouTube, retrying..")
+                return self.scrape(
+                    bestmatch=bestmatch, tries_remaining=tries_remaining - 1
+                )
 
-            youtubedetails = {'link': link, 'title': title, 'videotime': videotime,
-                              'seconds': internals.get_sec(videotime)}
+            youtubedetails = {
+                "link": link,
+                "title": title,
+                "videotime": videotime,
+                "seconds": internals.get_sec(videotime),
+            }
             videos.append(youtubedetails)
 
         if bestmatch:
@@ -198,43 +351,54 @@ class GenerateYouTubeURL:
 
         return videos
 
-
     def api(self, bestmatch=True):
         """ Use YouTube API to search and return a list of matching videos. """
 
-        query = { 'part'       : 'snippet',
-                  'maxResults' :  50,
-                  'type'       : 'video' }
+        query = {"part": "snippet", "maxResults": 50, "type": "video"}
 
         if const.args.music_videos_only:
-            query['videoCategoryId'] = '10'
+            query["videoCategoryId"] = "10"
 
         if not self.meta_tags:
             song = self.raw_song
-            query['q'] = song
+            query["q"] = song
         else:
-            query['q'] = self.search_query
-        log.debug('query: {0}'.format(query))
+            query["q"] = self.search_query
+        log.debug("query: {0}".format(query))
 
-        data = pafy.call_gdata('search', query)
-        data['items'] = list(filter(lambda x: x['id'].get('videoId') is not None,
-                                    data['items']))
-        query_results = {'part': 'contentDetails,snippet,statistics',
-                  'maxResults': 50,
-                  'id': ','.join(i['id']['videoId'] for i in data['items'])}
-        log.debug('query_results: {0}'.format(query_results))
+        data = pafy.call_gdata("search", query)
+        data["items"] = list(
+            filter(lambda x: x["id"].get("videoId") is not None, data["items"])
+        )
+        query_results = {
+            "part": "contentDetails,snippet,statistics",
+            "maxResults": 50,
+            "id": ",".join(i["id"]["videoId"] for i in data["items"]),
+        }
+        log.debug("query_results: {0}".format(query_results))
 
-        vdata = pafy.call_gdata('videos', query_results)
+        vdata = pafy.call_gdata("videos", query_results)
 
         videos = []
-        for x in vdata['items']:
-            duration_s = pafy.playlist.parseISO8591(x['contentDetails']['duration'])
-            youtubedetails = {'link': x['id'], 'title': x['snippet']['title'],
-                              'videotime':internals.videotime_from_seconds(duration_s),
-                              'seconds': duration_s}
+        for x in vdata["items"]:
+            duration_s = pafy.playlist.parseISO8591(x["contentDetails"]["duration"])
+            youtubedetails = {
+                "link": x["id"],
+                "title": x["snippet"]["title"],
+                "videotime": internals.videotime_from_seconds(duration_s),
+                "seconds": duration_s,
+            }
             videos.append(youtubedetails)
 
         if bestmatch:
             return self._best_match(videos)
 
         return videos
+
+    @staticmethod
+    def _fetch_response(url):
+        # XXX: This method exists only because it helps us indirectly
+        # monkey patch `urllib.request.open`, directly monkey patching
+        # `urllib.request.open` causes us to end up in an infinite recursion
+        # during the test since `urllib.request.open` would monkeypatch itself.
+        return urllib.request.urlopen(url)

test/loader.py

@@ -1,14 +1,26 @@
 from spotdl import const
 from spotdl import handle
 from spotdl import spotdl
+import urllib
+
 import pytest
 
 
 def load_defaults():
-    const.args = handle.get_arguments(raw_args='', to_group=False, to_merge=False)
-    const.args.overwrite = 'skip'
-    const.args.log_level = 10
+    const.args = handle.get_arguments(raw_args="", to_group=False, to_merge=False)
+    const.args.overwrite = "skip"
 
     spotdl.args = const.args
-    spotdl.log = const.logzero.setup_logger(formatter=const._formatter,
-                                      level=const.args.log_level)
+    spotdl.log = const.logzero.setup_logger(
+        formatter=const._formatter, level=const.args.log_level
+    )
+
+
+# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
+# so that we get same results even if YouTube changes the list/order of videos on their page.
+GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
+
+
+def monkeypatch_youtube_search_page(*args, **kwargs):
+    fake_urlopen = urllib.request.urlopen(GIST_URL)
+    return fake_urlopen

test/test_download_with_metadata.py

@@ -0,0 +1,244 @@
+import subprocess
+import os
+
+from spotdl import const
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+from spotdl import convert
+from spotdl import metadata
+from spotdl import downloader
+
+import pytest
+import loader
+
+loader.load_defaults()
+
+SPOTIFY_TRACK_URL = "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD"
+EXPECTED_YOUTUBE_TITLE = "Janji - Heroes Tonight (feat. Johnning) [NCS Release]"
+EXPECTED_SPOTIFY_TITLE = "Janji - Heroes Tonight"
+EXPECTED_YOUTUBE_URL = "http://youtube.com/watch?v=3nQNiWdeH2Q"
+# GIST_URL is the monkeypatched version of: https://www.youtube.com/results?search_query=janji+-+heroes
+# so that we get same results even if YouTube changes the list/order of videos on their page.
+GIST_URL = "https://gist.githubusercontent.com/ritiek/e731338e9810e31c2f00f13c249a45f5/raw/c11a27f3b5d11a8d082976f1cdd237bd605ec2c2/search_results.html"
+
+
+def pytest_namespace():
+    # XXX: We override the value of `content_fixture` later in the tests.
+    # We do not use an acutal @pytest.fixture because it does not accept
+    # the monkeypatch parameter and we need to monkeypatch the network
+    # request before creating the Pafy object.
+    return {"content_fixture": None}
+
+
+@pytest.fixture(scope="module")
+def metadata_fixture():
+    meta_tags = spotify_tools.generate_metadata(SPOTIFY_TRACK_URL)
+    return meta_tags
+
+
+def test_metadata(metadata_fixture):
+    expect_number = 24
+    assert len(metadata_fixture) == expect_number
+
+
+class TestFileFormat:
+    def test_with_spaces(self, metadata_fixture):
+        title = internals.format_string(const.args.file_format, metadata_fixture)
+        assert title == EXPECTED_SPOTIFY_TITLE
+
+    def test_without_spaces(self, metadata_fixture):
+        const.args.no_spaces = True
+        title = internals.format_string(const.args.file_format, metadata_fixture)
+        assert title == EXPECTED_SPOTIFY_TITLE.replace(" ", "_")
+
+
+def test_youtube_url(metadata_fixture, monkeypatch):
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
+    url = youtube_tools.generate_youtube_url(SPOTIFY_TRACK_URL, metadata_fixture)
+    assert url == EXPECTED_YOUTUBE_URL
+
+
+def test_youtube_title(metadata_fixture, monkeypatch):
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
+    content = youtube_tools.go_pafy(SPOTIFY_TRACK_URL, metadata_fixture)
+    pytest.content_fixture = content
+    title = youtube_tools.get_youtube_title(content)
+    assert title == EXPECTED_YOUTUBE_TITLE
+
+
+@pytest.fixture(scope="module")
+def filename_fixture(metadata_fixture):
+    songname = internals.format_string(const.args.file_format, metadata_fixture)
+    filename = internals.sanitize_title(songname)
+    return filename
+
+
+def test_check_track_exists_before_download(tmpdir, metadata_fixture, filename_fixture):
+    expect_check = False
+    const.args.folder = str(tmpdir)
+    # prerequisites for determining filename
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
+    assert check == expect_check
+
+
+class TestDownload:
+    def blank_audio_generator(self, filepath):
+        if filepath.endswith(".m4a"):
+            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a aac {}".format(filepath)
+        elif filepath.endswith(".webm"):
+            cmd = "ffmpeg -f lavfi -i anullsrc -t 1 -c:a libopus {}".format(filepath)
+        subprocess.call(cmd.split(" "))
+
+    def test_m4a(self, monkeypatch, filename_fixture):
+        expect_download = True
+        monkeypatch.setattr(
+            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
+        )
+        monkeypatch.setattr(
+            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
+        )
+        download = youtube_tools.download_song(
+            filename_fixture + ".m4a", pytest.content_fixture
+        )
+        assert download == expect_download
+
+    def test_webm(self, monkeypatch, filename_fixture):
+        expect_download = True
+        monkeypatch.setattr(
+            "pafy.backend_shared.BaseStream.download", self.blank_audio_generator
+        )
+        monkeypatch.setattr(
+            "pafy.backend_youtube_dl.YtdlStream.download", self.blank_audio_generator
+        )
+        download = youtube_tools.download_song(
+            filename_fixture + ".webm", pytest.content_fixture
+        )
+        assert download == expect_download
+
+
+class TestFFmpeg:
+    def test_convert_from_webm_to_mp3(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".webm", filename_fixture + ".mp3", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_webm_to_m4a(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.webm -cutoff 20000 -codec:a aac -ar 44100 -b:a 192k -vn {0}.m4a".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".webm", filename_fixture + ".m4a", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:v copy -codec:a libmp3lame -ar 44100 -b:a 192k -vn {0}.mp3".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".mp3", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_webm(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a libopus -vbr on -b:a 192k -vn {0}.webm".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".webm", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_convert_from_m4a_to_flac(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a -codec:a flac -ar 44100 -b:a 192k -vn {0}.flac".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        monkeypatch.setattr("os.remove", lambda x: None)
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".flac", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+    def test_correct_container_for_m4a(self, filename_fixture, monkeypatch):
+        expect_command = "ffmpeg -y -nostdin -hide_banner -nostats -v panic -i {0}.m4a.temp -acodec copy -b:a 192k -vn {0}.m4a".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        _, command = convert.song(
+            filename_fixture + ".m4a", filename_fixture + ".m4a", const.args.folder
+        )
+        assert " ".join(command) == expect_command
+
+
+class TestAvconv:
+    @pytest.mark.skip(reason="avconv is no longer provided with FFmpeg")
+    def test_convert_from_m4a_to_mp3(self, filename_fixture, monkeypatch):
+        monkeypatch.setattr("os.remove", lambda x: None)
+        expect_command = "avconv -loglevel 0 -i {0}.m4a -ab 192k {0}.mp3 -y".format(
+            os.path.join(const.args.folder, filename_fixture)
+        )
+        _, command = convert.song(
+            filename_fixture + ".m4a",
+            filename_fixture + ".mp3",
+            const.args.folder,
+            avconv=True,
+        )
+        assert " ".join(command) == expect_command
+
+
+@pytest.fixture(scope="module")
+def trackpath_fixture(filename_fixture):
+    trackpath = os.path.join(const.args.folder, filename_fixture)
+    return trackpath
+
+
+class TestEmbedMetadata:
+    def test_embed_in_mp3(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".mp3", metadata_fixture)
+        assert embed == expect_embed
+
+    def test_embed_in_m4a(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".m4a", metadata_fixture)
+        os.remove(trackpath_fixture + ".m4a")
+        assert embed == expect_embed
+
+    def test_embed_in_webm(self, metadata_fixture, trackpath_fixture):
+        expect_embed = False
+        embed = metadata.embed(trackpath_fixture + ".webm", metadata_fixture)
+        os.remove(trackpath_fixture + ".webm")
+        assert embed == expect_embed
+
+    def test_embed_in_flac(self, metadata_fixture, trackpath_fixture):
+        expect_embed = True
+        embed = metadata.embed(trackpath_fixture + ".flac", metadata_fixture)
+        os.remove(trackpath_fixture + ".flac")
+        assert embed == expect_embed
+
+
+def test_check_track_exists_after_download(
+    metadata_fixture, filename_fixture, trackpath_fixture
+):
+    expect_check = True
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(SPOTIFY_TRACK_URL)
+    os.remove(trackpath_fixture + ".mp3")
+    assert check == expect_check

test/test_dry_run.py

@@ -1,18 +1,21 @@
-from spotdl import const
-
-from spotdl import spotdl
-import loader
 import os
 
+from spotdl import const
+from spotdl import downloader
+
+import loader
+
 loader.load_defaults()
 
+TRACK_URL = "http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU"
+
 
 def test_dry_download_list(tmpdir):
-    song = 'http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU'
     const.args.folder = str(tmpdir)
     const.args.dry_run = True
-    file_path = os.path.join(const.args.folder, 'test_list.txt')
-    with open(file_path, 'w') as tin:
-        tin.write(song)
-    downloaded_song, *_ = spotdl.download_list(file_path)
-    assert downloaded_song == song
+    file_path = os.path.join(const.args.folder, "test_list.txt")
+    with open(file_path, "w") as f:
+        f.write(TRACK_URL)
+    list_dl = downloader.ListDownloader(file_path)
+    downloaded_song, *_ = list_dl.download_list()
+    assert downloaded_song == TRACK_URL

test/test_handle.py

@@ -1,46 +1,66 @@
-import yaml
-
-from spotdl import handle
-from spotdl import const
-
-import pytest
 import os
 import sys
 import argparse
 
+from spotdl import handle
+
+import pytest
+import yaml
+
+
+def test_error_m3u_without_list():
+    with pytest.raises(SystemExit):
+        handle.get_arguments(raw_args=("-s cool song", "--write-m3u"), to_group=True)
+
+
+def test_m3u_with_list():
+    handle.get_arguments(raw_args=("-l cool_list.txt", "--write-m3u"), to_group=True)
+
 
 def test_log_str_to_int():
     expect_levels = [20, 30, 40, 10]
-    levels = [handle.log_leveller(level)
-              for level in handle._LOG_LEVELS_STR]
+    levels = [handle.log_leveller(level) for level in handle._LOG_LEVELS_STR]
     assert levels == expect_levels
 
 
+@pytest.fixture(scope="module")
+def config_path_fixture(tmpdir_factory):
+    config_path = os.path.join(str(tmpdir_factory.mktemp("config")), "config.yml")
+    return config_path
+
+
+@pytest.fixture(scope="module")
+def modified_config_fixture():
+    modified_config = dict(handle.default_conf)
+    return modified_config
+
+
 class TestConfig:
-    def test_default_config(self, tmpdir):
-        expect_config = handle.default_conf['spotify-downloader']
-        global config_path
-        config_path = os.path.join(str(tmpdir), 'config.yml')
-        config = handle.get_config(config_path)
+    def test_default_config(self, config_path_fixture):
+        expect_config = handle.default_conf["spotify-downloader"]
+        config = handle.get_config(config_path_fixture)
         assert config == expect_config
 
-    def test_modified_config(self):
-        global modified_config
-        modified_config = dict(handle.default_conf)
-        modified_config['spotify-downloader']['file-format'] = 'just_a_test'
-        merged_config = handle.merge(handle.default_conf, modified_config)
-        assert merged_config == modified_config
+    def test_modified_config(self, modified_config_fixture):
+        modified_config_fixture["spotify-downloader"]["file-format"] = "just_a_test"
+        merged_config = handle.merge(handle.default_conf, modified_config_fixture)
+        assert merged_config == modified_config_fixture
 
-    def test_custom_config_path(self, tmpdir):
+    def test_custom_config_path(self, config_path_fixture, modified_config_fixture):
         parser = argparse.ArgumentParser()
-        with open(config_path, 'w') as config_file:
-            yaml.dump(modified_config, config_file, default_flow_style=False)
-        overridden_config = handle.override_config(config_path,
-                                                   parser,
-                                                   raw_args='')
-        modified_values = [ str(value) for value in modified_config['spotify-downloader'].values() ]
+        with open(config_path_fixture, "w") as config_file:
+            yaml.dump(modified_config_fixture, config_file, default_flow_style=False)
+        overridden_config = handle.override_config(
+            config_path_fixture, parser, raw_args=""
+        )
+        modified_values = [
+            str(value)
+            for value in modified_config_fixture["spotify-downloader"].values()
+        ]
         overridden_config.folder = os.path.realpath(overridden_config.folder)
-        overridden_values = [ str(value) for value in overridden_config.__dict__.values() ]
+        overridden_values = [
+            str(value) for value in overridden_config.__dict__.values()
+        ]
         assert sorted(overridden_values) == sorted(modified_values)
 
 

test/test_internals.py

@@ -1,82 +1,180 @@
-from spotdl import internals
-
 import sys
 import os
 import subprocess
+
+from spotdl import internals
+
 import pytest
 
 
+DUPLICATE_TRACKS_TEST_TABLE = [
+    (
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+        ),
+        ("https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",),
+    ),
+    (
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "",
+            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
+        ),
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
+        ),
+    ),
+    (
+        (
+            "ncs fade",
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "",
+            "ncs fade",
+        ),
+        ("ncs fade", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
+    ),
+    (
+        ("ncs spectre ", "  https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ", ""),
+        ("ncs spectre", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
+    ),
+]
+
+STRING_IDS_TEST_TABLE = [
+    (
+        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8?si=_cVm-FBRQmi7VWML7E49Ig",
+        "1feoGrmmD8QmNqtK2Gdwy8",
+    ),
+    (
+        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8",
+        "1feoGrmmD8QmNqtK2Gdwy8",
+    ),
+    ("spotify:artist:1feoGrmmD8QmNqtK2Gdwy8", "1feoGrmmD8QmNqtK2Gdwy8"),
+    (
+        "https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp?si=LkVQLJGGT--Lh8BWM4MGvg",
+        "1d1l3UkeAjtM7kVTDyR8yp",
+    ),
+    ("https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
+    ("spotify:album:1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
+    (
+        "https://open.spotify.com/user/5kkyy50uu8btnagp30pobxz2f/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "spotify:user:5kkyy50uu8btnagp30pobxz2f:playlist:3SFKRjUXm0IMQJMkEgPHeY",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/user/uqlakumu7wslkoen46s5bulq0",
+        "uqlakumu7wslkoen46s5bulq0",
+    ),
+]
+
+
+FROM_SECONDS_TEST_TABLE = [
+    (35, "35"),
+    (23, "23"),
+    (158, "2:38"),
+    (263, "4:23"),
+    (4562, "1:16:02"),
+    (26762, "7:26:02"),
+]
+
+
+TO_SECONDS_TEST_TABLE = [
+    ("0:23", 23),
+    ("0:45", 45),
+    ("2:19", 139),
+    ("3:33", 213),
+    ("7:38", 458),
+    ("1:30:05", 5405),
+]
+
+
 def test_default_music_directory():
-    if sys.platform.startswith('linux'):
-        output = subprocess.check_output(['xdg-user-dir', 'MUSIC'])
-        expect_directory = output.decode('utf-8').rstrip()
+    if sys.platform.startswith("linux"):
+        output = subprocess.check_output(["xdg-user-dir", "MUSIC"])
+        expect_directory = output.decode("utf-8").rstrip()
     else:
-        home = os.path.expanduser('~')
-        expect_directory = os.path.join(home, 'Music')
+        home = os.path.expanduser("~")
+        expect_directory = os.path.join(home, "Music")
 
     directory = internals.get_music_dir()
     assert directory == expect_directory
 
 
+@pytest.fixture(scope="module")
+def directory_fixture(tmpdir_factory):
+    dir_path = os.path.join(str(tmpdir_factory.mktemp("tmpdir")), "filter_this_folder")
+    return dir_path
+
+
 class TestPathFilterer:
-    def test_create_directory(self, tmpdir):
+    def test_create_directory(self, directory_fixture):
         expect_path = True
-        global folder_path
-        folder_path = os.path.join(str(tmpdir), 'filter_this_folder')
-        internals.filter_path(folder_path)
-        is_path = os.path.isdir(folder_path)
+        internals.filter_path(directory_fixture)
+        is_path = os.path.isdir(directory_fixture)
         assert is_path == expect_path
 
-    def test_remove_temp_files(self, tmpdir):
+    def test_remove_temp_files(self, directory_fixture):
         expect_file = False
-        file_path = os.path.join(folder_path, 'pesky_file.temp')
-        open(file_path, 'a')
-        internals.filter_path(folder_path)
+        file_path = os.path.join(directory_fixture, "pesky_file.temp")
+        open(file_path, "a")
+        internals.filter_path(directory_fixture)
         is_file = os.path.isfile(file_path)
         assert is_file == expect_file
 
 
-class TestVideoTimeFromSeconds:
-    def test_from_seconds(self):
-        expect_duration = '35'
-        duration = internals.videotime_from_seconds(35)
-        assert duration == expect_duration
-
-    def test_from_minutes(self):
-        expect_duration = '2:38'
-        duration = internals.videotime_from_seconds(158)
-        assert duration == expect_duration
-
-    def test_from_hours(self):
-        expect_duration = '1:16:02'
-        duration = internals.videotime_from_seconds(4562)
-        assert duration == expect_duration
+@pytest.mark.parametrize("sec_duration, str_duration", FROM_SECONDS_TEST_TABLE)
+def test_video_time_from_seconds(sec_duration, str_duration):
+    duration = internals.videotime_from_seconds(sec_duration)
+    assert duration == str_duration
 
 
-class TestGetSeconds:
-    def test_from_seconds(self):
-        expect_secs = 45
-        secs = internals.get_sec('0:45')
-        assert secs == expect_secs
-        secs = internals.get_sec('0.45')
-        assert secs == expect_secs
+@pytest.mark.parametrize("str_duration, sec_duration", TO_SECONDS_TEST_TABLE)
+def test_get_seconds_from_video_time(str_duration, sec_duration):
+    secs = internals.get_sec(str_duration)
+    assert secs == sec_duration
 
-    def test_from_minutes(self):
-        expect_secs = 213
-        secs = internals.get_sec('3.33')
-        assert secs == expect_secs
-        secs = internals.get_sec('3:33')
-        assert secs == expect_secs
 
-    def test_from_hours(self):
-        expect_secs = 5405
-        secs = internals.get_sec('1.30.05')
-        assert secs == expect_secs
-        secs = internals.get_sec('1:30:05')
-        assert secs == expect_secs
+@pytest.mark.parametrize("duplicates, expected", DUPLICATE_TRACKS_TEST_TABLE)
+def test_get_unique_tracks(tmpdir, duplicates, expected):
+    file_path = os.path.join(str(tmpdir), "test_duplicates.txt")
+    with open(file_path, "w") as f:
+        f.write("\n".join(duplicates))
 
-    def test_raise_error(self):
-        with pytest.raises(ValueError):
-            internals.get_sec('10*05')
-        with pytest.raises(ValueError):
-            internals.get_sec('02,28,46')
+    unique_tracks = internals.get_unique_tracks(file_path)
+    assert tuple(unique_tracks) == expected
+
+
+@pytest.mark.parametrize("input_str, expected_spotify_id", STRING_IDS_TEST_TABLE)
+def test_extract_spotify_id(input_str, expected_spotify_id):
+    spotify_id = internals.extract_spotify_id(input_str)
+    assert spotify_id == expected_spotify_id
+
+
+def test_trim(tmpdir):
+    text_file = os.path.join(str(tmpdir), "test_trim.txt")
+    with open(text_file, "w") as track_file:
+        track_file.write("ncs - spectre\nncs - heroes\nncs - hope")
+
+    with open(text_file, "r") as track_file:
+        tracks = track_file.readlines()
+
+    expect_number = len(tracks) - 1
+    expect_track = tracks[0]
+    track = internals.trim_song(text_file)
+
+    with open(text_file, "r") as track_file:
+        number = len(track_file.readlines())
+
+    assert expect_number == number and expect_track == track

test/test_list.py

@@ -1,50 +0,0 @@
-from spotdl import spotify_tools
-from spotdl import const
-
-from spotdl import spotdl
-
-import builtins
-import os
-
-
-def test_user_playlists(tmpdir, monkeypatch):
-    expect_tracks = 14
-    text_file = os.path.join(str(tmpdir), 'test_us.txt')
-    monkeypatch.setattr('builtins.input', lambda x: 1)
-    spotify_tools.write_user_playlist('alex', text_file)
-    with open(text_file, 'r') as tin:
-        tracks = len(tin.readlines())
-    assert tracks == expect_tracks
-
-
-def test_playlist(tmpdir):
-    expect_tracks = 14
-    text_file = os.path.join(str(tmpdir), 'test_pl.txt')
-    spotify_tools.write_playlist('https://open.spotify.com/user/alex/playlist/0iWOVoumWlkXIrrBTSJmN8', text_file)
-    with open(text_file, 'r') as tin:
-        tracks = len(tin.readlines())
-    assert tracks == expect_tracks
-
-
-def test_album(tmpdir):
-    expect_tracks = 15
-    global text_file
-    text_file = os.path.join(str(tmpdir), 'test_al.txt')
-    spotify_tools.write_album('https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg', text_file)
-    with open(text_file, 'r') as tin:
-        tracks = len(tin.readlines())
-    assert tracks == expect_tracks
-
-
-def test_trim():
-    with open(text_file, 'r') as track_file:
-        tracks = track_file.readlines()
-
-    expect_number = len(tracks) - 1
-    expect_track = tracks[0]
-    track = spotdl.internals.trim_song(text_file)
-
-    with open(text_file, 'r') as track_file:
-        number = len(track_file.readlines())
-
-    assert (expect_number == number and expect_track == track)

test/test_patcher.py

@@ -0,0 +1,36 @@
+from spotdl import patcher
+import pafy
+
+import pytest
+
+pafy_patcher = patcher.PatchPafy()
+pafy_patcher.patch_getbestthumb()
+
+
+class TestPafyContentAvailable:
+    pass
+
+
+class TestMethodAssignment:
+    def test_pafy_getbestthumb(self):
+        pafy.backend_shared.BasePafy.getbestthumb == patcher._getbestthumb
+
+
+class TestMethodCalls:
+    @pytest.fixture(scope="module")
+    def content_fixture(self):
+        content = pafy.new("http://youtube.com/watch?v=3nQNiWdeH2Q")
+        return content
+
+    def test_pafy_getbestthumb(self, content_fixture):
+        thumbnail = patcher._getbestthumb(content_fixture)
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/hqdefault.jpg"
+
+    def test_pafy_getbestthumb_without_ytdl(self, content_fixture):
+        content_fixture._ydl_info["thumbnails"][0]["url"] = None
+        thumbnail = patcher._getbestthumb(content_fixture)
+        assert thumbnail == "https://i.ytimg.com/vi/3nQNiWdeH2Q/sddefault.jpg"
+
+    def test_pafy_content_available(self):
+        TestPafyContentAvailable._content_available = patcher._content_available
+        assert TestPafyContentAvailable()._content_available("https://youtube.com/")

test/test_spotify_tools.py

@@ -0,0 +1,178 @@
+from spotdl import spotify_tools
+from spotdl import const
+
+import spotipy
+
+import os
+import pytest
+import loader
+
+loader.load_defaults()
+
+
+def test_generate_token():
+    token = spotify_tools.generate_token()
+    assert len(token) == 83
+
+
+class TestMustBeAuthorizedDecorator:
+    def test_spotify_instance_is_unset(self):
+        spotify_tools.spotify = None
+
+        @spotify_tools.must_be_authorized
+        def sample_func():
+            return True
+
+        assert sample_func()
+
+    def test_spotify_instance_forces_assertion_error(self):
+        @spotify_tools.must_be_authorized
+        def sample_func():
+            raise AssertionError
+
+        with pytest.raises(AssertionError):
+            sample_func()
+
+    def test_fake_token_generator(self, monkeypatch):
+        spotify_tools.spotify = None
+        monkeypatch.setattr(spotify_tools, "generate_token", lambda: 123123)
+
+        with pytest.raises(spotipy.client.SpotifyException):
+            spotify_tools.generate_metadata("ncs - spectre")
+
+    def test_correct_token(self):
+        assert spotify_tools.generate_metadata("ncs - spectre")
+
+
+class TestGenerateMetadata:
+    @pytest.fixture(scope="module")
+    def metadata_fixture(self):
+        metadata = spotify_tools.generate_metadata("ncs - spectre")
+        return metadata
+
+    def test_len(self, metadata_fixture):
+        assert len(metadata_fixture) == 24
+
+    def test_trackname(self, metadata_fixture):
+        assert metadata_fixture["name"] == "Spectre"
+
+    def test_artist(self, metadata_fixture):
+        assert metadata_fixture["artists"][0]["name"] == "Alan Walker"
+
+    def test_duration(self, metadata_fixture):
+        assert metadata_fixture["duration"] == 230.634
+
+
+def test_get_playlists():
+    expect_playlist_ids = [
+        "34gWCK8gVeYDPKcctB6BQJ",
+        "04wTU2c2WNQG9XE5oSLYfj",
+        "0fWBMhGh38y0wsYWwmM9Kt",
+    ]
+
+    expect_playlists = [
+        "https://open.spotify.com/playlist/" + playlist_id
+        for playlist_id in expect_playlist_ids
+    ]
+
+    playlists = spotify_tools.get_playlists("uqlakumu7wslkoen46s5bulq0")
+    assert playlists == expect_playlists
+
+
+def test_write_user_playlist(tmpdir, monkeypatch):
+    expect_tracks = 17
+    text_file = os.path.join(str(tmpdir), "test_us.txt")
+    monkeypatch.setattr("builtins.input", lambda x: 1)
+    spotify_tools.write_user_playlist("uqlakumu7wslkoen46s5bulq0", text_file)
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+class TestFetchPlaylist:
+    @pytest.fixture(scope="module")
+    def playlist_fixture(self):
+        playlist = spotify_tools.fetch_playlist(
+            "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt"
+        )
+        return playlist
+
+    def test_name(self, playlist_fixture):
+        assert playlist_fixture["name"] == "special_test_playlist"
+
+    def test_tracks(self, playlist_fixture):
+        assert playlist_fixture["tracks"]["total"] == 14
+
+
+def test_write_playlist(tmpdir):
+    expect_tracks = 14
+    text_file = os.path.join(str(tmpdir), "test_pl.txt")
+    spotify_tools.write_playlist(
+        "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+# XXX: Monkeypatch these tests if they fail in future
+class TestFetchAlbum:
+    @pytest.fixture(scope="module")
+    def album_fixture(self):
+        album = spotify_tools.fetch_album(
+            "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg"
+        )
+        return album
+
+    def test_name(self, album_fixture):
+        assert album_fixture["name"] == "NCS: Infinity"
+
+    def test_tracks(self, album_fixture):
+        assert album_fixture["tracks"]["total"] == 15
+
+
+# XXX: Monkeypatch these tests if they fail in future
+class TestFetchAlbumsFromArtist:
+    @pytest.fixture(scope="module")
+    def albums_from_artist_fixture(self):
+        albums = spotify_tools.fetch_albums_from_artist(
+            "https://open.spotify.com/artist/7oPftvlwr6VrsViSDV7fJY"
+        )
+        return albums
+
+    def test_len(self, albums_from_artist_fixture):
+        assert len(albums_from_artist_fixture) == 54
+
+    def test_zeroth_album_name(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["name"] == "Revolution Radio"
+
+    def test_zeroth_album_tracks(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["total_tracks"] == 12
+
+    def test_fist_album_name(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[1]["name"] == "Demolicious"
+
+    def test_first_album_tracks(self, albums_from_artist_fixture):
+        assert albums_from_artist_fixture[0]["total_tracks"] == 12
+
+
+def test_write_all_albums_from_artist(tmpdir):
+    expect_tracks = 282
+    text_file = os.path.join(str(tmpdir), "test_ab.txt")
+    spotify_tools.write_all_albums_from_artist(
+        "https://open.spotify.com/artist/4dpARuHxo51G3z768sgnrY", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+def test_write_album(tmpdir):
+    expect_tracks = 15
+    text_file = os.path.join(str(tmpdir), "test_al.txt")
+    spotify_tools.write_album(
+        "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg", text_file
+    )
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks

test/test_with_metadata.py

@@ -1,154 +0,0 @@
-
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-from spotdl import convert
-from spotdl import metadata
-
-from spotdl import spotdl
-
-import loader
-import os
-
-loader.load_defaults()
-raw_song = 'http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU'
-
-
-def test_metadata():
-    expect_number = 23
-    global meta_tags
-    meta_tags = spotify_tools.generate_metadata(raw_song)
-    assert len(meta_tags) == expect_number
-
-
-class TestFileFormat:
-    def test_with_spaces(self):
-        expect_title = 'David André Østby - Intro'
-        title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == expect_title
-
-    def test_without_spaces(self):
-        expect_title = 'David_André_Østby_-_Intro'
-        const.args.no_spaces = True
-        title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == expect_title
-
-
-def test_youtube_url():
-    expect_url = 'http://youtube.com/watch?v=rg1wfcty0BA'
-    url = youtube_tools.generate_youtube_url(raw_song, meta_tags)
-    assert url == expect_url
-
-
-def test_youtube_title():
-    expect_title = 'Intro - David André Østby'
-    global content
-    content = youtube_tools.go_pafy(raw_song, meta_tags)
-    title = youtube_tools.get_youtube_title(content)
-    assert title == expect_title
-
-
-def test_check_track_exists_before_download(tmpdir):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    songname = internals.format_string(const.args.file_format, meta_tags)
-    global file_name
-    file_name = internals.sanitize_title(songname)
-    check = spotdl.check_exists(file_name, raw_song, meta_tags)
-    assert check == expect_check
-
-
-class TestDownload:
-    def test_m4a(self):
-        expect_download = True
-        download = youtube_tools.download_song(file_name + '.m4a', content)
-        assert download == expect_download
-
-    def test_webm(self):
-        expect_download = True
-        download = youtube_tools.download_song(file_name + '.webm', content)
-        assert download == expect_download
-
-
-class TestFFmpeg():
-    def test_convert_from_webm_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.webm',
-                                   file_name + '.mp3',
-                                   const.args.folder)
-        assert return_code == expect_return_code
-
-    def test_convert_from_webm_to_m4a(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.webm',
-                                   file_name + '.m4a',
-                                   const.args.folder)
-        assert return_code == expect_return_code
-
-
-    def test_convert_from_m4a_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
-                                   file_name + '.mp3',
-                                   const.args.folder)
-        assert return_code == expect_return_code
-
-    def test_convert_from_m4a_to_webm(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
-                                   file_name + '.webm',
-                                   const.args.folder)
-        assert return_code == expect_return_code
-
-    def test_convert_from_m4a_to_flac(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
-                                   file_name + '.flac',
-                                   const.args.folder)
-        assert return_code == expect_return_code
-
-
-class TestAvconv:
-    def test_convert_from_m4a_to_mp3(self):
-        expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
-                                   file_name + '.mp3',
-                                   const.args.folder,
-                                   avconv=True)
-        assert return_code == expect_return_code
-
-
-class TestEmbedMetadata:
-    def test_embed_in_mp3(self):
-        expect_embed = True
-        global track_path
-        track_path = os.path.join(const.args.folder, file_name)
-        embed = metadata.embed(track_path + '.mp3', meta_tags)
-        assert embed == expect_embed
-
-    def test_embed_in_m4a(self):
-        expect_embed = True
-        embed = metadata.embed(track_path + '.m4a', meta_tags)
-        os.remove(track_path + '.m4a')
-        assert embed == expect_embed
-
-    def test_embed_in_webm(self):
-        expect_embed = False
-        embed = metadata.embed(track_path + '.webm', meta_tags)
-        os.remove(track_path + '.webm')
-        assert embed == expect_embed
-
-    def test_embed_in_flac(self):
-        expect_embed = True
-        embed = metadata.embed(track_path + '.flac', meta_tags)
-        os.remove(track_path + '.flac')
-        assert embed == expect_embed
-
-
-def test_check_track_exists_after_download():
-    expect_check = True
-    check = spotdl.check_exists(file_name, raw_song, meta_tags)
-    os.remove(track_path + '.mp3')
-    assert check == expect_check

test/test_without_metadata.py

@@ -1,129 +0,0 @@
-from spotdl import const
-from spotdl import internals
-from spotdl import spotify_tools
-from spotdl import youtube_tools
-
-from spotdl import spotdl
-import loader
-
-import os
-import builtins
-
-loader.load_defaults()
-raw_song = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
-
-
-class TestYouTubeAPIKeys:
-    def test_custom(self):
-        expect_key = 'some_api_key'
-        const.args.youtube_api_key = expect_key
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == expect_key
-
-    def test_default(self):
-        expect_key = 'AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0'
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        key = youtube_tools.pafy.g.api_key
-        assert key == expect_key
-
-
-def test_metadata():
-    expect_metadata = None
-    global metadata
-    metadata = spotify_tools.generate_metadata(raw_song)
-    assert metadata == expect_metadata
-
-
-class TestArgsManualResultCount:
-    # Regresson test for issue #264
-    def test_scrape(self):
-        const.args.manual = True
-        url = youtube_tools.GenerateYouTubeURL("she is still sleeping SAO",
-                                               meta_tags=None)
-        video_ids = url.scrape(bestmatch=False)
-        # Web scraping gives us all videos on the 1st page
-        assert len(video_ids) == 20
-
-    def test_api(self):
-        url = youtube_tools.GenerateYouTubeURL("she is still sleeping SAO",
-                                               meta_tags=None)
-        video_ids = url.api(bestmatch=False)
-        const.args.manual = False
-        # API gives us 50 videos (or as requested)
-        assert len(video_ids) == 50
-
-
-class TestYouTubeURL:
-    def test_only_music_category(self):
-        # YouTube keeps changing its results
-        expect_urls = ('http://youtube.com/watch?v=qOOcy2-tmbk',
-                       'http://youtube.com/watch?v=5USR1Omo7f0')
-        const.args.music_videos_only = True
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
-        assert url in expect_urls
-
-    def test_all_categories(self):
-        expect_url = 'http://youtube.com/watch?v=qOOcy2-tmbk'
-        const.args.music_videos_only = False
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
-        assert url == expect_url
-
-    def test_args_manual(self, monkeypatch):
-        expect_url = 'http://youtube.com/watch?v=qOOcy2-tmbk'
-        const.args.manual = True
-        monkeypatch.setattr('builtins.input', lambda x: '1')
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
-        assert url == expect_url
-
-    def test_args_manual_none(self, monkeypatch):
-        expect_url = None
-        monkeypatch.setattr('builtins.input', lambda x: '0')
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
-        const.args.manual = False
-        assert url == expect_url
-
-
-class TestYouTubeTitle:
-    def test_single_download_with_youtube_api(self):
-        global content
-        global title
-        expect_title = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
-        key = 'AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90'
-        const.args.youtube_api_key = key
-        youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(raw_song, metadata)
-        title = youtube_tools.get_youtube_title(content)
-        assert title == expect_title
-
-    def test_download_from_list_without_youtube_api(self):
-        expect_title = "1. Tony's Videos VERY SHORT VIDEO 28.10.2016"
-        const.args.youtube_api_key = None
-        youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(raw_song, metadata)
-        title = youtube_tools.get_youtube_title(content, 1)
-        assert title == expect_title
-
-
-def test_check_exists(tmpdir):
-    expect_check = False
-    const.args.folder = str(tmpdir)
-    # prerequisites for determining filename
-    global file_name
-    file_name = internals.sanitize_title(title)
-    check = spotdl.check_exists(file_name, raw_song, metadata)
-    assert check == expect_check
-
-
-class TestDownload:
-    def test_webm(self):
-        # content does not have any .webm audiostream
-        expect_download = False
-        download = youtube_tools.download_song(file_name + '.webm', content)
-        assert download == expect_download
-
-    def test_other(self):
-        expect_download = False
-        download = youtube_tools.download_song(file_name + '.fake_extension', content)
-        assert download == expect_download

test/test_youtube_tools.py

@@ -0,0 +1,243 @@
+import os
+import builtins
+
+from spotdl import const
+from spotdl import internals
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+from spotdl import downloader
+
+import loader
+import pytest
+
+loader.load_defaults()
+
+YT_API_KEY = "AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90"
+
+TRACK_SEARCH = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
+EXPECTED_TITLE = TRACK_SEARCH
+EXPECTED_YT_URL = "http://youtube.com/watch?v=qOOcy2-tmbk"
+
+RESULT_COUNT_SEARCH = "she is still sleeping SAO"
+
+EXPECTED_YT_API_KEY = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
+EXPECTED_YT_API_KEY_CUSTOM = "some_api_key"
+
+
+class TestYouTubeAPIKeys:
+    def test_custom(self):
+        const.args.youtube_api_key = EXPECTED_YT_API_KEY_CUSTOM
+        youtube_tools.set_api_key()
+        key = youtube_tools.pafy.g.api_key
+        assert key == EXPECTED_YT_API_KEY_CUSTOM
+
+    def test_default(self):
+        const.args.youtube_api_key = None
+        youtube_tools.set_api_key()
+        key = youtube_tools.pafy.g.api_key
+        assert key == EXPECTED_YT_API_KEY
+
+
+@pytest.fixture(scope="module")
+def metadata_fixture():
+    metadata = spotify_tools.generate_metadata(TRACK_SEARCH)
+    return metadata
+
+
+def test_metadata(metadata_fixture):
+    expect_metadata = None
+    assert metadata_fixture == expect_metadata
+
+
+class TestArgsManualResultCount:
+    # Regresson test for issue #264
+    def test_scrape(self):
+        const.args.manual = True
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
+        video_ids = url.scrape(bestmatch=False)
+        # Web scraping gives us all videos on the 1st page
+        assert len(video_ids) == 20
+
+    def test_api(self):
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
+        video_ids = url.api(bestmatch=False)
+        const.args.manual = False
+        # API gives us 50 videos (or as requested)
+        assert len(video_ids) == 50
+
+
+class TestYouTubeURL:
+    def test_only_music_category(self, metadata_fixture):
+        const.args.music_videos_only = True
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_all_categories(self, metadata_fixture):
+        const.args.music_videos_only = False
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_args_manual(self, metadata_fixture, monkeypatch):
+        const.args.manual = True
+        monkeypatch.setattr("builtins.input", lambda x: "1")
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        assert url == EXPECTED_YT_URL
+
+    def test_args_manual_none(self, metadata_fixture, monkeypatch):
+        expect_url = None
+        monkeypatch.setattr("builtins.input", lambda x: "0")
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata_fixture)
+        const.args.manual = False
+        assert url == expect_url
+
+
+@pytest.fixture(scope="module")
+def content_fixture(metadata_fixture):
+    content = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
+    return content
+
+
+# True = Metadata must be fetched from Spotify
+# False = Metadata must be fetched from YouTube
+# None = Metadata must be `None`
+
+MATCH_METADATA_NO_FALLBACK_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
+    ("Linux Talk | Working with Drives and Filesystems", None),
+]
+
+MATCH_METADATA_FALLBACK_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", True),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", False),
+    ("Linux Talk | Working with Drives and Filesystems", False),
+]
+
+MATCH_METADATA_NO_METADATA_TEST_TABLE = [
+    ("https://open.spotify.com/track/5nWduGwBGBn1PSqYTJUDbS", None),
+    ("http://youtube.com/watch?v=3nQNiWdeH2Q", None),
+    ("Linux Talk | Working with Drives and Filesystems", None),
+]
+
+
+class TestMetadataOrigin:
+    def match_metadata(self, track, metadata_type):
+        _, metadata = youtube_tools.match_video_and_metadata(track)
+        if metadata_type is None:
+            assert metadata == metadata_type
+        else:
+            assert metadata["spotify_metadata"] == metadata_type
+
+    @pytest.mark.parametrize(
+        "track, metadata_type", MATCH_METADATA_NO_FALLBACK_TEST_TABLE
+    )
+    def test_match_metadata_with_no_fallback(
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
+        const.args.no_fallback_metadata = True
+        self.match_metadata(track, metadata_type)
+
+    @pytest.mark.parametrize("track, metadata_type", MATCH_METADATA_FALLBACK_TEST_TABLE)
+    def test_match_metadata_with_fallback(
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
+        const.args.no_fallback_metadata = False
+        self.match_metadata(track, metadata_type)
+
+    @pytest.mark.parametrize(
+        "track, metadata_type", MATCH_METADATA_NO_METADATA_TEST_TABLE
+    )
+    def test_match_metadata_with_no_metadata(
+        self, track, metadata_type, content_fixture, monkeypatch
+    ):
+        monkeypatch.setattr(
+            youtube_tools, "go_pafy", lambda track, meta_tags: content_fixture
+        )
+        const.args.no_metadata = True
+        self.match_metadata(track, metadata_type)
+
+
+@pytest.fixture(scope="module")
+def title_fixture(content_fixture):
+    title = youtube_tools.get_youtube_title(content_fixture)
+    return title
+
+
+class TestYouTubeTitle:
+    def test_single_download_with_youtube_api(self, title_fixture):
+        const.args.youtube_api_key = YT_API_KEY
+        youtube_tools.set_api_key()
+        assert title_fixture == EXPECTED_TITLE
+
+    def test_download_from_list_without_youtube_api(
+        self, metadata_fixture, content_fixture
+    ):
+        const.args.youtube_api_key = None
+        youtube_tools.set_api_key()
+        content_fixture = youtube_tools.go_pafy(TRACK_SEARCH, metadata_fixture)
+        title = youtube_tools.get_youtube_title(content_fixture, 1)
+        assert title == "1. {0}".format(EXPECTED_TITLE)
+
+
+@pytest.fixture(scope="module")
+def filename_fixture(title_fixture):
+    filename = internals.sanitize_title(title_fixture)
+    return filename
+
+
+def test_check_exists(metadata_fixture, filename_fixture, tmpdir):
+    expect_check = False
+    const.args.folder = str(tmpdir)
+    # prerequisites for determining filename
+    track_existence = downloader.CheckExists(filename_fixture, metadata_fixture)
+    check = track_existence.already_exists(TRACK_SEARCH)
+    assert check == expect_check
+
+
+def test_generate_m3u(tmpdir, monkeypatch):
+    monkeypatch.setattr(
+        youtube_tools.GenerateYouTubeURL,
+        "_fetch_response",
+        loader.monkeypatch_youtube_search_page,
+    )
+    expect_m3u = (
+        "#EXTM3U\n\n"
+        "#EXTINF:208,Janji - Heroes Tonight (feat. Johnning) [NCS Release]\n"
+        "http://www.youtube.com/watch?v=3nQNiWdeH2Q\n"
+        "#EXTINF:226,Alan Walker - Spectre [NCS Release]\n"
+        "http://www.youtube.com/watch?v=AOeY-nDp7hI\n"
+    )
+    m3u_track_file = os.path.join(str(tmpdir), "m3u_test.txt")
+    with open(m3u_track_file, "w") as track_file:
+        track_file.write("\nhttps://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD")
+        track_file.write("\nhttp://www.youtube.com/watch?v=AOeY-nDp7hI")
+    youtube_tools.generate_m3u(m3u_track_file)
+    m3u_file = "{}.m3u".format(m3u_track_file.split(".")[0])
+    with open(m3u_file, "r") as m3u_in:
+        m3u = m3u_in.readlines()
+    assert "".join(m3u) == expect_m3u
+
+
+class TestDownload:
+    def test_webm(self, content_fixture, filename_fixture, monkeypatch):
+        # content_fixture does not have any .webm audiostream
+        expect_download = False
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
+        download = youtube_tools.download_song(
+            filename_fixture + ".webm", content_fixture
+        )
+        assert download == expect_download
+
+    def test_other(self, content_fixture, filename_fixture, monkeypatch):
+        expect_download = False
+        monkeypatch.setattr("pafy.backend_shared.BaseStream.download", lambda x: None)
+        download = youtube_tools.download_song(
+            filename_fixture + ".fake_extension", content_fixture
+        )
+        assert download == expect_download