Bump to v1.1.0 (#427)

Rere-fix tests

.gitignore

@@ -1,6 +1,7 @@
 config.yml
 Music/
 *.txt
+*.m3u
 upload.sh
 .pytest_cache/
 

CHANGES.md

@@ -6,8 +6,46 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+*Please keep this section up to date when creating a PR!*
+
+## [1.1.0] - 2018-11-13
+### Added
+- Output error details when track download fails from list file ([@ManveerBasra](https://github.com/ManveerBasra)) (#406)
+- Add support for `.m3u` playlists ([@ritiek](https://github.com/ritiek)) (#401)
+- Introduce usage of black (code formatter) ([@linusg](https://github.com/linusg)) (#393)
+- Added command line option for getting all artist's songs ([@AlfredoSequeida](https://github.com/AlfredoSequeida)) (#389)
+- Added command line options for skipping tracks file and successful downloads file and
+  place newline before track URL when appending to track file ([@linusg](https://github.com/linusg)) (#386)
+- Overwrite track file with unique tracks ([@ritiek](https://github.com/ritiek)) (#380)
+- Embed comment metadata in `.m4a` ([@ritiek](https://github.com/ritiek)) (#379)
+- Added check for publisher tag before adding publisher id3 tag to audio file ([@gnodar01](https://github.com/gnodar01)) (#377)
+
+### Changed
+- `--list` flag accepts only text files using mimetypes ([@ManveerBasra](https://github.com/ManveerBasra)) (#414)
+- Refactored Spotify token refresh ([@ManveerBasra](https://github.com/ManveerBasra)) (#408)
+- Don't search song on Spotify if `--no-metadata` is passed ([@ManveerBasra](https://github.com/ManveerBasra)) (#404)
+- Changed test track to one whose lyrics are found ([@ManveerBasra](https://github.com/ManveerBasra)) (#400)
+- Windows - 'My Music' folder won't be assumed to be on C drive but looked up in Registry ([@SillySam](https://github.com/SillySam)) (#387)
+- Updated `setup.py` (fix PyPI URL, add Python 3.7 modifier) ([@linusg](https://github.com/linusg)) (#383)
+- Updated dependencies to their newest versions (as of 2018-10-02) ([@linusg](https://github.com/linusg)) (#382)
+- Remove duplicates from track file while preserving order ([@ritiek](https://github.com/ritiek)) (#369)
+- Moved a lot of content from `README.md` to the [repository's GitHub wiki](https://github.com/ritiek/spotify-downloader/wiki) ([@sdhutchins](https://github.com/sdhutchins), [@ritiek](https://github.com/ritiek)) (#361)
+- Refactored internal use of logging ([@arryon](https://github.com/arryon)) (#358)
+
+### Fixed
+- Check and replace slashes with dashes to avoid directory creation error ([@ManveerBasra](https://github.com/ManveerBasra)) (#402)
+- Filter unwanted text from Spotify URLs when extracting information ([@ritiek](https://github.com/ritiek)) (#394)
+- Correctly embed metadata in `.m4a` ([@arryon](https://github.com/arryon)) (#372)
+- Slugify will not ignore the `'` character (single quotation mark) anymore ([@jimangel2001](https://github.com/jimangel2001)) (#357)
+
+## [1.0.0] - 2018-09-09
+### Added
+- Initial complete release, recommended way to install is now from PyPI
+
 ## 1.0.0-beta.1 - 2018-02-02
 ### Added
 - Initial release, prepare for 1.0.0
 
-[Unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v1.0.0-beta.1...HEAD
+[Unreleased]: https://github.com/ritiek/spotify-downloader/compare/v1.1.0...HEAD
+[1.1.0]: https://github.com/ritiek/spotify-downloader/compare/v1.0.0...v1.1.0
+[1.0.0]: https://github.com/ritiek/spotify-downloader/compare/v1.0.0-beta.1...v1.0.0

CONTRIBUTING.md

@@ -22,9 +22,11 @@ don't feel bad. Open an issue any way!
 [good-first-issue](https://github.com/ritiek/spotify-downloader/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
 - When making a PR, point it to the [master branch](https://github.com/ritiek/spotify-downloader/tree/master)
 unless mentioned otherwise.
+- Code should be formatted using [black](https://github.com/ambv/black). Don't worry if you forgot or don't know how to do this, the codebase will be black-formatted with each release.
 - All tests are placed in the [test directory](https://github.com/ritiek/spotify-downloader/tree/master/test). We use [pytest](https://github.com/pytest-dev/pytest)
 to run the test suite: `$ python3 -m pytest test`.
 If you don't have pytest, you can install it with `$ pip3 install pytest`.
+- Add a note about the changes, your GitHub username and a reference to the PR to the `Unreleased` section of the [`CHANGES.md`](CHANGES.md) file (see existing releases for examples), add the appropriate section ("Added", "Changed", "Fixed" etc.) if necessary. You don't have to increment version numbers. See https://keepachangelog.com/en/1.0.0/ for more information.
 - If you are planning to work on something big, let us know through an issue. So we can discuss more about it.
 - Lastly, please don't hesitate to ask if you have any questions!
 Let us know (through an issue) if you are facing any trouble making a PR, we'd be glad to help you out!

README.md

@@ -4,33 +4,24 @@
 [![Build Status](https://travis-ci.org/ritiek/spotify-downloader.svg?branch=master)](https://travis-ci.org/ritiek/spotify-downloader)
 [![Coverage Status](https://codecov.io/gh/ritiek/spotify-downloader/branch/master/graph/badge.svg)](https://codecov.io/gh/ritiek/spotify-downloader)
 [![Docker Build Status](https://img.shields.io/docker/build/ritiek/spotify-downloader.svg)](https://hub.docker.com/r/ritiek/spotify-downloader)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
 [![Gitter Chat](https://badges.gitter.im/ritiek/spotify-downloader/Lobby.svg)](https://gitter.im/spotify-downloader/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 - Downloads songs from YouTube in an MP3 format by using Spotify's HTTP link.
 - Can also download a song by entering its artist and song name (in case if you don't have the Spotify's HTTP link for some song).
-- Automatically applies metadata to the downloaded song which include:
+- Automatically applies metadata to the downloaded song which includes:
 
-  - Title
+  - `Title`, `Artist`, `Album`, `Album art`, `Lyrics` (if found on [lyrics wikia](http://lyrics.wikia.com)), `Album artist`, `Genre`, `Track number`, `Disc number`, `Release date`, and more...
-  - Artist
-  - Album
-  - Album art
-  - Lyrics (if found on http://lyrics.wikia.com)
-  - Album artist
-  - Genre
-  - Track number
-  - Disc number
-  - Release date
-  - And more...
 
-- Works straight out of the box and does not require to generate or mess with your API keys (already included).
+- Works straight out of the box and does not require you to generate or mess with your API keys (already included).
 
-That's how your music library will look like!
+Below is how your music library will look!
 
 <img src="http://i.imgur.com/Gpch7JI.png" width="290"><img src="http://i.imgur.com/5vhk3HY.png" width="290"><img src="http://i.imgur.com/RDTCCST.png" width="290">
 
 ## Installation
 
-**This tool works only with Python 3.**
+:heavy_exclamation_mark: **This tool works only with Python 3.**
 
 Python 2 compatibility was dropped because of the way it deals with unicode (2020 is coming soon too).
 If you still need to use Python 2 - check out the (outdated)
@@ -38,8 +29,13 @@ If you still need to use Python 2 - check out the (outdated)
 
 spotify-downloader works with all major distributions and even on low-powered devices such as a Raspberry Pi.
 
-Check out the [Installation](https://github.com/ritiek/spotify-downloader/wiki/Installation) wiki page
+spotify-downloader can be installed via pip with:
-for OS-specific instructions to get spotify-downloader working on your system.
+```
+$ pip3 install spotdl
+```
+
+but be sure to check out the [Installation](https://github.com/ritiek/spotify-downloader/wiki/Installation) wiki
+page for detailed OS-specific instructions to get it and other dependencies it relies on working on your system.
 
 ## Usage
 

setup.py

@@ -1,59 +1,64 @@
 from setuptools import setup
 
-with open('README.md', 'r') as f:
+with open("README.md", "r") as f:
     long_description = f.read()
 
 import spotdl
 
 setup(
     # 'spotify-downloader' was already taken :/
-    name='spotdl',
+    name="spotdl",
-    py_modules=['spotdl'],
     # Tests are included automatically:
     # https://docs.python.org/3.6/distutils/sourcedist.html#specifying-the-files-to-distribute
-    packages=['spotdl'],
+    packages=["spotdl"],
     version=spotdl.__version__,
     install_requires=[
-        'pathlib >= 1.0.1',
+        "pathlib >= 1.0.1",
-        'youtube_dl >= 2017.9.8',
+        "youtube_dl >= 2017.9.26",
-        'pafy >= 0.5.3.1',
+        "pafy >= 0.5.3.1",
-        'spotipy >= 2.4.4',
+        "spotipy >= 2.4.4",
-        'mutagen >= 1.37',
+        "mutagen >= 1.41.1",
-        'beautifulsoup4 >= 4.6.0',
+        "beautifulsoup4 >= 4.6.3",
-        'unicode-slugify >= 0.1.3',
+        "unicode-slugify >= 0.1.3",
-        'titlecase >= 0.10.0',
+        "titlecase >= 0.10.0",
-        'logzero >= 1.3.1',
+        "logzero >= 1.3.1",
-        'lyricwikia >= 0.1.8',
+        "lyricwikia >= 0.1.8",
-        'PyYAML >= 3.12',
+        "PyYAML >= 3.13",
-        'appdirs >= 1.4.3'
+        "appdirs >= 1.4.3",
     ],
-    description='Download songs from YouTube using Spotify song URLs or playlists with albumart and meta-tags.',
+    description="Download songs from YouTube using Spotify song URLs or playlists with albumart and meta-tags.",
     long_description=long_description,
-    long_description_content_type='text/markdown',
+    long_description_content_type="text/markdown",
-    author='Ritiek Malhotra and the spotify-downloader contributors',
+    author="Ritiek Malhotra and the spotify-downloader contributors",
-    author_email='ritiekmalhotra123@gmail.com',
+    author_email="ritiekmalhotra123@gmail.com",
-    license='MIT',
+    license="MIT",
-    python_requires='>=3.4',
+    python_requires=">=3.4",
-    url='https://github.com/ritiek/spotify-downloader',
+    url="https://github.com/ritiek/spotify-downloader",
-    download_url='https://pypi.org/project/spotify-downloader/',
+    download_url="https://pypi.org/project/spotdl/",
-    keywords=['spotify', 'downloader', 'download', 'music', 'youtube', 'mp3', 'album', 'metadata'],
+    keywords=[
-    classifiers=[
+        "spotify",
-        'Development Status :: 4 - Beta',
+        "downloader",
-        'Intended Audience :: End Users/Desktop',
+        "download",
-        'License :: OSI Approved :: MIT License',
+        "music",
-        'Programming Language :: Python',
+        "youtube",
-        'Programming Language :: Python :: 3',
+        "mp3",
-        'Programming Language :: Python :: 3.4',
+        "album",
-        'Programming Language :: Python :: 3.5',
+        "metadata",
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3 :: Only',
-        'Topic :: Multimedia',
-        'Topic :: Multimedia :: Sound/Audio',
-        'Topic :: Utilities'
     ],
-    entry_points={
+    classifiers=[
-        'console_scripts': [
+        "Development Status :: 4 - Beta",
-            'spotdl = spotdl.spotdl:main',
+        "Intended Audience :: End Users/Desktop",
-        ],
+        "License :: OSI Approved :: MIT License",
-    }
+        "Programming Language :: Python",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.4",
+        "Programming Language :: Python :: 3.5",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3 :: Only",
+        "Topic :: Multimedia",
+        "Topic :: Multimedia :: Sound/Audio",
+        "Topic :: Utilities",
+    ],
+    entry_points={"console_scripts": ["spotdl = spotdl.spotdl:main"]},
 )

spotdl/__init__.py

@@ -1 +1 @@
-__version__ = '1.0.0'
+__version__ = "1.1.0"

spotdl/const.py

@@ -1,32 +1,38 @@
 import logzero
 
-_log_format = ("%(color)s%(levelname)s:%(end_color)s %(message)s")
+_log_format = "%(color)s%(levelname)s:%(end_color)s %(message)s"
 _formatter = logzero.LogFormatter(fmt=_log_format)
 
+# Set up a temporary logger with default log level so that
+# it can be used before log level argument is determined
+logzero.setup_default_logger(formatter=_formatter)
+
 # options
-log = logzero.setup_logger(formatter=_formatter)
 args = None
 
 # Apple has specific tags - see mutagen docs -
 # http://mutagen.readthedocs.io/en/latest/api/mp4.html
-M4A_TAG_PRESET = { 'album'        : '\xa9alb',
+M4A_TAG_PRESET = {
-                   'artist'       : '\xa9ART',
+    "album": "\xa9alb",
-                   'date'         : '\xa9day',
+    "artist": "\xa9ART",
-                   'title'        : '\xa9nam',
+    "date": "\xa9day",
-                   'year'         : '\xa9day',
+    "title": "\xa9nam",
-                   'originaldate' : 'purd',
+    "year": "\xa9day",
-                   'comment'      : '\xa9cmt',
+    "originaldate": "purd",
-                   'group'        : '\xa9grp',
+    "comment": "\xa9cmt",
-                   'writer'       : '\xa9wrt',
+    "group": "\xa9grp",
-                   'genre'        : '\xa9gen',
+    "writer": "\xa9wrt",
-                   'tracknumber'  : 'trkn',
+    "genre": "\xa9gen",
-                   'albumartist'  : 'aART',
+    "tracknumber": "trkn",
-                   'discnumber'   : 'disk',
+    "albumartist": "aART",
-                   'cpil'         : 'cpil',
+    "discnumber": "disk",
-                   'albumart'     : 'covr',
+    "cpil": "cpil",
-                   'copyright'    : 'cprt',
+    "albumart": "covr",
-                   'tempo'        : 'tmpo',
+    "copyright": "cprt",
-                   'lyrics'       : '\xa9lyr' }
+    "tempo": "tmpo",
+    "lyrics": "\xa9lyr",
+    "comment": "\xa9cmt",
+}
 
 TAG_PRESET = {}
 for key in M4A_TAG_PRESET.keys():

spotdl/convert.py

@@ -1,6 +1,6 @@
 import subprocess
 import os
-from spotdl.const import log
+from logzero import logger as log
 
 
 """What are the differences and similarities between ffmpeg, libav, and avconv?
@@ -20,8 +20,7 @@ def song(input_song, output_song, folder, avconv=False, trim_silence=False):
     if input_song == output_song:
         return 0
     convert = Converter(input_song, output_song, folder, trim_silence)
-    log.info('Converting {0} to {1}'.format(
+    log.info("Converting {0} to {1}".format(input_song, output_song.split(".")[-1]))
-        input_song, output_song.split('.')[-1]))
     if avconv:
         exit_code = convert.with_avconv()
     else:
@@ -37,54 +36,67 @@ class Converter:
 
     def with_avconv(self):
         if log.level == 10:
-            level = 'debug'
+            level = "debug"
         else:
-            level = '0'
+            level = "0"
 
-        command = ['avconv', '-loglevel', level, '-i',
+        command = [
-                   self.input_file, '-ab', '192k',
+            "avconv",
-                   self.output_file, '-y']
+            "-loglevel",
+            level,
+            "-i",
+            self.input_file,
+            "-ab",
+            "192k",
+            self.output_file,
+            "-y",
+        ]
 
         if self.trim_silence:
-            log.warning('--trim-silence not supported with avconv')
+            log.warning("--trim-silence not supported with avconv")
 
         log.debug(command)
         return subprocess.call(command)
 
     def with_ffmpeg(self):
-        ffmpeg_pre = 'ffmpeg -y '
+        ffmpeg_pre = "ffmpeg -y "
 
         if not log.level == 10:
-            ffmpeg_pre += '-hide_banner -nostats -v panic '
+            ffmpeg_pre += "-hide_banner -nostats -v panic "
 
         _, input_ext = os.path.splitext(self.input_file)
         _, output_ext = os.path.splitext(self.output_file)
 
-        ffmpeg_params = ''
+        ffmpeg_params = ""
 
-        if input_ext == '.m4a':
+        if input_ext == ".m4a":
-            if output_ext == '.mp3':
+            if output_ext == ".mp3":
-                ffmpeg_params = '-codec:v copy -codec:a libmp3lame -ar 44100 '
+                ffmpeg_params = "-codec:v copy -codec:a libmp3lame -ar 44100 "
-            elif output_ext == '.webm':
+            elif output_ext == ".webm":
-                ffmpeg_params = '-codec:a libopus -vbr on '
+                ffmpeg_params = "-codec:a libopus -vbr on "
 
-        elif input_ext == '.webm':
+        elif input_ext == ".webm":
-            if output_ext == '.mp3':
+            if output_ext == ".mp3":
-                ffmpeg_params = '-codec:a libmp3lame -ar 44100 '
+                ffmpeg_params = "-codec:a libmp3lame -ar 44100 "
-            elif output_ext == '.m4a':
+            elif output_ext == ".m4a":
-                ffmpeg_params = '-cutoff 20000 -codec:a libfdk_aac -ar 44100 '
+                ffmpeg_params = "-cutoff 20000 -codec:a libfdk_aac -ar 44100 "
 
-        if output_ext == '.flac':
+        if output_ext == ".flac":
-            ffmpeg_params = '-codec:a flac -ar 44100 '
+            ffmpeg_params = "-codec:a flac -ar 44100 "
 
         # add common params for any of the above combination
-        ffmpeg_params += '-b:a 192k -vn '
+        ffmpeg_params += "-b:a 192k -vn "
-        ffmpeg_pre += ' -i'
+        ffmpeg_pre += " -i"
 
         if self.trim_silence:
-            ffmpeg_params += '-af silenceremove=start_periods=1 '
+            ffmpeg_params += "-af silenceremove=start_periods=1 "
 
-        command = ffmpeg_pre.split() + [self.input_file] + ffmpeg_params.split() + [self.output_file]
+        command = (
+            ffmpeg_pre.split()
+            + [self.input_file]
+            + ffmpeg_params.split()
+            + [self.output_file]
+        )
 
         log.debug(command)
         return subprocess.call(command)

spotdl/handle.py

@@ -1,36 +1,40 @@
 import appdirs
 from spotdl import internals, const
-
+from logzero import logger as log
-log = const.log
 
 import logging
 import yaml
 import argparse
+import mimetypes
 
 import os
 import sys
 
 
-_LOG_LEVELS_STR = ['INFO', 'WARNING', 'ERROR', 'DEBUG']
+_LOG_LEVELS_STR = ["INFO", "WARNING", "ERROR", "DEBUG"]
 
-default_conf = { 'spotify-downloader':
+default_conf = {
-                 { 'manual'                 : False,
+    "spotify-downloader": {
-                   'no-metadata'            : False,
+        "manual": False,
-                   'avconv'                 : False,
+        "no-metadata": False,
-                   'folder'                 : internals.get_music_dir(),
+        "avconv": False,
-                   'overwrite'              : 'prompt',
+        "folder": internals.get_music_dir(),
-                   'input-ext'              : '.m4a',
+        "overwrite": "prompt",
-                   'output-ext'             : '.mp3',
+        "input-ext": ".m4a",
-                   'trim-silence'           : False,
+        "output-ext": ".mp3",
-                   'download-only-metadata' : False,
+        "trim-silence": False,
-                   'dry-run'                : False,
+        "download-only-metadata": False,
-                   'music-videos-only'      : False,
+        "dry-run": False,
-                   'no-spaces'              : False,
+        "music-videos-only": False,
-                   'file-format'            : '{artist} - {track_name}',
+        "no-spaces": False,
-                   'search-format'          : '{artist} - {track_name} lyrics',
+        "file-format": "{artist} - {track_name}",
-                   'youtube-api-key'        : None,
+        "search-format": "{artist} - {track_name} lyrics",
-                   'log-level'              : 'INFO' }
+        "youtube-api-key": None,
-               }
+        "skip": None,
+        "write-successful": None,
+        "log-level": "INFO",
+    }
+}
 
 
 def log_leveller(log_level_str):
@@ -49,142 +53,224 @@ def merge(default, config):
 
 def get_config(config_file):
     try:
-        with open(config_file, 'r') as ymlfile:
+        with open(config_file, "r") as ymlfile:
             cfg = yaml.load(ymlfile)
     except FileNotFoundError:
-        log.info('Writing default configuration to {0}:'.format(config_file))
+        log.info("Writing default configuration to {0}:".format(config_file))
-        with open(config_file, 'w') as ymlfile:
+        with open(config_file, "w") as ymlfile:
             yaml.dump(default_conf, ymlfile, default_flow_style=False)
             cfg = default_conf
 
-        for line in yaml.dump(default_conf['spotify-downloader'], default_flow_style=False).split('\n'):
+        for line in yaml.dump(
+            default_conf["spotify-downloader"], default_flow_style=False
+        ).split("\n"):
             if line.strip():
                 log.info(line.strip())
-        log.info('Please note that command line arguments have higher priority '
+        log.info(
-                 'than their equivalents in the configuration file')
+            "Please note that command line arguments have higher priority "
+            "than their equivalents in the configuration file"
+        )
 
-    return cfg['spotify-downloader']
+    return cfg["spotify-downloader"]
 
 
 def override_config(config_file, parser, raw_args=None):
     """ Override default dict with config dict passed as comamnd line argument. """
     config_file = os.path.realpath(config_file)
-    config = merge(default_conf['spotify-downloader'], get_config(config_file))
+    config = merge(default_conf["spotify-downloader"], get_config(config_file))
     parser.set_defaults(**config)
     return parser.parse_args(raw_args)
 
 
 def get_arguments(raw_args=None, to_group=True, to_merge=True):
     parser = argparse.ArgumentParser(
-        description='Download and convert tracks from Spotify, Youtube etc.',
+        description="Download and convert tracks from Spotify, Youtube etc.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
 
     if to_merge:
-        config_dir = os.path.join(appdirs.user_config_dir(), 'spotdl')
+        config_dir = os.path.join(appdirs.user_config_dir(), "spotdl")
         os.makedirs(config_dir, exist_ok=True)
-        config_file = os.path.join(config_dir, 'config.yml')
+        config_file = os.path.join(config_dir, "config.yml")
-        config = merge(default_conf['spotify-downloader'], get_config(config_file))
+        config = merge(default_conf["spotify-downloader"], get_config(config_file))
     else:
-        config = default_conf['spotify-downloader']
+        config = default_conf["spotify-downloader"]
 
     if to_group:
         group = parser.add_mutually_exclusive_group(required=True)
 
         group.add_argument(
-            '-s', '--song',
+            "-s", "--song", help="download track by spotify link or name"
-            help='download track by spotify link or name')
+        )
+        group.add_argument("-l", "--list", help="download tracks from a file")
         group.add_argument(
-            '-l', '--list',
+            "-p",
-            help='download tracks from a file')
+            "--playlist",
+            help="load tracks from playlist URL into <playlist_name>.txt",
+        )
         group.add_argument(
-            '-p', '--playlist',
+            "-b", "--album", help="load tracks from album URL into <album_name>.txt"
-            help='load tracks from playlist URL into <playlist_name>.txt')
+        )
         group.add_argument(
-            '-b', '--album',
+            "-ab",
-            help='load tracks from album URL into <album_name>.txt')
+            "--all-albums",
+            help="load all tracks from artist URL into <artist_name>.txt"
+        )
         group.add_argument(
-            '-u', '--username',
+            "-u",
-            help="load tracks from user's playlist into <playlist_name>.txt")
+            "--username",
+            help="load tracks from user's playlist into <playlist_name>.txt",
+        )
         group.add_argument(
-            '-V', '--version',
+            "-V", "--version", help="show version and exit", action="store_true"
-            help="show version and exit",
+        )
-            action='store_true')
 
     parser.add_argument(
-        '-m', '--manual', default=config['manual'],
+        '--write-m3u',
-        help='choose the track to download manually from a list '
+        help="generate an .m3u playlist file with youtube links given "
-             'of matching tracks',
+             "a text file containing tracks",
-        action='store_true')
+        action='store_true'
+    )
     parser.add_argument(
-        '-nm', '--no-metadata', default=config['no-metadata'],
+        "-m",
-        help='do not embed metadata in tracks', action='store_true')
+        "--manual",
+        default=config["manual"],
+        help="choose the track to download manually from a list " "of matching tracks",
+        action="store_true",
+    )
     parser.add_argument(
-        '-a', '--avconv', default=config['avconv'],
+        "-nm",
-        help='use avconv for conversion (otherwise defaults to ffmpeg)',
+        "--no-metadata",
-        action='store_true')
+        default=config["no-metadata"],
+        help="do not embed metadata in tracks",
+        action="store_true",
+    )
     parser.add_argument(
-        '-f', '--folder', default=os.path.abspath(config['folder']),
+        "-a",
-        help='path to folder where downloaded tracks will be stored in')
+        "--avconv",
+        default=config["avconv"],
+        help="use avconv for conversion (otherwise defaults to ffmpeg)",
+        action="store_true",
+    )
     parser.add_argument(
-        '--overwrite', default=config['overwrite'],
+        "-f",
-        help='change the overwrite policy',
+        "--folder",
-        choices={'prompt', 'force', 'skip'})
+        default=os.path.abspath(config["folder"]),
+        help="path to folder where downloaded tracks will be stored in",
+    )
     parser.add_argument(
-        '-i', '--input-ext', default=config['input-ext'],
+        "--overwrite",
-        help='preferred input format .m4a or .webm (Opus)',
+        default=config["overwrite"],
-        choices={'.m4a', '.webm'})
+        help="change the overwrite policy",
+        choices={"prompt", "force", "skip"},
+    )
     parser.add_argument(
-        '-o', '--output-ext', default=config['output-ext'],
+        "-i",
-        help='preferred output format .mp3, .m4a (AAC), .flac, etc.')
+        "--input-ext",
+        default=config["input-ext"],
+        help="preferred input format .m4a or .webm (Opus)",
+        choices={".m4a", ".webm"},
+    )
     parser.add_argument(
-        '-ff', '--file-format', default=config['file-format'],
+        "-o",
-        help='file format to save the downloaded track with, each tag '
+        "--output-ext",
-             'is surrounded by curly braces. Possible formats: '
+        default=config["output-ext"],
-             '{}'.format([internals.formats[x] for x in internals.formats]))
+        help="preferred output format .mp3, .m4a (AAC), .flac, etc.",
+    )
     parser.add_argument(
-        '--trim-silence', default=config['trim-silence'],
+        "-ff",
-        help='remove silence from the start of the audio',
+        "--file-format",
-        action='store_true')
+        default=config["file-format"],
+        help="file format to save the downloaded track with, each tag "
+        "is surrounded by curly braces. Possible formats: "
+        "{}".format([internals.formats[x] for x in internals.formats]),
+    )
     parser.add_argument(
-        '-sf', '--search-format', default=config['search-format'],
+        "--trim-silence",
-        help='search format to search for on YouTube, each tag '
+        default=config["trim-silence"],
-             'is surrounded by curly braces. Possible formats: '
+        help="remove silence from the start of the audio",
-             '{}'.format([internals.formats[x] for x in internals.formats]))
+        action="store_true",
+    )
     parser.add_argument(
-        '-dm', '--download-only-metadata', default=config['download-only-metadata'],
+        "-sf",
-        help='download tracks only whose metadata is found',
+        "--search-format",
-        action='store_true')
+        default=config["search-format"],
+        help="search format to search for on YouTube, each tag "
+        "is surrounded by curly braces. Possible formats: "
+        "{}".format([internals.formats[x] for x in internals.formats]),
+    )
     parser.add_argument(
-        '-d', '--dry-run', default=config['dry-run'],
+        "-dm",
-        help='show only track title and YouTube URL, and then skip '
+        "--download-only-metadata",
-             'to the next track (if any)',
+        default=config["download-only-metadata"],
-        action='store_true')
+        help="download tracks only whose metadata is found",
+        action="store_true",
+    )
     parser.add_argument(
-        '-mo', '--music-videos-only', default=config['music-videos-only'],
+        "-d",
-        help='search only for music videos on Youtube (works only '
+        "--dry-run",
-             'when YouTube API key is set',
+        default=config["dry-run"],
-        action='store_true')
+        help="show only track title and YouTube URL, and then skip "
+        "to the next track (if any)",
+        action="store_true",
+    )
     parser.add_argument(
-        '-ns', '--no-spaces', default=config['no-spaces'],
+        "-mo",
-        help='replace spaces with underscores in file names',
+        "--music-videos-only",
-        action='store_true')
+        default=config["music-videos-only"],
+        help="search only for music videos on Youtube (works only "
+        "when YouTube API key is set",
+        action="store_true",
+    )
     parser.add_argument(
-        '-ll', '--log-level', default=config['log-level'],
+        "-ns",
+        "--no-spaces",
+        default=config["no-spaces"],
+        help="replace spaces with underscores in file names",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-ll",
+        "--log-level",
+        default=config["log-level"],
         choices=_LOG_LEVELS_STR,
         type=str.upper,
-        help='set log verbosity')
+        help="set log verbosity",
+    )
     parser.add_argument(
-        '-yk', '--youtube-api-key', default=config['youtube-api-key'],
+        "-yk",
-        help=argparse.SUPPRESS)
+        "--youtube-api-key",
+        default=config["youtube-api-key"],
+        help=argparse.SUPPRESS,
+    )
     parser.add_argument(
-        '-c', '--config', default=None,
+        "-sk",
-        help='path to custom config.yml file')
+        "--skip",
+        default=config["skip"],
+        help="path to file containing tracks to skip",
+    )
+    parser.add_argument(
+        "-w",
+        "--write-successful",
+        default=config["write-successful"],
+        help="path to file to write successful tracks to",
+    )
+    parser.add_argument(
+        "-c", "--config", default=None, help="path to custom config.yml file"
+    )
 
     parsed = parser.parse_args(raw_args)
 
     if parsed.config is not None and to_merge:
         parsed = override_config(parsed.config, parser)
 
+    if to_group and parsed.list and \
+            not mimetypes.MimeTypes().guess_type(parsed.list)[0] == "text/plain":
+        parser.error("{0} is not of a valid argument to --list, argument must be plain text file".format(parsed.list))
+
+    if parsed.write_m3u and not parsed.list:
+        parser.error('--write-m3u can only be used with --list')
+
     parsed.log_level = log_leveller(parsed.log_level)
 
     return parsed

spotdl/internals.py

@@ -1,52 +1,58 @@
 import os
 import sys
+from logzero import logger as log
 
 from spotdl import const
 
-log = const.log
+try:
+    import winreg
+except ImportError:
+    pass
 
 try:
     from slugify import SLUG_OK, slugify
 except ImportError:
-    log.error('Oops! `unicode-slugify` was not found.')
+    log.error("Oops! `unicode-slugify` was not found.")
-    log.info('Please remove any other slugify library and install `unicode-slugify`')
+    log.info("Please remove any other slugify library and install `unicode-slugify`")
     sys.exit(5)
 
-formats = { 0  : 'track_name',
+formats = {
-            1  : 'artist',
+    0: "track_name",
-            2  : 'album',
+    1: "artist",
-            3  : 'album_artist',
+    2: "album",
-            4  : 'genre',
+    3: "album_artist",
-            5  : 'disc_number',
+    4: "genre",
-            6  : 'duration',
+    5: "disc_number",
-            7  : 'year',
+    6: "duration",
-            8  : 'original_date',
+    7: "year",
-            9  : 'track_number',
+    8: "original_date",
-            10 : 'total_tracks',
+    9: "track_number",
-            11 : 'isrc' }
+    10: "total_tracks",
+    11: "isrc",
+}
 
 
 def input_link(links):
     """ Let the user input a choice. """
     while True:
         try:
-            log.info('Choose your number:')
+            log.info("Choose your number:")
-            the_chosen_one = int(input('> '))
+            the_chosen_one = int(input("> "))
             if 1 <= the_chosen_one <= len(links):
                 return links[the_chosen_one - 1]
             elif the_chosen_one == 0:
                 return None
             else:
-                log.warning('Choose a valid number!')
+                log.warning("Choose a valid number!")
         except ValueError:
-            log.warning('Choose a valid number!')
+            log.warning("Choose a valid number!")
 
 
 def trim_song(text_file):
     """ Remove the first song from file. """
-    with open(text_file, 'r') as file_in:
+    with open(text_file, "r") as file_in:
         data = file_in.read().splitlines(True)
-    with open(text_file, 'w') as file_out:
+    with open(text_file, "w") as file_out:
         file_out.writelines(data[1:])
     return data[0]
 
@@ -54,7 +60,7 @@ def trim_song(text_file):
 def is_spotify(raw_song):
     """ Check if the input song is a Spotify link. """
     status = len(raw_song) == 22 and raw_song.replace(" ", "%20") == raw_song
-    status = status or raw_song.find('spotify') > -1
+    status = status or raw_song.find("spotify") > -1
     return status
 
 
@@ -62,49 +68,49 @@ def is_youtube(raw_song):
     """ Check if the input song is a YouTube link. """
     status = len(raw_song) == 11 and raw_song.replace(" ", "%20") == raw_song
     status = status and not raw_song.lower() == raw_song
-    status = status or 'youtube.com/watch?v=' in raw_song
+    status = status or "youtube.com/watch?v=" in raw_song
     return status
 
 
 def format_string(string_format, tags, slugification=False, force_spaces=False):
     """ Generate a string of the format '[artist] - [song]' for the given spotify song. """
     format_tags = dict(formats)
-    format_tags[0]  = tags['name']
+    format_tags[0] = tags["name"]
-    format_tags[1]  = tags['artists'][0]['name']
+    format_tags[1] = tags["artists"][0]["name"]
-    format_tags[2]  = tags['album']['name']
+    format_tags[2] = tags["album"]["name"]
-    format_tags[3]  = tags['artists'][0]['name']
+    format_tags[3] = tags["artists"][0]["name"]
-    format_tags[4]  = tags['genre']
+    format_tags[4] = tags["genre"]
-    format_tags[5]  = tags['disc_number']
+    format_tags[5] = tags["disc_number"]
-    format_tags[6]  = tags['duration']
+    format_tags[6] = tags["duration"]
-    format_tags[7]  = tags['year']
+    format_tags[7] = tags["year"]
-    format_tags[8]  = tags['release_date']
+    format_tags[8] = tags["release_date"]
-    format_tags[9]  = tags['track_number']
+    format_tags[9] = tags["track_number"]
-    format_tags[10] = tags['total_tracks']
+    format_tags[10] = tags["total_tracks"]
-    format_tags[11] = tags['external_ids']['isrc']
+    format_tags[11] = tags["external_ids"]["isrc"]
 
-    for tag in format_tags:
+    format_tags_sanitized = {
-        if slugification:
+        k: sanitize_title(str(v), ok="'-_()[]{}") if slugification else str(v)
-            format_tags[tag] = sanitize_title(format_tags[tag],
+        for k, v in format_tags.items()
-                                              ok='-_()[]{}')
+    }
-        else:
-            format_tags[tag] = str(format_tags[tag])
 
     for x in formats:
-        format_tag = '{' + formats[x] + '}'
+        format_tag = "{" + formats[x] + "}"
-        string_format = string_format.replace(format_tag,
+        string_format = string_format.replace(format_tag, format_tags_sanitized[x])
-                                              format_tags[x])
 
     if const.args.no_spaces and not force_spaces:
-        string_format = string_format.replace(' ', '_')
+        string_format = string_format.replace(" ", "_")
 
     return string_format
 
 
-def sanitize_title(title, ok='-_()[]{}\/'):
+def sanitize_title(title, ok="-_()[]{}"):
     """ Generate filename of the song to be downloaded. """
 
     if const.args.no_spaces:
-        title = title.replace(' ', '_')
+        title = title.replace(" ", "_")
+
+    # replace slashes with "-" to avoid folder creation errors
+    title = title.replace("/", "-").replace("\\", "-")
 
     # slugify removes any special characters
     title = slugify(title, ok=ok, lower=False, spaces=True)
@@ -115,7 +121,7 @@ def filter_path(path):
     if not os.path.exists(path):
         os.makedirs(path)
     for temp in os.listdir(path):
-        if temp.endswith('.temp'):
+        if temp.endswith(".temp"):
             os.remove(os.path.join(path, temp))
 
 
@@ -123,19 +129,20 @@ def videotime_from_seconds(time):
     if time < 60:
         return str(time)
     if time < 3600:
-        return '{0}:{1:02}'.format(time//60, time % 60)
+        return "{0}:{1:02}".format(time // 60, time % 60)
 
-    return '{0}:{1:02}:{2:02}'.format((time//60)//60, (time//60) % 60, time % 60)
+    return "{0}:{1:02}:{2:02}".format((time // 60) // 60, (time // 60) % 60, time % 60)
 
 
 def get_sec(time_str):
-    if ':' in time_str:
+    if ":" in time_str:
-        splitter = ':'
+        splitter = ":"
-    elif '.' in time_str:
+    elif "." in time_str:
-        splitter = '.'
+        splitter = "."
     else:
-        raise ValueError("No expected character found in {} to split"
+        raise ValueError(
-                         "time values.".format(time_str))
+            "No expected character found in {} to split" "time values.".format(time_str)
+        )
     v = time_str.split(splitter, 3)
     v.reverse()
     sec = 0
@@ -148,35 +155,97 @@ def get_sec(time_str):
     return sec
 
 
-def get_splits(url):
+def extract_spotify_id(raw_string):
-    if '/' in url:
+    """
-        if url.endswith('/'):
+    Returns a Spotify ID of a playlist, album, etc. after extracting
-            url = url[:-1]
+    it from a given HTTP URL or Spotify URI.
-        splits = url.split('/')
+    """
+
+    if "/" in raw_string:
+        # Input string is an HTTP URL
+        if raw_string.endswith("/"):
+            raw_string = raw_string[:-1]
+        # We need to manually trim additional text from HTTP URLs
+        # We could skip this if https://github.com/plamere/spotipy/pull/324
+        # gets merged,
+        to_trim = raw_string.find("?")
+        if not to_trim == -1:
+            raw_string = raw_string[:to_trim]
+        splits = raw_string.split("/")
     else:
-        splits = url.split(':')
+        # Input string is a Spotify URI
-    return splits
+        splits = raw_string.split(":")
+
+    spotify_id = splits[-1]
+
+    return spotify_id
 
 
-# a hacky way to user's localized music directory
+def get_unique_tracks(text_file):
+    """
+    Returns a list of unique tracks given a path to a
+    file containing tracks.
+    """
+
+    with open(text_file, "r") as listed:
+        # Read tracks into a list and remove any duplicates
+        lines = listed.read().splitlines()
+
+    # Remove blank and strip whitespaces from lines (if any)
+    lines = [line.strip() for line in lines if line.strip()]
+    lines = remove_duplicates(lines)
+
+    return lines
+
+
+# a hacky way to get user's localized music directory
 # (thanks @linusg, issue #203)
 def get_music_dir():
-    home = os.path.expanduser('~')
+    home = os.path.expanduser("~")
 
     # On Linux, the localized folder names are the actual ones.
     # It's a freedesktop standard though.
-    if sys.platform.startswith('linux'):
+    if sys.platform.startswith("linux"):
-        for file_item in ('.config/user-dirs.dirs', 'user-dirs.dirs'):
+        for file_item in (".config/user-dirs.dirs", "user-dirs.dirs"):
             path = os.path.join(home, file_item)
             if os.path.isfile(path):
-                with open(path, 'r') as f:
+                with open(path, "r") as f:
                     for line in f:
-                        if line.startswith('XDG_MUSIC_DIR'):
+                        if line.startswith("XDG_MUSIC_DIR"):
-                            return os.path.expandvars(line.strip().split('=')[1].strip('"'))
+                            return os.path.expandvars(
+                                line.strip().split("=")[1].strip('"')
+                            )
+
+    # Windows / Cygwin
+    # Queries registry for 'My Music' folder path (as this can be changed)
+    if "win" in sys.platform:
+        try:
+            key = winreg.OpenKey(
+                winreg.HKEY_CURRENT_USER,
+                r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders",
+                0,
+                winreg.KEY_ALL_ACCESS,
+            )
+            return winreg.QueryValueEx(key, "My Music")[0]
+        except (FileNotFoundError, NameError):
+            pass
 
     # On both Windows and macOS, the localized folder names you see in
     # Explorer and Finder are actually in English on the file system.
     # So, defaulting to C:\Users\<user>\Music or /Users/<user>/Music
     # respectively is sufficient.
     # On Linux, default to /home/<user>/Music if the above method failed.
-    return os.path.join(home, 'Music')
+    return os.path.join(home, "Music")
+
+
+def remove_duplicates(tracks):
+    """
+    Removes duplicates from a list whilst preserving order.
+
+    We could directly call `set()` on the list but it changes
+    the order of elements.
+    """
+
+    local_set = set()
+    local_set_add = local_set.add
+    return [x for x in tracks if not (x in local_set or local_set_add(x))]

spotdl/metadata.py

@@ -2,7 +2,8 @@ from mutagen.easyid3 import EasyID3
 from mutagen.id3 import ID3, TORY, TYER, TPUB, APIC, USLT, COMM
 from mutagen.mp4 import MP4, MP4Cover
 from mutagen.flac import Picture, FLAC
-from spotdl.const import log, TAG_PRESET, M4A_TAG_PRESET
+from logzero import logger as log
+from spotdl.const import TAG_PRESET, M4A_TAG_PRESET
 
 import urllib.request
 
@@ -11,12 +12,12 @@ def compare(music_file, metadata):
     """Check if the input music file title matches the expected title."""
     already_tagged = False
     try:
-        if music_file.endswith('.mp3'):
+        if music_file.endswith(".mp3"):
             audiofile = EasyID3(music_file)
-            already_tagged = audiofile['title'][0] == metadata['name']
+            already_tagged = audiofile["title"][0] == metadata["name"]
-        elif music_file.endswith('.m4a'):
+        elif music_file.endswith(".m4a"):
             audiofile = MP4(music_file)
-            already_tagged = audiofile['\xa9nam'][0] == metadata['name']
+            already_tagged = audiofile["\xa9nam"][0] == metadata["name"]
     except (KeyError, TypeError):
         pass
 
@@ -26,17 +27,17 @@ def compare(music_file, metadata):
 def embed(music_file, meta_tags):
     """ Embed metadata. """
     embed = EmbedMetadata(music_file, meta_tags)
-    if music_file.endswith('.m4a'):
+    if music_file.endswith(".m4a"):
-        log.info('Applying metadata')
+        log.info("Applying metadata")
         return embed.as_m4a()
-    elif music_file.endswith('.mp3'):
+    elif music_file.endswith(".mp3"):
-        log.info('Applying metadata')
+        log.info("Applying metadata")
         return embed.as_mp3()
-    elif music_file.endswith('.flac'):
+    elif music_file.endswith(".flac"):
-        log.info('Applying metadata')
+        log.info("Applying metadata")
         return embed.as_flac()
     else:
-        log.warning('Cannot embed metadata into given output extension')
+        log.warning("Cannot embed metadata into given output extension")
         return False
 
 
@@ -55,33 +56,41 @@ class EmbedMetadata:
         # Check out somewhere at end of above linked file
         audiofile = EasyID3(music_file)
         self._embed_basic_metadata(audiofile, preset=TAG_PRESET)
-        audiofile['media'] = meta_tags['type']
+        audiofile["media"] = meta_tags["type"]
-        audiofile['author'] = meta_tags['artists'][0]['name']
+        audiofile["author"] = meta_tags["artists"][0]["name"]
-        audiofile['lyricist'] = meta_tags['artists'][0]['name']
+        audiofile["lyricist"] = meta_tags["artists"][0]["name"]
-        audiofile['arranger'] = meta_tags['artists'][0]['name']
+        audiofile["arranger"] = meta_tags["artists"][0]["name"]
-        audiofile['performer'] = meta_tags['artists'][0]['name']
+        audiofile["performer"] = meta_tags["artists"][0]["name"]
-        audiofile['website'] = meta_tags['external_urls']['spotify']
+        audiofile["website"] = meta_tags["external_urls"]["spotify"]
-        audiofile['length'] = str(meta_tags['duration'])
+        audiofile["length"] = str(meta_tags["duration"])
-        if meta_tags['publisher']:
+        if meta_tags["publisher"]:
-            audiofile['encodedby'] = meta_tags['publisher']
+            audiofile["encodedby"] = meta_tags["publisher"]
-        if meta_tags['external_ids']['isrc']:
+        if meta_tags["external_ids"]["isrc"]:
-            audiofile['isrc'] = meta_tags['external_ids']['isrc']
+            audiofile["isrc"] = meta_tags["external_ids"]["isrc"]
         audiofile.save(v2_version=3)
 
         # For supported id3 tags:
         # https://github.com/quodlibet/mutagen/blob/master/mutagen/id3/_frames.py
         # Each class represents an id3 tag
         audiofile = ID3(music_file)
-        audiofile['TORY'] = TORY(encoding=3, text=meta_tags['year'])
+        audiofile["TORY"] = TORY(encoding=3, text=meta_tags["year"])
-        audiofile['TYER'] = TYER(encoding=3, text=meta_tags['year'])
+        audiofile["TYER"] = TYER(encoding=3, text=meta_tags["year"])
-        audiofile['TPUB'] = TPUB(encoding=3, text=meta_tags['publisher'])
+        if meta_tags["publisher"]:
-        audiofile['COMM'] = COMM(encoding=3, text=meta_tags['external_urls']['spotify'])
+            audiofile["TPUB"] = TPUB(encoding=3, text=meta_tags["publisher"])
-        if meta_tags['lyrics']:
+        audiofile["COMM"] = COMM(encoding=3, text=meta_tags["external_urls"]["spotify"])
-            audiofile['USLT'] = USLT(encoding=3, desc=u'Lyrics', text=meta_tags['lyrics'])
+        if meta_tags["lyrics"]:
+            audiofile["USLT"] = USLT(
+                encoding=3, desc=u"Lyrics", text=meta_tags["lyrics"]
+            )
         try:
-            albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
+            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
-            audiofile['APIC'] = APIC(encoding=3, mime='image/jpeg', type=3,
+            audiofile["APIC"] = APIC(
-                                     desc=u'Cover', data=albumart.read())
+                encoding=3,
+                mime="image/jpeg",
+                type=3,
+                desc=u"Cover",
+                data=albumart.read(),
+            )
             albumart.close()
         except IndexError:
             pass
@@ -95,13 +104,15 @@ class EmbedMetadata:
         meta_tags = self.meta_tags
         audiofile = MP4(music_file)
         self._embed_basic_metadata(audiofile, preset=M4A_TAG_PRESET)
-        audiofile[M4A_TAG_PRESET['year']] = meta_tags['year']
+        audiofile[M4A_TAG_PRESET["year"]] = meta_tags["year"]
-        if meta_tags['lyrics']:
+        audiofile[M4A_TAG_PRESET["comment"]] = meta_tags["external_urls"]["spotify"]
-            audiofile['lyrics'] = meta_tags['lyrics']
+        if meta_tags["lyrics"]:
+            audiofile[M4A_TAG_PRESET["lyrics"]] = meta_tags["lyrics"]
         try:
-            albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
+            albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
-            audiofile[M4A_TAG_PRESET['albumart']] = [MP4Cover(
+            audiofile[M4A_TAG_PRESET["albumart"]] = [
-                albumart.read(), imageformat=MP4Cover.FORMAT_JPEG)]
+                MP4Cover(albumart.read(), imageformat=MP4Cover.FORMAT_JPEG)
+            ]
             albumart.close()
         except IndexError:
             pass
@@ -114,16 +125,16 @@ class EmbedMetadata:
         meta_tags = self.meta_tags
         audiofile = FLAC(music_file)
         self._embed_basic_metadata(audiofile)
-        audiofile['year'] = meta_tags['year']
+        audiofile["year"] = meta_tags["year"]
-        audiofile['comment'] = meta_tags['external_urls']['spotify']
+        audiofile["comment"] = meta_tags["external_urls"]["spotify"]
-        if meta_tags['lyrics']:
+        if meta_tags["lyrics"]:
-            audiofile['lyrics'] = meta_tags['lyrics']
+            audiofile["lyrics"] = meta_tags["lyrics"]
 
         image = Picture()
         image.type = 3
-        image.desc = 'Cover'
+        image.desc = "Cover"
-        image.mime = 'image/jpeg'
+        image.mime = "image/jpeg"
-        albumart = urllib.request.urlopen(meta_tags['album']['images'][0]['url'])
+        albumart = urllib.request.urlopen(meta_tags["album"]["images"][0]["url"])
         image.data = albumart.read()
         albumart.close()
         audiofile.add_picture(image)
@@ -133,27 +144,28 @@ class EmbedMetadata:
 
     def _embed_basic_metadata(self, audiofile, preset=TAG_PRESET):
         meta_tags = self.meta_tags
-        audiofile[preset['artist']] = meta_tags['artists'][0]['name']
+        audiofile[preset["artist"]] = meta_tags["artists"][0]["name"]
-        audiofile[preset['albumartist']] = meta_tags['artists'][0]['name']
+        audiofile[preset["albumartist"]] = meta_tags["artists"][0]["name"]
-        audiofile[preset['album']] = meta_tags['album']['name']
+        audiofile[preset["album"]] = meta_tags["album"]["name"]
-        audiofile[preset['title']] = meta_tags['name']
+        audiofile[preset["title"]] = meta_tags["name"]
-        audiofile[preset['date']] = meta_tags['release_date']
+        audiofile[preset["date"]] = meta_tags["release_date"]
-        audiofile[preset['originaldate']] = meta_tags['release_date']
+        audiofile[preset["originaldate"]] = meta_tags["release_date"]
-        if meta_tags['genre']:
+        if meta_tags["genre"]:
-            audiofile[preset['genre']] = meta_tags['genre']
+            audiofile[preset["genre"]] = meta_tags["genre"]
-        if meta_tags['copyright']:
+        if meta_tags["copyright"]:
-            audiofile[preset['copyright']] = meta_tags['copyright']
+            audiofile[preset["copyright"]] = meta_tags["copyright"]
-        if self.music_file.endswith('.flac'):
+        if self.music_file.endswith(".flac"):
-            audiofile[preset['discnumber']] = str(meta_tags['disc_number'])
+            audiofile[preset["discnumber"]] = str(meta_tags["disc_number"])
         else:
-            audiofile[preset['discnumber']] = [(meta_tags['disc_number'], 0)]
+            audiofile[preset["discnumber"]] = [(meta_tags["disc_number"], 0)]
-        if self.music_file.endswith('.flac'):
+        if self.music_file.endswith(".flac"):
-            audiofile[preset['tracknumber']] = str(meta_tags['track_number'])
+            audiofile[preset["tracknumber"]] = str(meta_tags["track_number"])
         else:
-            if preset['tracknumber'] == TAG_PRESET['tracknumber']:
+            if preset["tracknumber"] == TAG_PRESET["tracknumber"]:
-                audiofile[preset['tracknumber']] = '{}/{}'.format(meta_tags['track_number'],
+                audiofile[preset["tracknumber"]] = "{}/{}".format(
-                                                                  meta_tags['total_tracks'])
+                    meta_tags["track_number"], meta_tags["total_tracks"]
+                )
             else:
-                audiofile[preset['tracknumber']] = [
+                audiofile[preset["tracknumber"]] = [
-                    (meta_tags['track_number'], meta_tags['total_tracks'])
+                    (meta_tags["track_number"], meta_tags["total_tracks"])
                 ]

spotdl/spotdl.py

@@ -8,9 +8,11 @@ from spotdl import convert
 from spotdl import internals
 from spotdl import spotify_tools
 from spotdl import youtube_tools
+from logzero import logger as log
 from slugify import slugify
 import spotipy
 import urllib.request
+import logzero
 import os
 import sys
 import time
@@ -20,11 +22,13 @@ import pprint
 
 def check_exists(music_file, raw_song, meta_tags):
     """ Check if the input song already exists in the given folder. """
-    log.debug('Cleaning any temp files and checking '
+    log.debug(
-              'if "{}" already exists'.format(music_file))
+        "Cleaning any temp files and checking "
+        'if "{}" already exists'.format(music_file)
+    )
     songs = os.listdir(const.args.folder)
     for song in songs:
-        if song.endswith('.temp'):
+        if song.endswith(".temp"):
             os.remove(os.path.join(const.args.folder, song))
             continue
         # check if a song with the same name is already present in the given folder
@@ -33,116 +37,126 @@ def check_exists(music_file, raw_song, meta_tags):
             if internals.is_spotify(raw_song):
                 # check if the already downloaded song has correct metadata
                 # if not, remove it and download again without prompt
-                already_tagged = metadata.compare(os.path.join(const.args.folder, song),
+                already_tagged = metadata.compare(
-                                                  meta_tags)
+                    os.path.join(const.args.folder, song), meta_tags
-                log.debug('Checking if it is already tagged correctly? {}',
+                )
-                                                            already_tagged)
+                log.debug(
+                    "Checking if it is already tagged correctly? {}", already_tagged
+                )
                 if not already_tagged:
                     os.remove(os.path.join(const.args.folder, song))
                     return False
 
             log.warning('"{}" already exists'.format(song))
-            if const.args.overwrite == 'prompt':
+            if const.args.overwrite == "prompt":
-                log.info('"{}" has already been downloaded. '
+                log.info(
-                         'Re-download? (y/N): '.format(song))
+                    '"{}" has already been downloaded. '
-                prompt = input('> ')
+                    "Re-download? (y/N): ".format(song)
-                if prompt.lower() == 'y':
+                )
+                prompt = input("> ")
+                if prompt.lower() == "y":
                     os.remove(os.path.join(const.args.folder, song))
                     return False
                 else:
                     return True
-            elif const.args.overwrite == 'force':
+            elif const.args.overwrite == "force":
                 os.remove(os.path.join(const.args.folder, song))
                 log.info('Overwriting "{}"'.format(song))
                 return False
-            elif const.args.overwrite == 'skip':
+            elif const.args.overwrite == "skip":
                 log.info('Skipping "{}"'.format(song))
                 return True
     return False
 
 
-def download_list(text_file):
+def download_list(tracks_file, skip_file=None, write_successful_file=None):
     """ Download all songs from the list. """
-    with open(text_file, 'r') as listed:
-        # read tracks into a list and remove any duplicates
-        lines = listed.read().splitlines()
-        lines = list(set(lines))
-    # ignore blank lines in text_file (if any)
-    try:
-        lines.remove('')
-    except ValueError:
-        pass
 
-    log.info(u'Preparing to download {} songs'.format(len(lines)))
+    log.info("Checking and removing any duplicate tracks")
+    tracks = internals.get_unique_tracks(tracks_file)
+
+    # override file with unique tracks
+    with open(tracks_file, "w") as f:
+        f.write("\n".join(tracks))
+
+    # Remove tracks to skip from tracks list
+    if skip_file is not None:
+        skip_tracks = internals.get_unique_tracks(skip_file)
+        len_before = len(tracks)
+        tracks = [track for track in tracks if track not in skip_tracks]
+        log.info("Skipping {} tracks".format(len_before - len(tracks)))
+
+    log.info(u"Preparing to download {} songs".format(len(tracks)))
     downloaded_songs = []
 
-    for number, raw_song in enumerate(lines, 1):
+    for number, raw_song in enumerate(tracks, 1):
-        print('')
+        print("")
         try:
             download_single(raw_song, number=number)
         # token expires after 1 hour
         except spotipy.client.SpotifyException:
             # refresh token when it expires
-            log.debug('Token expired, generating new one and authorizing')
+            log.debug("Token expired, generating new one and authorizing")
-            new_token = spotify_tools.generate_token()
+            spotify_tools.refresh_token()
-            spotify_tools.spotify = spotipy.Spotify(auth=new_token)
             download_single(raw_song, number=number)
         # detect network problems
-        except (urllib.request.URLError, TypeError, IOError):
+        except (urllib.request.URLError, TypeError, IOError) as e:
-            lines.append(raw_song)
+            tracks.append(raw_song)
             # remove the downloaded song from file
-            internals.trim_song(text_file)
+            internals.trim_song(tracks_file)
             # and append it at the end of file
-            with open(text_file, 'a') as myfile:
+            with open(tracks_file, "a") as f:
-                myfile.write(raw_song + '\n')
+                f.write("\n" + raw_song)
-            log.warning('Failed to download song. Will retry after other songs\n')
+            log.exception(e)
+            log.warning("Failed to download song. Will retry after other songs\n")
             # wait 0.5 sec to avoid infinite looping
             time.sleep(0.5)
             continue
 
         downloaded_songs.append(raw_song)
-        log.debug('Removing downloaded song from text file')
+        # Add track to file of successful downloads
-        internals.trim_song(text_file)
+        log.debug("Adding downloaded song to write successful file")
+        if write_successful_file is not None:
+            with open(write_successful_file, "a") as f:
+                f.write("\n" + raw_song)
+        log.debug("Removing downloaded song from tracks file")
+        internals.trim_song(tracks_file)
 
     return downloaded_songs
 
 
 def download_single(raw_song, number=None):
     """ Logic behind downloading a song. """
-    if internals.is_youtube(raw_song):
+    content, meta_tags = youtube_tools.match_video_and_metadata(raw_song)
-        log.debug('Input song is a YouTube URL')
-        content = youtube_tools.go_pafy(raw_song, meta_tags=None)
-        raw_song = slugify(content.title).replace('-', ' ')
-        meta_tags = spotify_tools.generate_metadata(raw_song)
-    else:
-        meta_tags = spotify_tools.generate_metadata(raw_song)
-        content = youtube_tools.go_pafy(raw_song, meta_tags)
 
     if content is None:
-        log.debug('Found no matching video')
+        log.debug("Found no matching video")
         return
 
     if const.args.download_only_metadata and meta_tags is None:
-        log.info('Found no metadata. Skipping the download')
+        log.info("Found no metadata. Skipping the download")
         return
 
     # "[number]. [artist] - [song]" if downloading from list
     # otherwise "[artist] - [song]"
     youtube_title = youtube_tools.get_youtube_title(content, number)
-    log.info('{} ({})'.format(youtube_title, content.watchv_url))
+    log.info("{} ({})".format(youtube_title, content.watchv_url))
 
     # generate file name of the song to download
     songname = content.title
 
     if meta_tags is not None:
-        refined_songname = internals.format_string(const.args.file_format,
+        refined_songname = internals.format_string(
-                                                   meta_tags,
+            const.args.file_format, meta_tags, slugification=True
-                                                   slugification=True)
+        )
-        log.debug('Refining songname from "{0}" to "{1}"'.format(songname, refined_songname))
+        log.debug(
-        if not refined_songname == ' - ':
+            'Refining songname from "{0}" to "{1}"'.format(songname, refined_songname)
+        )
+        if not refined_songname == " - ":
             songname = refined_songname
     else:
-        log.warning('Could not find metadata')
+        if not const.args.no_metadata:
+            log.warning("Could not find metadata")
         songname = internals.sanitize_title(songname)
 
     if const.args.dry_run:
@@ -155,13 +169,18 @@ def download_single(raw_song, number=None):
         input_song = songname + const.args.input_ext
         output_song = songname + const.args.output_ext
         if youtube_tools.download_song(input_song, content):
-            print('')
+            print("")
             try:
-                convert.song(input_song, output_song, const.args.folder,
+                convert.song(
-                             avconv=const.args.avconv, trim_silence=const.args.trim_silence)
+                    input_song,
+                    output_song,
+                    const.args.folder,
+                    avconv=const.args.avconv,
+                    trim_silence=const.args.trim_silence,
+                )
             except FileNotFoundError:
-                encoder = 'avconv' if const.args.avconv else 'ffmpeg'
+                encoder = "avconv" if const.args.avconv else "ffmpeg"
-                log.warning('Could not find {0}, skipping conversion'.format(encoder))
+                log.warning("Could not find {0}, skipping conversion".format(encoder))
                 const.args.output_ext = const.args.input_ext
                 output_song = songname + const.args.output_ext
 
@@ -176,29 +195,36 @@ def main():
     const.args = handle.get_arguments()
 
     if const.args.version:
-        print('spotdl {version}'.format(version=__version__))
+        print("spotdl {version}".format(version=__version__))
         sys.exit()
 
     internals.filter_path(const.args.folder)
     youtube_tools.set_api_key()
 
-    const.log = const.logzero.setup_logger(formatter=const._formatter,
+    logzero.setup_default_logger(formatter=const._formatter, level=const.args.log_level)
-                                      level=const.args.log_level)
+
-    global log
+    log.debug("Python version: {}".format(sys.version))
-    log = const.log
+    log.debug("Platform: {}".format(platform.platform()))
-    log.debug('Python version: {}'.format(sys.version))
-    log.debug('Platform: {}'.format(platform.platform()))
     log.debug(pprint.pformat(const.args.__dict__))
 
     try:
         if const.args.song:
             download_single(raw_song=const.args.song)
         elif const.args.list:
-            download_list(text_file=const.args.list)
+            if const.args.write_m3u:
+                youtube_tools.generate_m3u(track_file=const.args.list)
+            else:
+                download_list(
+                    tracks_file=const.args.list,
+                    skip_file=const.args.skip,
+                    write_successful_file=const.args.write_successful,
+                )
         elif const.args.playlist:
             spotify_tools.write_playlist(playlist_url=const.args.playlist)
         elif const.args.album:
             spotify_tools.write_album(album_url=const.args.album)
+        elif const.args.all_albums:
+            spotify_tools.write_all_albums_from_artist(artist_url=const.args.all_albums)
         elif const.args.username:
             spotify_tools.write_user_playlist(username=const.args.username)
 
@@ -211,5 +237,5 @@ def main():
         sys.exit(3)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()

spotdl/spotify_tools.py

@@ -1,9 +1,9 @@
 import spotipy
 import spotipy.oauth2 as oauth2
 import lyricwikia
+from logzero import logger as log
 
 from spotdl import internals
-from spotdl.const import log
 
 from slugify import slugify
 from titlecase import titlecase
@@ -14,11 +14,19 @@ import sys
 def generate_token():
     """ Generate the token. Please respect these credentials :) """
     credentials = oauth2.SpotifyClientCredentials(
-        client_id='4fe3fecfe5334023a1472516cc99d805',
+        client_id="4fe3fecfe5334023a1472516cc99d805",
-        client_secret='0f02b7c483c04257984695007a4a8d5c')
+        client_secret="0f02b7c483c04257984695007a4a8d5c",
+    )
     token = credentials.get_access_token()
     return token
 
+
+def refresh_token():
+    """ Refresh expired token"""
+    global spotify
+    new_token = generate_token()
+    spotify = spotipy.Spotify(auth=new_token)
+
 # token is mandatory when using Spotify's API
 # https://developer.spotify.com/news-stories/2017/01/27/removing-unauthenticated-calls-to-the-web-api/
 token = generate_token()
@@ -29,51 +37,51 @@ def generate_metadata(raw_song):
     """ Fetch a song's metadata from Spotify. """
     if internals.is_spotify(raw_song):
         # fetch track information directly if it is spotify link
-        log.debug('Fetching metadata for given track URL')
+        log.debug("Fetching metadata for given track URL")
         meta_tags = spotify.track(raw_song)
     else:
         # otherwise search on spotify and fetch information from first result
         log.debug('Searching for "{}" on Spotify'.format(raw_song))
         try:
-            meta_tags = spotify.search(raw_song, limit=1)['tracks']['items'][0]
+            meta_tags = spotify.search(raw_song, limit=1)["tracks"]["items"][0]
         except IndexError:
             return None
-    artist = spotify.artist(meta_tags['artists'][0]['id'])
+    artist = spotify.artist(meta_tags["artists"][0]["id"])
-    album = spotify.album(meta_tags['album']['id'])
+    album = spotify.album(meta_tags["album"]["id"])
 
     try:
-        meta_tags[u'genre'] = titlecase(artist['genres'][0])
+        meta_tags[u"genre"] = titlecase(artist["genres"][0])
     except IndexError:
-        meta_tags[u'genre'] = None
+        meta_tags[u"genre"] = None
     try:
-        meta_tags[u'copyright'] = album['copyrights'][0]['text']
+        meta_tags[u"copyright"] = album["copyrights"][0]["text"]
     except IndexError:
-        meta_tags[u'copyright'] = None
+        meta_tags[u"copyright"] = None
     try:
-        meta_tags[u'external_ids'][u'isrc']
+        meta_tags[u"external_ids"][u"isrc"]
     except KeyError:
-        meta_tags[u'external_ids'][u'isrc'] = None
+        meta_tags[u"external_ids"][u"isrc"] = None
 
-    meta_tags[u'release_date'] = album['release_date']
+    meta_tags[u"release_date"] = album["release_date"]
-    meta_tags[u'publisher'] = album['label']
+    meta_tags[u"publisher"] = album["label"]
-    meta_tags[u'total_tracks'] = album['tracks']['total']
+    meta_tags[u"total_tracks"] = album["tracks"]["total"]
 
-    log.debug('Fetching lyrics')
+    log.debug("Fetching lyrics")
 
     try:
-        meta_tags['lyrics'] = lyricwikia.get_lyrics(
+        meta_tags["lyrics"] = lyricwikia.get_lyrics(
-                        meta_tags['artists'][0]['name'],
+            meta_tags["artists"][0]["name"], meta_tags["name"]
-                        meta_tags['name'])
+        )
     except lyricwikia.LyricsNotFound:
-        meta_tags['lyrics'] = None
+        meta_tags["lyrics"] = None
 
     # Some sugar
-    meta_tags['year'], *_ = meta_tags['release_date'].split('-')
+    meta_tags["year"], *_ = meta_tags["release_date"].split("-")
-    meta_tags['duration'] = meta_tags['duration_ms'] / 1000.0
+    meta_tags["duration"] = meta_tags["duration_ms"] / 1000.0
     # Remove unwanted parameters
-    del meta_tags['duration_ms']
+    del meta_tags["duration_ms"]
-    del meta_tags['available_markets']
+    del meta_tags["available_markets"]
-    del meta_tags['album']['available_markets']
+    del meta_tags["album"]["available_markets"]
 
     log.debug(pprint.pformat(meta_tags))
     return meta_tags
@@ -92,18 +100,20 @@ def get_playlists(username):
     check = 1
 
     while True:
-        for playlist in playlists['items']:
+        for playlist in playlists["items"]:
             # in rare cases, playlists may not be found, so playlists['next']
             # is None. Skip these. Also see Issue #91.
-            if playlist['name'] is not None:
+            if playlist["name"] is not None:
-                log.info(u'{0:>5}. {1:<30}  ({2} tracks)'.format(
+                log.info(
-                    check, playlist['name'],
+                    u"{0:>5}. {1:<30}  ({2} tracks)".format(
-                    playlist['tracks']['total']))
+                        check, playlist["name"], playlist["tracks"]["total"]
-                playlist_url = playlist['external_urls']['spotify']
+                    )
+                )
+                playlist_url = playlist["external_urls"]["spotify"]
                 log.debug(playlist_url)
                 links.append(playlist_url)
                 check += 1
-        if playlists['next']:
+        if playlists["next"]:
             playlists = spotify.next(playlists)
         else:
             break
@@ -112,20 +122,19 @@ def get_playlists(username):
 
 
 def fetch_playlist(playlist):
-    splits = internals.get_splits(playlist)
     try:
-        username = splits[-3]
+        playlist_id = internals.extract_spotify_id(playlist)
     except IndexError:
         # Wrong format, in either case
-        log.error('The provided playlist URL is not in a recognized format!')
+        log.error("The provided playlist URL is not in a recognized format!")
         sys.exit(10)
-    playlist_id = splits[-1]
     try:
-        results = spotify.user_playlist(username, playlist_id,
+        results = spotify.user_playlist(
-                                        fields='tracks,next,name')
+            user=None, playlist_id=playlist_id, fields="tracks,next,name"
+        )
     except spotipy.client.SpotifyException:
-        log.error('Unable to find playlist')
+        log.error("Unable to find playlist")
-        log.info('Make sure the playlist is set to publicly visible and then try again')
+        log.info("Make sure the playlist is set to publicly visible and then try again")
         sys.exit(11)
 
     return results
@@ -133,49 +142,107 @@ def fetch_playlist(playlist):
 
 def write_playlist(playlist_url, text_file=None):
     playlist = fetch_playlist(playlist_url)
-    tracks = playlist['tracks']
+    tracks = playlist["tracks"]
     if not text_file:
-        text_file = u'{0}.txt'.format(slugify(playlist['name'], ok='-_()[]{}'))
+        text_file = u"{0}.txt".format(slugify(playlist["name"], ok="-_()[]{}"))
     return write_tracks(tracks, text_file)
 
 
 def fetch_album(album):
-    splits = internals.get_splits(album)
+    album_id = internals.extract_spotify_id(album)
-    album_id = splits[-1]
     album = spotify.album(album_id)
     return album
 
 
+def fetch_album_from_artist(artist_url, album_type="album"):
+    """
+    This funcction returns all the albums from a give artist_url using the US
+    market
+    :param artist_url - spotify artist url
+    :param album_type - the type of album to fetch (ex: single) the default is
+                        a standard album
+    :param return - the album from the artist
+    """
+
+    # fetching artist's albums limitting the results to the US to avoid duplicate
+    # albums from multiple markets
+    artist_id = internals.extract_spotify_id(artist_url)
+    results = spotify.artist_albums(artist_id, album_type=album_type, country="US")
+
+    albums = results["items"]
+
+    # indexing all pages of results
+    while results["next"]:
+        results = spotify.next(results)
+        albums.extend(results["items"])
+
+    return albums
+
+
+def write_all_albums_from_artist(artist_url, text_file=None):
+    """
+    This function gets all albums from an artist and writes it to a file in the
+    current working directory called [ARTIST].txt, where [ARTIST] is the artist
+    of the album
+    :param artist_url - spotify artist url
+    :param text_file - file to write albums to
+    """
+
+    album_base_url = "https://open.spotify.com/album/"
+
+    # fetching all default albums
+    albums = fetch_album_from_artist(artist_url)
+
+    # if no file if given, the default save file is in the current working
+    # directory with the name of the artist
+    if text_file is None:
+        text_file = albums[0]["artists"][0]["name"] + ".txt"
+
+    for album in albums:
+        # logging album name
+        log.info("Fetching album: " + album["name"])
+        write_album(album_base_url + album["id"], text_file=text_file)
+
+    # fetching all single albums
+    singles = fetch_album_from_artist(artist_url, album_type="single")
+
+    for single in singles:
+        log.info("Fetching single: " + single["name"])
+        write_album(album_base_url + single["id"], text_file=text_file)
+
+
 def write_album(album_url, text_file=None):
     album = fetch_album(album_url)
-    tracks = spotify.album_tracks(album['id'])
+    tracks = spotify.album_tracks(album["id"])
     if not text_file:
-        text_file = u'{0}.txt'.format(slugify(album['name'], ok='-_()[]{}'))
+        text_file = u"{0}.txt".format(slugify(album["name"], ok="-_()[]{}"))
     return write_tracks(tracks, text_file)
 
 
 def write_tracks(tracks, text_file):
-    log.info(u'Writing {0} tracks to {1}'.format(
+    log.info(u"Writing {0} tracks to {1}".format(tracks["total"], text_file))
-               tracks['total'], text_file))
     track_urls = []
-    with open(text_file, 'a') as file_out:
+    with open(text_file, "a") as file_out:
         while True:
-            for item in tracks['items']:
+            for item in tracks["items"]:
-                if 'track' in item:
+                if "track" in item:
-                    track = item['track']
+                    track = item["track"]
                 else:
                     track = item
                 try:
-                    track_url = track['external_urls']['spotify']
+                    track_url = track["external_urls"]["spotify"]
                     log.debug(track_url)
-                    file_out.write(track_url + '\n')
+                    file_out.write(track_url + "\n")
                     track_urls.append(track_url)
                 except KeyError:
-                    log.warning(u'Skipping track {0} by {1} (local only?)'.format(
+                    log.warning(
-                        track['name'], track['artists'][0]['name']))
+                        u"Skipping track {0} by {1} (local only?)".format(
+                            track["name"], track["artists"][0]["name"]
+                        )
+                    )
             # 1 page = 50 results
             # check if there are more pages
-            if tracks['next']:
+            if tracks["next"]:
                 tracks = spotify.next(tracks)
             else:
                 break

spotdl/youtube_tools.py

@@ -1,18 +1,19 @@
 from bs4 import BeautifulSoup
 import urllib
 import pafy
+from slugify import slugify
+from logzero import logger as log
 
+from spotdl import spotify_tools
 from spotdl import internals
 from spotdl import const
 
 import os
 import pprint
 
-log = const.log
-
 # Fix download speed throttle on short duration tracks
 # Read more on mps-youtube/pafy#199
-pafy.g.opener.addheaders.append(('Range', 'bytes=0-'))
+pafy.g.opener.addheaders.append(("Range", "bytes=0-"))
 
 
 def set_api_key():
@@ -20,7 +21,7 @@ def set_api_key():
         key = const.args.youtube_api_key
     else:
         # Please respect this YouTube token :)
-        key = 'AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0'
+        key = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
     pafy.set_api_key(key)
 
 
@@ -39,32 +40,80 @@ def go_pafy(raw_song, meta_tags=None):
     return track_info
 
 
+def match_video_and_metadata(track, force_pafy=True):
+    """ Get and match track data from YouTube and Spotify. """
+    meta_tags = None
+
+    if internals.is_youtube(track):
+        log.debug("Input song is a YouTube URL")
+        content = go_pafy(track, meta_tags=None)
+        track = slugify(content.title).replace("-", " ")
+        if not const.args.no_metadata:
+            meta_tags = spotify_tools.generate_metadata(track)
+    else:
+        if not const.args.no_metadata:
+            meta_tags = spotify_tools.generate_metadata(track)
+        if force_pafy:
+            content = go_pafy(track, meta_tags)
+        else:
+            content = None
+    return content, meta_tags
+
+
 def get_youtube_title(content, number=None):
     """ Get the YouTube video's title. """
     title = content.title
     if number:
-        return '{0}. {1}'.format(number, title)
+        return "{0}. {1}".format(number, title)
     else:
         return title
 
 
+def generate_m3u(track_file):
+    tracks = internals.get_unique_tracks(track_file)
+    target_file = "{}.m3u".format(track_file.split(".")[0])
+    total_tracks = len(tracks)
+    log.info("Generating {0} from {1} YouTube URLs".format(target_file, total_tracks))
+    with open(target_file, "w") as output_file:
+        output_file.write("#EXTM3U\n\n")
+    for n, track in enumerate(tracks, 1):
+        content, _ = match_video_and_metadata(track)
+        if content is None:
+            log.warning("Skipping {}".format(track))
+        else:
+            log.info(
+                "Matched track {0}/{1} ({2})".format(
+                    n, total_tracks, content.watchv_url
+                )
+            )
+            log.debug(track)
+            m3u_key = "#EXTINF:{duration},{title}\n{youtube_url}\n".format(
+                duration=internals.get_sec(content.duration),
+                title=content.title,
+                youtube_url=content.watchv_url,
+            )
+            log.debug(m3u_key)
+            with open(target_file, "a") as output_file:
+                output_file.write(m3u_key)
+
+
 def download_song(file_name, content):
     """ Download the audio file from YouTube. """
     _, extension = os.path.splitext(file_name)
-    if extension in ('.webm', '.m4a'):
+    if extension in (".webm", ".m4a"):
         link = content.getbestaudio(preftype=extension[1:])
     else:
-        log.debug('No audio streams available for {} type'.format(extension))
+        log.debug("No audio streams available for {} type".format(extension))
         return False
 
     if link:
-        log.debug('Downloading from URL: ' + link.url)
+        log.debug("Downloading from URL: " + link.url)
         filepath = os.path.join(const.args.folder, file_name)
-        log.debug('Saving to: ' + filepath)
+        log.debug("Saving to: " + filepath)
         link.download(filepath=filepath)
         return True
     else:
-        log.debug('No audio streams available')
+        log.debug("No audio streams available")
         return False
 
 
@@ -73,23 +122,25 @@ def generate_search_url(query):
     # urllib.request.quote() encodes string with special characters
     quoted_query = urllib.request.quote(query)
     # Special YouTube URL filter to search only for videos
-    url = 'https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={0}'.format(quoted_query)
+    url = "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q={0}".format(
+        quoted_query
+    )
     return url
 
 
 def is_video(result):
     # ensure result is not a channel
-    not_video = result.find('channel') is not None or \
+    not_video = (
-                'yt-lockup-channel' in result.parent.attrs['class'] or \
+        result.find("channel") is not None
-                'yt-lockup-channel' in result.attrs['class']
+        or "yt-lockup-channel" in result.parent.attrs["class"]
+        or "yt-lockup-channel" in result.attrs["class"]
+    )
 
     # ensure result is not a mix/playlist
-    not_video = not_video or \
+    not_video = not_video or "yt-lockup-playlist" in result.parent.attrs["class"]
-               'yt-lockup-playlist' in result.parent.attrs['class']
 
     # ensure video result is not an advertisement
-    not_video = not_video or \
+    not_video = not_video or result.find("googleads") is not None
-                result.find('googleads') is not None
 
     video = not not_video
     return video
@@ -112,18 +163,25 @@ class GenerateYouTubeURL:
         if meta_tags is None:
             self.search_query = raw_song
         else:
-            self.search_query = internals.format_string(const.args.search_format,
+            self.search_query = internals.format_string(
-                                                        meta_tags, force_spaces=True)
+                const.args.search_format, meta_tags, force_spaces=True
+            )
 
     def _best_match(self, videos):
         """ Select the best matching video from a list of videos. """
         if const.args.manual:
             log.info(self.raw_song)
-            log.info('0. Skip downloading this song.\n')
+            log.info("0. Skip downloading this song.\n")
             # fetch all video links on first page on YouTube
             for i, v in enumerate(videos):
-                log.info(u'{0}. {1} {2} {3}'.format(i+1, v['title'], v['videotime'],
+                log.info(
-                      "http://youtube.com/watch?v="+v['link']))
+                    u"{0}. {1} {2} {3}".format(
+                        i + 1,
+                        v["title"],
+                        v["videotime"],
+                        "http://youtube.com/watch?v=" + v["link"],
+                    )
+                )
             # let user select the song to download
             result = internals.input_link(videos)
             if result is None:
@@ -133,7 +191,9 @@ class GenerateYouTubeURL:
                 # if the metadata could not be acquired, take the first result
                 # from Youtube because the proper song length is unknown
                 result = videos[0]
-                log.debug('Since no metadata found on Spotify, going with the first result')
+                log.debug(
+                    "Since no metadata found on Spotify, going with the first result"
+                )
             else:
                 # filter out videos that do not have a similar length to the Spotify song
                 duration_tolerance = 10
@@ -144,16 +204,27 @@ class GenerateYouTubeURL:
                 # until one of the Youtube results falls within the correct duration or
                 # the duration_tolerance has reached the max_duration_tolerance
                 while len(possible_videos_by_duration) == 0:
-                    possible_videos_by_duration = list(filter(lambda x: abs(x['seconds'] - self.meta_tags['duration']) <= duration_tolerance, videos))
+                    possible_videos_by_duration = list(
+                        filter(
+                            lambda x: abs(x["seconds"] - self.meta_tags["duration"])
+                            <= duration_tolerance,
+                            videos,
+                        )
+                    )
                     duration_tolerance += 1
                     if duration_tolerance > max_duration_tolerance:
-                        log.error("{0} by {1} was not found.\n".format(self.meta_tags['name'], self.meta_tags['artists'][0]['name']))
+                        log.error(
+                            "{0} by {1} was not found.".format(
+                                self.meta_tags["name"],
+                                self.meta_tags["artists"][0]["name"],
+                            )
+                        )
                         return None
 
                 result = possible_videos_by_duration[0]
 
         if result:
-            url = "http://youtube.com/watch?v={0}".format(result['link'])
+            url = "http://youtube.com/watch?v={0}".format(result["link"])
         else:
             url = None
 
@@ -164,33 +235,41 @@ class GenerateYouTubeURL:
 
         # prevents an infinite loop but allows for a few retries
         if tries_remaining == 0:
-            log.debug('No tries left. I quit.')
+            log.debug("No tries left. I quit.")
             return
 
         search_url = generate_search_url(self.search_query)
-        log.debug('Opening URL: {0}'.format(search_url))
+        log.debug("Opening URL: {0}".format(search_url))
 
         item = urllib.request.urlopen(search_url).read()
         items_parse = BeautifulSoup(item, "html.parser")
 
         videos = []
-        for x in items_parse.find_all('div', {'class': 'yt-lockup-dismissable yt-uix-tile'}):
+        for x in items_parse.find_all(
+            "div", {"class": "yt-lockup-dismissable yt-uix-tile"}
+        ):
 
             if not is_video(x):
                 continue
 
-            y = x.find('div', class_='yt-lockup-content')
+            y = x.find("div", class_="yt-lockup-content")
-            link = y.find('a')['href'][-11:]
+            link = y.find("a")["href"][-11:]
-            title = y.find('a')['title']
+            title = y.find("a")["title"]
 
             try:
-                videotime = x.find('span', class_="video-time").get_text()
+                videotime = x.find("span", class_="video-time").get_text()
             except AttributeError:
-                log.debug('Could not find video duration on YouTube, retrying..')
+                log.debug("Could not find video duration on YouTube, retrying..")
-                return self.scrape(bestmatch=bestmatch, tries_remaining=tries_remaining-1)
+                return self.scrape(
+                    bestmatch=bestmatch, tries_remaining=tries_remaining - 1
+                )
 
-            youtubedetails = {'link': link, 'title': title, 'videotime': videotime,
+            youtubedetails = {
-                              'seconds': internals.get_sec(videotime)}
+                "link": link,
+                "title": title,
+                "videotime": videotime,
+                "seconds": internals.get_sec(videotime),
+            }
             videos.append(youtubedetails)
 
         if bestmatch:
@@ -198,40 +277,43 @@ class GenerateYouTubeURL:
 
         return videos
 
-
     def api(self, bestmatch=True):
         """ Use YouTube API to search and return a list of matching videos. """
 
-        query = { 'part'       : 'snippet',
+        query = {"part": "snippet", "maxResults": 50, "type": "video"}
-                  'maxResults' :  50,
-                  'type'       : 'video' }
 
         if const.args.music_videos_only:
-            query['videoCategoryId'] = '10'
+            query["videoCategoryId"] = "10"
 
         if not self.meta_tags:
             song = self.raw_song
-            query['q'] = song
+            query["q"] = song
         else:
-            query['q'] = self.search_query
+            query["q"] = self.search_query
-        log.debug('query: {0}'.format(query))
+        log.debug("query: {0}".format(query))
 
-        data = pafy.call_gdata('search', query)
+        data = pafy.call_gdata("search", query)
-        data['items'] = list(filter(lambda x: x['id'].get('videoId') is not None,
+        data["items"] = list(
-                                    data['items']))
+            filter(lambda x: x["id"].get("videoId") is not None, data["items"])
-        query_results = {'part': 'contentDetails,snippet,statistics',
+        )
-                  'maxResults': 50,
+        query_results = {
-                  'id': ','.join(i['id']['videoId'] for i in data['items'])}
+            "part": "contentDetails,snippet,statistics",
-        log.debug('query_results: {0}'.format(query_results))
+            "maxResults": 50,
+            "id": ",".join(i["id"]["videoId"] for i in data["items"]),
+        }
+        log.debug("query_results: {0}".format(query_results))
 
-        vdata = pafy.call_gdata('videos', query_results)
+        vdata = pafy.call_gdata("videos", query_results)
 
         videos = []
-        for x in vdata['items']:
+        for x in vdata["items"]:
-            duration_s = pafy.playlist.parseISO8591(x['contentDetails']['duration'])
+            duration_s = pafy.playlist.parseISO8591(x["contentDetails"]["duration"])
-            youtubedetails = {'link': x['id'], 'title': x['snippet']['title'],
+            youtubedetails = {
-                              'videotime':internals.videotime_from_seconds(duration_s),
+                "link": x["id"],
-                              'seconds': duration_s}
+                "title": x["snippet"]["title"],
+                "videotime": internals.videotime_from_seconds(duration_s),
+                "seconds": duration_s,
+            }
             videos.append(youtubedetails)
 
         if bestmatch:

test/loader.py

@@ -1,14 +1,16 @@
 from spotdl import const
 from spotdl import handle
 from spotdl import spotdl
+
 import pytest
 
 
 def load_defaults():
-    const.args = handle.get_arguments(raw_args='', to_group=False, to_merge=False)
+    const.args = handle.get_arguments(raw_args="", to_group=False, to_merge=False)
-    const.args.overwrite = 'skip'
+    const.args.overwrite = "skip"
     const.args.log_level = 10
 
     spotdl.args = const.args
-    spotdl.log = const.logzero.setup_logger(formatter=const._formatter,
+    spotdl.log = const.logzero.setup_logger(
-                                      level=const.args.log_level)
+        formatter=const._formatter, level=const.args.log_level
+    )

test/test_dry_run.py

@@ -1,18 +1,20 @@
-from spotdl import const
-
-from spotdl import spotdl
-import loader
 import os
 
+from spotdl import const
+from spotdl import spotdl
+
+import loader
+
 loader.load_defaults()
 
+TRACK_URL = "http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU"
+
 
 def test_dry_download_list(tmpdir):
-    song = 'http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU'
     const.args.folder = str(tmpdir)
     const.args.dry_run = True
-    file_path = os.path.join(const.args.folder, 'test_list.txt')
+    file_path = os.path.join(const.args.folder, "test_list.txt")
-    with open(file_path, 'w') as tin:
+    with open(file_path, "w") as f:
-        tin.write(song)
+        f.write(TRACK_URL)
     downloaded_song, *_ = spotdl.download_list(file_path)
-    assert downloaded_song == song
+    assert downloaded_song == TRACK_URL

test/test_handle.py

@@ -1,46 +1,57 @@
-import yaml
-
-from spotdl import handle
-from spotdl import const
-
-import pytest
 import os
 import sys
 import argparse
 
+from spotdl import handle
+
+import pytest
+import yaml
+
+
+def test_error_m3u_without_list():
+    with pytest.raises(SystemExit):
+        handle.get_arguments(raw_args=('-s cool song', '--write-m3u',),
+                             to_group=True)
+
+
+def test_m3u_with_list():
+    handle.get_arguments(raw_args=('-l cool_list.txt', '--write-m3u',),
+                         to_group=True)
+
 
 def test_log_str_to_int():
     expect_levels = [20, 30, 40, 10]
-    levels = [handle.log_leveller(level)
+    levels = [handle.log_leveller(level) for level in handle._LOG_LEVELS_STR]
-              for level in handle._LOG_LEVELS_STR]
     assert levels == expect_levels
 
 
 class TestConfig:
     def test_default_config(self, tmpdir):
-        expect_config = handle.default_conf['spotify-downloader']
+        expect_config = handle.default_conf["spotify-downloader"]
         global config_path
-        config_path = os.path.join(str(tmpdir), 'config.yml')
+        config_path = os.path.join(str(tmpdir), "config.yml")
         config = handle.get_config(config_path)
         assert config == expect_config
 
     def test_modified_config(self):
         global modified_config
         modified_config = dict(handle.default_conf)
-        modified_config['spotify-downloader']['file-format'] = 'just_a_test'
+        modified_config["spotify-downloader"]["file-format"] = "just_a_test"
         merged_config = handle.merge(handle.default_conf, modified_config)
         assert merged_config == modified_config
 
     def test_custom_config_path(self, tmpdir):
         parser = argparse.ArgumentParser()
-        with open(config_path, 'w') as config_file:
+        with open(config_path, "w") as config_file:
             yaml.dump(modified_config, config_file, default_flow_style=False)
-        overridden_config = handle.override_config(config_path,
+        overridden_config = handle.override_config(config_path, parser, raw_args="")
-                                                   parser,
+        modified_values = [
-                                                   raw_args='')
+            str(value) for value in modified_config["spotify-downloader"].values()
-        modified_values = [ str(value) for value in modified_config['spotify-downloader'].values() ]
+        ]
         overridden_config.folder = os.path.realpath(overridden_config.folder)
-        overridden_values = [ str(value) for value in overridden_config.__dict__.values() ]
+        overridden_values = [
+            str(value) for value in overridden_config.__dict__.values()
+        ]
         assert sorted(overridden_values) == sorted(modified_values)
 
 

test/test_internals.py

@@ -1,18 +1,92 @@
-from spotdl import internals
-
 import sys
 import os
 import subprocess
+
+from spotdl import internals
+
 import pytest
 
 
+DUPLICATE_TRACKS_TEST_TABLE = [
+    (
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+        ),
+        ("https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",),
+    ),
+    (
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "",
+            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
+        ),
+        (
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD",
+        ),
+    ),
+    (
+        (
+            "ncs fade",
+            "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ",
+            "",
+            "ncs fade",
+        ),
+        ("ncs fade", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
+    ),
+    (
+        ("ncs spectre ", "  https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ", ""),
+        ("ncs spectre", "https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ"),
+    ),
+]
+
+STRING_IDS_TEST_TABLE = [
+    (
+        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8?si=_cVm-FBRQmi7VWML7E49Ig",
+        "1feoGrmmD8QmNqtK2Gdwy8",
+    ),
+    (
+        "https://open.spotify.com/artist/1feoGrmmD8QmNqtK2Gdwy8",
+        "1feoGrmmD8QmNqtK2Gdwy8",
+    ),
+    ("spotify:artist:1feoGrmmD8QmNqtK2Gdwy8", "1feoGrmmD8QmNqtK2Gdwy8"),
+    (
+        "https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp?si=LkVQLJGGT--Lh8BWM4MGvg",
+        "1d1l3UkeAjtM7kVTDyR8yp",
+    ),
+    ("https://open.spotify.com/album/1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
+    ("spotify:album:1d1l3UkeAjtM7kVTDyR8yp", "1d1l3UkeAjtM7kVTDyR8yp"),
+    (
+        "https://open.spotify.com/user/5kkyy50uu8btnagp30pobxz2f/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY?si=8Da4gbE2T9qMkd8Upg22ZA",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/playlist/3SFKRjUXm0IMQJMkEgPHeY",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "spotify:user:5kkyy50uu8btnagp30pobxz2f:playlist:3SFKRjUXm0IMQJMkEgPHeY",
+        "3SFKRjUXm0IMQJMkEgPHeY",
+    ),
+    (
+        "https://open.spotify.com/user/uqlakumu7wslkoen46s5bulq0",
+        "uqlakumu7wslkoen46s5bulq0",
+    ),
+]
+
+
 def test_default_music_directory():
-    if sys.platform.startswith('linux'):
+    if sys.platform.startswith("linux"):
-        output = subprocess.check_output(['xdg-user-dir', 'MUSIC'])
+        output = subprocess.check_output(["xdg-user-dir", "MUSIC"])
-        expect_directory = output.decode('utf-8').rstrip()
+        expect_directory = output.decode("utf-8").rstrip()
     else:
-        home = os.path.expanduser('~')
+        home = os.path.expanduser("~")
-        expect_directory = os.path.join(home, 'Music')
+        expect_directory = os.path.join(home, "Music")
 
     directory = internals.get_music_dir()
     assert directory == expect_directory
@@ -22,15 +96,15 @@ class TestPathFilterer:
     def test_create_directory(self, tmpdir):
         expect_path = True
         global folder_path
-        folder_path = os.path.join(str(tmpdir), 'filter_this_folder')
+        folder_path = os.path.join(str(tmpdir), "filter_this_folder")
         internals.filter_path(folder_path)
         is_path = os.path.isdir(folder_path)
         assert is_path == expect_path
 
     def test_remove_temp_files(self, tmpdir):
         expect_file = False
-        file_path = os.path.join(folder_path, 'pesky_file.temp')
+        file_path = os.path.join(folder_path, "pesky_file.temp")
-        open(file_path, 'a')
+        open(file_path, "a")
         internals.filter_path(folder_path)
         is_file = os.path.isfile(file_path)
         assert is_file == expect_file
@@ -38,17 +112,17 @@ class TestPathFilterer:
 
 class TestVideoTimeFromSeconds:
     def test_from_seconds(self):
-        expect_duration = '35'
+        expect_duration = "35"
         duration = internals.videotime_from_seconds(35)
         assert duration == expect_duration
 
     def test_from_minutes(self):
-        expect_duration = '2:38'
+        expect_duration = "2:38"
         duration = internals.videotime_from_seconds(158)
         assert duration == expect_duration
 
     def test_from_hours(self):
-        expect_duration = '1:16:02'
+        expect_duration = "1:16:02"
         duration = internals.videotime_from_seconds(4562)
         assert duration == expect_duration
 
@@ -56,27 +130,43 @@ class TestVideoTimeFromSeconds:
 class TestGetSeconds:
     def test_from_seconds(self):
         expect_secs = 45
-        secs = internals.get_sec('0:45')
+        secs = internals.get_sec("0:45")
         assert secs == expect_secs
-        secs = internals.get_sec('0.45')
+        secs = internals.get_sec("0.45")
         assert secs == expect_secs
 
     def test_from_minutes(self):
         expect_secs = 213
-        secs = internals.get_sec('3.33')
+        secs = internals.get_sec("3.33")
         assert secs == expect_secs
-        secs = internals.get_sec('3:33')
+        secs = internals.get_sec("3:33")
         assert secs == expect_secs
 
     def test_from_hours(self):
         expect_secs = 5405
-        secs = internals.get_sec('1.30.05')
+        secs = internals.get_sec("1.30.05")
         assert secs == expect_secs
-        secs = internals.get_sec('1:30:05')
+        secs = internals.get_sec("1:30:05")
         assert secs == expect_secs
 
     def test_raise_error(self):
         with pytest.raises(ValueError):
-            internals.get_sec('10*05')
+            internals.get_sec("10*05")
         with pytest.raises(ValueError):
-            internals.get_sec('02,28,46')
+            internals.get_sec("02,28,46")
+
+
+@pytest.mark.parametrize("duplicates, expected", DUPLICATE_TRACKS_TEST_TABLE)
+def test_get_unique_tracks(tmpdir, duplicates, expected):
+    file_path = os.path.join(str(tmpdir), "test_duplicates.txt")
+    with open(file_path, "w") as f:
+        f.write("\n".join(duplicates))
+
+    unique_tracks = internals.get_unique_tracks(file_path)
+    assert tuple(unique_tracks) == expected
+
+
+@pytest.mark.parametrize("input_str, expected_spotify_id", STRING_IDS_TEST_TABLE)
+def test_extract_spotify_id(input_str, expected_spotify_id):
+    spotify_id = internals.extract_spotify_id(input_str)
+    assert spotify_id == expected_spotify_id

test/test_list.py

@@ -1,50 +1,90 @@
-from spotdl import spotify_tools
-from spotdl import const
-
-from spotdl import spotdl
-
 import builtins
 import os
 
+from spotdl import spotify_tools
+from spotdl import youtube_tools
+from spotdl import const
+from spotdl import spotdl
+
+import loader
+
+
+USERNAME = "uqlakumu7wslkoen46s5bulq0"
+PLAYLIST_URL = "https://open.spotify.com/playlist/0fWBMhGh38y0wsYWwmM9Kt"
+ALBUM_URL = "https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg"
+ARTIST_URL = "https://open.spotify.com/artist/4dpARuHxo51G3z768sgnrY"
+
+loader.load_defaults()
+
 
 def test_user_playlists(tmpdir, monkeypatch):
-    expect_tracks = 14
+    expect_tracks = 17
-    text_file = os.path.join(str(tmpdir), 'test_us.txt')
+    text_file = os.path.join(str(tmpdir), "test_us.txt")
-    monkeypatch.setattr('builtins.input', lambda x: 1)
+    monkeypatch.setattr("builtins.input", lambda x: 1)
-    spotify_tools.write_user_playlist('alex', text_file)
+    spotify_tools.write_user_playlist(USERNAME, text_file)
-    with open(text_file, 'r') as tin:
+    with open(text_file, "r") as f:
-        tracks = len(tin.readlines())
+        tracks = len(f.readlines())
     assert tracks == expect_tracks
 
 
 def test_playlist(tmpdir):
     expect_tracks = 14
-    text_file = os.path.join(str(tmpdir), 'test_pl.txt')
+    text_file = os.path.join(str(tmpdir), "test_pl.txt")
-    spotify_tools.write_playlist('https://open.spotify.com/user/alex/playlist/0iWOVoumWlkXIrrBTSJmN8', text_file)
+    spotify_tools.write_playlist(PLAYLIST_URL, text_file)
-    with open(text_file, 'r') as tin:
+    with open(text_file, "r") as f:
-        tracks = len(tin.readlines())
+        tracks = len(f.readlines())
     assert tracks == expect_tracks
 
 
 def test_album(tmpdir):
     expect_tracks = 15
+    text_file = os.path.join(str(tmpdir), "test_al.txt")
+    spotify_tools.write_album(ALBUM_URL, text_file)
+    with open(text_file, "r") as f:
+        tracks = len(f.readlines())
+    assert tracks == expect_tracks
+
+
+def test_m3u(tmpdir):
+    expect_m3u = (
+        "#EXTM3U\n\n"
+        "#EXTINF:31,Eminem - 01 - Eminem - Curtains Up (Skit)\n"
+        "http://www.youtube.com/watch?v=qk13SFlwG9A\n"
+        "#EXTINF:226,Alan Walker - Spectre [NCS Release]\n"
+        "http://www.youtube.com/watch?v=AOeY-nDp7hI\n"
+    )
+    m3u_track_file = os.path.join(str(tmpdir), "m3u_test.txt")
+    with open(m3u_track_file, "w") as track_file:
+        track_file.write("\nhttps://open.spotify.com/track/2nT5m433s95hvYJH4S7ont")
+        track_file.write("\nhttp://www.youtube.com/watch?v=AOeY-nDp7hI")
+    youtube_tools.generate_m3u(m3u_track_file)
+    m3u_file = "{}.m3u".format(m3u_track_file.split(".")[0])
+    with open(m3u_file, "r") as m3u_in:
+        m3u = m3u_in.readlines()
+    assert "".join(m3u) == expect_m3u
+
+
+def test_all_albums(tmpdir):
+    # current number of tracks on spotify since as of 10/10/2018
+    # in US market only
+    expect_tracks = 49
     global text_file
-    text_file = os.path.join(str(tmpdir), 'test_al.txt')
+    text_file = os.path.join(str(tmpdir), "test_ab.txt")
-    spotify_tools.write_album('https://open.spotify.com/album/499J8bIsEnU7DSrosFDJJg', text_file)
+    spotify_tools.write_all_albums_from_artist(ARTIST_URL, text_file)
-    with open(text_file, 'r') as tin:
+    with open(text_file, "r") as f:
-        tracks = len(tin.readlines())
+        tracks = len(f.readlines())
     assert tracks == expect_tracks
 
 
 def test_trim():
-    with open(text_file, 'r') as track_file:
+    with open(text_file, "r") as track_file:
         tracks = track_file.readlines()
 
     expect_number = len(tracks) - 1
     expect_track = tracks[0]
     track = spotdl.internals.trim_song(text_file)
 
-    with open(text_file, 'r') as track_file:
+    with open(text_file, "r") as track_file:
         number = len(track_file.readlines())
 
-    assert (expect_number == number and expect_track == track)
+    assert expect_number == number and expect_track == track

test/test_with_metadata.py

@@ -1,3 +1,4 @@
+import os
 
 from spotdl import const
 from spotdl import internals
@@ -5,48 +6,46 @@ from spotdl import spotify_tools
 from spotdl import youtube_tools
 from spotdl import convert
 from spotdl import metadata
-
 from spotdl import spotdl
 
 import loader
-import os
 
 loader.load_defaults()
-raw_song = 'http://open.spotify.com/track/0JlS7BXXD07hRmevDnbPDU'
+
+TRACK_URL = "https://open.spotify.com/track/2nT5m433s95hvYJH4S7ont"
+EXPECTED_TITLE = "Eminem - Curtains Up"
+EXPECTED_YT_TITLE = "Eminem - 01 - Eminem - Curtains Up (Skit)"
+EXPECTED_YT_URL = "http://youtube.com/watch?v=qk13SFlwG9A"
 
 
 def test_metadata():
     expect_number = 23
     global meta_tags
-    meta_tags = spotify_tools.generate_metadata(raw_song)
+    meta_tags = spotify_tools.generate_metadata(TRACK_URL)
     assert len(meta_tags) == expect_number
 
 
 class TestFileFormat:
     def test_with_spaces(self):
-        expect_title = 'David André Østby - Intro'
         title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == expect_title
+        assert title == EXPECTED_TITLE
 
     def test_without_spaces(self):
-        expect_title = 'David_André_Østby_-_Intro'
         const.args.no_spaces = True
         title = internals.format_string(const.args.file_format, meta_tags)
-        assert title == expect_title
+        assert title == EXPECTED_TITLE.replace(" ", "_")
 
 
 def test_youtube_url():
-    expect_url = 'http://youtube.com/watch?v=rg1wfcty0BA'
+    url = youtube_tools.generate_youtube_url(TRACK_URL, meta_tags)
-    url = youtube_tools.generate_youtube_url(raw_song, meta_tags)
+    assert url == EXPECTED_YT_URL
-    assert url == expect_url
 
 
 def test_youtube_title():
-    expect_title = 'Intro - David André Østby'
     global content
-    content = youtube_tools.go_pafy(raw_song, meta_tags)
+    content = youtube_tools.go_pafy(TRACK_URL, meta_tags)
     title = youtube_tools.get_youtube_title(content)
-    assert title == expect_title
+    assert title == EXPECTED_YT_TITLE
 
 
 def test_check_track_exists_before_download(tmpdir):
@@ -56,67 +55,65 @@ def test_check_track_exists_before_download(tmpdir):
     songname = internals.format_string(const.args.file_format, meta_tags)
     global file_name
     file_name = internals.sanitize_title(songname)
-    check = spotdl.check_exists(file_name, raw_song, meta_tags)
+    check = spotdl.check_exists(file_name, TRACK_URL, meta_tags)
     assert check == expect_check
 
 
 class TestDownload:
     def test_m4a(self):
         expect_download = True
-        download = youtube_tools.download_song(file_name + '.m4a', content)
+        download = youtube_tools.download_song(file_name + ".m4a", content)
         assert download == expect_download
 
     def test_webm(self):
         expect_download = True
-        download = youtube_tools.download_song(file_name + '.webm', content)
+        download = youtube_tools.download_song(file_name + ".webm", content)
         assert download == expect_download
 
 
-class TestFFmpeg():
+class TestFFmpeg:
     def test_convert_from_webm_to_mp3(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.webm',
+        return_code = convert.song(
-                                   file_name + '.mp3',
+            file_name + ".webm", file_name + ".mp3", const.args.folder
-                                   const.args.folder)
+        )
         assert return_code == expect_return_code
 
     def test_convert_from_webm_to_m4a(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.webm',
+        return_code = convert.song(
-                                   file_name + '.m4a',
+            file_name + ".webm", file_name + ".m4a", const.args.folder
-                                   const.args.folder)
+        )
         assert return_code == expect_return_code
 
-
     def test_convert_from_m4a_to_mp3(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
+        return_code = convert.song(
-                                   file_name + '.mp3',
+            file_name + ".m4a", file_name + ".mp3", const.args.folder
-                                   const.args.folder)
+        )
         assert return_code == expect_return_code
 
     def test_convert_from_m4a_to_webm(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
+        return_code = convert.song(
-                                   file_name + '.webm',
+            file_name + ".m4a", file_name + ".webm", const.args.folder
-                                   const.args.folder)
+        )
         assert return_code == expect_return_code
 
     def test_convert_from_m4a_to_flac(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
+        return_code = convert.song(
-                                   file_name + '.flac',
+            file_name + ".m4a", file_name + ".flac", const.args.folder
-                                   const.args.folder)
+        )
         assert return_code == expect_return_code
 
 
 class TestAvconv:
     def test_convert_from_m4a_to_mp3(self):
         expect_return_code = 0
-        return_code = convert.song(file_name + '.m4a',
+        return_code = convert.song(
-                                   file_name + '.mp3',
+            file_name + ".m4a", file_name + ".mp3", const.args.folder, avconv=True
-                                   const.args.folder,
+        )
-                                   avconv=True)
         assert return_code == expect_return_code
 
 
@@ -125,30 +122,30 @@ class TestEmbedMetadata:
         expect_embed = True
         global track_path
         track_path = os.path.join(const.args.folder, file_name)
-        embed = metadata.embed(track_path + '.mp3', meta_tags)
+        embed = metadata.embed(track_path + ".mp3", meta_tags)
         assert embed == expect_embed
 
     def test_embed_in_m4a(self):
         expect_embed = True
-        embed = metadata.embed(track_path + '.m4a', meta_tags)
+        embed = metadata.embed(track_path + ".m4a", meta_tags)
-        os.remove(track_path + '.m4a')
+        os.remove(track_path + ".m4a")
         assert embed == expect_embed
 
     def test_embed_in_webm(self):
         expect_embed = False
-        embed = metadata.embed(track_path + '.webm', meta_tags)
+        embed = metadata.embed(track_path + ".webm", meta_tags)
-        os.remove(track_path + '.webm')
+        os.remove(track_path + ".webm")
         assert embed == expect_embed
 
     def test_embed_in_flac(self):
         expect_embed = True
-        embed = metadata.embed(track_path + '.flac', meta_tags)
+        embed = metadata.embed(track_path + ".flac", meta_tags)
-        os.remove(track_path + '.flac')
+        os.remove(track_path + ".flac")
         assert embed == expect_embed
 
 
 def test_check_track_exists_after_download():
     expect_check = True
-    check = spotdl.check_exists(file_name, raw_song, meta_tags)
+    check = spotdl.check_exists(file_name, TRACK_URL, meta_tags)
-    os.remove(track_path + '.mp3')
+    os.remove(track_path + ".mp3")
     assert check == expect_check

test/test_without_metadata.py

@@ -1,38 +1,47 @@
+import os
+import builtins
+
 from spotdl import const
 from spotdl import internals
 from spotdl import spotify_tools
 from spotdl import youtube_tools
-
 from spotdl import spotdl
+
 import loader
 
-import os
-import builtins
-
 loader.load_defaults()
-raw_song = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
+
+YT_API_KEY = "AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90"
+
+TRACK_SEARCH = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
+EXPECTED_TITLE = TRACK_SEARCH
+EXPECTED_YT_URL = "http://youtube.com/watch?v=qOOcy2-tmbk"
+EXPECTED_YT_URLS = (EXPECTED_YT_URL, "http://youtube.com/watch?v=5USR1Omo7f0")
+
+RESULT_COUNT_SEARCH = "she is still sleeping SAO"
+
+EXPECTED_YT_API_KEY = "AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0"
+EXPECTED_YT_API_KEY_CUSTOM = "some_api_key"
 
 
 class TestYouTubeAPIKeys:
     def test_custom(self):
-        expect_key = 'some_api_key'
+        const.args.youtube_api_key = EXPECTED_YT_API_KEY_CUSTOM
-        const.args.youtube_api_key = expect_key
         youtube_tools.set_api_key()
         key = youtube_tools.pafy.g.api_key
-        assert key == expect_key
+        assert key == EXPECTED_YT_API_KEY_CUSTOM
 
     def test_default(self):
-        expect_key = 'AIzaSyC6cEeKlxtOPybk9sEe5ksFN5sB-7wzYp0'
         const.args.youtube_api_key = None
         youtube_tools.set_api_key()
         key = youtube_tools.pafy.g.api_key
-        assert key == expect_key
+        assert key == EXPECTED_YT_API_KEY
 
 
 def test_metadata():
     expect_metadata = None
     global metadata
-    metadata = spotify_tools.generate_metadata(raw_song)
+    metadata = spotify_tools.generate_metadata(TRACK_SEARCH)
     assert metadata == expect_metadata
 
 
@@ -40,15 +49,13 @@ class TestArgsManualResultCount:
     # Regresson test for issue #264
     def test_scrape(self):
         const.args.manual = True
-        url = youtube_tools.GenerateYouTubeURL("she is still sleeping SAO",
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-                                               meta_tags=None)
         video_ids = url.scrape(bestmatch=False)
         # Web scraping gives us all videos on the 1st page
         assert len(video_ids) == 20
 
     def test_api(self):
-        url = youtube_tools.GenerateYouTubeURL("she is still sleeping SAO",
+        url = youtube_tools.GenerateYouTubeURL(RESULT_COUNT_SEARCH, meta_tags=None)
-                                               meta_tags=None)
         video_ids = url.api(bestmatch=False)
         const.args.manual = False
         # API gives us 50 videos (or as requested)
@@ -57,30 +64,26 @@ class TestArgsManualResultCount:
 
 class TestYouTubeURL:
     def test_only_music_category(self):
-        # YouTube keeps changing its results
-        expect_urls = ('http://youtube.com/watch?v=qOOcy2-tmbk',
-                       'http://youtube.com/watch?v=5USR1Omo7f0')
         const.args.music_videos_only = True
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        assert url in expect_urls
+        # YouTube keeps changing its results
+        assert url in EXPECTED_YT_URLS
 
     def test_all_categories(self):
-        expect_url = 'http://youtube.com/watch?v=qOOcy2-tmbk'
         const.args.music_videos_only = False
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        assert url == expect_url
+        assert url == EXPECTED_YT_URL
 
     def test_args_manual(self, monkeypatch):
-        expect_url = 'http://youtube.com/watch?v=qOOcy2-tmbk'
         const.args.manual = True
-        monkeypatch.setattr('builtins.input', lambda x: '1')
+        monkeypatch.setattr("builtins.input", lambda x: "1")
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
-        assert url == expect_url
+        assert url == EXPECTED_YT_URL
 
     def test_args_manual_none(self, monkeypatch):
         expect_url = None
-        monkeypatch.setattr('builtins.input', lambda x: '0')
+        monkeypatch.setattr("builtins.input", lambda x: "0")
-        url = youtube_tools.generate_youtube_url(raw_song, metadata)
+        url = youtube_tools.generate_youtube_url(TRACK_SEARCH, metadata)
         const.args.manual = False
         assert url == expect_url
 
@@ -89,21 +92,18 @@ class TestYouTubeTitle:
     def test_single_download_with_youtube_api(self):
         global content
         global title
-        expect_title = "Tony's Videos VERY SHORT VIDEO 28.10.2016"
+        const.args.youtube_api_key = YT_API_KEY
-        key = 'AIzaSyAnItl3udec-Q1d5bkjKJGL-RgrKO_vU90'
-        const.args.youtube_api_key = key
         youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(raw_song, metadata)
+        content = youtube_tools.go_pafy(TRACK_SEARCH, metadata)
         title = youtube_tools.get_youtube_title(content)
-        assert title == expect_title
+        assert title == EXPECTED_TITLE
 
     def test_download_from_list_without_youtube_api(self):
-        expect_title = "1. Tony's Videos VERY SHORT VIDEO 28.10.2016"
         const.args.youtube_api_key = None
         youtube_tools.set_api_key()
-        content = youtube_tools.go_pafy(raw_song, metadata)
+        content = youtube_tools.go_pafy(TRACK_SEARCH, metadata)
         title = youtube_tools.get_youtube_title(content, 1)
-        assert title == expect_title
+        assert title == "1. {0}".format(EXPECTED_TITLE)
 
 
 def test_check_exists(tmpdir):
@@ -112,7 +112,7 @@ def test_check_exists(tmpdir):
     # prerequisites for determining filename
     global file_name
     file_name = internals.sanitize_title(title)
-    check = spotdl.check_exists(file_name, raw_song, metadata)
+    check = spotdl.check_exists(file_name, TRACK_SEARCH, metadata)
     assert check == expect_check
 
 
@@ -120,10 +120,10 @@ class TestDownload:
     def test_webm(self):
         # content does not have any .webm audiostream
         expect_download = False
-        download = youtube_tools.download_song(file_name + '.webm', content)
+        download = youtube_tools.download_song(file_name + ".webm", content)
         assert download == expect_download
 
     def test_other(self):
         expect_download = False
-        download = youtube_tools.download_song(file_name + '.fake_extension', content)
+        download = youtube_tools.download_song(file_name + ".fake_extension", content)
         assert download == expect_download