mirror of
https://github.com/KevinMidboe/bulk-downloader-for-reddit.git
synced 2026-01-09 18:55:36 +00:00
Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0ec4bb3008 | ||
|
|
0dbe2ed917 | ||
|
|
9f831e1b78 | ||
|
|
59012077e1 | ||
|
|
5e3c79160b | ||
|
|
1e8eaa1a8d | ||
|
|
7dbc83fdce | ||
|
|
50a77f6ba5 | ||
|
|
4f7e406cd6 | ||
|
|
ded3cece8c | ||
|
|
dd671fd738 | ||
|
|
b357dff52c | ||
|
|
32ffd3b861 | ||
|
|
02673c3950 | ||
|
|
8448e47080 | ||
|
|
39f2c73f4c | ||
|
|
fe942b4734 | ||
|
|
205617e051 | ||
|
|
b93b206a96 | ||
|
|
b84684f786 | ||
|
|
68558950ca | ||
|
|
795965f754 |
7
.gitignore
vendored
7
.gitignore
vendored
@@ -1,4 +1,5 @@
|
||||
build/
|
||||
dist/
|
||||
MANIFEST
|
||||
__pycache__/
|
||||
src/__pycache__/
|
||||
logs/
|
||||
*.json
|
||||
src/__pycache__/
|
||||
50
README.md
50
README.md
@@ -3,20 +3,29 @@ This program downloads imgur, gfycat and direct image and video links of saved p
|
||||
|
||||
**PLEASE** post any issue you have with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
||||
|
||||
## What can it do?
|
||||
### It...
|
||||
- can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- sorts posts by hot, top, new and so on
|
||||
- downloads imgur albums, gfycat links, [self posts](#i-cant-open-the-self-posts) and any link to a direct image
|
||||
- skips the existing ones
|
||||
- puts post titles to file's name
|
||||
- puts every post to its subreddit's folder
|
||||
- saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||
- logs failed ones in a file to so that you can try to download them later
|
||||
- can be run with double-clicking on Windows (but I don't recommend it)
|
||||
## What it can do
|
||||
- Can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- Sorts posts by hot, top, new and so on
|
||||
- Downloads imgur albums, gfycat links, [self posts](#i-cant-open-the-self-post-files) and any link to a direct image
|
||||
- Skips the existing ones
|
||||
- Puts post titles to file's name
|
||||
- Puts every post to its subreddit's folder
|
||||
- Saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||
- Logs failed ones in a file to so that you can try to download them later
|
||||
- Can run with double-clicking on Windows
|
||||
|
||||
## [Download the latest release](https://github.com/aliparlakci/bulk-downloader-for-reddit/releases/latest)
|
||||
|
||||
## How it works
|
||||
|
||||
- For **Windows** and **Linux** users, there are executable files to run easily without installing a third party program. But if you are a paranoid like me, you can **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
- In Windows, double click on script.exe file
|
||||
- In Linux, extract files to a folder and open terminal inside it. Type **`./script`**
|
||||
|
||||
- **MacOS** users have to **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
|
||||
Script also accepts **command-line arguments**, get further information from **[`python script.py --help`](docs/COMMAND_LINE_ARGUMENTS.md)**
|
||||
|
||||
## Setting up the script
|
||||
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
||||
|
||||
@@ -32,20 +41,21 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
|
||||
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||
|
||||
## Running the script
|
||||
|
||||
For **Windows** users, there is an *EXE* file to run easily.
|
||||
|
||||
**Linux** and **MacOS** users have to install Python 3 and run it from the *source code* through terminal.
|
||||
|
||||
To get further information about that and **using command-line arguments to run the script**, see **[`python script.py --help`](docs/help_page.md)**
|
||||
|
||||
## FAQ
|
||||
### I can't open the self post files.
|
||||
- Self posts are held at reddit as styled with markdown. So, the script downloads them as they are in order not to lose their stylings.
|
||||
However, there is a great Chrome extension [here](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with Chrome.
|
||||
However, there is a [great Chrome extension](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with [Chrome](https://www.google.com/intl/tr/chrome/).
|
||||
|
||||
## Changelog
|
||||
### [13/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/9f831e1b784a770c82252e909462871401a05c11)
|
||||
- Change config.json file's path to home directory
|
||||
|
||||
### [12/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/50a77f6ba54c24f5647d5ea4e177400b71ff04a7)
|
||||
- Added binaries for Windows and Linux
|
||||
- Wait on KeyboardInterrupt
|
||||
- Accept multiple subreddit input
|
||||
- Fixed the bug that prevents choosing "[0] exit" with typing "exit"
|
||||
|
||||
### [11/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a28a7776ab826dea2a8d93873a94cd46db3a339b)
|
||||
- Improvements on UX and UI
|
||||
- Added logging errors to CONSOLE_LOG.txt
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
## python script.py --help
|
||||
# Using command-line arguments
|
||||
|
||||
See **[compiling from source](COMPILE_FROM_SOURCE.md)** page first unless you are using an executable file. If you are using an executable file, see [using terminal](COMPILE_FROM_SOURCE.md#using-terminal) and come back.
|
||||
|
||||
***Use*** `.\script.exe` ***or*** `./script` ***if you are using the executable***.
|
||||
```console
|
||||
$ python script.py --help
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||
[--submitted] [--upvoted] [--log LOG FILE]
|
||||
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
@@ -37,7 +41,7 @@ optional arguments:
|
||||
downloading later
|
||||
```
|
||||
|
||||
## Examples
|
||||
# Examples
|
||||
|
||||
- **Use `python3` instead of `python` if you are using *MacOS* or *Linux***
|
||||
|
||||
@@ -46,40 +50,49 @@ python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
.\script.exe
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
.\script.exe -- directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
./script --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
python script.py --directory .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
python script.py --directory .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
python script.py --directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
## FAQ
|
||||
### I can't startup the script no matter what.
|
||||
- Try these:
|
||||
- **`python`**
|
||||
- **`python3`**
|
||||
- **`python3.7`**
|
||||
- **`python3.6`**
|
||||
- **`py -3`**
|
||||
|
||||
Python have real issues about naming their program
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
```
|
||||
|
||||
# FAQ
|
||||
## I can't startup the script no matter what.
|
||||
See **[finding the correct keyword for Python](COMPILE_FROM_SOURCE.md#finding-the-correct-keyword-for-python)**
|
||||
42
docs/COMPILE_FROM_SOURCE.md
Normal file
42
docs/COMPILE_FROM_SOURCE.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Compiling from source code
|
||||
## Requirements
|
||||
### Python 3 Interpreter
|
||||
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option is mandatory.
|
||||
|
||||
\* *Use Python 3.6.5 if you encounter an issue*
|
||||
## Using terminal
|
||||
### To open it...
|
||||
- **On Windows 8/8.1/10**: Press the File tab on **Windows Explorer**, click on **Open Windows PowerShell** or **Open Windows Command Prompt** or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Windows 7**: Press **WindowsKey+R**, type **cmd** and hit Enter or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T** or look for **Terminal** in the programs.
|
||||
|
||||
- **On MacOS**: Look for an app called **Terminal**.
|
||||
|
||||
### Navigating to the directory where script is downloaded
|
||||
Go inside the folder where script.py is located. If you are not familier with changing directories on command-prompt and terminal read *Changing Directories* in [this article](https://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything)
|
||||
|
||||
## Finding the correct keyword for Python
|
||||
Enter these lines to terminal window until it prints out the version you have downloaded and installed:
|
||||
|
||||
- `python --version`
|
||||
- `python3 --version`
|
||||
- `python3.7 --version`
|
||||
- `python3.6 --version`
|
||||
- `py --version`
|
||||
- `py -3 --version`
|
||||
- `py -3.6 --version`
|
||||
- `py -3.7 --version`
|
||||
|
||||
Once it does, your keyword is without the `--version` part.
|
||||
|
||||
## Installing dependencies
|
||||
Enter the line below to terminal window when you are in the directory where script.py is, use your keyword for Python:
|
||||
```console
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
|
||||
53
script.py
53
script.py
@@ -22,7 +22,7 @@ from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
||||
|
||||
__author__ = "Ali Parlakci"
|
||||
__license__ = "GPL"
|
||||
__version__ = "1.1.0"
|
||||
__version__ = "1.1.2"
|
||||
__maintainer__ = "Ali Parlakci"
|
||||
__email__ = "parlakciali@gmail.com"
|
||||
|
||||
@@ -194,12 +194,12 @@ class PromptUser:
|
||||
))
|
||||
print(" "*4+"[0] exit\n")
|
||||
choice = input("> ")
|
||||
while not choice.lower() in choices+choicesByIndex:
|
||||
while not choice.lower() in choices+choicesByIndex+["exit"]:
|
||||
print("Invalid input\n")
|
||||
programModeIndex = input("> ")
|
||||
|
||||
if choice == "0":
|
||||
quit()
|
||||
if choice == "0" or choice == "exit":
|
||||
sys.exit()
|
||||
elif choice in choicesByIndex:
|
||||
return choices[int(choice)-1]
|
||||
else:
|
||||
@@ -232,10 +232,21 @@ class PromptUser:
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
|
||||
if programMode == "subreddit":
|
||||
GLOBAL.arguments.subreddit = input("\nsubreddit: ")
|
||||
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit = subredditInput
|
||||
|
||||
while not subredditInput == "":
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit += "+" + subredditInput
|
||||
|
||||
if " " in GLOBAL.arguments.subreddit:
|
||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit.split())
|
||||
|
||||
# DELETE THE PLUS (+) AT THE END
|
||||
GLOBAL.arguments.subreddit = GLOBAL.arguments.subreddit[:-1]
|
||||
|
||||
print(GLOBAL.arguments.subreddit)
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"hot","top","new","rising","controversial"
|
||||
@@ -389,7 +400,7 @@ def postFromLog(fileName):
|
||||
content = jsonFile(fileName).read()
|
||||
else:
|
||||
print("File not found")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
del content["HEADER"]
|
||||
@@ -497,7 +508,7 @@ def download(submissions):
|
||||
"Imgur login failed. Quitting the program "\
|
||||
"as unexpected errors might occur."
|
||||
)
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
except Exception as exception:
|
||||
print(exception)
|
||||
@@ -596,46 +607,47 @@ def main():
|
||||
PromptUser()
|
||||
except Exception as err:
|
||||
print(err)
|
||||
quit()
|
||||
|
||||
GLOBAL.config = getConfig("config.json")
|
||||
sys.exit()
|
||||
|
||||
if not Path(GLOBAL.configDirectory).is_dir():
|
||||
os.makedirs(GLOBAL.configDirectory)
|
||||
GLOBAL.config = getConfig(GLOBAL.configDirectory / "config.json")
|
||||
|
||||
if GLOBAL.arguments.log is not None:
|
||||
logDir = Path(GLOBAL.arguments.log)
|
||||
download(postFromLog(logDir))
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
POSTS = getPosts(prepareAttributes())
|
||||
except InsufficientPermission:
|
||||
print("You do not have permission to do that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoMatchingSubmissionFound:
|
||||
print("No matching submission was found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoRedditSupoort:
|
||||
print("Reddit does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoPrawSupport:
|
||||
print("PRAW does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except MultiredditNotFound:
|
||||
print("Multireddit not found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidSortingType:
|
||||
print("Invalid sorting type has given")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidRedditLink:
|
||||
print("Invalid reddit link")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if POSTS is None:
|
||||
print("I could not find any posts in that URL")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if GLOBAL.arguments.NoDownload:
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
else:
|
||||
download(POSTS)
|
||||
@@ -654,7 +666,6 @@ if __name__ == "__main__":
|
||||
if GLOBAL.directory is None:
|
||||
GLOBAL.directory = Path(".\\")
|
||||
print("\nQUITTING...")
|
||||
quit()
|
||||
except Exception as exception:
|
||||
logging.error("Runtime error!", exc_info=full_exc_info(sys.exc_info()))
|
||||
print(log_stream.getvalue())
|
||||
|
||||
50
setup.py
Normal file
50
setup.py
Normal file
@@ -0,0 +1,50 @@
|
||||
#!C:\Users\Ali\AppData\Local\Programs\Python\Python36\python.exe
|
||||
|
||||
## python setup.py build
|
||||
import sys
|
||||
from cx_Freeze import setup, Executable
|
||||
from script import __version__
|
||||
|
||||
options = {
|
||||
"build_exe": {
|
||||
"packages":[
|
||||
"idna","imgurpython", "praw", "requests"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
if sys.platform == "win32":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit.exe",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
elif sys.platform == "linux":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
setup(
|
||||
name = "Bulk Downloader for Reddit",
|
||||
version = __version__,
|
||||
description = "Bulk Downloader for Reddit",
|
||||
author = "Ali Parlakci",
|
||||
author_email="parlakciali@gmail.com",
|
||||
url="https://github.com/aliparlakci/bulk-downloader-for-reddit",
|
||||
classifiers=(
|
||||
"Programming Language :: Python :: 3",
|
||||
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)"
|
||||
"Natural Language :: English",
|
||||
"Environment :: Console",
|
||||
"Operating System :: OS Independent",
|
||||
),
|
||||
executables = executables,
|
||||
options = options
|
||||
)
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
else:
|
||||
@@ -98,7 +98,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
return reddit
|
||||
|
||||
@@ -14,6 +14,7 @@ class GLOBAL:
|
||||
config = None
|
||||
arguments = None
|
||||
directory = None
|
||||
configDirectory = Path.home() / "Bulk Downloader for Reddit"
|
||||
reddit_client_id = "BSyphDdxYZAgVQ"
|
||||
reddit_client_secret = "bfqNJaRh8NMh-9eAr-t4TRz-Blk"
|
||||
printVanilla = print
|
||||
|
||||
Reference in New Issue
Block a user