mirror of
https://github.com/KevinMidboe/bulk-downloader-for-reddit.git
synced 2026-01-09 18:55:36 +00:00
Compare commits
51 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
49920cc457 | ||
|
|
c70e7c2ebb | ||
|
|
3931dfff54 | ||
|
|
4a8c2377f9 | ||
|
|
8a18a42a9a | ||
|
|
6c2d748fbc | ||
|
|
8c966df105 | ||
|
|
2adf2c0451 | ||
|
|
3e3a2df4d1 | ||
|
|
7548a01019 | ||
|
|
2ab16608d5 | ||
|
|
e15f33b97a | ||
|
|
27211f993c | ||
|
|
87d3b294f7 | ||
|
|
8128378dcd | ||
|
|
cc93aa3012 | ||
|
|
50c4a8d6d7 | ||
|
|
5737904a54 | ||
|
|
f6eba6c5b0 | ||
|
|
41cbb58db3 | ||
|
|
c569124406 | ||
|
|
1a3836a8e1 | ||
|
|
fde6a1fac4 | ||
|
|
6bba2c4dbb | ||
|
|
a078d44236 | ||
|
|
deae0be769 | ||
|
|
3cf0203e6b | ||
|
|
0b31db0e2e | ||
|
|
d3f2b1b08e | ||
|
|
0ec4bb3008 | ||
|
|
0dbe2ed917 | ||
|
|
9f831e1b78 | ||
|
|
59012077e1 | ||
|
|
5e3c79160b | ||
|
|
1e8eaa1a8d | ||
|
|
7dbc83fdce | ||
|
|
50a77f6ba5 | ||
|
|
4f7e406cd6 | ||
|
|
ded3cece8c | ||
|
|
dd671fd738 | ||
|
|
b357dff52c | ||
|
|
32ffd3b861 | ||
|
|
02673c3950 | ||
|
|
8448e47080 | ||
|
|
39f2c73f4c | ||
|
|
fe942b4734 | ||
|
|
205617e051 | ||
|
|
b93b206a96 | ||
|
|
b84684f786 | ||
|
|
68558950ca | ||
|
|
795965f754 |
7
.gitignore
vendored
7
.gitignore
vendored
@@ -1,4 +1,5 @@
|
||||
build/
|
||||
dist/
|
||||
MANIFEST
|
||||
__pycache__/
|
||||
src/__pycache__/
|
||||
logs/
|
||||
*.json
|
||||
src/__pycache__/
|
||||
70
README.md
70
README.md
@@ -3,20 +3,29 @@ This program downloads imgur, gfycat and direct image and video links of saved p
|
||||
|
||||
**PLEASE** post any issue you have with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
||||
|
||||
## What can it do?
|
||||
### It...
|
||||
- can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- sorts posts by hot, top, new and so on
|
||||
- downloads imgur albums, gfycat links, [self posts](#i-cant-open-the-self-posts) and any link to a direct image
|
||||
- skips the existing ones
|
||||
- puts post titles to file's name
|
||||
- puts every post to its subreddit's folder
|
||||
- saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||
- logs failed ones in a file to so that you can try to download them later
|
||||
- can be run with double-clicking on Windows (but I don't recommend it)
|
||||
## What it can do
|
||||
- Can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- Sorts posts by hot, top, new and so on
|
||||
- Downloads imgur albums, gfycat links, [self posts](#how-do-i-open-self-post-files) and any link to a direct image
|
||||
- Skips the existing ones
|
||||
- Puts post titles to file's name
|
||||
- Puts every post to its subreddit's folder
|
||||
- Saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||
- Logs failed ones in a file to so that you can try to download them later
|
||||
- Can run with double-clicking on Windows
|
||||
|
||||
## [Download the latest release](https://github.com/aliparlakci/bulk-downloader-for-reddit/releases/latest)
|
||||
|
||||
## How it works
|
||||
|
||||
- For **Windows** and **Linux** users, there are executable files to run easily without installing a third party program. But if you are a paranoid like me, you can **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
- In Windows, double click on bulk-downloader-for-reddit file
|
||||
- In Linux, extract files to a folder and open terminal inside it. Type **`./bulk-downloader-for-reddit`**
|
||||
|
||||
- **MacOS** users have to **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
|
||||
Script also accepts **command-line arguments**, get further information from **[`--help`](docs/COMMAND_LINE_ARGUMENTS.md)**
|
||||
|
||||
## Setting up the script
|
||||
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
||||
|
||||
@@ -32,20 +41,39 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
|
||||
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||
|
||||
## Running the script
|
||||
|
||||
For **Windows** users, there is an *EXE* file to run easily.
|
||||
|
||||
**Linux** and **MacOS** users have to install Python 3 and run it from the *source code* through terminal.
|
||||
|
||||
To get further information about that and **using command-line arguments to run the script**, see **[`python script.py --help`](docs/help_page.md)**
|
||||
|
||||
## FAQ
|
||||
### I can't open the self post files.
|
||||
### How do I open self post files?
|
||||
- Self posts are held at reddit as styled with markdown. So, the script downloads them as they are in order not to lose their stylings.
|
||||
However, there is a great Chrome extension [here](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with Chrome.
|
||||
However, there is a [great Chrome extension](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with [Chrome](https://www.google.com/intl/tr/chrome/).
|
||||
|
||||
However, they are basically text files. You can also view them with any text editor such as Notepad on Windows, gedit on Linux or Text Editor on MacOS
|
||||
|
||||
### How can I change my credentials?
|
||||
- All of the user data is held in **config.json** file which is in a folder named "Bulk Downloader for Reddit" in your **Home** directory. You can edit
|
||||
them, there.
|
||||
|
||||
## Changelog
|
||||
### [21/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/4a8c2377f9fb4d60ed7eeb8d50aaf9a26492462a)
|
||||
- Added exclude mode
|
||||
|
||||
### [20/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/commit/7548a010198fb693841ca03654d2c9bdf5742139)
|
||||
- "0" input for no limit
|
||||
- Fixed the bug that recognizes none image direct links as image links
|
||||
|
||||
### [19/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/41cbb58db34f500a8a5ecc3ac4375bf6c3b275bb)
|
||||
- Added v.redd.it support
|
||||
- Added custom exception descriptions to FAILED.json file
|
||||
- Fixed the bug that prevents downloading some gfycat URLs
|
||||
|
||||
### [13/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/9f831e1b784a770c82252e909462871401a05c11)
|
||||
- Change config.json file's path to home directory
|
||||
|
||||
### [12/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/50a77f6ba54c24f5647d5ea4e177400b71ff04a7)
|
||||
- Added binaries for Windows and Linux
|
||||
- Wait on KeyboardInterrupt
|
||||
- Accept multiple subreddit input
|
||||
- Fixed the bug that prevents choosing "[0] exit" with typing "exit"
|
||||
|
||||
### [11/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a28a7776ab826dea2a8d93873a94cd46db3a339b)
|
||||
- Improvements on UX and UI
|
||||
- Added logging errors to CONSOLE_LOG.txt
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
theme: jekyll-theme-minimal
|
||||
show_downloads: false
|
||||
#title: Bulk Downloader for Reddit
|
||||
description: Code written by Ali PARLAKCI
|
||||
google_analytics: UA-80780721-3
|
||||
101
docs/COMMAND_LINE_ARGUMENTS.md
Normal file
101
docs/COMMAND_LINE_ARGUMENTS.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Using command-line arguments
|
||||
|
||||
See **[compiling from source](COMPILE_FROM_SOURCE.md)** page first unless you are using an executable file. If you are using an executable file, see [using terminal](COMPILE_FROM_SOURCE.md#using-terminal) and come back.
|
||||
|
||||
***Use*** `.\bulk-downloader-for-reddit.exe` ***or*** `./bulk-downloader-for-reddit` ***if you are using the executable***.
|
||||
```console
|
||||
$ python script.py --help
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||
[--submitted] [--upvoted] [--log LOG FILE]
|
||||
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
[--multireddit MULTIREDDIT] [--user redditor]
|
||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||
[--time TIME_LIMIT] [--NoDownload]
|
||||
|
||||
This program downloads media from reddit posts
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--directory DIRECTORY
|
||||
Specifies the directory where posts will be downloaded
|
||||
to
|
||||
--link link, -l link Get posts from link
|
||||
--saved Triggers saved mode
|
||||
--submitted Gets posts of --user
|
||||
--upvoted Gets upvoted posts of --user
|
||||
--log LOG FILE Takes a log file which created by itself (json files),
|
||||
reads posts and tries downloading them again.
|
||||
--subreddit SUBREDDIT [SUBREDDIT ...]
|
||||
Triggers subreddit mode and takes subreddit's name
|
||||
without r/. use "frontpage" for frontpage
|
||||
--multireddit MULTIREDDIT
|
||||
Triggers multireddit mode and takes multireddit's name
|
||||
without m/
|
||||
--user redditor reddit username if needed. use "me" for current user
|
||||
--search query Searches for given query in given subreddits
|
||||
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
||||
relevance default: hot
|
||||
--limit Limit default: unlimited
|
||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||
all
|
||||
--NoDownload Just gets the posts and store them in a file for
|
||||
downloading later
|
||||
--exclude {imgur,gfycat,direct,self} [{imgur,gfycat,direct,self} ...]
|
||||
Do not download specified links
|
||||
```
|
||||
|
||||
# Examples
|
||||
|
||||
- **Use `python3` instead of `python` if you are using *MacOS* or *Linux***
|
||||
|
||||
```console
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
.\bulk-downloader-for-reddit.exe
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
.\bulk-downloader-for-reddit.exe -- directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
./bulk-downloader-for-reddit --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
```
|
||||
|
||||
# FAQ
|
||||
## I can't startup the script no matter what.
|
||||
See **[finding the correct keyword for Python](COMPILE_FROM_SOURCE.md#finding-the-correct-keyword-for-python)**
|
||||
42
docs/COMPILE_FROM_SOURCE.md
Normal file
42
docs/COMPILE_FROM_SOURCE.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Compiling from source code
|
||||
## Requirements
|
||||
### Python 3 Interpreter
|
||||
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option is mandatory.
|
||||
|
||||
\* *Use Python 3.6.5 if you encounter an issue*
|
||||
## Using terminal
|
||||
### To open it...
|
||||
- **On Windows 8/8.1/10**: Press the File tab on **Windows Explorer**, click on **Open Windows PowerShell** or **Open Windows Command Prompt** or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Windows 7**: Press **WindowsKey+R**, type **cmd** and hit Enter or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T** or look for **Terminal** in the programs.
|
||||
|
||||
- **On MacOS**: Look for an app called **Terminal**.
|
||||
|
||||
### Navigating to the directory where script is downloaded
|
||||
Go inside the folder where script.py is located. If you are not familiar with changing directories on command-prompt and terminal read *Changing Directories* in [this article](https://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything)
|
||||
|
||||
## Finding the correct keyword for Python
|
||||
Enter these lines to terminal window until it prints out the version you have downloaded and installed:
|
||||
|
||||
- `python --version`
|
||||
- `python3 --version`
|
||||
- `python3.7 --version`
|
||||
- `python3.6 --version`
|
||||
- `py --version`
|
||||
- `py -3 --version`
|
||||
- `py -3.6 --version`
|
||||
- `py -3.7 --version`
|
||||
|
||||
Once it does, your keyword is without the `--version` part.
|
||||
|
||||
## Installing dependencies
|
||||
Enter the line below to terminal window when you are in the directory where script.py is, use your keyword for Python:
|
||||
```console
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
|
||||
@@ -1,85 +0,0 @@
|
||||
## python script.py --help
|
||||
|
||||
```console
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||
[--submitted] [--upvoted] [--log LOG FILE]
|
||||
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
[--multireddit MULTIREDDIT] [--user redditor]
|
||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||
[--time TIME_LIMIT] [--NoDownload]
|
||||
|
||||
This program downloads media from reddit posts
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--directory DIRECTORY
|
||||
Specifies the directory where posts will be downloaded
|
||||
to
|
||||
--link link, -l link Get posts from link
|
||||
--saved Triggers saved mode
|
||||
--submitted Gets posts of --user
|
||||
--upvoted Gets upvoted posts of --user
|
||||
--log LOG FILE Triggers log read mode and takes a log file
|
||||
--subreddit SUBREDDIT [SUBREDDIT ...]
|
||||
Triggers subreddit mode and takes subreddit's name
|
||||
without r/. use "frontpage" for frontpage
|
||||
--multireddit MULTIREDDIT
|
||||
Triggers multireddit mode and takes multireddit's name
|
||||
without m/
|
||||
--user redditor reddit username if needed. use "me" for current user
|
||||
--search query Searches for given query in given subreddits
|
||||
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
||||
relevance default: hot
|
||||
--limit Limit default: unlimited
|
||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||
all
|
||||
--NoDownload Just gets the posts and store them in a file for
|
||||
downloading later
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
- **Use `python3` instead of `python` if you are using *MacOS* or *Linux***
|
||||
|
||||
```console
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
```
|
||||
|
||||
## FAQ
|
||||
### I can't startup the script no matter what.
|
||||
- Try these:
|
||||
- **`python`**
|
||||
- **`python3`**
|
||||
- **`python3.7`**
|
||||
- **`python3.6`**
|
||||
- **`py -3`**
|
||||
|
||||
Python have real issues about naming their program
|
||||
107
script.py
107
script.py
@@ -22,7 +22,7 @@ from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
||||
|
||||
__author__ = "Ali Parlakci"
|
||||
__license__ = "GPL"
|
||||
__version__ = "1.1.0"
|
||||
__version__ = "1.3.0"
|
||||
__maintainer__ = "Ali Parlakci"
|
||||
__email__ = "parlakciali@gmail.com"
|
||||
|
||||
@@ -143,6 +143,12 @@ def parseArguments(arguments=[]):
|
||||
" for downloading later",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
parser.add_argument("--exclude",
|
||||
nargs="+",
|
||||
help="Do not download specified links",
|
||||
choices=["imgur","gfycat","direct","self"],
|
||||
type=str)
|
||||
|
||||
if arguments == []:
|
||||
return parser.parse_args()
|
||||
@@ -194,12 +200,12 @@ class PromptUser:
|
||||
))
|
||||
print(" "*4+"[0] exit\n")
|
||||
choice = input("> ")
|
||||
while not choice.lower() in choices+choicesByIndex:
|
||||
while not choice.lower() in choices+choicesByIndex+["exit"]:
|
||||
print("Invalid input\n")
|
||||
programModeIndex = input("> ")
|
||||
|
||||
if choice == "0":
|
||||
quit()
|
||||
if choice == "0" or choice == "exit":
|
||||
sys.exit()
|
||||
elif choice in choicesByIndex:
|
||||
return choices[int(choice)-1]
|
||||
else:
|
||||
@@ -232,10 +238,20 @@ class PromptUser:
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
|
||||
if programMode == "subreddit":
|
||||
GLOBAL.arguments.subreddit = input("\nsubreddit: ")
|
||||
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit = subredditInput
|
||||
|
||||
while not subredditInput == "":
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit += "+" + subredditInput
|
||||
|
||||
if " " in GLOBAL.arguments.subreddit:
|
||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit.split())
|
||||
|
||||
# DELETE THE PLUS (+) AT THE END
|
||||
GLOBAL.arguments.subreddit = GLOBAL.arguments.subreddit[:-1]
|
||||
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"hot","top","new","rising","controversial"
|
||||
@@ -308,9 +324,37 @@ class PromptUser:
|
||||
if Path(GLOBAL.arguments.log ).is_file():
|
||||
break
|
||||
|
||||
GLOBAL.arguments.exclude = []
|
||||
|
||||
sites = ["imgur","gfycat","direct","self"]
|
||||
|
||||
excludeInput = input("exclude: ").lower()
|
||||
if excludeInput in sites and excludeInput != "":
|
||||
GLOBAL.arguments.exclude = [excludeInput]
|
||||
|
||||
while not excludeInput == "":
|
||||
while True:
|
||||
excludeInput = input("exclude: ").lower()
|
||||
if not excludeInput in sites or excludeInput in GLOBAL.arguments.exclude:
|
||||
break
|
||||
elif excludeInput == "":
|
||||
break
|
||||
else:
|
||||
GLOBAL.arguments.exclude.append(excludeInput)
|
||||
|
||||
for i in range(len(GLOBAL.arguments.exclude)):
|
||||
if " " in GLOBAL.arguments.exclude[i]:
|
||||
inputWithWhitespace = GLOBAL.arguments.exclude[i]
|
||||
del GLOBAL.arguments.exclude[i]
|
||||
for siteInput in inputWithWhitespace.split():
|
||||
if siteInput in sites and siteInput not in GLOBAL.arguments.exclude:
|
||||
GLOBAL.arguments.exclude.append(siteInput)
|
||||
|
||||
while True:
|
||||
try:
|
||||
GLOBAL.arguments.limit = int(input("\nlimit: "))
|
||||
GLOBAL.arguments.limit = int(input("\nlimit (0 for none): "))
|
||||
if GLOBAL.arguments.limit == 0:
|
||||
GLOBAL.arguments.limit = None
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
@@ -389,7 +433,7 @@ def postFromLog(fileName):
|
||||
content = jsonFile(fileName).read()
|
||||
else:
|
||||
print("File not found")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
del content["HEADER"]
|
||||
@@ -432,6 +476,10 @@ def download(submissions):
|
||||
downloadedCount = subsLenght
|
||||
duplicates = 0
|
||||
BACKUP = {}
|
||||
if GLOBAL.arguments.exclude is not None:
|
||||
ToBeDownloaded = GLOBAL.arguments.exclude
|
||||
else:
|
||||
ToBeDownloaded = []
|
||||
|
||||
FAILED_FILE = createLogFile("FAILED")
|
||||
|
||||
@@ -454,7 +502,7 @@ def download(submissions):
|
||||
|
||||
directory = GLOBAL.directory / submissions[i]['postSubreddit']
|
||||
|
||||
if submissions[i]['postType'] == 'imgur':
|
||||
if submissions[i]['postType'] == 'imgur' and not 'imgur' in ToBeDownloaded:
|
||||
print("IMGUR",end="")
|
||||
|
||||
while int(time.time() - lastRequestTime) <= 2:
|
||||
@@ -497,7 +545,7 @@ def download(submissions):
|
||||
"Imgur login failed. Quitting the program "\
|
||||
"as unexpected errors might occur."
|
||||
)
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
except Exception as exception:
|
||||
print(exception)
|
||||
@@ -517,7 +565,7 @@ def download(submissions):
|
||||
)
|
||||
downloadedCount -= 1
|
||||
|
||||
elif submissions[i]['postType'] == 'gfycat':
|
||||
elif submissions[i]['postType'] == 'gfycat' and not 'gfycat' in ToBeDownloaded:
|
||||
print("GFYCAT")
|
||||
try:
|
||||
Gfycat(directory,submissions[i])
|
||||
@@ -528,7 +576,7 @@ def download(submissions):
|
||||
downloadedCount -= 1
|
||||
|
||||
except NotADownloadableLinkError as exception:
|
||||
print("Could not read the page source")
|
||||
print(exception)
|
||||
FAILED_FILE.add({int(i+1):[str(exception),submissions[i]]})
|
||||
downloadedCount -= 1
|
||||
|
||||
@@ -537,7 +585,7 @@ def download(submissions):
|
||||
FAILED_FILE.add({int(i+1):[str(exception),submissions[i]]})
|
||||
downloadedCount -= 1
|
||||
|
||||
elif submissions[i]['postType'] == 'direct':
|
||||
elif submissions[i]['postType'] == 'direct' and not 'direct' in ToBeDownloaded:
|
||||
print("DIRECT")
|
||||
try:
|
||||
Direct(directory,submissions[i])
|
||||
@@ -552,7 +600,7 @@ def download(submissions):
|
||||
FAILED_FILE.add({int(i+1):[str(exception),submissions[i]]})
|
||||
downloadedCount -= 1
|
||||
|
||||
elif submissions[i]['postType'] == 'self':
|
||||
elif submissions[i]['postType'] == 'self' and not 'self' in ToBeDownloaded:
|
||||
print("SELF")
|
||||
try:
|
||||
Self(directory,submissions[i])
|
||||
@@ -596,46 +644,47 @@ def main():
|
||||
PromptUser()
|
||||
except Exception as err:
|
||||
print(err)
|
||||
quit()
|
||||
|
||||
GLOBAL.config = getConfig("config.json")
|
||||
sys.exit()
|
||||
|
||||
if not Path(GLOBAL.configDirectory).is_dir():
|
||||
os.makedirs(GLOBAL.configDirectory)
|
||||
GLOBAL.config = getConfig(GLOBAL.configDirectory / "config.json")
|
||||
|
||||
if GLOBAL.arguments.log is not None:
|
||||
logDir = Path(GLOBAL.arguments.log)
|
||||
download(postFromLog(logDir))
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
POSTS = getPosts(prepareAttributes())
|
||||
except InsufficientPermission:
|
||||
print("You do not have permission to do that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoMatchingSubmissionFound:
|
||||
print("No matching submission was found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoRedditSupoort:
|
||||
print("Reddit does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoPrawSupport:
|
||||
print("PRAW does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except MultiredditNotFound:
|
||||
print("Multireddit not found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidSortingType:
|
||||
print("Invalid sorting type has given")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidRedditLink:
|
||||
print("Invalid reddit link")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if POSTS is None:
|
||||
print("I could not find any posts in that URL")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if GLOBAL.arguments.NoDownload:
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
else:
|
||||
download(POSTS)
|
||||
@@ -654,9 +703,11 @@ if __name__ == "__main__":
|
||||
if GLOBAL.directory is None:
|
||||
GLOBAL.directory = Path(".\\")
|
||||
print("\nQUITTING...")
|
||||
quit()
|
||||
except Exception as exception:
|
||||
logging.error("Runtime error!", exc_info=full_exc_info(sys.exc_info()))
|
||||
if GLOBAL.directory is None:
|
||||
GLOBAL.directory = Path(".\\")
|
||||
logging.error(sys.exc_info()[0].__name__,
|
||||
exc_info=full_exc_info(sys.exc_info()))
|
||||
print(log_stream.getvalue())
|
||||
|
||||
input("Press enter to quit\n")
|
||||
|
||||
50
setup.py
Normal file
50
setup.py
Normal file
@@ -0,0 +1,50 @@
|
||||
#!C:\Users\Ali\AppData\Local\Programs\Python\Python36\python.exe
|
||||
|
||||
## python setup.py build
|
||||
import sys
|
||||
from cx_Freeze import setup, Executable
|
||||
from script import __version__
|
||||
|
||||
options = {
|
||||
"build_exe": {
|
||||
"packages":[
|
||||
"idna","imgurpython", "praw", "requests"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
if sys.platform == "win32":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit.exe",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
elif sys.platform == "linux":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
setup(
|
||||
name = "Bulk Downloader for Reddit",
|
||||
version = __version__,
|
||||
description = "Bulk Downloader for Reddit",
|
||||
author = "Ali Parlakci",
|
||||
author_email="parlakciali@gmail.com",
|
||||
url="https://github.com/aliparlakci/bulk-downloader-for-reddit",
|
||||
classifiers=(
|
||||
"Programming Language :: Python :: 3",
|
||||
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)"
|
||||
"Natural Language :: English",
|
||||
"Environment :: Console",
|
||||
"Operating System :: OS Independent",
|
||||
),
|
||||
executables = executables,
|
||||
options = options
|
||||
)
|
||||
|
||||
|
||||
@@ -36,7 +36,10 @@ def getExtension(link):
|
||||
if TYPE in parsed:
|
||||
return "."+parsed[-1]
|
||||
else:
|
||||
return '.jpg'
|
||||
if not "v.redd.it" in link:
|
||||
return '.jpg'
|
||||
else:
|
||||
return '.mp4'
|
||||
|
||||
def getFile(fileDir,tempDir,imageURL,indent=0):
|
||||
"""Downloads given file to given directory.
|
||||
@@ -169,7 +172,9 @@ class Imgur:
|
||||
if duplicates == imagesLenght:
|
||||
raise FileAlreadyExistsError
|
||||
elif howManyDownloaded < imagesLenght:
|
||||
raise AlbumNotDownloadedCompletely
|
||||
raise AlbumNotDownloadedCompletely(
|
||||
"Album Not Downloaded Completely"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def initImgur():
|
||||
@@ -217,9 +222,9 @@ class Gfycat:
|
||||
try:
|
||||
POST['mediaURL'] = self.getLink(POST['postURL'])
|
||||
except IndexError:
|
||||
raise NotADownloadableLinkError
|
||||
raise NotADownloadableLinkError("Could not read the page source")
|
||||
except Exception as exception:
|
||||
raise NotADownloadableLinkError
|
||||
raise NotADownloadableLinkError("Could not read the page source")
|
||||
|
||||
POST['postExt'] = getExtension(POST['mediaURL'])
|
||||
|
||||
@@ -248,8 +253,7 @@ class Gfycat:
|
||||
if url[-1:] == '/':
|
||||
url = url[:-1]
|
||||
|
||||
if 'gifs' in url:
|
||||
url = "https://gfycat.com/" + url.split('/')[-1]
|
||||
url = "https://gfycat.com/" + url.split('/')[-1]
|
||||
|
||||
pageSource = (urllib.request.urlopen(url).read().decode().split('\n'))
|
||||
|
||||
@@ -266,7 +270,7 @@ class Gfycat:
|
||||
break
|
||||
|
||||
if "".join(link) == "":
|
||||
raise NotADownloadableLinkError
|
||||
raise NotADownloadableLinkError("Could not read the page source")
|
||||
|
||||
return "".join(link)
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
else:
|
||||
@@ -98,7 +98,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
return reddit
|
||||
@@ -397,8 +397,9 @@ def checkIfMatching(submission):
|
||||
imgurCount += 1
|
||||
return details
|
||||
|
||||
elif isDirectLink(submission.url):
|
||||
elif isDirectLink(submission.url) is not False:
|
||||
details['postType'] = 'direct'
|
||||
details['postURL'] = isDirectLink(submission.url)
|
||||
directCount += 1
|
||||
return details
|
||||
|
||||
@@ -435,7 +436,7 @@ def printSubmission(SUB,validNumber,totalNumber):
|
||||
|
||||
def isDirectLink(URL):
|
||||
"""Check if link is a direct image link.
|
||||
If so, return True,
|
||||
If so, return URL,
|
||||
if not, return False
|
||||
"""
|
||||
|
||||
@@ -444,10 +445,13 @@ def isDirectLink(URL):
|
||||
URL = URL[:-1]
|
||||
|
||||
if "i.reddituploads.com" in URL:
|
||||
return True
|
||||
return URL
|
||||
|
||||
elif "v.redd.it" in URL:
|
||||
return URL+"/DASH_600_K"
|
||||
|
||||
for extension in imageTypes:
|
||||
if extension in URL:
|
||||
return True
|
||||
return URL
|
||||
else:
|
||||
return False
|
||||
|
||||
@@ -14,6 +14,7 @@ class GLOBAL:
|
||||
config = None
|
||||
arguments = None
|
||||
directory = None
|
||||
configDirectory = Path.home() / "Bulk Downloader for Reddit"
|
||||
reddit_client_id = "BSyphDdxYZAgVQ"
|
||||
reddit_client_secret = "bfqNJaRh8NMh-9eAr-t4TRz-Blk"
|
||||
printVanilla = print
|
||||
|
||||
Reference in New Issue
Block a user