mirror of
https://github.com/KevinMidboe/bulk-downloader-for-reddit.git
synced 2026-01-10 19:25:41 +00:00
Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d685860c22 | ||
|
|
dcf9f35273 | ||
|
|
7fdf03aa24 | ||
|
|
25d61a4c78 | ||
|
|
558eb107f4 | ||
|
|
6e74630050 | ||
|
|
2fd9248715 | ||
|
|
457b8cd21c | ||
|
|
e953456ead | ||
|
|
ed0564fba0 | ||
|
|
5378555f74 | ||
|
|
95ef308915 | ||
|
|
436f867f2e |
28
README.md
28
README.md
@@ -23,7 +23,8 @@ This program downloads imgur, gfycat and direct image and video links of saved p
|
||||
|
||||
- **MacOS** users have to **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
|
||||
Script also accepts **command-line arguments**, get further information from **[`--help`](docs/COMMAND_LINE_ARGUMENTS.md)**
|
||||
### Additional options
|
||||
Script also accepts additional options via command-line arguments, get further information from **[`--help`](docs/COMMAND_LINE_ARGUMENTS.md)**
|
||||
|
||||
## Setting up the script
|
||||
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
||||
@@ -36,9 +37,9 @@ Because this is not a commercial app, you need to create an imgur developer app
|
||||
* Correct CHAPTCHA
|
||||
* Click **submit** button
|
||||
|
||||
It should redirect to a page which shows your **imgur_client_id** and **imgur_client_secret**
|
||||
It should redirect you to a page which shows your **imgur_client_id** and **imgur_client_secret**
|
||||
|
||||
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*. If this does not work, it is safe to proceed with **OAuth 2 authorization without a callback URL**.
|
||||
|
||||
## FAQ
|
||||
### What do the dots resemble when getting posts?
|
||||
@@ -48,7 +49,7 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
- You can press Ctrl+C to interrupt it and start downloading.
|
||||
|
||||
### How downloaded files' names are formatted?
|
||||
- Images that are not belong to an album or self posts are formatted as **`[SUBMITTER NAME]_[POST TITLE]_[REDDIT ID]`**.
|
||||
- Self posts and images that are not belong to an album are formatted as **`[SUBMITTER NAME]_[POST TITLE]_[REDDIT ID]`**.
|
||||
You can use *reddit id* to go to post's reddit page by going to link **reddit.com/[REDDIT ID]**
|
||||
|
||||
- An image in an imgur album is formatted as **`[ITEM NUMBER]_[IMAGE TITLE]_[IMGUR ID]`**
|
||||
@@ -65,9 +66,14 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
them, there.
|
||||
|
||||
## Changes on *master*
|
||||
### [26/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/1623722138bad80ae39ffcd5fb38baf80680deac)
|
||||
- Improved verbose mode
|
||||
- Minimalized the console output
|
||||
- Added quit option for auto quitting the program after process finished
|
||||
|
||||
### [25/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/1623722138bad80ae39ffcd5fb38baf80680deac)
|
||||
- Added verbose mode
|
||||
- Stylize the console output
|
||||
- Stylized the console output
|
||||
|
||||
### [24/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7a68ff3efac9939f9574c2cef6184b92edb135f4)
|
||||
- Added OP's name to file names (backwards compatible)
|
||||
@@ -75,19 +81,19 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
- Improved exception handling
|
||||
|
||||
### [23/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7314e17125aa78fd4e6b28e26fda7ec7db7e0147)
|
||||
- Split download() function
|
||||
- Splited download() function
|
||||
- Added erome support
|
||||
- Remove exclude feature
|
||||
- Bug fix
|
||||
- Removed exclude feature
|
||||
- Bug fixes
|
||||
|
||||
### [22/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a67da461d2fcd70672effcb20c8179e3224091bb)
|
||||
### [22/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/6e7463005051026ad64006a8580b0b5dc9536b8c)
|
||||
- Put log files in a folder named "LOG_FILES"
|
||||
- Fixed the bug that makes multireddit mode unusable
|
||||
|
||||
### [21/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/4a8c2377f9fb4d60ed7eeb8d50aaf9a26492462a)
|
||||
- Added exclude mode
|
||||
|
||||
### [20/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/commit/7548a010198fb693841ca03654d2c9bdf5742139)
|
||||
### [20/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7548a010198fb693841ca03654d2c9bdf5742139)
|
||||
- "0" input for no limit
|
||||
- Fixed the bug that recognizes none image direct links as image links
|
||||
|
||||
@@ -97,7 +103,7 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
|
||||
- Fixed the bug that prevents downloading some gfycat URLs
|
||||
|
||||
### [13/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/9f831e1b784a770c82252e909462871401a05c11)
|
||||
- Change config.json file's path to home directory
|
||||
- Changed config.json file's path to home directory
|
||||
|
||||
### [12/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/50a77f6ba54c24f5647d5ea4e177400b71ff04a7)
|
||||
- Added binaries for Windows and Linux
|
||||
|
||||
@@ -5,12 +5,12 @@ See **[compiling from source](COMPILE_FROM_SOURCE.md)** page first unless you ar
|
||||
***Use*** `.\bulk-downloader-for-reddit.exe` ***or*** `./bulk-downloader-for-reddit` ***if you are using the executable***.
|
||||
```console
|
||||
$ python script.py --help
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||
[--submitted] [--upvoted] [--log LOG FILE]
|
||||
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--NoDownload] [--verbose]
|
||||
[--quit] [--link link] [--saved] [--submitted] [--upvoted]
|
||||
[--log LOG FILE] [--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
[--multireddit MULTIREDDIT] [--user redditor]
|
||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||
[--time TIME_LIMIT] [--NoDownload] [--verbose]
|
||||
[--time TIME_LIMIT]
|
||||
|
||||
This program downloads media from reddit posts
|
||||
|
||||
@@ -19,6 +19,10 @@ optional arguments:
|
||||
--directory DIRECTORY, -d DIRECTORY
|
||||
Specifies the directory where posts will be downloaded
|
||||
to
|
||||
--NoDownload Just gets the posts and stores them in a file for
|
||||
downloading later
|
||||
--verbose, -v Verbose Mode
|
||||
--quit, -q Auto quit afer the process finishes
|
||||
--link link, -l link Get posts from link
|
||||
--saved Triggers saved mode
|
||||
--submitted Gets posts of --user
|
||||
@@ -38,9 +42,6 @@ optional arguments:
|
||||
--limit Limit default: unlimited
|
||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||
all
|
||||
--NoDownload Just gets the posts and store them in a file for
|
||||
downloading later
|
||||
--verbose, -v Verbose Mode
|
||||
```
|
||||
|
||||
# Examples
|
||||
|
||||
@@ -1,16 +1,14 @@
|
||||
# Compiling from source code
|
||||
## Requirements
|
||||
### Python 3 Interpreter
|
||||
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option is mandatory.
|
||||
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option when installing the software is mandatory.
|
||||
|
||||
\* *Use Python 3.6.5 if you encounter an issue*
|
||||
## Using terminal
|
||||
### To open it...
|
||||
- **On Windows 8/8.1/10**: Press the File tab on **Windows Explorer**, click on **Open Windows PowerShell** or **Open Windows Command Prompt** or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
- **On Windows**: Press **Shift+Right Click**, select **Open Powershell window here** or **Open Command Prompt window here**
|
||||
|
||||
- **On Windows 7**: Press **WindowsKey+R**, type **cmd** and hit Enter or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T** or look for **Terminal** in the programs.
|
||||
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T**.
|
||||
|
||||
- **On MacOS**: Look for an app called **Terminal**.
|
||||
|
||||
@@ -39,4 +37,4 @@ python -m pip install -r requirements.txt
|
||||
|
||||
---
|
||||
|
||||
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
|
||||
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
|
||||
|
||||
58
script.py
58
script.py
@@ -22,7 +22,7 @@ from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
||||
|
||||
__author__ = "Ali Parlakci"
|
||||
__license__ = "GPL"
|
||||
__version__ = "1.5.2"
|
||||
__version__ = "1.6.0"
|
||||
__maintainer__ = "Ali Parlakci"
|
||||
__email__ = "parlakciali@gmail.com"
|
||||
|
||||
@@ -66,6 +66,22 @@ def parseArguments(arguments=[]):
|
||||
help="Specifies the directory where posts will be " \
|
||||
"downloaded to",
|
||||
metavar="DIRECTORY")
|
||||
|
||||
parser.add_argument("--NoDownload",
|
||||
help="Just gets the posts and stores them in a file" \
|
||||
" for downloading later",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
parser.add_argument("--verbose","-v",
|
||||
help="Verbose Mode",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
parser.add_argument("--quit","-q",
|
||||
help="Auto quit afer the process finishes",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
parser.add_argument("--link","-l",
|
||||
help="Get posts from link",
|
||||
@@ -137,18 +153,6 @@ def parseArguments(arguments=[]):
|
||||
choices=["all","hour","day","week","month","year"],
|
||||
metavar="TIME_LIMIT",
|
||||
type=str)
|
||||
|
||||
parser.add_argument("--NoDownload",
|
||||
help="Just gets the posts and store them in a file" \
|
||||
" for downloading later",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
parser.add_argument("--verbose","-v",
|
||||
help="Verbose Mode",
|
||||
action="store_true",
|
||||
default=False)
|
||||
|
||||
|
||||
if arguments == []:
|
||||
return parser.parse_args()
|
||||
@@ -486,15 +490,19 @@ def downloadPost(SUBMISSION):
|
||||
+ " Minutes " \
|
||||
+ str(int(IMGUR_RESET_TIME%60)) \
|
||||
+ " Seconds")
|
||||
|
||||
|
||||
if credit['ClientRemaining'] < 25 or credit['UserRemaining'] < 25:
|
||||
print(
|
||||
"==> Client: {} - User: {} - Reset {}".format(
|
||||
credit['ClientRemaining'],
|
||||
credit['UserRemaining'],
|
||||
USER_RESET
|
||||
),end=""
|
||||
)
|
||||
printCredit = {"noPrint":False}
|
||||
else:
|
||||
printCredit = {"noPrint":True}
|
||||
|
||||
print(
|
||||
"==> Client: {} - User: {} - Reset {}\n".format(
|
||||
credit['ClientRemaining'],
|
||||
credit['UserRemaining'],
|
||||
USER_RESET
|
||||
),end="",**printCredit
|
||||
)
|
||||
|
||||
if not (credit['UserRemaining'] == 0 or \
|
||||
credit['ClientRemaining'] == 0):
|
||||
@@ -536,8 +544,8 @@ def download(submissions):
|
||||
|
||||
for i in range(subsLenght):
|
||||
print(
|
||||
f"\n({i+1}/{subsLenght}) ({submissions[i]['postType'].upper()}) " \
|
||||
f"(r/{submissions[i]['postSubreddit']})",end=""
|
||||
f"\n({i+1}/{subsLenght}) – {submissions[i]['postType'].upper()} " \
|
||||
f"– r/{submissions[i]['postSubreddit']}",end=""
|
||||
)
|
||||
|
||||
if isPostExists(submissions[i]):
|
||||
@@ -620,7 +628,7 @@ def main():
|
||||
else:
|
||||
GLOBAL.directory = Path(input("download directory: "))
|
||||
|
||||
print("\n"," ".join(sys.argv),"\n")
|
||||
print("\n"," ".join(sys.argv),"\n",noPrint=True)
|
||||
|
||||
try:
|
||||
checkConflicts()
|
||||
@@ -692,4 +700,4 @@ if __name__ == "__main__":
|
||||
exc_info=full_exc_info(sys.exc_info()))
|
||||
print(log_stream.getvalue())
|
||||
|
||||
input("\nPress enter to quit\n")
|
||||
if not GLOBAL.arguments.quit: input("\nPress enter to quit\n")
|
||||
|
||||
@@ -126,8 +126,6 @@ def getPosts(args):
|
||||
if args["user"] == "me":
|
||||
args["user"] = str(reddit.user.me())
|
||||
|
||||
# print("\nGETTING POSTS\n.\n.\n.\n")
|
||||
|
||||
if not "search" in args:
|
||||
if args["sort"] == "top" or args["sort"] == "controversial":
|
||||
keyword_params = {
|
||||
@@ -159,7 +157,7 @@ def getPosts(args):
|
||||
sort=args["sort"],
|
||||
subreddit=args["subreddit"],
|
||||
time=args["time"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
return redditSearcher(
|
||||
reddit.subreddit(args["subreddit"]).search(
|
||||
@@ -187,7 +185,7 @@ def getPosts(args):
|
||||
"saved posts\nuser:{username}\nlimit={limit}\n".format(
|
||||
username=reddit.user.me(),
|
||||
limit=args["limit"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
return redditSearcher(reddit.user.me().saved(limit=args["limit"]))
|
||||
|
||||
@@ -202,7 +200,7 @@ def getPosts(args):
|
||||
sort=args["sort"],
|
||||
subreddit=args["subreddit"],
|
||||
time=args["time"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
return redditSearcher(
|
||||
getattr(reddit.front,args["sort"]) (**keyword_params)
|
||||
@@ -216,7 +214,7 @@ def getPosts(args):
|
||||
sort=args["sort"],
|
||||
subreddit=args["subreddit"],
|
||||
time=args["time"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
return redditSearcher(
|
||||
getattr(
|
||||
@@ -234,7 +232,7 @@ def getPosts(args):
|
||||
sort=args["sort"],
|
||||
multireddit=args["multireddit"],
|
||||
time=args["time"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
try:
|
||||
return redditSearcher(
|
||||
@@ -255,7 +253,7 @@ def getPosts(args):
|
||||
sort=args["sort"],
|
||||
user=args["user"],
|
||||
time=args["time"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
return redditSearcher(
|
||||
getattr(
|
||||
@@ -268,7 +266,7 @@ def getPosts(args):
|
||||
"upvoted posts of {user}\nlimit: {limit}\n".format(
|
||||
user=args["user"],
|
||||
limit=args["limit"]
|
||||
).upper()
|
||||
).upper(),noPrint=True
|
||||
)
|
||||
try:
|
||||
return redditSearcher(
|
||||
@@ -278,7 +276,7 @@ def getPosts(args):
|
||||
raise InsufficientPermission
|
||||
|
||||
elif "post" in args:
|
||||
print("post: {post}\n".format(post=args["post"]).upper())
|
||||
print("post: {post}\n".format(post=args["post"]).upper(),noPrint=True)
|
||||
return redditSearcher(
|
||||
reddit.submission(url=args["post"]),SINGLE_POST=True
|
||||
)
|
||||
@@ -307,7 +305,8 @@ def redditSearcher(posts,SINGLE_POST=False):
|
||||
|
||||
allPosts = {}
|
||||
|
||||
print("GETTING POSTS")
|
||||
print("\nGETTING POSTS")
|
||||
if GLOBAL.arguments.verbose: print("\n")
|
||||
postsFile = createLogFile("POSTS")
|
||||
|
||||
if SINGLE_POST:
|
||||
@@ -344,7 +343,7 @@ def redditSearcher(posts,SINGLE_POST=False):
|
||||
sys.stdout.flush()
|
||||
|
||||
if subCount % 1000 == 0:
|
||||
sys.stdout.write("\n")
|
||||
sys.stdout.write("\n"+" "*14)
|
||||
sys.stdout.flush()
|
||||
|
||||
try:
|
||||
@@ -372,13 +371,18 @@ def redditSearcher(posts,SINGLE_POST=False):
|
||||
|
||||
postsFile.add(allPosts)
|
||||
|
||||
if not len(subList) == 0:
|
||||
print(
|
||||
f"\n\nTotal of {len(subList)} submissions found!\n"\
|
||||
f"{gfycatCount} GFYCATs, {imgurCount} IMGURs, " \
|
||||
f"{eromeCount} EROMEs, {directCount} DIRECTs " \
|
||||
f"and {selfCount} SELF POSTS"
|
||||
)
|
||||
if not len(subList) == 0:
|
||||
if GLOBAL.arguments.NoDownload or GLOBAL.arguments.verbose:
|
||||
print(
|
||||
f"\n\nTotal of {len(subList)} submissions found!"
|
||||
)
|
||||
print(
|
||||
f"{gfycatCount} GFYCATs, {imgurCount} IMGURs, " \
|
||||
f"{eromeCount} EROMEs, {directCount} DIRECTs " \
|
||||
f"and {selfCount} SELF POSTS",noPrint=True
|
||||
)
|
||||
else:
|
||||
print()
|
||||
return subList
|
||||
else:
|
||||
raise NoMatchingSubmissionFound
|
||||
|
||||
@@ -90,7 +90,7 @@ def createLogFile(TITLE):
|
||||
|
||||
return FILE
|
||||
|
||||
def printToFile(*args, **kwargs):
|
||||
def printToFile(*args, noPrint=False,**kwargs):
|
||||
"""Print to both CONSOLE and
|
||||
CONSOLE LOG file in a folder time stampt in the name
|
||||
"""
|
||||
@@ -98,7 +98,12 @@ def printToFile(*args, **kwargs):
|
||||
TIME = str(time.strftime("%d-%m-%Y_%H-%M-%S",
|
||||
time.localtime(GLOBAL.RUN_TIME)))
|
||||
folderDirectory = GLOBAL.directory / "LOG_FILES" / TIME
|
||||
print(*args,**kwargs)
|
||||
|
||||
if not noPrint or \
|
||||
GLOBAL.arguments.verbose or \
|
||||
"file" in kwargs:
|
||||
|
||||
print(*args,**kwargs)
|
||||
|
||||
if not path.exists(folderDirectory):
|
||||
makedirs(folderDirectory)
|
||||
|
||||
Reference in New Issue
Block a user