13 Commits

Author SHA1 Message Date
Ali Parlakci
d685860c22 Update version 2018-07-26 12:25:54 +03:00
Ali Parlakci
dcf9f35273 Merge branch 'master' of https://github.com/aliparlakci/bulk-downloader-for-reddit 2018-07-26 12:25:50 +03:00
Ali Parlakci
7fdf03aa24 Added new line after 'GETTING POSTS' 2018-07-26 12:25:23 +03:00
Ali Parlakçı
25d61a4c78 Update README.md 2018-07-26 12:23:08 +03:00
Ali Parlakci
558eb107f4 Update changelog 2018-07-26 12:01:00 +03:00
Ali Parlakci
6e74630050 Typo fix 2018-07-26 11:59:29 +03:00
Ali Parlakci
2fd9248715 Added quit after finish option 2018-07-26 11:15:13 +03:00
Ali Parlakci
457b8cd21c Added remaining credits to log file 2018-07-26 11:08:37 +03:00
Ali Parlakci
e953456ead Merge branch 'master' of https://github.com/aliparlakci/bulk-downloader-for-reddit 2018-07-26 10:10:48 +03:00
Ali Parlakci
ed0564fba0 Improve verbose mode 2018-07-26 10:08:57 +03:00
Ali Parlakçı
5378555f74 Update COMPILE_FROM_SOURCE.md 2018-07-26 09:24:50 +03:00
Ali Parlakçı
95ef308915 Update COMPILE_FROM_SOURCE.md 2018-07-26 09:22:14 +03:00
Ali Parlakçı
436f867f2e Update COMPILE_FROM_SOURCE.md 2018-07-26 09:22:03 +03:00
6 changed files with 92 additions and 70 deletions

View File

@@ -23,7 +23,8 @@ This program downloads imgur, gfycat and direct image and video links of saved p
- **MacOS** users have to **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
Script also accepts **command-line arguments**, get further information from **[`--help`](docs/COMMAND_LINE_ARGUMENTS.md)**
### Additional options
Script also accepts additional options via command-line arguments, get further information from **[`--help`](docs/COMMAND_LINE_ARGUMENTS.md)**
## Setting up the script
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
@@ -36,9 +37,9 @@ Because this is not a commercial app, you need to create an imgur developer app
* Correct CHAPTCHA
* Click **submit** button
It should redirect to a page which shows your **imgur_client_id** and **imgur_client_secret**
It should redirect you to a page which shows your **imgur_client_id** and **imgur_client_secret**
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*. If this does not work, it is safe to proceed with **OAuth 2 authorization without a callback URL**.
## FAQ
### What do the dots resemble when getting posts?
@@ -48,7 +49,7 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
- You can press Ctrl+C to interrupt it and start downloading.
### How downloaded files' names are formatted?
- Images that are not belong to an album or self posts are formatted as **`[SUBMITTER NAME]_[POST TITLE]_[REDDIT ID]`**.
- Self posts and images that are not belong to an album are formatted as **`[SUBMITTER NAME]_[POST TITLE]_[REDDIT ID]`**.
You can use *reddit id* to go to post's reddit page by going to link **reddit.com/[REDDIT ID]**
- An image in an imgur album is formatted as **`[ITEM NUMBER]_[IMAGE TITLE]_[IMGUR ID]`**
@@ -65,9 +66,14 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
them, there.
## Changes on *master*
### [26/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/1623722138bad80ae39ffcd5fb38baf80680deac)
- Improved verbose mode
- Minimalized the console output
- Added quit option for auto quitting the program after process finished
### [25/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/1623722138bad80ae39ffcd5fb38baf80680deac)
- Added verbose mode
- Stylize the console output
- Stylized the console output
### [24/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7a68ff3efac9939f9574c2cef6184b92edb135f4)
- Added OP's name to file names (backwards compatible)
@@ -75,19 +81,19 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
- Improved exception handling
### [23/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7314e17125aa78fd4e6b28e26fda7ec7db7e0147)
- Split download() function
- Splited download() function
- Added erome support
- Remove exclude feature
- Bug fix
- Removed exclude feature
- Bug fixes
### [22/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a67da461d2fcd70672effcb20c8179e3224091bb)
### [22/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/6e7463005051026ad64006a8580b0b5dc9536b8c)
- Put log files in a folder named "LOG_FILES"
- Fixed the bug that makes multireddit mode unusable
### [21/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/4a8c2377f9fb4d60ed7eeb8d50aaf9a26492462a)
- Added exclude mode
### [20/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/commit/7548a010198fb693841ca03654d2c9bdf5742139)
### [20/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/7548a010198fb693841ca03654d2c9bdf5742139)
- "0" input for no limit
- Fixed the bug that recognizes none image direct links as image links
@@ -97,7 +103,7 @@ It should redirect to a page which shows your **imgur_client_id** and **imgur_cl
- Fixed the bug that prevents downloading some gfycat URLs
### [13/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/9f831e1b784a770c82252e909462871401a05c11)
- Change config.json file's path to home directory
- Changed config.json file's path to home directory
### [12/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/50a77f6ba54c24f5647d5ea4e177400b71ff04a7)
- Added binaries for Windows and Linux

View File

@@ -5,12 +5,12 @@ See **[compiling from source](COMPILE_FROM_SOURCE.md)** page first unless you ar
***Use*** `.\bulk-downloader-for-reddit.exe` ***or*** `./bulk-downloader-for-reddit` ***if you are using the executable***.
```console
$ python script.py --help
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
[--submitted] [--upvoted] [--log LOG FILE]
[--subreddit SUBREDDIT [SUBREDDIT ...]]
usage: script.py [-h] [--directory DIRECTORY] [--NoDownload] [--verbose]
[--quit] [--link link] [--saved] [--submitted] [--upvoted]
[--log LOG FILE] [--subreddit SUBREDDIT [SUBREDDIT ...]]
[--multireddit MULTIREDDIT] [--user redditor]
[--search query] [--sort SORT TYPE] [--limit Limit]
[--time TIME_LIMIT] [--NoDownload] [--verbose]
[--time TIME_LIMIT]
This program downloads media from reddit posts
@@ -19,6 +19,10 @@ optional arguments:
--directory DIRECTORY, -d DIRECTORY
Specifies the directory where posts will be downloaded
to
--NoDownload Just gets the posts and stores them in a file for
downloading later
--verbose, -v Verbose Mode
--quit, -q Auto quit afer the process finishes
--link link, -l link Get posts from link
--saved Triggers saved mode
--submitted Gets posts of --user
@@ -38,9 +42,6 @@ optional arguments:
--limit Limit default: unlimited
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
all
--NoDownload Just gets the posts and store them in a file for
downloading later
--verbose, -v Verbose Mode
```
# Examples

View File

@@ -1,16 +1,14 @@
# Compiling from source code
## Requirements
### Python 3 Interpreter
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option is mandatory.
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option when installing the software is mandatory.
\* *Use Python 3.6.5 if you encounter an issue*
## Using terminal
### To open it...
- **On Windows 8/8.1/10**: Press the File tab on **Windows Explorer**, click on **Open Windows PowerShell** or **Open Windows Command Prompt** or look for *Command Prompt* or *PowerShell* in *Start Menu*.
- **On Windows**: Press **Shift+Right Click**, select **Open Powershell window here** or **Open Command Prompt window here**
- **On Windows 7**: Press **WindowsKey+R**, type **cmd** and hit Enter or look for *Command Prompt* or *PowerShell* in *Start Menu*.
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T** or look for **Terminal** in the programs.
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T**.
- **On MacOS**: Look for an app called **Terminal**.
@@ -39,4 +37,4 @@ python -m pip install -r requirements.txt
---
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)

View File

@@ -22,7 +22,7 @@ from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
__author__ = "Ali Parlakci"
__license__ = "GPL"
__version__ = "1.5.2"
__version__ = "1.6.0"
__maintainer__ = "Ali Parlakci"
__email__ = "parlakciali@gmail.com"
@@ -66,6 +66,22 @@ def parseArguments(arguments=[]):
help="Specifies the directory where posts will be " \
"downloaded to",
metavar="DIRECTORY")
parser.add_argument("--NoDownload",
help="Just gets the posts and stores them in a file" \
" for downloading later",
action="store_true",
default=False)
parser.add_argument("--verbose","-v",
help="Verbose Mode",
action="store_true",
default=False)
parser.add_argument("--quit","-q",
help="Auto quit afer the process finishes",
action="store_true",
default=False)
parser.add_argument("--link","-l",
help="Get posts from link",
@@ -137,18 +153,6 @@ def parseArguments(arguments=[]):
choices=["all","hour","day","week","month","year"],
metavar="TIME_LIMIT",
type=str)
parser.add_argument("--NoDownload",
help="Just gets the posts and store them in a file" \
" for downloading later",
action="store_true",
default=False)
parser.add_argument("--verbose","-v",
help="Verbose Mode",
action="store_true",
default=False)
if arguments == []:
return parser.parse_args()
@@ -486,15 +490,19 @@ def downloadPost(SUBMISSION):
+ " Minutes " \
+ str(int(IMGUR_RESET_TIME%60)) \
+ " Seconds")
if credit['ClientRemaining'] < 25 or credit['UserRemaining'] < 25:
print(
"==> Client: {} - User: {} - Reset {}".format(
credit['ClientRemaining'],
credit['UserRemaining'],
USER_RESET
),end=""
)
printCredit = {"noPrint":False}
else:
printCredit = {"noPrint":True}
print(
"==> Client: {} - User: {} - Reset {}\n".format(
credit['ClientRemaining'],
credit['UserRemaining'],
USER_RESET
),end="",**printCredit
)
if not (credit['UserRemaining'] == 0 or \
credit['ClientRemaining'] == 0):
@@ -536,8 +544,8 @@ def download(submissions):
for i in range(subsLenght):
print(
f"\n({i+1}/{subsLenght}) ({submissions[i]['postType'].upper()}) " \
f"(r/{submissions[i]['postSubreddit']})",end=""
f"\n({i+1}/{subsLenght}) {submissions[i]['postType'].upper()} " \
f" r/{submissions[i]['postSubreddit']}",end=""
)
if isPostExists(submissions[i]):
@@ -620,7 +628,7 @@ def main():
else:
GLOBAL.directory = Path(input("download directory: "))
print("\n"," ".join(sys.argv),"\n")
print("\n"," ".join(sys.argv),"\n",noPrint=True)
try:
checkConflicts()
@@ -692,4 +700,4 @@ if __name__ == "__main__":
exc_info=full_exc_info(sys.exc_info()))
print(log_stream.getvalue())
input("\nPress enter to quit\n")
if not GLOBAL.arguments.quit: input("\nPress enter to quit\n")

View File

@@ -126,8 +126,6 @@ def getPosts(args):
if args["user"] == "me":
args["user"] = str(reddit.user.me())
# print("\nGETTING POSTS\n.\n.\n.\n")
if not "search" in args:
if args["sort"] == "top" or args["sort"] == "controversial":
keyword_params = {
@@ -159,7 +157,7 @@ def getPosts(args):
sort=args["sort"],
subreddit=args["subreddit"],
time=args["time"]
).upper()
).upper(),noPrint=True
)
return redditSearcher(
reddit.subreddit(args["subreddit"]).search(
@@ -187,7 +185,7 @@ def getPosts(args):
"saved posts\nuser:{username}\nlimit={limit}\n".format(
username=reddit.user.me(),
limit=args["limit"]
).upper()
).upper(),noPrint=True
)
return redditSearcher(reddit.user.me().saved(limit=args["limit"]))
@@ -202,7 +200,7 @@ def getPosts(args):
sort=args["sort"],
subreddit=args["subreddit"],
time=args["time"]
).upper()
).upper(),noPrint=True
)
return redditSearcher(
getattr(reddit.front,args["sort"]) (**keyword_params)
@@ -216,7 +214,7 @@ def getPosts(args):
sort=args["sort"],
subreddit=args["subreddit"],
time=args["time"]
).upper()
).upper(),noPrint=True
)
return redditSearcher(
getattr(
@@ -234,7 +232,7 @@ def getPosts(args):
sort=args["sort"],
multireddit=args["multireddit"],
time=args["time"]
).upper()
).upper(),noPrint=True
)
try:
return redditSearcher(
@@ -255,7 +253,7 @@ def getPosts(args):
sort=args["sort"],
user=args["user"],
time=args["time"]
).upper()
).upper(),noPrint=True
)
return redditSearcher(
getattr(
@@ -268,7 +266,7 @@ def getPosts(args):
"upvoted posts of {user}\nlimit: {limit}\n".format(
user=args["user"],
limit=args["limit"]
).upper()
).upper(),noPrint=True
)
try:
return redditSearcher(
@@ -278,7 +276,7 @@ def getPosts(args):
raise InsufficientPermission
elif "post" in args:
print("post: {post}\n".format(post=args["post"]).upper())
print("post: {post}\n".format(post=args["post"]).upper(),noPrint=True)
return redditSearcher(
reddit.submission(url=args["post"]),SINGLE_POST=True
)
@@ -307,7 +305,8 @@ def redditSearcher(posts,SINGLE_POST=False):
allPosts = {}
print("GETTING POSTS")
print("\nGETTING POSTS")
if GLOBAL.arguments.verbose: print("\n")
postsFile = createLogFile("POSTS")
if SINGLE_POST:
@@ -344,7 +343,7 @@ def redditSearcher(posts,SINGLE_POST=False):
sys.stdout.flush()
if subCount % 1000 == 0:
sys.stdout.write("\n")
sys.stdout.write("\n"+" "*14)
sys.stdout.flush()
try:
@@ -372,13 +371,18 @@ def redditSearcher(posts,SINGLE_POST=False):
postsFile.add(allPosts)
if not len(subList) == 0:
print(
f"\n\nTotal of {len(subList)} submissions found!\n"\
f"{gfycatCount} GFYCATs, {imgurCount} IMGURs, " \
f"{eromeCount} EROMEs, {directCount} DIRECTs " \
f"and {selfCount} SELF POSTS"
)
if not len(subList) == 0:
if GLOBAL.arguments.NoDownload or GLOBAL.arguments.verbose:
print(
f"\n\nTotal of {len(subList)} submissions found!"
)
print(
f"{gfycatCount} GFYCATs, {imgurCount} IMGURs, " \
f"{eromeCount} EROMEs, {directCount} DIRECTs " \
f"and {selfCount} SELF POSTS",noPrint=True
)
else:
print()
return subList
else:
raise NoMatchingSubmissionFound

View File

@@ -90,7 +90,7 @@ def createLogFile(TITLE):
return FILE
def printToFile(*args, **kwargs):
def printToFile(*args, noPrint=False,**kwargs):
"""Print to both CONSOLE and
CONSOLE LOG file in a folder time stampt in the name
"""
@@ -98,7 +98,12 @@ def printToFile(*args, **kwargs):
TIME = str(time.strftime("%d-%m-%Y_%H-%M-%S",
time.localtime(GLOBAL.RUN_TIME)))
folderDirectory = GLOBAL.directory / "LOG_FILES" / TIME
print(*args,**kwargs)
if not noPrint or \
GLOBAL.arguments.verbose or \
"file" in kwargs:
print(*args,**kwargs)
if not path.exists(folderDirectory):
makedirs(folderDirectory)