mirror of
https://github.com/KevinMidboe/bulk-downloader-for-reddit.git
synced 2026-01-22 00:55:37 +00:00
Compare commits
48 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d2ed8327df | ||
|
|
a28a7776ab | ||
|
|
1a50bcecb0 | ||
|
|
d8a2204024 | ||
|
|
0577c332b5 | ||
|
|
e09da8db73 | ||
|
|
fd507870e1 | ||
|
|
c434d346a7 | ||
|
|
fcc3e53da7 | ||
|
|
7452af8bd0 | ||
|
|
1fd6951420 | ||
|
|
bddac14804 | ||
|
|
6e2c69d053 | ||
|
|
bedd481195 | ||
|
|
2994adea38 | ||
|
|
f1a9f1d3e4 | ||
|
|
9849c0439b | ||
|
|
ef9f0c543f | ||
|
|
26531fd6c0 | ||
|
|
cae044f7b6 | ||
|
|
4daa9cd66a | ||
|
|
e83d2626d3 | ||
|
|
f67c5f62c6 | ||
|
|
fef559cd40 | ||
|
|
42672dc15d | ||
|
|
3b15141b49 | ||
|
|
d080ca17bc | ||
|
|
7f97bd212a | ||
|
|
65592c5d3a | ||
|
|
57a5f0c85c | ||
|
|
f6240f402e | ||
|
|
fd179c0e4b | ||
|
|
d9dc3132f6 | ||
|
|
45191e2c60 | ||
|
|
319a6c82a3 | ||
|
|
2c62337fac | ||
|
|
7c9c84c411 | ||
|
|
8922c27f9b | ||
|
|
cc59710f33 | ||
|
|
49129bea24 | ||
|
|
a701444d5b | ||
|
|
5a030a156c | ||
|
|
ffe3839aee | ||
|
|
a796038b71 | ||
|
|
ddccebbc70 | ||
|
|
5cfa443f83 | ||
|
|
2103c62218 | ||
|
|
10128b63ac |
151
README.md
151
README.md
@@ -1,41 +1,21 @@
|
|||||||
# Bulk Downloader for Reddit
|
# Bulk Downloader for Reddit
|
||||||
This program downloads imgur, gfycat and direct image and video links of saved posts from a reddit account. It is written in Python 3.
|
This program downloads imgur, gfycat and direct image and video links of saved posts from a reddit account. It is written in Python 3.
|
||||||
|
|
||||||
**PLEASE** post any issue you had with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
**PLEASE** post any issue you have with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
||||||
|
|
||||||
## Table of Contents
|
## What can it do?
|
||||||
|
|
||||||
- [What it can do?](#what-it-can-do)
|
|
||||||
- [Requirements](#requirements)
|
|
||||||
- [Setting up the script](#setting-up-the-script)
|
|
||||||
- [Creating an imgur app](#creating-an-imgur-app)
|
|
||||||
- [Program Modes](#program-modes)
|
|
||||||
- [Running the script](#running-the-script)
|
|
||||||
- [Using the command line arguments](#using-the-command-line-arguments)
|
|
||||||
- [Examples](#examples)
|
|
||||||
- [FAQ](#faq)
|
|
||||||
- [Changelog](#changelog)
|
|
||||||
|
|
||||||
## What it can do?
|
|
||||||
### It...
|
### It...
|
||||||
- can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
- can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||||
- sorts post by hot, top, new and so on
|
- sorts posts by hot, top, new and so on
|
||||||
- downloads imgur albums, gfycat links, [self posts](#i-can-t-open-the-self-posts-) and any link to a direct image
|
- downloads imgur albums, gfycat links, [self posts](#i-cant-open-the-self-posts) and any link to a direct image
|
||||||
- skips the existing ones
|
- skips the existing ones
|
||||||
- puts post titles to file's name
|
- puts post titles to file's name
|
||||||
- puts every post to its subreddit's folder
|
- puts every post to its subreddit's folder
|
||||||
- saves reusable a copy of posts' details that are found so that they can be re-downloaded again
|
- saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||||
- logs failed ones in a file to so that you can try to download them later
|
- logs failed ones in a file to so that you can try to download them later
|
||||||
- can be run with double-clicking on Windows (but I don't recommend it)
|
- can be run with double-clicking on Windows (but I don't recommend it)
|
||||||
|
|
||||||
## Requirements
|
## [Download the latest release](https://github.com/aliparlakci/bulk-downloader-for-reddit/releases/latest)
|
||||||
- Python 3.x*
|
|
||||||
|
|
||||||
You can install Python 3 here: [https://www.python.org/downloads/](https://www.python.org/downloads/)
|
|
||||||
|
|
||||||
You have to check "**Add Python 3 to PATH**" option when installing in order it to run correctly.
|
|
||||||
|
|
||||||
*\*Although the latest version of python is suggested, you can use 3.6.5 since it runs perfectly on that version*
|
|
||||||
|
|
||||||
## Setting up the script
|
## Setting up the script
|
||||||
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
||||||
@@ -50,120 +30,27 @@ Because this is not a commercial app, you need to create an imgur developer app
|
|||||||
|
|
||||||
It should redirect to a page which shows your **imgur_client_id** and **imgur_client_secret**
|
It should redirect to a page which shows your **imgur_client_id** and **imgur_client_secret**
|
||||||
|
|
||||||
\*Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||||
|
|
||||||
## Program Modes
|
|
||||||
All the program modes are activated with command-line arguments as shown [here](#using-the-command-line-arguments)
|
|
||||||
- **saved mode**
|
|
||||||
- Gets posts from given user's saved posts.
|
|
||||||
- **submitted mode**
|
|
||||||
- Gets posts from given user's submitted posts.
|
|
||||||
- **upvoted mode**
|
|
||||||
- Gets posts from given user's upvoted posts.
|
|
||||||
- **subreddit mode**
|
|
||||||
- Gets posts from given subreddit or subreddits that is sorted by given type and limited by given number.
|
|
||||||
- You may also use search in this mode. See [`py -3 script.py --help`](#using-the-command-line-arguments).
|
|
||||||
- **multireddit mode**
|
|
||||||
- Gets posts from given user's given multireddit that is sorted by given type and limited by given number.
|
|
||||||
- **link mode**
|
|
||||||
- Gets posts from given reddit link.
|
|
||||||
- You may customize the behaviour with `--sort`, `--time`, `--limit`.
|
|
||||||
- You may also use search in this mode. See [`py -3 script.py --help`](#using-the-command-line-arguments).
|
|
||||||
- **log read mode**
|
|
||||||
- Takes a log file which created by itself (json files), reads posts and tries downloading them again.
|
|
||||||
- Running log read mode for FAILED.json file once after the download is complete is **HIGHLY** recommended as unexpected problems may occur.
|
|
||||||
|
|
||||||
## Running the script
|
## Running the script
|
||||||
**DO NOT** let more than one instance of the script run as it interferes with IMGUR Request Rate.
|
|
||||||
|
|
||||||
### Using the command line arguments
|
For **Windows** users, there is an *EXE* file to run easily.
|
||||||
If no arguments are passed program will prompt you for arguments below which means you may start up the script with double-clicking on it (at least on Windows for sure).
|
|
||||||
|
|
||||||
Open up the [terminal](https://www.reddit.com/r/NSFW411/comments/8vtnl8/meta_i_made_reddit_downloader_that_can_download/e1rnbnl) and navigate to where script.py is. If you are unfamiliar with changing directories in terminal see Change Directories in [this article](https://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything).
|
**Linux** and **MacOS** users have to install Python 3 and run it from the *source code* through terminal.
|
||||||
|
|
||||||
Run the script.py file from terminal with command-line arguments. Here is the help page:
|
|
||||||
|
|
||||||
Use `.\` for current directory and `..\` for upper directory when using short directories, otherwise it might act weird.
|
|
||||||
|
|
||||||
```console
|
To get further information about that and **using command-line arguments to run the script**, see **[`python script.py --help`](docs/help_page.md)**
|
||||||
$ py -3 script.py --help
|
|
||||||
usage: script.py [-h] [--link link] [--saved] [--submitted] [--upvoted]
|
|
||||||
[--log LOG FILE] [--subreddit SUBREDDIT [SUBREDDIT ...]]
|
|
||||||
[--multireddit MULTIREDDIT] [--user redditor]
|
|
||||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
|
||||||
[--time TIME_LIMIT] [--NoDownload]
|
|
||||||
DIRECTORY
|
|
||||||
|
|
||||||
This program downloads media from reddit posts
|
|
||||||
|
|
||||||
positional arguments:
|
|
||||||
DIRECTORY Specifies the directory where posts will be downloaded
|
|
||||||
to
|
|
||||||
|
|
||||||
optional arguments:
|
|
||||||
-h, --help show this help message and exit
|
|
||||||
--link link, -l link Get posts from link
|
|
||||||
--saved Triggers saved mode
|
|
||||||
--submitted Gets posts of --user
|
|
||||||
--upvoted Gets upvoted posts of --user
|
|
||||||
--log LOG FILE Triggers log read mode and takes a log file
|
|
||||||
--subreddit SUBREDDIT [SUBREDDIT ...]
|
|
||||||
Triggers subreddit mode and takes subreddit's name
|
|
||||||
without r/. use "frontpage" for frontpage
|
|
||||||
--multireddit MULTIREDDIT
|
|
||||||
Triggers multireddit mode and takes multireddit's name
|
|
||||||
without m/
|
|
||||||
--user redditor reddit username if needed. use "me" for current user
|
|
||||||
--search query Searches for given query in given subreddits
|
|
||||||
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
|
||||||
relevance default: hot
|
|
||||||
--limit Limit default: unlimited
|
|
||||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
|
||||||
all
|
|
||||||
--NoDownload Just gets the posts and store them in a file for
|
|
||||||
downloading later
|
|
||||||
```
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
#### Don't include `py -3 script.py` part if you start the script by double-clicking
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
|
||||||
```
|
|
||||||
|
|
||||||
```console
|
|
||||||
py -3 script.py C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
|
||||||
```
|
|
||||||
|
|
||||||
## FAQ
|
## FAQ
|
||||||
### I can't startup the script no matter what.
|
### I can't open the self post files.
|
||||||
- Try `python3` or `python` or `py -3` as python have real issues about naming their program
|
- Self posts are held at reddit as styled with markdown. So, the script downloads them as they are in order not to lose their stylings.
|
||||||
|
However, there is a great Chrome extension [here](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with Chrome.
|
||||||
### I can't open the self posts.
|
|
||||||
- Self posts are held at subreddit as Markdown. So, the script downloads them as Markdown in order not to lose their stylings. However, there is a great Chrome extension [here](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with Chrome.
|
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
### 10/07/2018
|
### [11/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a28a7776ab826dea2a8d93873a94cd46db3a339b)
|
||||||
|
- Improvements on UX and UI
|
||||||
|
- Added logging errors to CONSOLE_LOG.txt
|
||||||
|
- Using current directory if directory has not been given yet.
|
||||||
|
|
||||||
|
### [10/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/ffe3839aee6dc1a552d95154d817aefc2b66af81)
|
||||||
- Added support for *self* post
|
- Added support for *self* post
|
||||||
- Now getting posts is quicker
|
- Now getting posts is quicker
|
||||||
|
|||||||
85
docs/help_page.md
Normal file
85
docs/help_page.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
## python script.py --help
|
||||||
|
|
||||||
|
```console
|
||||||
|
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||||
|
[--submitted] [--upvoted] [--log LOG FILE]
|
||||||
|
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||||
|
[--multireddit MULTIREDDIT] [--user redditor]
|
||||||
|
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||||
|
[--time TIME_LIMIT] [--NoDownload]
|
||||||
|
|
||||||
|
This program downloads media from reddit posts
|
||||||
|
|
||||||
|
optional arguments:
|
||||||
|
-h, --help show this help message and exit
|
||||||
|
--directory DIRECTORY
|
||||||
|
Specifies the directory where posts will be downloaded
|
||||||
|
to
|
||||||
|
--link link, -l link Get posts from link
|
||||||
|
--saved Triggers saved mode
|
||||||
|
--submitted Gets posts of --user
|
||||||
|
--upvoted Gets upvoted posts of --user
|
||||||
|
--log LOG FILE Triggers log read mode and takes a log file
|
||||||
|
--subreddit SUBREDDIT [SUBREDDIT ...]
|
||||||
|
Triggers subreddit mode and takes subreddit's name
|
||||||
|
without r/. use "frontpage" for frontpage
|
||||||
|
--multireddit MULTIREDDIT
|
||||||
|
Triggers multireddit mode and takes multireddit's name
|
||||||
|
without m/
|
||||||
|
--user redditor reddit username if needed. use "me" for current user
|
||||||
|
--search query Searches for given query in given subreddits
|
||||||
|
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
||||||
|
relevance default: hot
|
||||||
|
--limit Limit default: unlimited
|
||||||
|
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||||
|
all
|
||||||
|
--NoDownload Just gets the posts and store them in a file for
|
||||||
|
downloading later
|
||||||
|
```
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
- **Use `python3` instead of `python` if you are using *MacOS* or *Linux***
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
python script.py C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## FAQ
|
||||||
|
### I can't startup the script no matter what.
|
||||||
|
- Try these:
|
||||||
|
- **`python`**
|
||||||
|
- **`python3`**
|
||||||
|
- **`python3.7`**
|
||||||
|
- **`python3.6`**
|
||||||
|
- **`py -3`**
|
||||||
|
|
||||||
|
Python have real issues about naming their program
|
||||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
requests
|
||||||
|
praw
|
||||||
|
imgurpython
|
||||||
292
script.py
292
script.py
@@ -6,30 +6,26 @@ saved posts from a reddit account. It is written in Python 3.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import logging
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
|
from io import StringIO
|
||||||
from pathlib import Path, PurePath
|
from pathlib import Path, PurePath
|
||||||
|
|
||||||
from src.downloader import Direct, Gfycat, Imgur, Self
|
from src.downloader import Direct, Gfycat, Imgur, Self
|
||||||
|
from src.errors import *
|
||||||
from src.parser import LinkDesigner
|
from src.parser import LinkDesigner
|
||||||
from src.searcher import getPosts
|
from src.searcher import getPosts
|
||||||
from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
||||||
printToFile)
|
printToFile)
|
||||||
from src.errors import *
|
|
||||||
|
|
||||||
__author__ = "Ali Parlakci"
|
__author__ = "Ali Parlakci"
|
||||||
__license__ = "GPL"
|
__license__ = "GPL"
|
||||||
__version__ = "1.0.0"
|
__version__ = "1.1.0"
|
||||||
__maintainer__ = "Ali Parlakci"
|
__maintainer__ = "Ali Parlakci"
|
||||||
__email__ = "parlakciali@gmail.com"
|
__email__ = "parlakciali@gmail.com"
|
||||||
|
|
||||||
def debug(*post):
|
|
||||||
GLOBAL.config = getConfig('config.json')
|
|
||||||
GLOBAL.directory = Path(".\\debug\\")
|
|
||||||
download([*post])
|
|
||||||
quit()
|
|
||||||
|
|
||||||
def getConfig(configFileName):
|
def getConfig(configFileName):
|
||||||
"""Read credentials from config.json file"""
|
"""Read credentials from config.json file"""
|
||||||
|
|
||||||
@@ -66,7 +62,7 @@ def parseArguments(arguments=[]):
|
|||||||
description="This program downloads " \
|
description="This program downloads " \
|
||||||
"media from reddit " \
|
"media from reddit " \
|
||||||
"posts")
|
"posts")
|
||||||
parser.add_argument("directory",
|
parser.add_argument("--directory",
|
||||||
help="Specifies the directory where posts will be " \
|
help="Specifies the directory where posts will be " \
|
||||||
"downloaded to",
|
"downloaded to",
|
||||||
metavar="DIRECTORY")
|
metavar="DIRECTORY")
|
||||||
@@ -88,7 +84,9 @@ def parseArguments(arguments=[]):
|
|||||||
help="Gets upvoted posts of --user")
|
help="Gets upvoted posts of --user")
|
||||||
|
|
||||||
parser.add_argument("--log",
|
parser.add_argument("--log",
|
||||||
help="Triggers log read mode and takes a log file",
|
help="Takes a log file which created by itself " \
|
||||||
|
"(json files), reads posts and tries downloadin" \
|
||||||
|
"g them again.",
|
||||||
# type=argparse.FileType('r'),
|
# type=argparse.FileType('r'),
|
||||||
metavar="LOG FILE")
|
metavar="LOG FILE")
|
||||||
|
|
||||||
@@ -131,7 +129,6 @@ def parseArguments(arguments=[]):
|
|||||||
parser.add_argument("--limit",
|
parser.add_argument("--limit",
|
||||||
help="default: unlimited",
|
help="default: unlimited",
|
||||||
metavar="Limit",
|
metavar="Limit",
|
||||||
default=None,
|
|
||||||
type=int)
|
type=int)
|
||||||
|
|
||||||
parser.add_argument("--time",
|
parser.add_argument("--time",
|
||||||
@@ -157,88 +154,166 @@ def checkConflicts():
|
|||||||
if not, raise errors
|
if not, raise errors
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if GLOBAL.arguments.saved is False:
|
|
||||||
saved = 0
|
|
||||||
else:
|
|
||||||
saved = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.subreddit is None:
|
|
||||||
subreddit = 0
|
|
||||||
else:
|
|
||||||
subreddit = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.submitted is False:
|
|
||||||
submitted = 0
|
|
||||||
else:
|
|
||||||
submitted = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.search is None:
|
|
||||||
search = 0
|
|
||||||
else:
|
|
||||||
search = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.log is None:
|
|
||||||
log = 0
|
|
||||||
else:
|
|
||||||
log = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.link is None:
|
|
||||||
link = 0
|
|
||||||
else:
|
|
||||||
link = 1
|
|
||||||
|
|
||||||
if GLOBAL.arguments.user is None:
|
if GLOBAL.arguments.user is None:
|
||||||
user = 0
|
user = 0
|
||||||
else:
|
else:
|
||||||
user = 1
|
user = 1
|
||||||
|
|
||||||
if GLOBAL.arguments.upvoted is False:
|
modes = ["saved","subreddit","submitted","search","log","link","upvoted"]
|
||||||
upvoted = 0
|
|
||||||
else:
|
|
||||||
upvoted = 1
|
|
||||||
|
|
||||||
if not saved+subreddit+log+link+submitted+upvoted == 1:
|
values = {
|
||||||
print("Program mode is invalid")
|
x: 0 if getattr(GLOBAL.arguments,x) is None or \
|
||||||
quit()
|
getattr(GLOBAL.arguments,x) is False \
|
||||||
|
else 1 \
|
||||||
|
for x in modes
|
||||||
|
}
|
||||||
|
|
||||||
|
if not sum(values[x] for x in values) == 1:
|
||||||
|
raise ProgramModeError("Invalid program mode")
|
||||||
|
|
||||||
if search+subreddit == 2:
|
if values["search"]+values["saved"] == 2:
|
||||||
print("You cannot search in your saved posts")
|
raise SearchModeError("You cannot search in your saved posts")
|
||||||
quit()
|
|
||||||
|
|
||||||
if search+submitted == 2:
|
if values["search"]+values["submitted"] == 2:
|
||||||
print("You cannot search in submitted posts")
|
raise SearchModeError("You cannot search in submitted posts")
|
||||||
quit()
|
|
||||||
|
|
||||||
if search+upvoted == 2:
|
if values["search"]+values["upvoted"] == 2:
|
||||||
print("You cannot search in upvoted posts")
|
raise SearchModeError("You cannot search in upvoted posts")
|
||||||
quit()
|
|
||||||
|
|
||||||
if upvoted+submitted == 1 and user == 0:
|
if values["upvoted"]+values["submitted"] == 1 and user == 0:
|
||||||
print("No redditor name given")
|
raise RedditorNameError("No redditor name given")
|
||||||
quit()
|
|
||||||
|
|
||||||
def postFromLog(fileName):
|
class PromptUser:
|
||||||
"""Analyze a log file and return a list of dictionaries containing
|
@staticmethod
|
||||||
submissions
|
def chooseFrom(choices):
|
||||||
"""
|
print()
|
||||||
if Path.is_file(Path(fileName)):
|
choicesByIndex = list(str(x) for x in range(len(choices)+1))
|
||||||
content = jsonFile(fileName).read()
|
for i in range(len(choices)):
|
||||||
else:
|
print("{indent}[{order}] {mode}".format(
|
||||||
print("File not found")
|
indent=" "*4,order=i+1,mode=choices[i]
|
||||||
quit()
|
))
|
||||||
|
print(" "*4+"[0] exit\n")
|
||||||
|
choice = input("> ")
|
||||||
|
while not choice.lower() in choices+choicesByIndex:
|
||||||
|
print("Invalid input\n")
|
||||||
|
programModeIndex = input("> ")
|
||||||
|
|
||||||
try:
|
if choice == "0":
|
||||||
del content["HEADER"]
|
quit()
|
||||||
except KeyError:
|
elif choice in choicesByIndex:
|
||||||
pass
|
return choices[int(choice)-1]
|
||||||
|
else:
|
||||||
|
return choice
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
print("select program mode:")
|
||||||
|
programModes = [
|
||||||
|
"search","subreddit","multireddit",
|
||||||
|
"submitted","upvoted","saved","log"
|
||||||
|
]
|
||||||
|
programMode = self.chooseFrom(programModes)
|
||||||
|
|
||||||
posts = []
|
if programMode == "search":
|
||||||
|
GLOBAL.arguments.search = input("\nquery: ")
|
||||||
|
GLOBAL.arguments.subreddit = input("\nsubreddit: ")
|
||||||
|
|
||||||
for post in content:
|
print("\nselect sort type:")
|
||||||
if not content[post][-1]['postType'] == None:
|
sortTypes = [
|
||||||
posts.append(content[post][-1])
|
"relevance","top","new"
|
||||||
|
]
|
||||||
|
sortType = self.chooseFrom(sortTypes)
|
||||||
|
GLOBAL.arguments.sort = sortType
|
||||||
|
|
||||||
return posts
|
print("\nselect time filter:")
|
||||||
|
timeFilters = [
|
||||||
|
"hour","day","week","month","year","all"
|
||||||
|
]
|
||||||
|
timeFilter = self.chooseFrom(timeFilters)
|
||||||
|
GLOBAL.arguments.time = timeFilter
|
||||||
|
|
||||||
|
if programMode == "subreddit":
|
||||||
|
GLOBAL.arguments.subreddit = input("\nsubreddit: ")
|
||||||
|
if " " in GLOBAL.arguments.subreddit:
|
||||||
|
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit.split())
|
||||||
|
|
||||||
|
print("\nselect sort type:")
|
||||||
|
sortTypes = [
|
||||||
|
"hot","top","new","rising","controversial"
|
||||||
|
]
|
||||||
|
sortType = self.chooseFrom(sortTypes)
|
||||||
|
GLOBAL.arguments.sort = sortType
|
||||||
|
|
||||||
|
if sortType in ["top","controversial"]:
|
||||||
|
print("\nselect time filter:")
|
||||||
|
timeFilters = [
|
||||||
|
"hour","day","week","month","year","all"
|
||||||
|
]
|
||||||
|
timeFilter = self.chooseFrom(timeFilters)
|
||||||
|
GLOBAL.arguments.time = timeFilter
|
||||||
|
else:
|
||||||
|
GLOBAL.arguments.time = "all"
|
||||||
|
|
||||||
|
elif programMode == "multireddit":
|
||||||
|
GLOBAL.arguments.user = input("\nredditor: ")
|
||||||
|
GLOBAL.arguments.subreddit = input("\nmultireddit: ")
|
||||||
|
|
||||||
|
print("\nselect sort type:")
|
||||||
|
sortTypes = [
|
||||||
|
"hot","top","new","rising","controversial"
|
||||||
|
]
|
||||||
|
sortType = self.chooseFrom(sortTypes)
|
||||||
|
GLOBAL.arguments.sort = sortType
|
||||||
|
|
||||||
|
if sortType in ["top","controversial"]:
|
||||||
|
print("\nselect time filter:")
|
||||||
|
timeFilters = [
|
||||||
|
"hour","day","week","month","year","all"
|
||||||
|
]
|
||||||
|
timeFilter = self.chooseFrom(timeFilters)
|
||||||
|
GLOBAL.arguments.time = timeFilter
|
||||||
|
else:
|
||||||
|
GLOBAL.arguments.time = "all"
|
||||||
|
|
||||||
|
elif programMode == "submitted":
|
||||||
|
GLOBAL.arguments.submitted = True
|
||||||
|
GLOBAL.arguments.user = input("\nredditor: ")
|
||||||
|
|
||||||
|
print("\nselect sort type:")
|
||||||
|
sortTypes = [
|
||||||
|
"hot","top","new","controversial"
|
||||||
|
]
|
||||||
|
sortType = self.chooseFrom(sortTypes)
|
||||||
|
GLOBAL.arguments.sort = sortType
|
||||||
|
|
||||||
|
if sortType == "top":
|
||||||
|
print("\nselect time filter:")
|
||||||
|
timeFilters = [
|
||||||
|
"hour","day","week","month","year","all"
|
||||||
|
]
|
||||||
|
timeFilter = self.chooseFrom(timeFilters)
|
||||||
|
GLOBAL.arguments.time = timeFilter
|
||||||
|
else:
|
||||||
|
GLOBAL.arguments.time = "all"
|
||||||
|
|
||||||
|
elif programMode == "upvoted":
|
||||||
|
GLOBAL.arguments.upvoted = True
|
||||||
|
GLOBAL.arguments.user = input("\nredditor: ")
|
||||||
|
|
||||||
|
elif programMode == "saved":
|
||||||
|
GLOBAL.arguments.saved = True
|
||||||
|
|
||||||
|
elif programMode == "log":
|
||||||
|
while True:
|
||||||
|
GLOBAL.arguments.log = input("\nlog file directory:")
|
||||||
|
if Path(GLOBAL.arguments.log ).is_file():
|
||||||
|
break
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
GLOBAL.arguments.limit = int(input("\nlimit: "))
|
||||||
|
break
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
def prepareAttributes():
|
def prepareAttributes():
|
||||||
ATTRIBUTES = {}
|
ATTRIBUTES = {}
|
||||||
@@ -285,7 +360,8 @@ def prepareAttributes():
|
|||||||
ATTRIBUTES["time"] = GLOBAL.arguments.time
|
ATTRIBUTES["time"] = GLOBAL.arguments.time
|
||||||
|
|
||||||
elif GLOBAL.arguments.subreddit is not None:
|
elif GLOBAL.arguments.subreddit is not None:
|
||||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit)
|
if type(GLOBAL.arguments.subreddit) == list:
|
||||||
|
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit)
|
||||||
|
|
||||||
ATTRIBUTES["subreddit"] = GLOBAL.arguments.subreddit
|
ATTRIBUTES["subreddit"] = GLOBAL.arguments.subreddit
|
||||||
|
|
||||||
@@ -305,13 +381,36 @@ def prepareAttributes():
|
|||||||
|
|
||||||
return ATTRIBUTES
|
return ATTRIBUTES
|
||||||
|
|
||||||
|
def postFromLog(fileName):
|
||||||
|
"""Analyze a log file and return a list of dictionaries containing
|
||||||
|
submissions
|
||||||
|
"""
|
||||||
|
if Path.is_file(Path(fileName)):
|
||||||
|
content = jsonFile(fileName).read()
|
||||||
|
else:
|
||||||
|
print("File not found")
|
||||||
|
quit()
|
||||||
|
|
||||||
|
try:
|
||||||
|
del content["HEADER"]
|
||||||
|
except KeyError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
posts = []
|
||||||
|
|
||||||
|
for post in content:
|
||||||
|
if not content[post][-1]['postType'] == None:
|
||||||
|
posts.append(content[post][-1])
|
||||||
|
|
||||||
|
return posts
|
||||||
|
|
||||||
def postExists(POST):
|
def postExists(POST):
|
||||||
"""Figure out a file's name and checks if the file already exists"""
|
"""Figure out a file's name and checks if the file already exists"""
|
||||||
|
|
||||||
title = nameCorrector(POST['postTitle'])
|
title = nameCorrector(POST['postTitle'])
|
||||||
FILENAME = title + "_" + POST['postId']
|
FILENAME = title + "_" + POST['postId']
|
||||||
PATH = GLOBAL.directory / POST["postSubreddit"]
|
PATH = GLOBAL.directory / POST["postSubreddit"]
|
||||||
possibleExtensions = [".jpg",".png",".mp4",".gif",".webm"]
|
possibleExtensions = [".jpg",".png",".mp4",".gif",".webm",".md"]
|
||||||
|
|
||||||
for i in range(2):
|
for i in range(2):
|
||||||
for extension in possibleExtensions:
|
for extension in possibleExtensions:
|
||||||
@@ -347,6 +446,7 @@ def download(submissions):
|
|||||||
|
|
||||||
if postExists(submissions[i]):
|
if postExists(submissions[i]):
|
||||||
result = False
|
result = False
|
||||||
|
print(submissions[i]['postType'].upper())
|
||||||
print("It already exists")
|
print("It already exists")
|
||||||
duplicates += 1
|
duplicates += 1
|
||||||
downloadedCount -= 1
|
downloadedCount -= 1
|
||||||
@@ -481,20 +581,25 @@ def download(submissions):
|
|||||||
print(" Total of {} links downloaded!".format(downloadedCount))
|
print(" Total of {} links downloaded!".format(downloadedCount))
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
if sys.argv[-1].endswith(__file__):
|
GLOBAL.arguments = parseArguments()
|
||||||
GLOBAL.arguments = parseArguments(input("> ").split())
|
|
||||||
else:
|
|
||||||
GLOBAL.arguments = parseArguments()
|
|
||||||
if GLOBAL.arguments.directory is not None:
|
if GLOBAL.arguments.directory is not None:
|
||||||
GLOBAL.directory = Path(GLOBAL.arguments.directory)
|
GLOBAL.directory = Path(GLOBAL.arguments.directory)
|
||||||
else:
|
else:
|
||||||
print("Invalid directory")
|
GLOBAL.directory = Path(input("download directory: "))
|
||||||
|
|
||||||
|
print("\n"," ".join(sys.argv),"\n")
|
||||||
|
|
||||||
|
try:
|
||||||
|
checkConflicts()
|
||||||
|
except ProgramModeError as err:
|
||||||
|
PromptUser()
|
||||||
|
except Exception as err:
|
||||||
|
print(err)
|
||||||
quit()
|
quit()
|
||||||
GLOBAL.config = getConfig(Path(PurePath(__file__).parent / 'config.json'))
|
|
||||||
|
|
||||||
checkConflicts()
|
GLOBAL.config = getConfig("config.json")
|
||||||
|
|
||||||
print(sys.argv)
|
|
||||||
|
|
||||||
if GLOBAL.arguments.log is not None:
|
if GLOBAL.arguments.log is not None:
|
||||||
logDir = Path(GLOBAL.arguments.log)
|
logDir = Path(GLOBAL.arguments.log)
|
||||||
@@ -536,11 +641,22 @@ def main():
|
|||||||
download(POSTS)
|
download(POSTS)
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|
||||||
|
log_stream = StringIO()
|
||||||
|
logging.basicConfig(stream=log_stream, level=logging.INFO)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
VanillaPrint = print
|
VanillaPrint = print
|
||||||
print = printToFile
|
print = printToFile
|
||||||
GLOBAL.RUN_TIME = time.time()
|
GLOBAL.RUN_TIME = time.time()
|
||||||
main()
|
main()
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
|
if GLOBAL.directory is None:
|
||||||
|
GLOBAL.directory = Path(".\\")
|
||||||
print("\nQUITTING...")
|
print("\nQUITTING...")
|
||||||
quit()
|
quit()
|
||||||
|
except Exception as exception:
|
||||||
|
logging.error("Runtime error!", exc_info=full_exc_info(sys.exc_info()))
|
||||||
|
print(log_stream.getvalue())
|
||||||
|
|
||||||
|
input("Press enter to quit\n")
|
||||||
|
|||||||
@@ -4,19 +4,14 @@ import sys
|
|||||||
import urllib.request
|
import urllib.request
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import imgurpython
|
||||||
|
from multiprocessing import Queue
|
||||||
|
|
||||||
from src.errors import (AlbumNotDownloadedCompletely, FileAlreadyExistsError,
|
from src.errors import (AlbumNotDownloadedCompletely, FileAlreadyExistsError,
|
||||||
FileNameTooLong, ImgurLoginError,
|
FileNameTooLong, ImgurLoginError,
|
||||||
NotADownloadableLinkError)
|
NotADownloadableLinkError)
|
||||||
from src.tools import GLOBAL, nameCorrector, printToFile
|
from src.tools import GLOBAL, nameCorrector, printToFile
|
||||||
|
|
||||||
try:
|
|
||||||
from imgurpython import *
|
|
||||||
except ModuleNotFoundError:
|
|
||||||
print("\nimgurpython not found on your computer, installing...\n")
|
|
||||||
from src.tools import install
|
|
||||||
install("imgurpython")
|
|
||||||
from imgurpython import *
|
|
||||||
|
|
||||||
VanillaPrint = print
|
VanillaPrint = print
|
||||||
print = printToFile
|
print = printToFile
|
||||||
|
|
||||||
@@ -181,7 +176,7 @@ class Imgur:
|
|||||||
"""Initialize imgur api"""
|
"""Initialize imgur api"""
|
||||||
|
|
||||||
config = GLOBAL.config
|
config = GLOBAL.config
|
||||||
return ImgurClient(
|
return imgurpython.ImgurClient(
|
||||||
config['imgur_client_id'],
|
config['imgur_client_id'],
|
||||||
config['imgur_client_secret']
|
config['imgur_client_secret']
|
||||||
)
|
)
|
||||||
@@ -213,7 +208,7 @@ class Imgur:
|
|||||||
elif identity['type'] == 'album':
|
elif identity['type'] == 'album':
|
||||||
return {'object':self.imgurClient.get_album(identity['id']),
|
return {'object':self.imgurClient.get_album(identity['id']),
|
||||||
'type':'album'}
|
'type':'album'}
|
||||||
|
@staticmethod
|
||||||
def get_credits():
|
def get_credits():
|
||||||
return Imgur.initImgur().get_credits()
|
return Imgur.initImgur().get_credits()
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,36 @@
|
|||||||
|
import sys
|
||||||
|
|
||||||
|
class FauxTb(object):
|
||||||
|
def __init__(self, tb_frame, tb_lineno, tb_next):
|
||||||
|
self.tb_frame = tb_frame
|
||||||
|
self.tb_lineno = tb_lineno
|
||||||
|
self.tb_next = tb_next
|
||||||
|
|
||||||
|
def current_stack(skip=0):
|
||||||
|
try: 1/0
|
||||||
|
except ZeroDivisionError:
|
||||||
|
f = sys.exc_info()[2].tb_frame
|
||||||
|
for i in range(skip + 2):
|
||||||
|
f = f.f_back
|
||||||
|
lst = []
|
||||||
|
while f is not None:
|
||||||
|
lst.append((f, f.f_lineno))
|
||||||
|
f = f.f_back
|
||||||
|
return lst
|
||||||
|
|
||||||
|
def extend_traceback(tb, stack):
|
||||||
|
"""Extend traceback with stack info."""
|
||||||
|
head = tb
|
||||||
|
for tb_frame, tb_lineno in stack:
|
||||||
|
head = FauxTb(tb_frame, tb_lineno, head)
|
||||||
|
return head
|
||||||
|
|
||||||
|
def full_exc_info(exc_info):
|
||||||
|
"""Like sys.exc_info, but includes the full traceback."""
|
||||||
|
t, v, tb = exc_info
|
||||||
|
full_tb = extend_traceback(tb, current_stack(1))
|
||||||
|
return t, v, full_tb
|
||||||
|
|
||||||
class RedditLoginFailed(Exception):
|
class RedditLoginFailed(Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@@ -19,6 +52,15 @@ class FileNameTooLong(Exception):
|
|||||||
class InvalidRedditLink(Exception):
|
class InvalidRedditLink(Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
class ProgramModeError(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
class SearchModeError(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
class RedditorNameError(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
class NoMatchingSubmissionFound(Exception):
|
class NoMatchingSubmissionFound(Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|||||||
@@ -3,14 +3,7 @@ import random
|
|||||||
import socket
|
import socket
|
||||||
import webbrowser
|
import webbrowser
|
||||||
|
|
||||||
try:
|
import praw
|
||||||
import praw
|
|
||||||
except ModuleNotFoundError:
|
|
||||||
print("\nPRAW not found on your computer, installing...\n")
|
|
||||||
from src.tools import install
|
|
||||||
install("praw")
|
|
||||||
import praw
|
|
||||||
|
|
||||||
from prawcore.exceptions import NotFound, ResponseException, Forbidden
|
from prawcore.exceptions import NotFound, ResponseException, Forbidden
|
||||||
|
|
||||||
from src.tools import GLOBAL, createLogFile, jsonFile, printToFile
|
from src.tools import GLOBAL, createLogFile, jsonFile, printToFile
|
||||||
@@ -360,7 +353,7 @@ def redditSearcher(posts,SINGLE_POST=False):
|
|||||||
printSubmission(submission,subCount,orderCount)
|
printSubmission(submission,subCount,orderCount)
|
||||||
subList.append(details)
|
subList.append(details)
|
||||||
|
|
||||||
allPosts = {**allPosts,**details}
|
allPosts[subCount] = [details]
|
||||||
|
|
||||||
postsFile.add(allPosts)
|
postsFile.add(allPosts)
|
||||||
|
|
||||||
|
|||||||
@@ -2,20 +2,11 @@ import io
|
|||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
|
|
||||||
try:
|
|
||||||
from pip import main as pipmain
|
|
||||||
except:
|
|
||||||
from pip._internal import main as pipmain
|
|
||||||
|
|
||||||
from os import makedirs, path, remove
|
from os import makedirs, path, remove
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from src.errors import FileNotFoundError
|
from src.errors import FileNotFoundError
|
||||||
|
|
||||||
def install(package):
|
|
||||||
pipmain(['install', package])
|
|
||||||
|
|
||||||
class GLOBAL:
|
class GLOBAL:
|
||||||
"""Declare global variables"""
|
"""Declare global variables"""
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user