mirror of
https://github.com/KevinMidboe/bulk-downloader-for-reddit.git
synced 2026-01-08 10:15:36 +00:00
Compare commits
70 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0ec4bb3008 | ||
|
|
0dbe2ed917 | ||
|
|
9f831e1b78 | ||
|
|
59012077e1 | ||
|
|
5e3c79160b | ||
|
|
1e8eaa1a8d | ||
|
|
7dbc83fdce | ||
|
|
50a77f6ba5 | ||
|
|
4f7e406cd6 | ||
|
|
ded3cece8c | ||
|
|
dd671fd738 | ||
|
|
b357dff52c | ||
|
|
32ffd3b861 | ||
|
|
02673c3950 | ||
|
|
8448e47080 | ||
|
|
39f2c73f4c | ||
|
|
fe942b4734 | ||
|
|
205617e051 | ||
|
|
b93b206a96 | ||
|
|
b84684f786 | ||
|
|
68558950ca | ||
|
|
795965f754 | ||
|
|
d2ed8327df | ||
|
|
a28a7776ab | ||
|
|
1a50bcecb0 | ||
|
|
d8a2204024 | ||
|
|
0577c332b5 | ||
|
|
e09da8db73 | ||
|
|
fd507870e1 | ||
|
|
c434d346a7 | ||
|
|
fcc3e53da7 | ||
|
|
7452af8bd0 | ||
|
|
1fd6951420 | ||
|
|
bddac14804 | ||
|
|
6e2c69d053 | ||
|
|
bedd481195 | ||
|
|
2994adea38 | ||
|
|
f1a9f1d3e4 | ||
|
|
9849c0439b | ||
|
|
ef9f0c543f | ||
|
|
26531fd6c0 | ||
|
|
cae044f7b6 | ||
|
|
4daa9cd66a | ||
|
|
e83d2626d3 | ||
|
|
f67c5f62c6 | ||
|
|
fef559cd40 | ||
|
|
42672dc15d | ||
|
|
3b15141b49 | ||
|
|
d080ca17bc | ||
|
|
7f97bd212a | ||
|
|
65592c5d3a | ||
|
|
57a5f0c85c | ||
|
|
f6240f402e | ||
|
|
fd179c0e4b | ||
|
|
d9dc3132f6 | ||
|
|
45191e2c60 | ||
|
|
319a6c82a3 | ||
|
|
2c62337fac | ||
|
|
7c9c84c411 | ||
|
|
8922c27f9b | ||
|
|
cc59710f33 | ||
|
|
49129bea24 | ||
|
|
a701444d5b | ||
|
|
5a030a156c | ||
|
|
ffe3839aee | ||
|
|
a796038b71 | ||
|
|
ddccebbc70 | ||
|
|
5cfa443f83 | ||
|
|
2103c62218 | ||
|
|
10128b63ac |
7
.gitignore
vendored
7
.gitignore
vendored
@@ -1,4 +1,5 @@
|
||||
build/
|
||||
dist/
|
||||
MANIFEST
|
||||
__pycache__/
|
||||
src/__pycache__/
|
||||
logs/
|
||||
*.json
|
||||
src/__pycache__/
|
||||
179
README.md
179
README.md
@@ -1,41 +1,30 @@
|
||||
# Bulk Downloader for Reddit
|
||||
This program downloads imgur, gfycat and direct image and video links of saved posts from a reddit account. It is written in Python 3.
|
||||
|
||||
**PLEASE** post any issue you had with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
||||
**PLEASE** post any issue you have with the script to [Issues](https://github.com/aliparlakci/bulk-downloader-for-reddit/issues) tab. Since I don't have any testers or contributers I need your feedback.
|
||||
|
||||
## Table of Contents
|
||||
## What it can do
|
||||
- Can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- Sorts posts by hot, top, new and so on
|
||||
- Downloads imgur albums, gfycat links, [self posts](#i-cant-open-the-self-post-files) and any link to a direct image
|
||||
- Skips the existing ones
|
||||
- Puts post titles to file's name
|
||||
- Puts every post to its subreddit's folder
|
||||
- Saves a reusable copy of posts' details that are found so that they can be re-downloaded again
|
||||
- Logs failed ones in a file to so that you can try to download them later
|
||||
- Can run with double-clicking on Windows
|
||||
|
||||
- [What it can do?](#what-it-can-do)
|
||||
- [Requirements](#requirements)
|
||||
- [Setting up the script](#setting-up-the-script)
|
||||
- [Creating an imgur app](#creating-an-imgur-app)
|
||||
- [Program Modes](#program-modes)
|
||||
- [Running the script](#running-the-script)
|
||||
- [Using the command line arguments](#using-the-command-line-arguments)
|
||||
- [Examples](#examples)
|
||||
- [FAQ](#faq)
|
||||
- [Changelog](#changelog)
|
||||
## [Download the latest release](https://github.com/aliparlakci/bulk-downloader-for-reddit/releases/latest)
|
||||
|
||||
## What it can do?
|
||||
### It...
|
||||
- can get posts from: frontpage, subreddits, multireddits, redditor's submissions, upvoted and saved posts; search results or just plain reddit links
|
||||
- sorts post by hot, top, new and so on
|
||||
- downloads imgur albums, gfycat links, [self posts](#i-can-t-open-the-self-posts-) and any link to a direct image
|
||||
- skips the existing ones
|
||||
- puts post titles to file's name
|
||||
- puts every post to its subreddit's folder
|
||||
- saves reusable a copy of posts' details that are found so that they can be re-downloaded again
|
||||
- logs failed ones in a file to so that you can try to download them later
|
||||
- can be run with double-clicking on Windows (but I don't recommend it)
|
||||
|
||||
## Requirements
|
||||
- Python 3.x*
|
||||
|
||||
You can install Python 3 here: [https://www.python.org/downloads/](https://www.python.org/downloads/)
|
||||
## How it works
|
||||
|
||||
You have to check "**Add Python 3 to PATH**" option when installing in order it to run correctly.
|
||||
- For **Windows** and **Linux** users, there are executable files to run easily without installing a third party program. But if you are a paranoid like me, you can **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
- In Windows, double click on script.exe file
|
||||
- In Linux, extract files to a folder and open terminal inside it. Type **`./script`**
|
||||
|
||||
- **MacOS** users have to **[compile it from source code](docs/COMPILE_FROM_SOURCE.md)**.
|
||||
|
||||
*\*Although the latest version of python is suggested, you can use 3.6.5 since it runs perfectly on that version*
|
||||
Script also accepts **command-line arguments**, get further information from **[`python script.py --help`](docs/COMMAND_LINE_ARGUMENTS.md)**
|
||||
|
||||
## Setting up the script
|
||||
Because this is not a commercial app, you need to create an imgur developer app in order API to work.
|
||||
@@ -50,120 +39,28 @@ Because this is not a commercial app, you need to create an imgur developer app
|
||||
|
||||
It should redirect to a page which shows your **imgur_client_id** and **imgur_client_secret**
|
||||
|
||||
\*Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||
|
||||
## Program Modes
|
||||
All the program modes are activated with command-line arguments as shown [here](#using-the-command-line-arguments)
|
||||
- **saved mode**
|
||||
- Gets posts from given user's saved posts.
|
||||
- **submitted mode**
|
||||
- Gets posts from given user's submitted posts.
|
||||
- **upvoted mode**
|
||||
- Gets posts from given user's upvoted posts.
|
||||
- **subreddit mode**
|
||||
- Gets posts from given subreddit or subreddits that is sorted by given type and limited by given number.
|
||||
- You may also use search in this mode. See [`py -3 script.py --help`](#using-the-command-line-arguments).
|
||||
- **multireddit mode**
|
||||
- Gets posts from given user's given multireddit that is sorted by given type and limited by given number.
|
||||
- **link mode**
|
||||
- Gets posts from given reddit link.
|
||||
- You may customize the behaviour with `--sort`, `--time`, `--limit`.
|
||||
- You may also use search in this mode. See [`py -3 script.py --help`](#using-the-command-line-arguments).
|
||||
- **log read mode**
|
||||
- Takes a log file which created by itself (json files), reads posts and tries downloading them again.
|
||||
- Running log read mode for FAILED.json file once after the download is complete is **HIGHLY** recommended as unexpected problems may occur.
|
||||
|
||||
## Running the script
|
||||
**DO NOT** let more than one instance of the script run as it interferes with IMGUR Request Rate.
|
||||
|
||||
### Using the command line arguments
|
||||
If no arguments are passed program will prompt you for arguments below which means you may start up the script with double-clicking on it (at least on Windows for sure).
|
||||
|
||||
Open up the [terminal](https://www.reddit.com/r/NSFW411/comments/8vtnl8/meta_i_made_reddit_downloader_that_can_download/e1rnbnl) and navigate to where script.py is. If you are unfamiliar with changing directories in terminal see Change Directories in [this article](https://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything).
|
||||
|
||||
Run the script.py file from terminal with command-line arguments. Here is the help page:
|
||||
|
||||
Use `.\` for current directory and `..\` for upper directory when using short directories, otherwise it might act weird.
|
||||
|
||||
```console
|
||||
$ py -3 script.py --help
|
||||
usage: script.py [-h] [--link link] [--saved] [--submitted] [--upvoted]
|
||||
[--log LOG FILE] [--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
[--multireddit MULTIREDDIT] [--user redditor]
|
||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||
[--time TIME_LIMIT] [--NoDownload]
|
||||
DIRECTORY
|
||||
|
||||
This program downloads media from reddit posts
|
||||
|
||||
positional arguments:
|
||||
DIRECTORY Specifies the directory where posts will be downloaded
|
||||
to
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--link link, -l link Get posts from link
|
||||
--saved Triggers saved mode
|
||||
--submitted Gets posts of --user
|
||||
--upvoted Gets upvoted posts of --user
|
||||
--log LOG FILE Triggers log read mode and takes a log file
|
||||
--subreddit SUBREDDIT [SUBREDDIT ...]
|
||||
Triggers subreddit mode and takes subreddit's name
|
||||
without r/. use "frontpage" for frontpage
|
||||
--multireddit MULTIREDDIT
|
||||
Triggers multireddit mode and takes multireddit's name
|
||||
without m/
|
||||
--user redditor reddit username if needed. use "me" for current user
|
||||
--search query Searches for given query in given subreddits
|
||||
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
||||
relevance default: hot
|
||||
--limit Limit default: unlimited
|
||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||
all
|
||||
--NoDownload Just gets the posts and store them in a file for
|
||||
downloading later
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
#### Don't include `py -3 script.py` part if you start the script by double-clicking
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
py -3 script.py C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
```
|
||||
\* Select **OAuth 2 authorization without a callback URL** first then select **Anonymous usage without user authorization** if it says *Authorization callback URL: required*
|
||||
|
||||
## FAQ
|
||||
### I can't startup the script no matter what.
|
||||
- Try `python3` or `python` or `py -3` as python have real issues about naming their program
|
||||
|
||||
### I can't open the self posts.
|
||||
- Self posts are held at subreddit as Markdown. So, the script downloads them as Markdown in order not to lose their stylings. However, there is a great Chrome extension [here](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with Chrome.
|
||||
### I can't open the self post files.
|
||||
- Self posts are held at reddit as styled with markdown. So, the script downloads them as they are in order not to lose their stylings.
|
||||
However, there is a [great Chrome extension](https://chrome.google.com/webstore/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk) for viewing Markdown files with its styling. Install it and open the files with [Chrome](https://www.google.com/intl/tr/chrome/).
|
||||
|
||||
## Changelog
|
||||
### 10/07/2018
|
||||
### [13/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/9f831e1b784a770c82252e909462871401a05c11)
|
||||
- Change config.json file's path to home directory
|
||||
|
||||
### [12/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/50a77f6ba54c24f5647d5ea4e177400b71ff04a7)
|
||||
- Added binaries for Windows and Linux
|
||||
- Wait on KeyboardInterrupt
|
||||
- Accept multiple subreddit input
|
||||
- Fixed the bug that prevents choosing "[0] exit" with typing "exit"
|
||||
|
||||
### [11/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/a28a7776ab826dea2a8d93873a94cd46db3a339b)
|
||||
- Improvements on UX and UI
|
||||
- Added logging errors to CONSOLE_LOG.txt
|
||||
- Using current directory if directory has not been given yet.
|
||||
|
||||
### [10/07/2018](https://github.com/aliparlakci/bulk-downloader-for-reddit/tree/ffe3839aee6dc1a552d95154d817aefc2b66af81)
|
||||
- Added support for *self* post
|
||||
- Now getting posts is quicker
|
||||
|
||||
98
docs/COMMAND_LINE_ARGUMENTS.md
Normal file
98
docs/COMMAND_LINE_ARGUMENTS.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Using command-line arguments
|
||||
|
||||
See **[compiling from source](COMPILE_FROM_SOURCE.md)** page first unless you are using an executable file. If you are using an executable file, see [using terminal](COMPILE_FROM_SOURCE.md#using-terminal) and come back.
|
||||
|
||||
***Use*** `.\script.exe` ***or*** `./script` ***if you are using the executable***.
|
||||
```console
|
||||
$ python script.py --help
|
||||
usage: script.py [-h] [--directory DIRECTORY] [--link link] [--saved]
|
||||
[--submitted] [--upvoted] [--log LOG FILE]
|
||||
[--subreddit SUBREDDIT [SUBREDDIT ...]]
|
||||
[--multireddit MULTIREDDIT] [--user redditor]
|
||||
[--search query] [--sort SORT TYPE] [--limit Limit]
|
||||
[--time TIME_LIMIT] [--NoDownload]
|
||||
|
||||
This program downloads media from reddit posts
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--directory DIRECTORY
|
||||
Specifies the directory where posts will be downloaded
|
||||
to
|
||||
--link link, -l link Get posts from link
|
||||
--saved Triggers saved mode
|
||||
--submitted Gets posts of --user
|
||||
--upvoted Gets upvoted posts of --user
|
||||
--log LOG FILE Triggers log read mode and takes a log file
|
||||
--subreddit SUBREDDIT [SUBREDDIT ...]
|
||||
Triggers subreddit mode and takes subreddit's name
|
||||
without r/. use "frontpage" for frontpage
|
||||
--multireddit MULTIREDDIT
|
||||
Triggers multireddit mode and takes multireddit's name
|
||||
without m/
|
||||
--user redditor reddit username if needed. use "me" for current user
|
||||
--search query Searches for given query in given subreddits
|
||||
--sort SORT TYPE Either hot, top, new, controversial, rising or
|
||||
relevance default: hot
|
||||
--limit Limit default: unlimited
|
||||
--time TIME_LIMIT Either hour, day, week, month, year or all. default:
|
||||
all
|
||||
--NoDownload Just gets the posts and store them in a file for
|
||||
downloading later
|
||||
```
|
||||
|
||||
# Examples
|
||||
|
||||
- **Use `python3` instead of `python` if you are using *MacOS* or *Linux***
|
||||
|
||||
```console
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
.\script.exe
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py
|
||||
```
|
||||
|
||||
```console
|
||||
.\script.exe -- directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
./script --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --sort new --time all --limit 10 --link "https://www.reddit.com/r/gifs/search?q=dogs&restrict_sr=on&type=link&sort=new&t=month"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --link "https://www.reddit.com/r/learnprogramming/comments/7mjw12/"
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --search cats --sort new --time all --subreddit gifs pics --NoDownload
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --user [USER_NAME] --submitted --limit 10
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER --multireddit good_subs --user [USER_NAME] --sort top --time week --limit 250
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory .\\NEW_FOLDER\\ANOTHER_FOLDER --saved --limit 1000
|
||||
```
|
||||
|
||||
```console
|
||||
python script.py --directory C:\\NEW_FOLDER\\ANOTHER_FOLDER --log UNNAMED_FOLDER\\FAILED.json
|
||||
```
|
||||
|
||||
# FAQ
|
||||
## I can't startup the script no matter what.
|
||||
See **[finding the correct keyword for Python](COMPILE_FROM_SOURCE.md#finding-the-correct-keyword-for-python)**
|
||||
42
docs/COMPILE_FROM_SOURCE.md
Normal file
42
docs/COMPILE_FROM_SOURCE.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Compiling from source code
|
||||
## Requirements
|
||||
### Python 3 Interpreter
|
||||
Latest* version of **Python 3** is needed. See if it is already installed [here](#finding-the-correct-keyword-for-python). If not, download the matching release for your platform [here](https://www.python.org/downloads/) and install it. If you are a *Windows* user, selecting **Add Python 3 to PATH** option is mandatory.
|
||||
|
||||
\* *Use Python 3.6.5 if you encounter an issue*
|
||||
## Using terminal
|
||||
### To open it...
|
||||
- **On Windows 8/8.1/10**: Press the File tab on **Windows Explorer**, click on **Open Windows PowerShell** or **Open Windows Command Prompt** or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Windows 7**: Press **WindowsKey+R**, type **cmd** and hit Enter or look for *Command Prompt* or *PowerShell* in *Start Menu*.
|
||||
|
||||
- **On Linux**: Right-click in a folder and select **Open Terminal** or press **Ctrl+Alt+T** or look for **Terminal** in the programs.
|
||||
|
||||
- **On MacOS**: Look for an app called **Terminal**.
|
||||
|
||||
### Navigating to the directory where script is downloaded
|
||||
Go inside the folder where script.py is located. If you are not familier with changing directories on command-prompt and terminal read *Changing Directories* in [this article](https://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything)
|
||||
|
||||
## Finding the correct keyword for Python
|
||||
Enter these lines to terminal window until it prints out the version you have downloaded and installed:
|
||||
|
||||
- `python --version`
|
||||
- `python3 --version`
|
||||
- `python3.7 --version`
|
||||
- `python3.6 --version`
|
||||
- `py --version`
|
||||
- `py -3 --version`
|
||||
- `py -3.6 --version`
|
||||
- `py -3.7 --version`
|
||||
|
||||
Once it does, your keyword is without the `--version` part.
|
||||
|
||||
## Installing dependencies
|
||||
Enter the line below to terminal window when you are in the directory where script.py is, use your keyword for Python:
|
||||
```console
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Now, you can go to [Using command-line arguments](COMMAND_LINE_ARGUMENTS.md)
|
||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
||||
requests
|
||||
praw
|
||||
imgurpython
|
||||
329
script.py
329
script.py
@@ -6,30 +6,26 @@ saved posts from a reddit account. It is written in Python 3.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from io import StringIO
|
||||
from pathlib import Path, PurePath
|
||||
|
||||
from src.downloader import Direct, Gfycat, Imgur, Self
|
||||
from src.errors import *
|
||||
from src.parser import LinkDesigner
|
||||
from src.searcher import getPosts
|
||||
from src.tools import (GLOBAL, createLogFile, jsonFile, nameCorrector,
|
||||
printToFile)
|
||||
from src.errors import *
|
||||
|
||||
__author__ = "Ali Parlakci"
|
||||
__license__ = "GPL"
|
||||
__version__ = "1.0.0"
|
||||
__version__ = "1.1.2"
|
||||
__maintainer__ = "Ali Parlakci"
|
||||
__email__ = "parlakciali@gmail.com"
|
||||
|
||||
def debug(*post):
|
||||
GLOBAL.config = getConfig('config.json')
|
||||
GLOBAL.directory = Path(".\\debug\\")
|
||||
download([*post])
|
||||
quit()
|
||||
|
||||
def getConfig(configFileName):
|
||||
"""Read credentials from config.json file"""
|
||||
|
||||
@@ -66,7 +62,7 @@ def parseArguments(arguments=[]):
|
||||
description="This program downloads " \
|
||||
"media from reddit " \
|
||||
"posts")
|
||||
parser.add_argument("directory",
|
||||
parser.add_argument("--directory",
|
||||
help="Specifies the directory where posts will be " \
|
||||
"downloaded to",
|
||||
metavar="DIRECTORY")
|
||||
@@ -88,7 +84,9 @@ def parseArguments(arguments=[]):
|
||||
help="Gets upvoted posts of --user")
|
||||
|
||||
parser.add_argument("--log",
|
||||
help="Triggers log read mode and takes a log file",
|
||||
help="Takes a log file which created by itself " \
|
||||
"(json files), reads posts and tries downloadin" \
|
||||
"g them again.",
|
||||
# type=argparse.FileType('r'),
|
||||
metavar="LOG FILE")
|
||||
|
||||
@@ -131,7 +129,6 @@ def parseArguments(arguments=[]):
|
||||
parser.add_argument("--limit",
|
||||
help="default: unlimited",
|
||||
metavar="Limit",
|
||||
default=None,
|
||||
type=int)
|
||||
|
||||
parser.add_argument("--time",
|
||||
@@ -157,88 +154,177 @@ def checkConflicts():
|
||||
if not, raise errors
|
||||
"""
|
||||
|
||||
if GLOBAL.arguments.saved is False:
|
||||
saved = 0
|
||||
else:
|
||||
saved = 1
|
||||
|
||||
if GLOBAL.arguments.subreddit is None:
|
||||
subreddit = 0
|
||||
else:
|
||||
subreddit = 1
|
||||
|
||||
if GLOBAL.arguments.submitted is False:
|
||||
submitted = 0
|
||||
else:
|
||||
submitted = 1
|
||||
|
||||
if GLOBAL.arguments.search is None:
|
||||
search = 0
|
||||
else:
|
||||
search = 1
|
||||
|
||||
if GLOBAL.arguments.log is None:
|
||||
log = 0
|
||||
else:
|
||||
log = 1
|
||||
|
||||
if GLOBAL.arguments.link is None:
|
||||
link = 0
|
||||
else:
|
||||
link = 1
|
||||
|
||||
if GLOBAL.arguments.user is None:
|
||||
user = 0
|
||||
else:
|
||||
user = 1
|
||||
|
||||
if GLOBAL.arguments.upvoted is False:
|
||||
upvoted = 0
|
||||
else:
|
||||
upvoted = 1
|
||||
modes = ["saved","subreddit","submitted","search","log","link","upvoted"]
|
||||
|
||||
if not saved+subreddit+log+link+submitted+upvoted == 1:
|
||||
print("Program mode is invalid")
|
||||
quit()
|
||||
values = {
|
||||
x: 0 if getattr(GLOBAL.arguments,x) is None or \
|
||||
getattr(GLOBAL.arguments,x) is False \
|
||||
else 1 \
|
||||
for x in modes
|
||||
}
|
||||
|
||||
if not sum(values[x] for x in values) == 1:
|
||||
raise ProgramModeError("Invalid program mode")
|
||||
|
||||
if search+subreddit == 2:
|
||||
print("You cannot search in your saved posts")
|
||||
quit()
|
||||
if values["search"]+values["saved"] == 2:
|
||||
raise SearchModeError("You cannot search in your saved posts")
|
||||
|
||||
if search+submitted == 2:
|
||||
print("You cannot search in submitted posts")
|
||||
quit()
|
||||
if values["search"]+values["submitted"] == 2:
|
||||
raise SearchModeError("You cannot search in submitted posts")
|
||||
|
||||
if search+upvoted == 2:
|
||||
print("You cannot search in upvoted posts")
|
||||
quit()
|
||||
if values["search"]+values["upvoted"] == 2:
|
||||
raise SearchModeError("You cannot search in upvoted posts")
|
||||
|
||||
if upvoted+submitted == 1 and user == 0:
|
||||
print("No redditor name given")
|
||||
quit()
|
||||
if values["upvoted"]+values["submitted"] == 1 and user == 0:
|
||||
raise RedditorNameError("No redditor name given")
|
||||
|
||||
def postFromLog(fileName):
|
||||
"""Analyze a log file and return a list of dictionaries containing
|
||||
submissions
|
||||
"""
|
||||
if Path.is_file(Path(fileName)):
|
||||
content = jsonFile(fileName).read()
|
||||
else:
|
||||
print("File not found")
|
||||
quit()
|
||||
class PromptUser:
|
||||
@staticmethod
|
||||
def chooseFrom(choices):
|
||||
print()
|
||||
choicesByIndex = list(str(x) for x in range(len(choices)+1))
|
||||
for i in range(len(choices)):
|
||||
print("{indent}[{order}] {mode}".format(
|
||||
indent=" "*4,order=i+1,mode=choices[i]
|
||||
))
|
||||
print(" "*4+"[0] exit\n")
|
||||
choice = input("> ")
|
||||
while not choice.lower() in choices+choicesByIndex+["exit"]:
|
||||
print("Invalid input\n")
|
||||
programModeIndex = input("> ")
|
||||
|
||||
try:
|
||||
del content["HEADER"]
|
||||
except KeyError:
|
||||
pass
|
||||
if choice == "0" or choice == "exit":
|
||||
sys.exit()
|
||||
elif choice in choicesByIndex:
|
||||
return choices[int(choice)-1]
|
||||
else:
|
||||
return choice
|
||||
|
||||
def __init__(self):
|
||||
print("select program mode:")
|
||||
programModes = [
|
||||
"search","subreddit","multireddit",
|
||||
"submitted","upvoted","saved","log"
|
||||
]
|
||||
programMode = self.chooseFrom(programModes)
|
||||
|
||||
posts = []
|
||||
if programMode == "search":
|
||||
GLOBAL.arguments.search = input("\nquery: ")
|
||||
GLOBAL.arguments.subreddit = input("\nsubreddit: ")
|
||||
|
||||
for post in content:
|
||||
if not content[post][-1]['postType'] == None:
|
||||
posts.append(content[post][-1])
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"relevance","top","new"
|
||||
]
|
||||
sortType = self.chooseFrom(sortTypes)
|
||||
GLOBAL.arguments.sort = sortType
|
||||
|
||||
return posts
|
||||
print("\nselect time filter:")
|
||||
timeFilters = [
|
||||
"hour","day","week","month","year","all"
|
||||
]
|
||||
timeFilter = self.chooseFrom(timeFilters)
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
|
||||
if programMode == "subreddit":
|
||||
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit = subredditInput
|
||||
|
||||
while not subredditInput == "":
|
||||
subredditInput = input("subreddit: ")
|
||||
GLOBAL.arguments.subreddit += "+" + subredditInput
|
||||
|
||||
if " " in GLOBAL.arguments.subreddit:
|
||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit.split())
|
||||
|
||||
# DELETE THE PLUS (+) AT THE END
|
||||
GLOBAL.arguments.subreddit = GLOBAL.arguments.subreddit[:-1]
|
||||
|
||||
print(GLOBAL.arguments.subreddit)
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"hot","top","new","rising","controversial"
|
||||
]
|
||||
sortType = self.chooseFrom(sortTypes)
|
||||
GLOBAL.arguments.sort = sortType
|
||||
|
||||
if sortType in ["top","controversial"]:
|
||||
print("\nselect time filter:")
|
||||
timeFilters = [
|
||||
"hour","day","week","month","year","all"
|
||||
]
|
||||
timeFilter = self.chooseFrom(timeFilters)
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
else:
|
||||
GLOBAL.arguments.time = "all"
|
||||
|
||||
elif programMode == "multireddit":
|
||||
GLOBAL.arguments.user = input("\nredditor: ")
|
||||
GLOBAL.arguments.subreddit = input("\nmultireddit: ")
|
||||
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"hot","top","new","rising","controversial"
|
||||
]
|
||||
sortType = self.chooseFrom(sortTypes)
|
||||
GLOBAL.arguments.sort = sortType
|
||||
|
||||
if sortType in ["top","controversial"]:
|
||||
print("\nselect time filter:")
|
||||
timeFilters = [
|
||||
"hour","day","week","month","year","all"
|
||||
]
|
||||
timeFilter = self.chooseFrom(timeFilters)
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
else:
|
||||
GLOBAL.arguments.time = "all"
|
||||
|
||||
elif programMode == "submitted":
|
||||
GLOBAL.arguments.submitted = True
|
||||
GLOBAL.arguments.user = input("\nredditor: ")
|
||||
|
||||
print("\nselect sort type:")
|
||||
sortTypes = [
|
||||
"hot","top","new","controversial"
|
||||
]
|
||||
sortType = self.chooseFrom(sortTypes)
|
||||
GLOBAL.arguments.sort = sortType
|
||||
|
||||
if sortType == "top":
|
||||
print("\nselect time filter:")
|
||||
timeFilters = [
|
||||
"hour","day","week","month","year","all"
|
||||
]
|
||||
timeFilter = self.chooseFrom(timeFilters)
|
||||
GLOBAL.arguments.time = timeFilter
|
||||
else:
|
||||
GLOBAL.arguments.time = "all"
|
||||
|
||||
elif programMode == "upvoted":
|
||||
GLOBAL.arguments.upvoted = True
|
||||
GLOBAL.arguments.user = input("\nredditor: ")
|
||||
|
||||
elif programMode == "saved":
|
||||
GLOBAL.arguments.saved = True
|
||||
|
||||
elif programMode == "log":
|
||||
while True:
|
||||
GLOBAL.arguments.log = input("\nlog file directory:")
|
||||
if Path(GLOBAL.arguments.log ).is_file():
|
||||
break
|
||||
|
||||
while True:
|
||||
try:
|
||||
GLOBAL.arguments.limit = int(input("\nlimit: "))
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def prepareAttributes():
|
||||
ATTRIBUTES = {}
|
||||
@@ -285,7 +371,8 @@ def prepareAttributes():
|
||||
ATTRIBUTES["time"] = GLOBAL.arguments.time
|
||||
|
||||
elif GLOBAL.arguments.subreddit is not None:
|
||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit)
|
||||
if type(GLOBAL.arguments.subreddit) == list:
|
||||
GLOBAL.arguments.subreddit = "+".join(GLOBAL.arguments.subreddit)
|
||||
|
||||
ATTRIBUTES["subreddit"] = GLOBAL.arguments.subreddit
|
||||
|
||||
@@ -305,13 +392,36 @@ def prepareAttributes():
|
||||
|
||||
return ATTRIBUTES
|
||||
|
||||
def postFromLog(fileName):
|
||||
"""Analyze a log file and return a list of dictionaries containing
|
||||
submissions
|
||||
"""
|
||||
if Path.is_file(Path(fileName)):
|
||||
content = jsonFile(fileName).read()
|
||||
else:
|
||||
print("File not found")
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
del content["HEADER"]
|
||||
except KeyError:
|
||||
pass
|
||||
|
||||
posts = []
|
||||
|
||||
for post in content:
|
||||
if not content[post][-1]['postType'] == None:
|
||||
posts.append(content[post][-1])
|
||||
|
||||
return posts
|
||||
|
||||
def postExists(POST):
|
||||
"""Figure out a file's name and checks if the file already exists"""
|
||||
|
||||
title = nameCorrector(POST['postTitle'])
|
||||
FILENAME = title + "_" + POST['postId']
|
||||
PATH = GLOBAL.directory / POST["postSubreddit"]
|
||||
possibleExtensions = [".jpg",".png",".mp4",".gif",".webm"]
|
||||
possibleExtensions = [".jpg",".png",".mp4",".gif",".webm",".md"]
|
||||
|
||||
for i in range(2):
|
||||
for extension in possibleExtensions:
|
||||
@@ -347,6 +457,7 @@ def download(submissions):
|
||||
|
||||
if postExists(submissions[i]):
|
||||
result = False
|
||||
print(submissions[i]['postType'].upper())
|
||||
print("It already exists")
|
||||
duplicates += 1
|
||||
downloadedCount -= 1
|
||||
@@ -397,7 +508,7 @@ def download(submissions):
|
||||
"Imgur login failed. Quitting the program "\
|
||||
"as unexpected errors might occur."
|
||||
)
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
except Exception as exception:
|
||||
print(exception)
|
||||
@@ -481,66 +592,82 @@ def download(submissions):
|
||||
print(" Total of {} links downloaded!".format(downloadedCount))
|
||||
|
||||
def main():
|
||||
if sys.argv[-1].endswith(__file__):
|
||||
GLOBAL.arguments = parseArguments(input("> ").split())
|
||||
else:
|
||||
GLOBAL.arguments = parseArguments()
|
||||
GLOBAL.arguments = parseArguments()
|
||||
|
||||
if GLOBAL.arguments.directory is not None:
|
||||
GLOBAL.directory = Path(GLOBAL.arguments.directory)
|
||||
else:
|
||||
print("Invalid directory")
|
||||
quit()
|
||||
GLOBAL.config = getConfig(Path(PurePath(__file__).parent / 'config.json'))
|
||||
GLOBAL.directory = Path(input("download directory: "))
|
||||
|
||||
checkConflicts()
|
||||
print("\n"," ".join(sys.argv),"\n")
|
||||
|
||||
print(sys.argv)
|
||||
try:
|
||||
checkConflicts()
|
||||
except ProgramModeError as err:
|
||||
PromptUser()
|
||||
except Exception as err:
|
||||
print(err)
|
||||
sys.exit()
|
||||
|
||||
if not Path(GLOBAL.configDirectory).is_dir():
|
||||
os.makedirs(GLOBAL.configDirectory)
|
||||
GLOBAL.config = getConfig(GLOBAL.configDirectory / "config.json")
|
||||
|
||||
if GLOBAL.arguments.log is not None:
|
||||
logDir = Path(GLOBAL.arguments.log)
|
||||
download(postFromLog(logDir))
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
try:
|
||||
POSTS = getPosts(prepareAttributes())
|
||||
except InsufficientPermission:
|
||||
print("You do not have permission to do that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoMatchingSubmissionFound:
|
||||
print("No matching submission was found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoRedditSupoort:
|
||||
print("Reddit does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except NoPrawSupport:
|
||||
print("PRAW does not support that")
|
||||
quit()
|
||||
sys.exit()
|
||||
except MultiredditNotFound:
|
||||
print("Multireddit not found")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidSortingType:
|
||||
print("Invalid sorting type has given")
|
||||
quit()
|
||||
sys.exit()
|
||||
except InvalidRedditLink:
|
||||
print("Invalid reddit link")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if POSTS is None:
|
||||
print("I could not find any posts in that URL")
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
if GLOBAL.arguments.NoDownload:
|
||||
quit()
|
||||
sys.exit()
|
||||
|
||||
else:
|
||||
download(POSTS)
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
log_stream = StringIO()
|
||||
logging.basicConfig(stream=log_stream, level=logging.INFO)
|
||||
|
||||
try:
|
||||
VanillaPrint = print
|
||||
print = printToFile
|
||||
GLOBAL.RUN_TIME = time.time()
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
if GLOBAL.directory is None:
|
||||
GLOBAL.directory = Path(".\\")
|
||||
print("\nQUITTING...")
|
||||
quit()
|
||||
except Exception as exception:
|
||||
logging.error("Runtime error!", exc_info=full_exc_info(sys.exc_info()))
|
||||
print(log_stream.getvalue())
|
||||
|
||||
input("Press enter to quit\n")
|
||||
|
||||
50
setup.py
Normal file
50
setup.py
Normal file
@@ -0,0 +1,50 @@
|
||||
#!C:\Users\Ali\AppData\Local\Programs\Python\Python36\python.exe
|
||||
|
||||
## python setup.py build
|
||||
import sys
|
||||
from cx_Freeze import setup, Executable
|
||||
from script import __version__
|
||||
|
||||
options = {
|
||||
"build_exe": {
|
||||
"packages":[
|
||||
"idna","imgurpython", "praw", "requests"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
if sys.platform == "win32":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit.exe",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
elif sys.platform == "linux":
|
||||
executables = [Executable(
|
||||
"script.py",
|
||||
targetName="bulk-downloader-for-reddit",
|
||||
shortcutName="Bulk Downloader for Reddit",
|
||||
shortcutDir="DesktopFolder"
|
||||
)]
|
||||
|
||||
setup(
|
||||
name = "Bulk Downloader for Reddit",
|
||||
version = __version__,
|
||||
description = "Bulk Downloader for Reddit",
|
||||
author = "Ali Parlakci",
|
||||
author_email="parlakciali@gmail.com",
|
||||
url="https://github.com/aliparlakci/bulk-downloader-for-reddit",
|
||||
classifiers=(
|
||||
"Programming Language :: Python :: 3",
|
||||
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)"
|
||||
"Natural Language :: English",
|
||||
"Environment :: Console",
|
||||
"Operating System :: OS Independent",
|
||||
),
|
||||
executables = executables,
|
||||
options = options
|
||||
)
|
||||
|
||||
|
||||
@@ -4,19 +4,14 @@ import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import imgurpython
|
||||
from multiprocessing import Queue
|
||||
|
||||
from src.errors import (AlbumNotDownloadedCompletely, FileAlreadyExistsError,
|
||||
FileNameTooLong, ImgurLoginError,
|
||||
NotADownloadableLinkError)
|
||||
from src.tools import GLOBAL, nameCorrector, printToFile
|
||||
|
||||
try:
|
||||
from imgurpython import *
|
||||
except ModuleNotFoundError:
|
||||
print("\nimgurpython not found on your computer, installing...\n")
|
||||
from src.tools import install
|
||||
install("imgurpython")
|
||||
from imgurpython import *
|
||||
|
||||
VanillaPrint = print
|
||||
print = printToFile
|
||||
|
||||
@@ -181,7 +176,7 @@ class Imgur:
|
||||
"""Initialize imgur api"""
|
||||
|
||||
config = GLOBAL.config
|
||||
return ImgurClient(
|
||||
return imgurpython.ImgurClient(
|
||||
config['imgur_client_id'],
|
||||
config['imgur_client_secret']
|
||||
)
|
||||
@@ -213,7 +208,7 @@ class Imgur:
|
||||
elif identity['type'] == 'album':
|
||||
return {'object':self.imgurClient.get_album(identity['id']),
|
||||
'type':'album'}
|
||||
|
||||
@staticmethod
|
||||
def get_credits():
|
||||
return Imgur.initImgur().get_credits()
|
||||
|
||||
|
||||
@@ -1,3 +1,36 @@
|
||||
import sys
|
||||
|
||||
class FauxTb(object):
|
||||
def __init__(self, tb_frame, tb_lineno, tb_next):
|
||||
self.tb_frame = tb_frame
|
||||
self.tb_lineno = tb_lineno
|
||||
self.tb_next = tb_next
|
||||
|
||||
def current_stack(skip=0):
|
||||
try: 1/0
|
||||
except ZeroDivisionError:
|
||||
f = sys.exc_info()[2].tb_frame
|
||||
for i in range(skip + 2):
|
||||
f = f.f_back
|
||||
lst = []
|
||||
while f is not None:
|
||||
lst.append((f, f.f_lineno))
|
||||
f = f.f_back
|
||||
return lst
|
||||
|
||||
def extend_traceback(tb, stack):
|
||||
"""Extend traceback with stack info."""
|
||||
head = tb
|
||||
for tb_frame, tb_lineno in stack:
|
||||
head = FauxTb(tb_frame, tb_lineno, head)
|
||||
return head
|
||||
|
||||
def full_exc_info(exc_info):
|
||||
"""Like sys.exc_info, but includes the full traceback."""
|
||||
t, v, tb = exc_info
|
||||
full_tb = extend_traceback(tb, current_stack(1))
|
||||
return t, v, full_tb
|
||||
|
||||
class RedditLoginFailed(Exception):
|
||||
pass
|
||||
|
||||
@@ -19,6 +52,15 @@ class FileNameTooLong(Exception):
|
||||
class InvalidRedditLink(Exception):
|
||||
pass
|
||||
|
||||
class ProgramModeError(Exception):
|
||||
pass
|
||||
|
||||
class SearchModeError(Exception):
|
||||
pass
|
||||
|
||||
class RedditorNameError(Exception):
|
||||
pass
|
||||
|
||||
class NoMatchingSubmissionFound(Exception):
|
||||
pass
|
||||
|
||||
|
||||
@@ -3,14 +3,7 @@ import random
|
||||
import socket
|
||||
import webbrowser
|
||||
|
||||
try:
|
||||
import praw
|
||||
except ModuleNotFoundError:
|
||||
print("\nPRAW not found on your computer, installing...\n")
|
||||
from src.tools import install
|
||||
install("praw")
|
||||
import praw
|
||||
|
||||
import praw
|
||||
from prawcore.exceptions import NotFound, ResponseException, Forbidden
|
||||
|
||||
from src.tools import GLOBAL, createLogFile, jsonFile, printToFile
|
||||
@@ -96,7 +89,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
else:
|
||||
@@ -105,7 +98,7 @@ def beginPraw(config,user_agent = str(socket.gethostname())):
|
||||
authorizedInstance = GetAuth(reddit,port).getRefreshToken(*scopes)
|
||||
reddit = authorizedInstance[0]
|
||||
refresh_token = authorizedInstance[1]
|
||||
jsonFile("config.json").add({
|
||||
jsonFile(GLOBAL.configDirectory / "config.json").add({
|
||||
"reddit_refresh_token":refresh_token
|
||||
})
|
||||
return reddit
|
||||
@@ -360,7 +353,7 @@ def redditSearcher(posts,SINGLE_POST=False):
|
||||
printSubmission(submission,subCount,orderCount)
|
||||
subList.append(details)
|
||||
|
||||
allPosts = {**allPosts,**details}
|
||||
allPosts[subCount] = [details]
|
||||
|
||||
postsFile.add(allPosts)
|
||||
|
||||
|
||||
10
src/tools.py
10
src/tools.py
@@ -2,20 +2,11 @@ import io
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
|
||||
try:
|
||||
from pip import main as pipmain
|
||||
except:
|
||||
from pip._internal import main as pipmain
|
||||
|
||||
from os import makedirs, path, remove
|
||||
from pathlib import Path
|
||||
|
||||
from src.errors import FileNotFoundError
|
||||
|
||||
def install(package):
|
||||
pipmain(['install', package])
|
||||
|
||||
class GLOBAL:
|
||||
"""Declare global variables"""
|
||||
|
||||
@@ -23,6 +14,7 @@ class GLOBAL:
|
||||
config = None
|
||||
arguments = None
|
||||
directory = None
|
||||
configDirectory = Path.home() / "Bulk Downloader for Reddit"
|
||||
reddit_client_id = "BSyphDdxYZAgVQ"
|
||||
reddit_client_secret = "bfqNJaRh8NMh-9eAr-t4TRz-Blk"
|
||||
printVanilla = print
|
||||
|
||||
Reference in New Issue
Block a user