Forum Access • Support • Help


Postby KonnoMate » 04 Aug 2016, 18:30

I'm sure I'm not the only one that creates scripts to scrap sites. Feel free to share ones you've made here.
This script is for I used it to download around 200GB within a few hours. It will crawl a specified page, create a folder for the model name and dump all the images in the set folder.

Requires: Python2, BeautifulSoup4 and Requests

Example: python ttl 1 6 0
(Note: Get the agency name from the url.

This means the script will crawl "[url]*new-1/[/url]" up to page 6, starting at the first model on the page.(model index 0) Occasionally the script may fuck up due to server issues, in which case you can just restart it where you left off.

I'd also be happy to fulfill any other customized scrappers/site rippers requests, if for whatever reasons BID/IGH/DownthemALL don't suffice.

User avatar

Advanced Member
Posts: 26
Joined: 27 Jul 2016, 11:13
Country: America


Re: Scripts

Postby buzzer2 » 04 Aug 2016, 18:48

Do not link to other sites/forums as backlinks can be traced.

If you absolutely must post a URL to an external resource, please ensure the link is disabled by removing the protocol. I have edited your post above. :)

User avatar

Posts: 4968
Joined: 13 Nov 2015, 09:15
Country: Canada

Return to Help • Tutorials

Who Is Online

Users browsing this forum: No registered users and 1 guest