NSFW Data Scraper

Scripts that gather tens of thousands of labeled images for training an NSFW image classifier.

Stars: 12.6k
Forks: 2.8k
Licence: MIT
Language: Shell
Last update: 2.8 years ago
Type: Dev tool

No commits in over a year — check for forks or alternatives before adopting.

What it is

This is a set of shell scripts that collect a large number of images from the web, sorted into rough categories such as porn, hentai, sexy, neutral and drawings. The sorted images are meant to train an image classifier that can recognize sexually explicit content, which is useful for content moderation. The README warns that the dataset is noisy and should be used with caution.

Run the scripts in a Docker container to build a folder of images sorted by category
Split collected images into train and test folders with a configurable number per class
Add optional SFW anime images from Danbooru2018 or neutral images from Caltech256
Train a CNN classifier using the included train_model.ipynb notebook with fastai

Great for

Machine learning practitioners building a content moderation classifier
Researchers who want a starting point for NSFW image datasets
People comfortable running Docker and editing shell scripts
Those who plan to clean and review the data before training

Not ideal for

Anyone wanting a ready-to-use NSFW filter, since this only builds training data and the README does not describe a finished tool
Beginners without Docker or command-line experience, since setup depends on them
Users who need clean, verified labels, because the README says the dataset is noisy

Step-by-step guide: how to use NSFW Data ScraperInstall, first run, examples, tips and FAQs

How to get it

1
Run with Docker
You'll need: Docker
Build the Docker image
docker build . -t docker_nsfw_data_scraper
Run all scripts in the container (may take several hours)
docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh
2
Run it
- Clone the repository and open its folder
- Build the Docker image with the build command
- Run the container with scripts/runall.sh and leave it running, as the README suggests overnight
- Check that data/train and data/test contain the five class folders
- Install fastai with conda and run train_model.ipynb from top to bottom to train a model
3
Get the most out of it
- Edit files under scripts/source_urls to change which sources are scraped, then rerun step 1
- Change N=2000 inside 6_create_test_.sh to adjust the train/test split size
- Run the optional scripts 3 and 4 to add drawings and neutral images
- Review the noisy data before training, as the README recommends caution

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

Automates collection of a large image set that would otherwise be slow to assemble by hand
Pre-collected raw data is included, so the URL-gathering step does not need to be rerun
Docker setup keeps the required tools (wget, rsync, ImageMagick, Java) in one environment

Trade-offs

The README itself says the dataset is noisy, so labels may be wrong and need manual review
Source material comes largely from subreddits and other sites, which raises content and licensing questions the README does not address
Last push was January 2024 and the README is brief, so upkeep and current compatibility are unclear

How it compares

Project	Stars	Type	Level	Updated	Best for
nsfw_data_scraper (this)	12.6k	Dev tool	intermediate	2.8 years ago	Machine learning practitioners building a content moderation classifier
deepface	23.6k	Library	intermediate	5 days ago	Python developers prototyping face verification or face search
ai-engineering-from-scratch	76.3k	Learning resource	intermediate	4 days ago	Students who want to understand how LLMs, retrieval, and agents work internally
AI-For-Beginners	70.5k	Learning resource	beginner	13 days ago	Beginners who want a structured, self-paced introduction to AI
annotated_deep_learning_paper_implementations	67.6k	Learning resource	intermediate	4 days ago	Students learning how transformers, GANs, or optimizers work internally
Made-With-ML	49.9k	Learning resource	intermediate	8 months ago	Software engineers who want to learn how ML models get deployed and maintained

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.