Skip to content
Get packs

Guide · Open source

How to use nsfw_data_scraper: build the Docker image and run the scripts

Updated · written with AI from the project's own documentation and checked against our catalogue data

nsfw_data_scraper is an MIT-licensed collection of shell scripts that gathers images from the web into five rough categories (porn, hentai, sexy, neutral and drawings) for training an NSFW image classifier. This guide is for people who want to build that training data with Docker and then train a model on it. By the end you will know what the scripts do, how to build and run the Docker container, and what the output folders contain.

NSFW Data Scraperalex000kim/nsfw_data_scraper · ★ 12.6k · MIT

Quick start

  1. 1Install Docker. Install Docker, the only prerequisite listed in the README, and confirm it runs from your terminal.
  2. 2Get the repository. Download the nsfw_data_scraper project so you have the Dockerfile and the scripts folder.
  3. 3Build the image. Run docker build . -t docker_nsfw_data_scraper from the project root.
  4. 4Run the scripts. Run docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh and allow several hours.
  5. 5Check the data folders. Confirm that data/train and data/test each contain drawings, hentai, neutral, porn and sexy folders.
  6. 6Review the data. Inspect and clean the images, because the README says the dataset is noisy.
  7. 7Train the model. Install fastai and run train_model.ipynb from top to bottom.

What nsfw_data_scraper does

The project is a set of scripts that automatically collects a large number of images for five loosely defined categories: porn, hentai, sexy, neutral and drawings. The images are meant to be used later for training an image classifier.

The README warns that the dataset is noisy and should be used with caution. Plan to review and clean the data before you train on it. The project only builds training data; the README does not describe a finished NSFW filter.

  • porn: pornography images
  • hentai: hentai images, including pornographic drawings
  • sexy: sexually explicit images that are not pornography, such as nude photos, playboy or bikini images
  • neutral: safe-for-work images of everyday things and people
  • drawings: safe-for-work drawings, including anime

Prerequisites: what you need before you start

The only prerequisite listed in the README is Docker. The setup is built around a Docker container, so you need Docker installed and working from the command line before you begin.

Because the project is a set of shell scripts and Docker commands, it is best suited to people who are comfortable with the command line.

  • Docker

Build the Docker image

From the root of the repository, build the image with the command below. The build uses the Dockerfile in the project and tags the result as docker_nsfw_data_scraper.

The build installs wget, rsync, imagemagick and default-jre inside an Ubuntu 18.04 base image, according to the README's build output.

docker build . -t docker_nsfw_data_scraper

Run the scripts in the container

Run all scripts with the command below. The README notes that this may take several hours and recommends leaving it overnight. The -v flag mounts your current directory into the container so the output is saved on your machine.

The scripts run in order. Script 1 downloads image URLs with the ripme application, using text files under scripts/source_urls. The README says the URL outputs were already generated and are stored in the raw_data directory, so you do not need to rerun script 1 unless you edit the files under scripts/source_urls.

docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh

Understand the individual scripts

You can read the scripts in the scripts directory to see what each step does. The README lists them as follows. Scripts 3 and 4 are optional.

Script 6 creates data/test by moving N=2000 random files per class from data/train. The README says you can change N inside the script, and that running it again moves another N images per class.

  • 1_get_urls_.sh: reads text files under scripts/source_urls and downloads image URLs for the five categories using ripme. The source URLs are mostly subreddits but can be any site ripme supports.
  • 2_download_from_urls_.sh: downloads the actual images for URLs found in text files in raw_data.
  • 3_optional_download_drawings_.sh (optional): downloads SFW anime images from the Danbooru2018 database.
  • 4_optional_download_neutral_.sh (optional): downloads SFW neutral images from the Caltech256 dataset.
  • 5_create_train_.sh: creates data/train, copies all *.jpg and *.jpeg files from raw_data into it, and removes corrupted images.
  • 6_create_test_.sh: creates data/test and moves N=2000 random files per class from data/train into it.

Check the output folders

When the run finishes, data should contain two folders, test and train. Each contains one folder per class. The README's example listing shows the same five class folders in both places.

These folders are the input for training. Review the images in them before you train, since the README warns that the data is noisy.

ls data ls data/train/ ls data/test/

Train a CNN model on the data

The README describes training as a separate step. Install fastai using the command below, then run train_model.ipynb from top to bottom.

The README reports that its author trained a CNN classifier to 91% accuracy and shows a confusion matrix. It notes that drawings and hentai are confused with each other more often than with other classes, and the same happens with porn and sexy. Treat this as the author's result, not a guarantee for your run.

  • Run train_model.ipynb top to bottom after installing fastai.
conda install -c pytorch -c fastai fastai

What people use it for

  • Building a starting dataset for a content moderation image classifier
  • Studying how a CNN separates visually similar classes such as drawings and hentai, using the README's confusion matrix as a reference
  • Creating a train and test split with the 2000-images-per-class default in script 6
  • Adding optional SFW anime or neutral images through scripts 3 and 4

Frequently asked questions

Is nsfw_data_scraper free to use?

The repository is licensed under the MIT licence, according to its listing. The README does not mention any paid service or cost.

What do I need to install nsfw_data_scraper?

The README lists Docker as the only prerequisite. You build an image with docker build and run the scripts inside a container, so you do not install the scripts directly.

How long does nsfw_data_scraper take to run?

The README says the runall.sh command may run for several hours and recommends leaving it overnight.

Is nsfw_data_scraper a ready-to-use NSFW filter?

No. The project builds training data for an image classifier. The README does not describe a finished filtering tool.

Why is the dataset described as noisy?

The README says the dataset is noisy and should be used with caution. The images come from web sources such as subreddits, so you should review and clean them before training.

Can I change the train and test split size?

Yes. Script 6 moves N=2000 files per class to data/test, and the README says you can change this number inside the script.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

nsfw_data_scraper suits machine learning practitioners and researchers who are comfortable with Docker and shell scripts and who plan to clean the data before training a content moderation classifier. It is not a good choice if you want a ready-made NSFW filter, if you lack Docker or command-line experience, or if you need clean, verified labels.

Install options & downloads