How to use the Bulk Ingestion Pipeline agent skill

What is an agent skill, and what does this one make the agent do?

An agent skill is a folder with a SKILL.md file that gives an AI agent a repeatable procedure for a task. When a request matches the skill, the agent reads the instructions and follows them.

This skill covers the full lifecycle of a bulk import. The phases are: define the schema, verify access and build the manifest, run a trial of 5-10 diverse examples, evaluate and improve, codify the pipeline, test it, promote it to a skill, run the bulk job, and monitor failures. The skill says not to skip to the bulk run until the trial results meet your quality bar.

• Schema: a page template, filing rules, an entity propagation spec and a dedup key, written before the first trial.
• Manifest: a durable state file at projects/<pipeline-name>/manifest.json, so a crash or session break resumes from what is actually on disk.
• Dedup: source plus source_id is used to skip items already ingested. A separate check with brain-ingest-gate catches duplicates that arrive from a different source.
• Untrusted text: fetched content is treated as data. Agent-directed instructions found inside it are flagged, not followed.

How do you install the bulk-ingestion skill?

Download the ZIP file from this page. You can install it in one of two ways.

Option A, personal install: unzip the file into ~/.claude/skills/ so the bulk-ingestion folder is available to you across projects.

Option B, project install: unzip the file into the .claude/skills/ folder inside a specific project, so it only applies there.

Option C, Claude apps: upload the ZIP in the Claude apps under Settings, then Capabilities, then Skills.

How do you trigger the skill with a natural request?

You do not need a special command. Describe the job in plain language and include one of the trigger phrases the skill listens for. The skill's triggers include bulk ingest, bulk import, ingest all, ingestion pipeline, mass ingestion, bulk backfill, make a manifest, processing manifest and track a large ingest.

Be specific about the source and the destination. Name the data (for example, an email export folder or a set of audio files) and say which brain or project it should go into.

• Bulk ingest the email export in ~/Downloads/takeout into my brain.
• Make a manifest for the 2,000 PDFs in the research archive and start a trial with 8 varied files.
• Ingest all the chat logs from this API dump, and track progress so we can resume if the session ends.

What do you need before running it?

The skill depends on other parts of a GBrain setup. It expects a brain or repository that already contains the referenced conventions and the ingest, brain-ingest-gate and archive-crawler skills. You also need read access to the source data, either export files or API credentials depending on the source.

The reviewed files reference supporting material, including MANIFEST-PATTERN.md and a conventions folder, but those were not included in the files reviewed for this page. Confirm that they are present in your setup before you start a run.

The skill runs no scripts and uses no network access, according to its listing. It works through the agent and the files in your brain.

• A GBrain brain or repository with the ingest, brain-ingest-gate and archive-crawler skills.
• Read access to the source files or API credentials.
• The MANIFEST-PATTERN.md file and the conventions folder referenced by the skill.

What should you watch for when running a bulk import?

The skill is built to limit damage from a bad run, but it cannot stop a rushed one. A bulk run can write large amounts of content into your brain if you skip or hurry the trial phase. Review the trial output before approving the full run.

Use the skill for large or multi-session jobs. For a single item, the skill points you to the simpler ingest skill instead. For messy personal archives, run archive-crawler first to decide what is worth importing.

• Do not approve the bulk run until the 5-10 trial items pass your review.
• Check that the manifest total matches the count from the source enumeration.
• Be cautious with third-party corpora you have not reviewed. Flagged imperatives are a warning sign to inspect.

Tips for getting good results

Choose trial examples that are different from each other: a clean one, a messy one, one with many people or companies, one with very little text, and one with missing fields. Variety shows problems that a set of easy examples will hide.

Keep the manifest in the project folder and let the agent read it at the start of each session. The skill is designed so the agent works from disk, not memory.

• Set the visibility field in the page template for anything remote agents must not read.
• Let the agent re-scan outputs before it marks a subagent job as complete. The skill does not trust a completion message alone.