Skip to content
Get packs

Guide · Agent skill

How to use the CLIP image-text skill for AI agents

Updated · written with AI from the project's own documentation and checked against our catalogue data

The CLIP skill teaches an AI agent to use OpenAI's CLIP model, which links pictures and words. Once it is installed, you can ask the agent to label images with plain-language categories, score how well an image matches a sentence, or build a simple search over a folder of pictures. This guide explains what the skill does, how to install it, how to ask for it, and what it needs to run.

CLIP Image-Text UnderstandingOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the skill. Get the ZIP from this skill's page and unzip it.
  2. 2Place the folder. Move the unzipped clip folder into ~/.claude/skills/ for personal use or .claude/skills/ inside a project.
  3. 3Install Python packages. Install PyTorch, torchvision, transformers, pillow, ftfy, regex, tqdm, and the OpenAI CLIP package from GitHub.
  4. 4Ask in plain language. Describe the task, name your images or folder, and list the categories you want.
  5. 5Review the code. Check the Python the agent writes and runs, especially any package installs or network calls.
  6. 6Check the results. Look at the scores and spot-check labels, since zero-shot output can be wrong.

What the CLIP skill makes the agent do

An agent skill is a folder of instructions and supporting files that Claude loads when a task matches it. The CLIP skill covers zero-shot image classification, image-text similarity, semantic image search, and cross-modal retrieval in both directions (image to text and text to image).

The agent writes and runs Python code for these tasks rather than only describing them. The skill notes that CLIP works best with broad categories and is trained on 400M image-text pairs.

  • Zero-shot classification: label an image using categories you describe in plain language, with no training data.
  • Image-text matching: score how well an image fits a sentence.
  • Semantic image search: rank a folder of images against a text query.
  • Batch processing and a sketch for storing embeddings in a vector database.

Install the CLIP skill

Download the ZIP from the skill's page in this directory and unzip it. For a personal setup, place the folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

In the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/
  • Project: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills, then upload

Install the Python requirements

The skill needs Python, PyTorch with torchvision, transformers, and pillow. It also uses ftfy, regex, and tqdm, and the OpenAI CLIP package installed from GitHub. A GPU with CUDA is optional and speeds up processing. The chromadb package is optional and only needed for the vector database example.

  • Required: Python, PyTorch and torchvision, transformers, pillow, ftfy, regex, tqdm
  • Required: the OpenAI CLIP package from GitHub
  • Optional: a CUDA GPU for faster processing
  • Optional: chromadb for the vector database example
pip install git+https://github.com/openai/CLIP.git pip install torch torchvision ftfy regex tqdm

Trigger the skill with natural requests

You do not need special syntax. Describe the task in everyday words and mention your images. The agent should pick up the skill when the request matches its description, such as image search, image tagging, or matching pictures to text.

Name your categories or query explicitly. The skill advises using descriptive labels, because vague labels perform poorly.

  • "Label every photo in this folder as beach, city, forest or mountain."
  • "Find the three images in my folder that best match 'a sunset over the ocean'."
  • "How well does photo.jpg match the sentence 'a dog running on grass'?"

Choose a model and understand the trade-offs

The skill lists five models: RN50, RN101, ViT-B/32, ViT-B/16 and ViT-L/14. It recommends ViT-B/32 for most cases as a good balance. ViT-B/16 gives better quality but is slower, and ViT-L/14 is the best quality and slowest.

  • RN50: fast, good quality
  • ViT-B/32: medium speed, better quality (recommended default)
  • ViT-L/14: slow, best quality

Safety notes: scripts and network access

This skill runs scripts and uses the network. The agent installs packages from GitHub and loads models, so review the commands it proposes before approving them, especially on a machine with sensitive files.

The skill is not suitable for safety-critical content moderation or legal decisions. Its own content moderation example uses broad categories such as safe for work and violent content, and zero-shot labels can be wrong or biased because CLIP was trained on web data.

  • Review package installs and scripts before running them.
  • Do not rely on it alone for safety-critical moderation or legal decisions.
  • Expect weak results for counting, fine-grained recognition, and locating objects; it returns whole-image results and no bounding boxes.

What people use it for

  • Tagging a photo library with broad categories such as beach, city, or forest.
  • Searching a folder of pictures with a sentence like 'a sunset over the ocean'.
  • Scoring how well product photos match their descriptions.
  • Prototyping image search without collecting labelled training data.
  • Learning how vision-language models work through runnable examples.

Frequently asked questions

How do I install the CLIP skill?

Download the ZIP from this skill's page and unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills.

What do I need to run CLIP?

You need Python, PyTorch with torchvision, transformers, pillow, ftfy, regex, tqdm, and the OpenAI CLIP package installed from GitHub. A CUDA GPU is optional for faster processing.

Is CLIP free to use?

The skill and the CLIP project are listed under the MIT License. Check the license file in the source repository for the full terms.

Can CLIP be used for content moderation?

The skill includes a basic content moderation example, but it is not suited to safety-critical or legal decisions. The skill notes that zero-shot labels can be wrong or biased.

What are the alternatives to CLIP?

The skill suggests BLIP-2 for better captioning, LLaVA for vision-language chat, and Segment Anything for image segmentation.

Why are CLIP results poor for counting or finding objects?

The skill lists fine-grained recognition, counting and spatial position as weak points, and it gives no bounding boxes. CLIP works on the whole image and broad categories.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Choose this skill if you are a developer, data team member or student who wants an agent to run image labelling or semantic search with CLIP and you can run Python and PyTorch. Pick another tool if you need safety-critical moderation, precise counting or object location, or if you cannot run code on your machine.

Download the skill