Skip to content
Get packs

Guide · Agent skill

How to use the BLIP-2 Vision-Language agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The BLIP-2 Vision-Language skill teaches an AI agent to work with Salesforce's BLIP-2 model, which can describe an image, answer questions about it, or match images to text. It is written for developers, data teams and researchers who want the agent to write working Python for these tasks. After following this guide you will know how to install the skill, ask for BLIP-2 work in plain language, and check whether your hardware and data suit it.

BLIP-2 Vision-Language GuideOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the blip-2-vision-language skill as a ZIP from our page.
  2. 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder.
  3. 3Or upload it in the apps. In the Claude apps, go to Settings → Capabilities → Skills and upload the ZIP.
  4. 4Install Python dependencies. Install transformers, accelerate, torch and Pillow, plus salesforce-lavis if you want the LAVIS path.
  5. 5Ask in plain language. Describe your image task, such as captioning a folder or asking a question about a photo.
  6. 6Check the code before running. Review any script the agent writes, especially if the images are private, since the code downloads model weights.

What is an agent skill, and what does this one do?

An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches it. This skill is named blip-2-vision-language and its main file is SKILL.md. Once it is installed, the agent knows how to use Salesforce's BLIP-2 for image captioning, visual question answering (VQA), and image-text matching.

  • Write Python that loads BLIP-2 with Hugging Face Transformers or the LAVIS library.
  • Caption a photo, ask a question about an image, or process a batch of images.
  • Choose a model variant (OPT 2.7B, OPT 6.7B, FlanT5-XL, FlanT5-XXL) and generation settings suited to the task.
  • Apply memory options such as 8-bit or 4-bit quantization for large models.

How do you install the blip-2 skill?

Download the skill as a ZIP from our page and unzip it. Put the resulting folder in one of two places: ~/.claude/skills/ for personal use across projects, or .claude/skills/ inside a specific project folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

  • Personal install: unzip into ~/.claude/skills/.
  • Project install: unzip into the project's .claude/skills/ folder.
  • Claude apps: upload the ZIP under Settings → Capabilities → Skills.

How do you trigger the skill with a natural request?

You do not need a special command. Describe the task in plain language and mention images or BLIP-2, and the agent can match the request to this skill. The skill's description covers image captioning, visual question answering, image-text retrieval and multimodal chat, so requests using those terms are the most likely to match.

  • "Use BLIP-2 to write a caption for every JPG in ./photos."
  • "Write a Python script that answers 'What color is the car?' for this image."
  • "Set up BLIP-2 with Transformers to match these images against these sentences."

What do you need before you start?

The skill's examples assume a Python environment with PyTorch, transformers, Pillow and accelerate (recommended). The examples also expect a CUDA-capable GPU. The model weights are downloaded from Hugging Face the first time they are used, so you need internet access then, and the variants are several gigabytes each (about 4GB for OPT-2.7B up to about 13GB for FlanT5-XXL).

  • Python with PyTorch (torch>=1.10.0), transformers>=4.30.0 and Pillow.
  • accelerate (recommended).
  • A CUDA-capable GPU, which the examples assume.
  • Optional: the salesforce-lavis library.
  • Enough disk space for the model weights, and internet access on first use.
pip install transformers accelerate torch Pillow

What does the basic image caption code look like?

The skill's core example loads the Salesforce/blip2-opt-2.7b model and its processor, opens an image, and generates a caption. You can ask the agent for this pattern directly, and it will adapt the file name and settings to your task.

  • Load the processor and model with Blip2Processor and Blip2ForConditionalGeneration.
  • Open the image with Pillow and convert it to RGB.
  • Generate tokens and decode them with skip_special_tokens=True.
processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b") model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16, device_map="auto")

What safety and privacy points should you check?

This skill runs scripts and uses the network. The code downloads model weights from Hugging Face, and the agent may run that code on your data. Check the scripts before running them on private images. If your photos are sensitive, confirm that downloading the model and running it locally meets your privacy requirements, and do not send images to a third-party service through this workflow unless you have decided that is acceptable.

  • Review generated scripts before the agent runs them on private data.
  • Expect outbound requests to Hugging Face on first model load.
  • Keep sensitive images on a machine you control.
  • The skill is MIT licensed.

What tips help the results?

Pick the smallest model that meets your quality needs, since larger variants need more memory and disk. Use the troubleshooting and advanced reference files in the skill when installs or CUDA errors occur. For fine-tuning, the advanced reference describes LoRA on the language model and training only the Q-Former.

  • Start with blip2-opt-2.7b and move up only if quality is not enough.
  • Use 8-bit or 4-bit quantization if memory is tight, as described in the memory optimization section.
  • Read references/troubleshooting.md for import errors, LAVIS install problems and CUDA mismatches.
  • Read references/advanced-usage.md for batch processing, generation controls and fine-tuning.

What people use it for

  • Generate captions for a folder of product or catalogue photos.
  • Ask visual questions about images, such as the colour of an object.
  • Tag or describe a large local image collection with a model that runs on your own GPU.
  • Compare BLIP-2 against other vision-language models for zero-shot image understanding.
  • Build image-text matching checks with the LAVIS library.

Frequently asked questions

Is the blip-2 agent skill free to use?

The skill is released under the MIT licence, as stated in its file. The model weights are downloaded from Hugging Face, so check the terms of the model source before commercial use.

What do I need to run the BLIP-2 skill?

You need Python, PyTorch (torch>=1.10.0), transformers>=4.30.0, Pillow and accelerate (recommended). The examples assume a CUDA-capable GPU and internet access to download model weights on first use.

Where do I put the skill folder after downloading?

Unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

Can I run BLIP-2 without a GPU?

The examples expect CUDA, so a machine without a GPU will need code changes and will be slow for the larger variants. The model variants are several gigabytes, so disk space matters too.

What are the alternatives to BLIP-2?

The skill itself points to LLaVA for instruction-following chat, InstructBLIP as a successor, GPT-4V or Claude 3 for production multimodal chat, CLIP for simple image-text similarity, and Flamingo for few-shot learning.

Why does my BLIP-2 install or import fail?

Check the troubleshooting reference file in the skill. It covers ModuleNotFoundError for transformers, LAVIS installation failures and CUDA version mismatches with example commands.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are a developer, data team or researcher who wants an agent to write BLIP-2 code for captioning, VQA or image matching on a machine with a GPU. Pick something else if you need a production chatbot with the best modern multimodal quality, or if you cannot send private images through a workflow that downloads and runs code.

Download the skill