Guide · Agent skill

How to use the LLaVA Vision Assistant agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The LLaVA skill gives an AI agent step-by-step guidance for LLaVA, an open-source model that looks at an image and answers questions about it in a back-and-forth conversation. This guide explains what the skill does, how to install it from a ZIP file, how to trigger it with plain requests, and what you need on your machine before it works. By the end you will know how to set up LLaVA locally and ask it questions about your own images.

LLaVA Vision AssistantOrchestra-Research/AI-Research-SKILLs · MIT Download the skill

Quick start

1Download and unzip the skill. Download the ZIP from this page and unzip it. Place the folder in ~/.claude/skills/ or a project's .claude/skills/ folder.
2Clone and install LLaVA. Clone the LLaVA repository from GitHub, move into the folder and run pip install -e . inside it.
3Check your GPU and libraries. Confirm you have an NVIDIA GPU with enough VRAM for your chosen model and that PyTorch, transformers and pillow are installed.
4Ask a question about an image. Ask the agent in plain language, for example to describe a photo, and let it run the LLaVA command or Python example with your image file.
5Continue the conversation or launch the web UI. Ask follow-up questions about the same image in multi-turn mode, or start the Gradio interface at http://localhost:7860.

What does the LLaVA skill make the agent do?

An agent skill is a folder of instructions and supporting files that an AI agent loads when a task matches the skill's description. This one covers LLaVA (Large Language and Vision Assistant), which combines a CLIP vision encoder with Vicuna or LLaMA language models.

Once installed, the agent knows how to set up LLaVA, load a model, pass in an image with a question, run multi-turn chats and launch a local web UI. It also explains how to reduce memory use with quantization and how to train a custom model.

Visual question answering about a single image
Multi-turn conversations about the same image
Image description and captioning
Document understanding with images
Visual instruction following

How do you install the LLaVA skill?

Download the skill ZIP from this page and unzip it. You have two options depending on how you use your agent.

For Claude Code, put the unzipped folder in ~/.claude/skills/ to make it available everywhere, or in a project's .claude/skills/ folder to limit it to that project.

In the Claude apps, upload the ZIP under Settings → Capabilities → Skills.

Personal install: ~/.claude/skills/
Project install: .claude/skills/ inside your project folder
Claude apps: Settings → Capabilities → Skills, then upload

What do you need before LLaVA runs?

The skill itself does not include the model. It tells the agent how to use the LLaVA project, which you set up on your own machine. The skill's files list these requirements.

You need Python with pip, plus the PyTorch, transformers and pillow libraries. You also need the LLaVA repository cloned from GitHub and installed. You need an NVIDIA GPU with roughly 4 to 70 GB of VRAM, depending on the model and whether you use quantization. Model weights are downloaded from Hugging Face (the liuhaotian/llava models).

Python with pip, PyTorch, transformers and pillow
LLaVA repository cloned from GitHub and installed with pip install -e .
NVIDIA GPU: about 14 GB VRAM for the 7B model in FP16, about 4 GB in 4-bit according to the performance table
Internet access to download model weights from Hugging Face

git clone https://github.com/haotian-liu/LLaVA
cd LLaVA
pip install -e .

How do you ask LLaVA about an image?

The basic workflow loads a model with load_pretrained_model, opens an image with PIL, builds a conversation prompt with the image token and your question, and then calls model.generate. The skill's basic example uses liuhaotian/llava-v1.5-7b and asks 'What is in this image?'.

For quick checks you can use the command-line tool instead. It takes a model path, an image file and a query. Leave out --query and you can type questions interactively about the same image.

Start with the 7B model for good quality at manageable VRAM
Specific questions get better answers than vague ones
Temperature 0.2 to 0.7 and max_new_tokens of 512 to 1024 are the ranges the source suggests

python -m llava.serve.cli \
    --model-path liuhaotian/llava-v1.5-7b \
    --image-file image.jpg \
    --query "What is in this image?"

How do you run a local web UI and reduce VRAM use?

The skill can launch a Gradio web interface on your machine. Add --load-4bit to reduce VRAM use. The source says the interface is then reachable at http://localhost:7860.

Quantization lowers memory use. The source says 4-bit loading reduces VRAM by about 4 times and 8-bit by about 2 times. In Python you pass load_4bit=True or load_8bit=True to load_pretrained_model.

4-bit: load_4bit=True, reduces VRAM by about 4 times
8-bit: load_8bit=True, reduces VRAM by about 2 times

python -m llava.serve.gradio_web_server \
    --model-path liuhaotian/llava-v1.5-7b \
    --load-4bit

What are the safety notes and limitations?

This skill runs scripts and uses the network. Setup clones a repository from GitHub, installs it with pip, and downloads model weights from Hugging Face. Review those sources before you run the commands, especially on a shared machine.

The source lists known limitations. LLaVA may describe things that are not in the image, struggles with precise locations, has difficulty reading small text, and is imprecise when counting many objects. It is also slower than CLIP. The source says CPU inference is extremely slow, so plan on a GPU.

The file also lists the LLaVA project's license as Apache 2.0, while this skill's own metadata says MIT. Check the license of the model weights you download before any commercial use.

Hallucinations: may describe things not in the image
Weak at precise spatial positions, fine print and counting
CPU inference is described as extremely slow
Review the GitHub and Hugging Face sources before running setup

What people use it for

Building a vision-language chatbot that answers questions about user-uploaded photos
Running image captioning or visual Q&A locally on your own GPU for a data team
Comparing open-source multimodal models in a research setting
Extracting textual descriptions of scenes for review, with the source's accuracy limits in mind

Frequently asked questions

Is the LLaVA skill free to use?

The skill's metadata lists the MIT license. The LLaVA project's own resources section lists Apache 2.0. Model weights come from Hugging Face, so check each model's terms before using it commercially.

What GPU do I need to run LLaVA?

The source says you need an NVIDIA GPU with roughly 4 to 70 GB of VRAM, depending on the model and quantization. The 7B model needs about 14 GB in FP16, and 4-bit loading lowers that. CPU inference is described as extremely slow.

How do I install the LLaVA agent skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills.

Can I run LLaVA on a low-VRAM GPU?

You can use 4-bit or 8-bit quantization. The source says 4-bit reduces VRAM by about 4 times and 8-bit by about 2 times. Quality may change, and the source does not give exact numbers for every setup.

How does LLaVA compare to GPT-4V or BLIP-2?

The skill's alternatives list says GPT-4V gives the highest quality but is API-based, while BLIP-2 is better for captioning only. LLaVA is open-source and runs on your own GPU, so it suits local work. The source does not provide a direct benchmark against these models.

Why does LLaVA get document or counting questions wrong?

The source lists these as known limitations: hallucinations, weak spatial reasoning, difficulty with small text, and imprecise object counting. For those tasks, verify the answer yourself or use a different tool.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you have a suitable NVIDIA GPU and want an agent to set up and run a local, open-source image chatbot. Pick a hosted vision API instead if you need zero setup, guaranteed accuracy on documents or counting, or no local model downloads.

Download the skill