Guide · Agent skill
How to use the Vaex large dataset analysis agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The Vaex skill teaches an AI agent to analyze tabular scientific datasets that are too large to fit in memory, using the Vaex library on a single machine. This guide explains what the skill changes, how to install it from the ZIP on this page, how to ask for it in plain language, and what it needs to run safely. By the end you will know how to set it up and which requests suit it.
Vaex Large Dataset AnalysisK-Dense-AI/scientific-agent-skills · MIT Download the skill Quick start
- 1Download and unzip. Download the skill ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder.
- 2Create the environment. Use uv to create a Python 3.12 virtual environment and install the pinned vaex-core, vaex-hdf5 and vaex-viz packages.
- 3Add optional ML support. Install vaex-ml 0.19.0 only if you need machine learning features.
- 4Ask in plain language. Describe the file, its format and the analysis you want, such as a mean over a filtered subset.
- 5Check the output. Ask the agent to validate counts and a small sample against an independent calculation.
What the Vaex skill makes the agent do
Agent skills are folders of instructions and reference files that an AI agent reads when a task matches them. This one covers larger-than-RAM analysis of HDF5, Arrow, CSV and Parquet files with Vaex. Vaex processes data in chunks, so the agent can compute statistics and histograms without loading the full table at once.
The skill asks the agent to follow a workflow: check the schema and missing values first, use expressions and virtual columns for derived values, batch statistics, plot aggregated grids instead of raw points, and convert between file formats. It also warns where operations such as sorting, joins, large group dictionaries or model fitting can still need substantial RAM.
Install the Vaex skill
Download the skill ZIP from this page and unzip it into one of two places. For a personal install, use ~/.claude/skills/. For a single project, use the project's .claude/skills/ folder.
In the Claude apps you can instead upload the ZIP under Settings → Capabilities → Skills.
- Personal: unzip into ~/.claude/skills/
- Project: unzip into .claude/skills/ inside the project folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Set up a Python environment for Vaex
The skill recommends a separate environment, because the core Vaex release supports Python 3.9 to 3.12 and was tested on 3.12. It recommends uv for setup. The skill's install commands create a virtual environment and install the pinned packages: vaex-core, vaex-hdf5 and vaex-viz, with vaex-ml as an optional add-on for machine learning features.
Do not change these version pins independently. Arrow support is included in core, while FITS files need vaex-astro.
- Required: Python 3.9 to 3.12, NumPy, pandas, PyArrow, Matplotlib
- Optional integrations: vaex-hdf5, vaex-viz, vaex-ml
- On Windows, the interpreter path is .venv-vaex\Scripts\python.exe
uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex/bin/python "vaex-core==4.19.0" "vaex-hdf5==0.15.0" "vaex-viz==0.6.0"
# Optional ML:
uv pip install --python .venv-vaex/bin/python "vaex-ml==0.19.0"Trigger the skill with natural requests
You do not need special syntax. Describe a large tabular file and the analysis you want, and mention Vaex or out-of-core work if you want to be sure the skill is used. The skill's description covers Vaex expressions, filtered views, streamed statistics, binned plots and file conversion.
Good requests name the file, the format and the question. Include the columns and what counts as a missing value if you know it.
- Compute the mean and count of energy for rows where x is at least 3 in a 20 GB HDF5 file.
- Make a binned heatmap of temperature against depth from this Parquet folder without loading it all.
- Convert this large CSV to HDF5 in chunks and check the row count afterwards.
- Prepare virtual features for a Vaex ML pipeline from these columns.
Run the example and check the results
The skill includes a small executable example. It builds a synthetic table, creates a virtual column, runs delayed statistics, exports to HDF5 in chunks and reopens the file to compare results. It writes to a new file and raises an error if the output already exists, so it will not overwrite your data.
For a real input, the skill replaces the in-memory fixture with vaex.open on your file and compares sampled rows and streamed summaries instead of converting the whole dataset to NumPy.
vaex.open('input.hdf5')Safety notes: scripts and network access
This skill runs scripts and uses network access. Network access is needed only for installing packages or fetching remote data. Local workflows need no credentials.
Be careful with state files. The skill notes that Vaex state files carry transformations and can contain serialized executable objects, so load only trusted artifacts. State files also do not carry the original dataset or prove where it came from.
- Read the scripts the agent proposes before running them on sensitive data
- Load state files only from sources you trust
- Write exports to new paths and verify them before replacing source data
Tips for working with large files
Expect that opening CSV and Parquet files is not instant. CSV opening performs indexing and schema work, and Parquet must decode compressed data. Inspect CSV headers before parsing, and give explicit types for ID columns.
Use approximate percentiles when exact ones are not needed. The skill recommends df.percentile_approx for approximate percentiles, and warns that .values, .to_numpy(), .materialize() and unchunked .to_pandas_df() can allocate full arrays.
- Choose a plot type that matches the question: count heatmaps and mean heatmaps show different things
- Validate counts and units against a small independently computed subset
- Keep dates, time zones and sampling cadence explicit
What people use it for
- Computing filtered means and counts over a multi-gigabyte HDF5 table on a laptop
- Building binned heatmaps of large scientific measurements without plotting raw points
- Converting large CSV files to HDF5, Arrow or Parquet in chunks
- Creating virtual feature columns before Vaex ML preprocessing
Frequently asked questions
How do I install the Vaex agent skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder. In the Claude apps you can upload the ZIP under Settings → Capabilities → Skills.
What Python version does the Vaex skill need?
The core vaex-core 4.19.0 release supports Python 3.9 to 3.12, and the skill was tested on Python 3.12. Use a separate environment rather than a newer default Python.
Is the Vaex skill free to use?
The skill is licensed under MIT. The Vaex library itself is installed separately with pip-style tools through uv, as shown in the skill's install commands.
Does Vaex work on files bigger than my RAM?
Vaex is designed for columnar analysis when data exceeds RAM, especially repeated reductions and histograms over local HDF5 or Arrow files. The skill notes that sorting, joins, large group dictionaries and many model fits still need substantial memory.
Is Vaex better than pandas for large datasets?
Vaex suits large local tables that pandas struggles with. The skill says that for small datasets pandas handles easily, Vaex adds setup complexity for no gain, so pandas is the better choice there.
Why does the agent say a Vaex state file is unsafe?
Vaex state files can contain serialized executable objects, so the skill says to load only trusted artifacts. They also do not carry the original dataset or prove its provenance.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you analyze multi-gigabyte HDF5, Parquet or CSV files on one laptop or workstation and want an agent to follow a careful, chunked workflow. Pick something else for small datasets pandas handles easily, for distributed cluster jobs across several machines, or if you must load untrusted state files.
Download the skill