Skip to content
Get packs

Embedding anisotropy check

Checks whether a cosine similarity score is meaningful by comparing it against an unrelated-pair floor

Source Download skill
Repository
github/awesome-copilot
Stars
40,703
Licence
MIT
Files
3 files

What it does

Embeddings are lists of numbers that represent text or images, and a cosine similarity score says how alike two of them are. Often these scores are inflated because most vectors point in one shared direction, so even unrelated items score high. With this skill installed, the agent checks any similarity figure it is about to report against a floor measured from unrelated pairs in the same comparison, and reports the matched score beside that floor, with and without centring on the mean.

Great for

  • ML engineers comparing embeddings across model precisions, runtimes or layers
  • Researchers evaluating retrieval, probes or adapters who need to justify a similarity claim
  • Anyone writing a results table or README that calls a similarity score high or two conditions equal

Skip it if

  • Comparing embeddings from two independently trained models without first fitting an explicit mapping, since the script's cosine is then meaningless (the skill itself says so)
  • Quick non-technical lookups where no similarity number is being reported

Try asking

  • “Before I report that the 4-bit export matches bf16 at cosine 0.91, check it against an unrelated-pair floor”
  • “Run the anisotropy check on these two layer outputs saved as A.npy and B.npy”
  • “Write a retrieval eval for our index and report the chance-level floor beside the recall figure”
Step-by-step guide: how to use Embedding anisotropy checkInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    embedding-anisotropy-check.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip embedding-anisotropy-check.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip embedding-anisotropy-check.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Supply a train split from the same domain and model via --fit-a and --fit-b rather than fitting the mean on the compared rows, to avoid mis-centring
  • Report both raw and centred readings and the partner_ranked_first rate against its 1/n chance rate
  • Average signed differences across seeds before taking absolute value, and state the smallest effect the test could detect

What's inside

  • scripts/anisotropy_check.py
  • scripts/example_scifact_halves.py
  • SKILL.md

Redistributed unmodified from github/awesome-copilot under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-11-02. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.