Skip to content
Get packs

Spark Job Optimization

Makes the agent tune Apache Spark jobs for partitioning, caching, shuffles, skew and memory.

Source Download skill
Repository
wshobson/agents
Stars
42,095
Licence
MIT
Files
2 files

What it does

This skill gives an AI agent a set of established Apache Spark performance guidelines so it can review and improve slow data-processing jobs. Once installed, the agent knows to check things like shuffle cost, data skew, partition sizes, serialization and memory settings, and it can suggest or write Spark code and configuration that follow these patterns. It is a knowledge guide, not a tool that runs Spark for you.

Great for

  • Data engineers whose Spark jobs run slowly or fail with memory errors
  • Developers scaling Spark pipelines to larger datasets
  • Teams learning which Spark settings (adaptive query execution, broadcast joins, Parquet) matter most

Skip it if

  • People not using Apache Spark (the guidance is specific to it and would be misleading for other engines)
  • Production changes made without testing: the advice is general and the right settings depend on the cluster and data, so changes should be benchmarked first
  • Anyone expecting automatic profiling; the agent reads code and described symptoms, not your live Spark UI or cluster metrics unless you provide them

Try asking

  • “My Spark job spends most of its time in one stage after a join. How do I find and fix the skew?”
  • “Review this PySpark script and suggest partitioning and caching improvements.”
  • “Which Spark configs should I set for a job that shuffles about 500 GB of Parquet data?”
  • “Why does my Spark job run out of executor memory, and how should I tune it?”
Step-by-step guide: how to use Spark Job OptimizationInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    spark-optimization.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip spark-optimization.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip spark-optimization.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Share the Spark UI details, job stages, or config you are using so the agent can give specific rather than generic advice.
  • Test each suggested configuration change on a representative data sample and compare run times before applying it to production.
  • Ask the agent to explain the trade-off of each recommendation, such as memory versus parallelism, so you can judge it for your workload.

What's inside

  • references/details.md
  • SKILL.md

Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.