Skip to content
Get packs

Spark Engineer

Guides writing, debugging and tuning Apache Spark jobs, SQL queries and streaming pipelines.

Source Download skill
Repository
Jeffallan/claude-skills
Stars
12,176
Licence
MIT
Files
6 files

What it does

This skill turns the AI agent into a Spark specialist for big data processing. When you ask for help with Apache Spark, the agent follows a structured workflow: it plans partitioning, picks DataFrames over RDDs, writes code with explicit schemas, and checks the Spark UI for shuffle spill and data skew before calling the job done. It also loads reference notes on SQL, RDDs, partitioning and caching, performance tuning and structured streaming when relevant.

Great for

  • Data engineers writing or fixing PySpark or Spark SQL pipelines
  • Developers whose Spark jobs are slow, skewed or running out of memory
  • Teams configuring executor memory and shuffle partitions for large Parquet workloads

Skip it if

  • Small datasets that fit in a single machine, where pandas or plain SQL is simpler and Spark adds overhead
  • Non-Spark big data tools such as Flink, Dask or warehouse-native SQL, where the guidance may not apply
  • Running generated code unreviewed against production clusters or data, since examples such as writes to S3 can overwrite data

Try asking

  • “Write a PySpark job that reads Parquet events from S3, aggregates by user and writes results with an explicit schema”
  • “My Spark join is stuck on one task; help me diagnose and fix the data skew”
  • “Tune shuffle partitions and executor memory for a 2 TB nightly ETL job”
  • “Convert this slow Spark UDF pipeline to built-in DataFrame functions”
Step-by-step guide: how to use Spark EngineerInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 6 original files, the MIT licence and an attribution note.

    spark-engineer.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip spark-engineer.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip spark-engineer.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent your data volume, cluster size and Spark version so it can recommend partition counts and memory settings.
  • Paste Spark UI details (shuffle spill, task skew, GC time) so the agent can follow its optimize-and-validate loop.
  • Review generated write operations and paths before running them, especially overwrite modes, against real data.

What's inside

  • references/partitioning-caching.md
  • references/performance-tuning.md
  • references/rdd-operations.md
  • references/spark-sql-dataframes.md
  • references/streaming-patterns.md
  • SKILL.md

Redistributed unmodified from Jeffallan/claude-skills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.