Skip to content
SkillModel-led1,307 tokens in use

Spark Engineer

Writes and tunes Apache Spark jobs: DataFrame pipelines, partitioning, broadcast joins, skew handling, caching and structured streaming.

12,471 stars on the repository MIT LicenseJeffallan/claude-skills · skills/spark-engineer
/plugin install fullstack-dev-skills@fullstack-dev-skills
View source

Install command as written in the repository's README

Skill label

measured from its files

What installing Spark Engineer costs

74tokens idle
1,307tokens in use
22,010tokens, up to

Idle is what sits in context while the skill is installed. In use is its instructions, loaded when it runs. Up to adds the files its instructions link. Counts are approximate, made offline with cl100k_base.

Model dependence Model-led

Script-ledMixedModel-led

Mostly instructions, so how well it works depends on the model that follows them.

This describes the skill's structure, how much of it is scripts. It does not say how well it works.

Touches

Nothing detected: no shell, network, file writes, git or credentials in its files.

Declared tags come from the skill's frontmatter; detected ones from its scripts, its code blocks and the commands it names in inline code. This is detected, not audited.

How it scores

Quality

How well the skill is written, counted from its own files: its frontmatter, its steps, examples, stated output and constraints, its size and whether every file it ships is one it names.How we score it →
Excellent
  • Frontmatter15/15
  • Structure22/25
  • Examples12/15
  • Output15/15
  • Size15/15
  • Resources15/15

Security

Starts at 100 and loses points for each thing in the skill's files that a reader should look at first.How we score it →
Excellent

No findings.

Repository health

Pre-computed score based on development activity, maintenance, community, maturity, and trend momentum.How we score it →
Excellent
  • Development activity100
  • Maintenance100
  • Community60
  • Maturity36
  • Momentum40

Spark Engineer gives the agent a workflow for distributed data processing with Apache Spark: analyze data volume and latency needs, design the pipeline, implement with optimized transformations, tune shuffles and joins, and validate against the Spark UI.

The skill includes PySpark examples for a schema-first pipeline, a broadcast join, key salting for skewed data and a correct caching pattern, and lists what to always do and what to avoid, such as collect() on large datasets.

What You Get

  • A five-step workflow with a validation loop on shuffle spill and skew
  • PySpark examples for an explicit-schema pipeline, a broadcast join, salting and caching
  • A must-do list: DataFrame API, explicit schemas, partition sizing, broadcast joins for small tables
  • A must-not list: collect() on large data, UDFs where built-ins exist, caching everything, small files
  • Output templates with code, configuration, partitioning strategy and monitoring advice
  • Reference files for Spark SQL, RDDs, partitioning and caching, performance tuning and streaming

Common Use Cases

  • Speeding up a slow join with a broadcast hint or key salting
  • Tuning shuffle partitions and executor memory for a large ETL job
  • Building a structured streaming pipeline with watermarks
  • Reading and writing parquet with explicit schemas

What Makes It Unique

The skill gives numbers to work with: 200 to 1000 partitions per executor core, broadcast joins for tables under 200 MB, and a warning that UDFs run far slower than built-in functions. It ties its validation step to what the Spark UI reports, so tuning follows measurements.

Under The Hood

The skill is a SKILL.md with five reference files that the agent loads by topic. It ships no scripts. The examples are PySpark code that reads and writes parquet in S3 buckets and sets cluster configuration, written for you to adapt; the skill itself does not run a cluster. The instructions ask the agent to inspect the Spark UI and test with production-scale data, which means running jobs against your own cluster and storage when you permit it.

Join founders building with open source

Opinionated takes, migration guides, cost-saving tips, and insights from the open source ecosystem.

Subscribe on SubstackJoin 750+ subscribers