Spark Engineer gives the agent a workflow for distributed data processing with Apache Spark: analyze data volume and latency needs, design the pipeline, implement with optimized transformations, tune shuffles and joins, and validate against the Spark UI.
The skill includes PySpark examples for a schema-first pipeline, a broadcast join, key salting for skewed data and a correct caching pattern, and lists what to always do and what to avoid, such as collect() on large datasets.
What You Get
- A five-step workflow with a validation loop on shuffle spill and skew
- PySpark examples for an explicit-schema pipeline, a broadcast join, salting and caching
- A must-do list: DataFrame API, explicit schemas, partition sizing, broadcast joins for small tables
- A must-not list: collect() on large data, UDFs where built-ins exist, caching everything, small files
- Output templates with code, configuration, partitioning strategy and monitoring advice
- Reference files for Spark SQL, RDDs, partitioning and caching, performance tuning and streaming
Common Use Cases
- Speeding up a slow join with a broadcast hint or key salting
- Tuning shuffle partitions and executor memory for a large ETL job
- Building a structured streaming pipeline with watermarks
- Reading and writing parquet with explicit schemas
What Makes It Unique
The skill gives numbers to work with: 200 to 1000 partitions per executor core, broadcast joins for tables under 200 MB, and a warning that UDFs run far slower than built-in functions. It ties its validation step to what the Spark UI reports, so tuning follows measurements.
Under The Hood
The skill is a SKILL.md with five reference files that the agent loads by topic. It ships no scripts. The examples are PySpark code that reads and writes parquet in S3 buckets and sets cluster configuration, written for you to adapt; the skill itself does not run a cluster. The instructions ask the agent to inspect the Spark UI and test with production-scale data, which means running jobs against your own cluster and storage when you permit it.