Do this after #132 (the CLAUDE.md -> AGENTS.md migration), so the skill hangs off AGENTS.md and not CLAUDE.md.
Write an agent skill that captures how Spark code is actually written in this repo, so agents stop producing round-trip-heavy or accidentally-eager Spark.
Must cover
- Spark is lazy. Transformations only build a plan; nothing executes until an action.
- Spark Connect is extra lazy. Even metadata resolution is deferred:
spark.read... / spark.table(...) does not hit the catalog or files until something forces it. Call out the peripheral triggers (Metadata information: .schema, .columns, .printSchema(), or an Action .count(), .collect(), .toPandas()).
- Actions and
collect(). When collecting is correct, what it costs (every row lands on the driver), and the caps IceGraph already enforces (MAX_DATA_FILES_TO_COLLECT, MAX_SNAPSHOTS_TO_COMPUTE).
- One collect, many things. Prefer building a single DataFrame (union / aggregate / join) and collecting once over collecting per snapshot, per manifest, or per file.
- Repo conventions. Where Spark calls belong (
backend/collectors/, backend/extractors/, backend/base_classes/), the shared session, and the read-only + Iceberg v2 constraints.
Write an agent skill that captures how Spark code is actually written in this repo, so agents stop producing round-trip-heavy or accidentally-eager Spark.
Must cover
spark.read.../spark.table(...)does not hit the catalog or files until something forces it. Call out the peripheral triggers (Metadata information:.schema,.columns,.printSchema(), or an Action.count(),.collect(),.toPandas()).collect(). When collecting is correct, what it costs (every row lands on the driver), and the caps IceGraph already enforces (MAX_DATA_FILES_TO_COLLECT,MAX_SNAPSHOTS_TO_COMPUTE).backend/collectors/,backend/extractors/,backend/base_classes/), the shared session, and the read-only + Iceberg v2 constraints.