Spark Driver Memory

Where this is already explained

• Spark Broadcast Hash Join

  That broadcast() is a free speedup for any small-looking table and that raising spark.sql.autoBroadcastJoinThreshold makes more joins fast. The planner tests a compressed on-disk estimate, the driver pays the decompressed in-memory price, and every executor holds a full copy for the life of the query — so the setting that looks like a speed dial is really a driver-heap and cluster-memory dial.

• Spark Executor Memory

  That an executor out of memory, or spilling, means the executor is too small, so raising spark.executor.memory fixes it. Raising memory raises every task's ceiling equally, so it stops the skewed task spilling without making it any less skewed — the stage is still gated by the one task holding 40x the rows, and the only visible change is that the spill metric disappears.