Run Spark on your laptop

What you need

Run the labs in a Jupyter notebook, a Python script, or the pyspark shell. Each lab assumes a variable called spark, created by one of the sessions below, and writes its files under a lab/ folder in your working directory.

A plain Spark session

# pip install pyspark==4.2.0
from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .master("local[4]")          # run on this machine, with 4 cores (task slots)
    .appName("labs")
    .getOrCreate())
spark.sparkContext.setLogLevel("WARN")
print(spark.version, spark.sparkContext.uiWebUrl)   # the Spark UI, while the session runs

• local[4] runs the driverdriver: The process that runs your program, plans the work and sends tasks to executors. Learn more → and a single executorexecutor: A worker process on a cluster machine that runs tasks and holds data in memory. Learn more → inside one JVM on your machine, with 4 tasktask: The work for one partition in one stage, run on one CPU core. Learn more → slots. local[*] uses every core; local uses one, which hides parallelism, so avoid it for these labs.
• Everything else is real Spark: the same planner, shufflesshuffle: Moving rows between machines so that all rows with the same key end up together. Needed by joins, groupBy and sorting, and usually the most expensive step of a job. Learn more →, AQEAQE: Adaptive Query Execution: Spark re-plans a running query using the real data sizes it has measured. Learn more → and Spark UI as on a cluster. Only the network and the number of machines are missing.
• getOrCreate() returns the existing session if there is one. Settings that size the application (memory, cores, packages) only apply when the session is first created: restart Python to change them.

The Spark UI

While the session is running, open the address the setup prints (usually http://localhost:4040; 4041 and up if 4040 is taken). The labs refer to its tabs: Jobs and Stagesstage: A group of steps Spark can run without moving data between machines. A new stage starts at every shuffle. Learn more → for task times, shuffle and spillspill: Writing data to local disk because it does not fit in memory. Slower, but the job keeps running. Learn more →; Storage for cached data; SQL / DataFrame for the final plan of each query, with row counts per operator. The UI disappears when the session stops.

With Delta Lake

# pip install pyspark==4.2.0 delta-spark==4.4.1
from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession

builder = (SparkSession.builder
    .master("local[4]")
    .appName("delta-labs")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog"))
spark = configure_spark_with_delta_pip(builder).getOrCreate()   # downloads the Delta jars on first run
spark.sparkContext.setLogLevel("WARN")
print(spark.version)

The delta-spark package adds the Python API (DeltaTable), and configure_spark_with_delta_pip adds the matching Delta jars, downloaded once into ~/.ivy2. The two configs enable Delta's SQL commands (MERGE, OPTIMIZE, VACUUM, ...) and let saveAsTable create Delta tables. To use them on a table by path, write delta.`/absolute/path`: a relative path there is not recognised, which is why the labs wrap paths in os.path.abspath. Delta and Spark versions must match: each Delta release lists the Spark versions it supports.

With Apache Iceberg

# pip install pyspark==4.1.3   (Iceberg 1.12 ships a runtime for Spark 4.1, not yet for 4.2)
from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .master("local[4]")
    .appName("iceberg-labs")
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")   # a catalog named "local"
    .config("spark.sql.catalog.local.type", "hadoop")                             # metadata in plain files
    .config("spark.sql.catalog.local.warehouse", "lab-warehouse")                 # a folder next to your script
    .getOrCreate())
spark.sparkContext.setLogLevel("WARN")
print(spark.version)

Iceberg ships one runtime jar per Spark minor version, and spark.jars.packages downloads it on first start. The local catalog stores table metadata as files under lab-warehouse/, so tables are named local.db.table. Production setups point the catalog at a REST catalog, Glue, Hive or Nessie instead; the table commands stay the same. Use a separate virtual environment from the Delta one, since the two need different Spark versions here.

With Apache Hudi

# pip install pyspark==4.1.3   (Hudi 1.2 ships a bundle for Spark 4.1)
from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .master("local[4]")
    .appName("hudi-labs")
    .config("spark.jars.packages", "org.apache.hudi:hudi-spark4.1-bundle_2.13:1.2.1")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")       # Hudi requires Kryo
    .config("spark.kryo.registrator", "org.apache.spark.HoodieSparkKryoRegistrar")
    .config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
    .getOrCreate())
spark.sparkContext.setLogLevel("WARN")
print(spark.version)

Hudi needs Kryo serialization and its own catalog and SQL extensions. It also expects absolute table paths, so the Hudi lab wraps its path in os.path.abspath. It runs on the same Spark 4.1 environment as Iceberg.

With Spark Connect

# pip install "pyspark[connect]==4.2.0"
from pyspark.sql import SparkSession

# remote("local") starts a Spark Connect server on this machine and connects to it over gRPC.
spark = SparkSession.builder.remote("local[4]").appName("connect-labs").getOrCreate()
print(spark.version, type(spark).__module__)

Spark Connect separates your Python process from the Spark driver. remote("local[4]") starts a Connect server on your machine and connects to it, which is the quickest way to see what changes for client code. Start it in a fresh Python process: a session cannot switch between classic and Connect mode.

When it does not start

• ✗JAVA_GATEWAY_EXITED or "Java not found" — No Java, or an unsupported version. Install Java 17 and set JAVA_HOME to it.
• ✗Python worker version mismatch — Spark started workers with a different Python than your notebook. Set PYSPARK_PYTHON to your virtual environment's Python.
• ✗ClassNotFoundException for Delta or Iceberg classes — The package did not load: a typo in the package name, no internet on first start, or a Spark and Delta/Iceberg version mismatch.
• ✗Port 4040 in use — Another session is running. Spark picks 4041 and up; stop old sessions with spark.stop().
• ✗LOCATION_ALREADY_EXISTS on saveAsTable — The local catalog forgets tables when Python exits, but the files in spark-warehouse/ stay. Delete that folder, or the table's subfolder, and run again.

Common mistakes

• ✗Using local (one core) for performance experiments — With a single task slot, nothing runs in parallel and partitionpartition: A chunk of a DataFrame's rows. Spark processes each partition as one task, so partitions decide how much work runs in parallel. Learn more → effects disappear. Use local[4] or local[*].
• ✗Changing spark.driver.memory with spark.conf.set — Memory and cores are fixed when the JVM starts. Set them in the builder and restart Python.
• ✗Mixing Spark and Delta or Iceberg versions — Each Delta release and Iceberg runtime supports specific Spark versions; mismatches fail at startup or on first use.