cast and schemas

What it does

Data read from CSV, JSON or an API often arrives as strings. cast converts a column to another type so you can do maths, compare dates and sort numbers numerically ("10" sorts before "9" as a string).

Run the example

from pyspark.sql import functions as F
result = raw_orders.select(
    F.col("order_id").cast("int").alias("order_id"),
    F.col("amount").cast("double").alias("amount"),
    F.col("qty").cast("int").alias("qty"),
    F.col("paid").cast("boolean").alias("paid"),
)

SELECT CAST(order_id AS INT)   AS order_id,
       CAST(amount AS DOUBLE)  AS amount,
       CAST(qty AS INT)        AS qty,
       CAST(paid AS BOOLEAN)   AS paid
FROM raw_orders

No error, no warning: three values quietly became null. In the running example this is classic Spark behaviour, which the in-browser engine follows.

Spark 4.0: ANSI mode and try_cast

Spark 4.0 turns on spark.sql.ansi.enabled by default. With it, CAST('abc' AS INT) fails the query with a CAST_INVALID_INPUT error instead of returning null. When bad values are expected, say so explicitly with try_cast, which returns null:

clean = raw_orders.select(F.col("amount").try_cast("double").alias("amount"))   # Spark 4.0+
# Spark 3.x: F.expr("try_cast(amount AS DOUBLE)")

SELECT try_cast(amount AS DOUBLE) AS amount FROM raw_orders

Counting what a cast lost

A value that was present before the cast and null after it is a conversion failure. Count those before you trust the result:

bad = raw_orders.filter(F.col("amount").isNotNull() & F.col("amount").cast("double").isNull())
print(bad.count())   # 1 row: "abc"

SELECT * FROM raw_orders
WHERE amount IS NOT NULL AND CAST(amount AS DOUBLE) IS NULL

Reading a schema

• df.printSchema() prints the tree of column names, types and nullability.
• df.dtypes returns a list such as [("order_id", "int"), ("amount", "double")].
• df.schema returns the StructType object; df.schema["amount"].dataType gives one column's type.
• To cast many columns at once, build one select: df.select([F.col(c).cast("double") for c in cols]).

double vs decimal for money

A double is a binary fraction: 0.1 + 0.2 is 0.30000000000000004. Summing millions of prices as doubles drifts by cents. decimal(p, s) stores exact digits: p total digits, s after the point. Use decimal(18,2) or similar for money. Values that do not fit the precision become null (or an error under ANSI mode), so leave room for totals.

Common mistakes