PySpark functions

26 in-depth lessons. One function per lesson: what it does, how it works step by step, the gotchas, and every example in PySpark with its SQL equivalent.

Everyone knows

• select · Everyone knows · 4 min read. Choose, rename and compute columns, and why select beats a chain of withColumn calls.
• filter / where · Everyone knows · 4 min read. Keep the rows that match a condition, and the null and operator traps that silently drop data.
• withColumn · Everyone knows · 3 min read. Add or replace one column, and why calling it in a long loop slows Spark down.
• groupBy + agg · Everyone knows · 3 min read. Aggregate rows per group: counts, sums, averages, and how count(*) differs from count(col).
• join · Everyone knows · 4 min read. Combine DataFrames on a key: inner, left, right and full joins, and the duplicate-key fan-out trap.
• orderBy · Everyone knows · 3 min read. Sort rows ascending or descending, control where nulls go, and know what a global sort costs.
• when / otherwise · Everyone knows · 3 min read. CASE WHEN logic inside a column: first match wins, and a missing otherwise means null.
• dropDuplicates · Everyone knows · 3 min read. Remove duplicate rows or keys, and why "keep the latest row" needs a window instead.

Good engineers know

• row_number / rank · Good engineers know · 3 min read. Number and rank rows inside a window, and how the three ranking functions treat ties.
• lag / lead · Good engineers know · 3 min read. Read the previous or next row in a window to compute changes, gaps and streaks.
• coalesce · Good engineers know · 3 min read. Take the first non-null value from several columns, and why it is not the same as fillna.
• explode · Good engineers know · 3 min read. Turn each array element into its own row, and the rows explode silently drops.
• collect_list / collect_set · Good engineers know · 3 min read. Gather the values of a group into an array, with or without duplicates, and in which order.
• pivot · Good engineers know · 3 min read. Turn row values into columns, and why passing the values list up front makes it faster.
• left_semi / left_anti · Good engineers know · 3 min read. Keep the rows that have a match, or the rows that do not, without duplicating anything.
• trunc / add_months · Good engineers know · 3 min read. Truncate dates to the month or year and shift them by whole months, including month-end rules.
• regexp_extract · Good engineers know · 3 min read. Pull fields out of text with a regular expression, and what it returns when nothing matches.

Great engineers know

• rangeBetween · Great engineers know · 4 min read. Window frames by value instead of row count: the last 7 calendar days, not the last 7 rows.
• max_by / min_by · Great engineers know · 4 min read. Return the value from the row where another column is largest or smallest, in one aggregate.
• rollup / cube · Great engineers know · 3 min read. Subtotals and grand totals in one pass, and how grouping() tells a total row from a real null.
• unpivot · Great engineers know · 3 min read. Turn columns into rows (wide to long) with unpivot or the SQL stack() function.
• eqNullSafe (<=>) · Great engineers know · 3 min read. Equality where null matches null: the fix for joins and comparisons that silently drop null keys.
• posexplode · Great engineers know · 3 min read. Explode an array and keep each element position, plus the _outer variant that keeps empty arrays.
• percentile_approx · Great engineers know · 3 min read. Medians and p90s that scale to billions of rows, and how the accuracy setting trades memory for error.
• unionByName · Great engineers know · 3 min read. Stack DataFrames by column name instead of position, and survive schema drift with allowMissingColumns.
• last(ignorenulls) · Great engineers know · 3 min read. Forward-fill missing values with last over a window, and the frame that makes it work.