Everyone knows 4 min read · Selection 37 practice problems ↓

select

Choose, rename and compute columns, and why select beats a chain of withColumn calls.

On this page

Show code in

Every code block on the page follows this.

You will learn

The four ways to refer to a column, and when each one is needed
How to compute and rename columns in one pass
Why one select beats a long chain of withColumn
How selecting fewer columns makes Spark read less data

Read first

Nothing. This lesson starts from scratch.

TL;DR select returns a new DataFrame with exactly the columns you list. Columns can be existing names or expressions, renamed with alias. It is the PySpark version of the SQL SELECT list.

What it does

A DataFrame is immutable, so select never changes the original. It builds a new DataFrame whose columns are the expressions you pass, in that order. Anything you do not list is dropped.

PySparkSpark SQL

employees.select("name", "dept")

SELECT name, dept FROM employees

Step by step

Here we keep two columns and compute a third: a 10% raise, renamed with alias.

Input

name	dept	salary
Asha	Engineering	72000
Ben	Sales	48000
Chitra	Engineering	95000

Output

name	salary	raised
Asha	72000	79200
Ben	48000	52800
Chitra	95000	104500

Every output row comes from exactly one input row: select never adds or removes rows, only columns.

1
Spark resolves each name against the input schema. A typo fails here, before any data is read, with an AnalysisException.
2
Each expression is evaluated once per row. F.col("salary") * 1.1 is an expression, not a value: nothing is computed until an action runs.
3
alias("raised") only renames the result. Without it the column would be called (salary * 1.1).

Run the example

PySparkSpark SQL

from pyspark.sql import functions as F
result = employees.select(
    "name",
    "salary",
    F.round(F.col("salary") * 1.1).alias("raised"),
)

SELECT name,
       salary,
       ROUND(salary * 1.1) AS raised
FROM employees

Switch to PySpark to edit and run this example in your browser.

Four ways to name a column

Form	Example	Use it when
String	`"salary"`	You only need the column as is. Shortest and clearest.
F.col	`F.col("salary")`	You need an expression: arithmetic, comparisons, `.alias`, `.cast`. Works before the DataFrame variable exists.
Attribute	`employees.salary`	Quick exploration. Breaks for names with spaces or that clash with DataFrame methods, like `count`.
Item	`employees["salary"]`	You must say which DataFrame a column comes from, typically after a join where both sides have the same name.

Tip: In production code, prefer F.col("x") for expressions and plain strings for pass-through columns. It reads well and never depends on a variable name.

select vs withColumn

Both add computed columns, but they scale differently. Every withColumn call adds one projection on top of the plan. Ten calls are fine. Hundreds, generated in a loop, make the plan so deep that Spark spends seconds or minutes just analysing it before any data moves, and can even fail with a StackOverflowError.

PySparkSpark SQL · Building many columns at once

cols = [F.col(c).cast("double").alias(c) for c in numeric_cols]
clean = df.select("id", *cols)

SELECT id,
       CAST(a AS DOUBLE) AS a,
       CAST(b AS DOUBLE) AS b
FROM df

One select with a list comprehension produces a single projection however many columns you add. Spark 3.3 also added withColumns({...}), which takes a dictionary and does the same.

Under the hood: column pruning

When the source is a columnar format like Parquet or Delta, Spark only reads the columns that survive to the end of the plan. Selecting 5 columns from a 200-column table can cut the bytes read by 97%. You can see it in explain() as ReadSchema listing only the columns you used.

Part of explain() output

FileScan parquet [name#1,salary#3]
  ReadSchema: struct<name:string,salary:int>

This is why select("*") early in a pipeline is not free: if you keep every column until the end, Spark has to read every column.

Common mistakes

Forgetting alias on a computed column

The column gets a generated name like (salary * 1.1), which is awkward to reference later and breaks writes to tables with fixed column names.

Ambiguous columns after a join

If both sides have id, select("id") fails with "Reference id is ambiguous". Use a["id"], or join on a list of names (on=["id"]) so Spark keeps one copy.

Mixing up select and filter

select(F.col("salary") > 50000) returns one boolean column. It does not remove rows; that is filter.

Key takeaways

select returns a new DataFrame with only the columns you list; rows are unchanged.
Use alias to name computed columns.
Prefer one select over long loops of withColumn.
Selecting fewer columns lets Spark read fewer columns from Parquet and Delta.

Check yourself

3 questions

1. What does employees.select(F.col("salary") > 50000) return?

Show the answer

One boolean column, with one row per employee. select evaluates the expression for every row and returns it as a column. Removing rows is filter's job.

2. Why can 500 chained withColumn calls be slow before any data is read?

Show the answer

Each call adds a projection, and analysing a very deep plan is expensive. withColumn is lazy and does not scan data, but each call adds a layer to the logical plan. Very deep plans take a long time to analyse and optimise.

3. You select 4 columns from a 120-column Parquet table. How many columns does Spark read from disk?

Show the answer

Only the 4 that are used. Column pruning pushes the projection into the Parquet scan, so only the referenced columns are read.

Practice it

Interview problems that use select: write the PySpark, run it, and get graded on hidden tests.

Solve: Build a Price List →

See all 37 problems →

Go deeper

PySpark functions
withColumn PySpark functions
filter / where Spark internals
Catalyst and physical plans

Primary sources: DataFrame.select · SELECT (Spark SQL)

Next →
filter / where