33. Learning Path per Student

Difficulty: Medium · Topics: Arrays, Aggregations, Strings, Filtering & Selection

enrollments has student, course and enrolled_on ('YYYY-MM-DD'). Students can retake a course.

For each student return:

path: their courses in enrollment order, joined with ' > ' (retakes appear again; enrollments on the same day are ordered by course name)
distinct_courses: how many different courses they took

Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.

Sample data

enrollments

student	course	enrolled_on
asha	Spark	2025-03-10
asha	SQL	2025-01-05
asha	Delta	2025-06-01
ben	Python	2025-02-01
ben	SQL	2025-02-01

Expected output

student	path	distinct_courses
asha	SQL > Spark > Delta	3
ben	Python > SQL	2

Hints

Hint 1

collect_list does not guarantee any order, even if you sort the DataFrame first: the groupBy shuffle reorders rows.

Hint 2

Collect values that sort correctly on their own, such as "2025-03-01|SQL", then F.sort_array them.

Hint 3

Join the sorted array into one string with F.array_join, and strip the date prefixes with F.regexp_replace. F.size(F.collect_set("course")) counts distinct courses.

Learn the concepts

collect_list / collect_set · 3 min read. Gather the values of a group into an array, with or without duplicates, and in which order.
regexp_extract · 3 min read. Pull fields out of text with a regular expression, and what it returns when nothing matches.
groupBy + agg · 3 min read. Aggregate rows per group: counts, sums, averages, and how count(*) differs from count(col).

PySpark functions you'll practise

groupBy
agg
collect_list
collect_set
sort_array
size
select
regexp_replace

Browse

Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings

Difficulty: Easy · Medium · Hard · PySpark interview roadmap · Learn · All problems