Lead the migration from Glue/Redshift to Databricks
10+ years of qualified data engineering experience, with at least 2+ years in a tech‑lead or lead engineer capacity.
5‑6 years of experience in each of the following areas:
Deep, hands‑on experience with Apache Spark (PySpark and Spark SQL), including performance tuning (partitioning, shuffles, skew, caching, file sizing).
Hands‑on production experience with Databricks — Delta Lake, Unity Catalog, Workflows, and either Delta Live Tables or a comparable declarative pipeline framework.
Strong production experience with AWS Glue (jobs, crawlers, Data Catalog, triggers) and Amazon Redshift (including Redshift Spectrum, workload management, distribution and sort keys, performance tuning, and Redshift SQL / stored procedures).
Proven experience leading at least one substantial data platform migration or re‑platforming effort end‑to‑end.