Sr Databricks Engineer

Role Overview
We are looking for an experienced Data Engineer (5 to 9 years) with deep hands-on expertise in Apache Spark, Databricks, Delta Lake, and Unity Catalog

Key Responsibilities

Design, build, and maintain batch and streaming data pipelines on Databricks using Spark (SQL, PySpark, or Scala)
Implement robust data models and ETL/ELT workflows on top of Delta Lake, including schema evolution, time travel, and CDC patterns
Configure and manage Unity Catalog (workspaces, catalogs, schemas, permissions, lineages) to enforce data governance and security across the lakehouse
Optimize pipeline and query performance using Spark internals knowledge, including: Partitioning, Z-Ordering, Liquid Clustering, and file layout tuning Caching strategies, broadcast joins, and efficient shuffle management Cluster sizing/auto-scaling and cost/performance trade-offs
Work with multiple table and file formats (Delta, Iceberg, Parquet, ORC, Avro, JSON, etc.) and choose the right format based on workload and governance needs
Contribute to and maintain CI/CD pipelines for Databricks jobs, notebooks, and workflows ,and/or Databricks Asset Bundles (DABs)

Requirements

Required Qualifications
Strong hands-on experience building data pipelines using Apache Spark with Min 5 years
(PySpark/Scala/SQL) in production
Practical experience with Databricks on at least one major cloud (AWS, Azure, or GCP)
Deep understanding of: Spark internals (execution model, DAGs, stages, tasks, shuffles, Catalyst optimizer)
Performance tuning and troubleshooting (e.g., skew mitigation, spill, shuffle tuning, join strategies)
Solid experience with Delta Lake (ACID transactions, schema enforcement, time travel, OPTIMIZE/VACUUM)
Knowledge of table and file formats including Delta, Iceberg, and Parquet, and when to use which
Hands-on experience with:
Liquid Clustering and Z-Ordering for improving query performance and cost efficiency
Other storage/layout optimization techniques (partitioning, compaction, small-file handling)