Senior Data Engineer - Databricks & CI/CD
Hyderabad, IndiaFull-timeMid · 5-18 years
About this role
Job Title: Senior Data Engineer
Experience: 5 - 18 Years
Employment Type: Full-Time
Job Description
We are looking for a highly skilled Senior/Lead Data Engineer with expertise in Data Engineering, and CI/CD automation. The ideal candidate will be responsible for building scalable data solutions, developing deployment frameworks, and implementing automated release processes across enterprise data platforms.
Key Responsibilities
- Design, develop, and maintain scalable data engineering solutions using Python, PySpark, Spark SQL, and SQL.
- Build and manage Databricks-based applications utilizing Unity Catalog, Delta Lake, Databricks Workflows, and Asset Bundles.
- Develop and maintain CI/CD pipelines using Jenkins, Git/Bitbucket, and Docker.
- Design metadata-driven and configuration-driven deployment frameworks for enterprise data platforms.
- Manage deployment automation, dependency orchestration, validation, and release processes across environments.
- Integrate enterprise platforms and services using REST APIs and metadata management tools.
- Collaborate with cross-functional teams to deliver reliable, scalable, and governed data solutions.
Required Skills
- 4+ years of experience in Data Engineering, Data Platform Engineering, or DevOps.
- Strong experience with Python, PySpark, Spark SQL, SQL, JSON, and YAML.
- Hands-on expertise in Databricks, Unity Catalog, Delta Lake, Databricks Workflows, and Databricks Asset Bundles.
- Strong experience in Jenkins CI/CD pipelines, deployment automation, and release management.
- Experience with Docker, Git/Bitbucket, containerization, and source control best practices.
- Knowledge of metadata-driven deployment frameworks, dependency management, workflow orchestration, and CI/CD automation.
- Experience integrating enterprise platforms such as Alation, SharePoint, Knowledge Graphs, Vector Databases, Semantic/Ontology platforms, and REST APIs.
Preferred Skills
- Experience working with cloud-based data platforms.
- Strong problem-solving and analytical skills.
- Excellent communication and collaboration abilities.