Data Engineer

The Data Engineer supports, develops, and maintains data and analytics platforms that enable reliable, scalable, and efficient access to data. This role partners with Business and IT teams to understand requirements and leverage modern data engineering technologies to deliver high-quality data solutions at scale. The Data Engineer will design, develop, and maintain data pipelines and data storage solutions, ensure data quality and integrity, and contribute to data governance, analytics, and cloud-based data platforms. The role works within Agile delivery environments and applies modern engineering practices to continuously improve data solutions.

Key Responsibilities

• Develop and maintain reliable, scalable, and efficient ETL/ELT data pipelines using technologies such as Azure Databricks, PySpark, Python/Scala, and SQL.
• Design and implement data ingestion and transformation solutions for a variety of data sources, including relational, event-based, and unstructured data.
• Build and maintain scalable Data Lake and Lakehouse solutions and optimize data processing and storage performance.
• Develop physical data models and implement data storage architectures in accordance with established design and engineering guidelines.
• Implement data quality checks, monitoring, alerting, and troubleshooting mechanisms to identify and resolve data quality and data integrity issues.
• Analyze complex data elements, data flows, dependencies, and relationships to contribute to conceptual, logical, and physical data models.
• Develop and operate large-scale data storage and processing solutions using distributed and cloud-based technologies.
• Work with Azure services such as Azure Data Lake Storage (ADLS), Event Hubs, and Azure Functions to support scalable data solutions.
• Implement and support data governance practices, including metadata management, data access, retention, and availability.
• Participate in testing, validation, troubleshooting, and continuous improvement of data pipelines and solutions.
• Collaborate with business stakeholders, analysts, data scientists, and IT teams to understand requirements and deliver analytics and data solutions.
• Apply Agile development practices, including Scrum, Kanban, DevOps, and continuous improvement, to deliver data-driven solutions.
• Document technical solutions, processes, data flows, and system dependencies to support knowledge sharing and effective solution maintenance.
• Apply appropriate engineering, security, governance, and compliance practices throughout the data development lifecycle.