Data Engineer – Data Warehouse

1. Introduction

2. Scope of Work

On-Premise Data Warehouse Support

• Monitor and re-run failed data pipelines and workflows to ensure seamless operations on SSIS & T-SQL.
• Troubleshoot and resolve production issues, focusing exclusively on bug fixes.
• Maintain data accuracy and integrity across all pipelines.
• Conduct root cause analysis for recurring failures and document resolutions.
• Provide daily status reports on pipeline execution, highlighting failures, root causes, and applied fixes.
• Collaborate with onshore teams to address critical production incidents.
• Implement temporary workarounds to minimize downtime while working toward permanent solutions.

Cloud Data Warehouse Development

• Gather and analyze business requirements for business datasets in the cloud.
• Design, develop, and test data pipelines to support Cloud Data Warehouse initiatives.
• Build and optimize batch processing and streaming jobs to ensure timely data refresh.
• Implement Medallion Architecture and design models for structured data processing across different layers.
• Collaborate with onshore teams to establish best practices and reduce technical debt.
• Define data models and ETL strategies in alignment with stakeholder requirements.
• Participate in regular stand-ups and discussions with onshore and offshore teams to ensure seamless collaboration.
• Ensure compliance with data governance policies, security standards, and best practices.
• Develop and maintain comprehensive runbooks to document cloud operations, ensuring clear, step-by-step guidance for troubleshooting and maintenance.
• Standardize and optimize cloud workflows by creating detailed documentation that enhances efficiency, reduces downtime, and supports seamless knowledge transfer across teams.