Weekday AI·2 days ago
2 days ago
Data quality engineer
Sign up free to see how well your resume matches this role.
What you'll do
- Validate ETL pipelines, data transformations, business logic, and data quality across Snowflake, Databricks, Hive, and other data platforms.
- Design and execute comprehensive test scenarios for data pipelines, data warehouses, and analytical datasets.
- Translate business and technical requirements into effective test cases covering KPIs, metrics, calculations, and business rules.
- Use advanced SQL to validate large datasets, identify anomalies, reconcile data, and investigate data-quality issues.
- Partner with Data Engineers to identify potential failure points and proactively detect defects before production releases.
- Develop automated and reusable tests for data pipelines to improve coverage and reduce regression risk.
- Contribute to and enhance existing data test automation frameworks with a focus on scalability, reliability, and maintainability.
- Validate data accuracy, completeness, consistency, and integrity across source, transformation, and target systems.
- Perform regression testing and release validation for data platform changes.
- Collaborate with Data Analysts, Product Managers, Data Engineers, and Engineering teams to resolve data-quality issues.
- Support testing across batch and distributed data-processing environments.
- Use Python to develop automation scripts, validation utilities, and data-quality testing solutions.
What they're looking for
- 5+ years of experience in data quality, data QA, ETL testing, data engineering testing, or a closely related role.
- Strong hands-on experience validating data pipelines, ETL processes, and data warehouses in production environments.
- Expert-level SQL skills with experience working with very large datasets, including terabyte-scale data.
- Proven ability to identify data anomalies, inconsistencies, and quality issues through efficient SQL analysis.
- Strong experience with Snowflake, Databricks, Hive, or similar modern data platforms.
- Solid proficiency in Python and experience developing automated tests for data pipelines.
- Good understanding of Apache Spark, Airflow, and modern data-processing workflows.
- Strong understanding of data warehousing, ETL/ELT concepts, data transformations, and data validation.
- Familiarity with CI/CD principles and integrating automated testing into development and deployment workflows.
- Experience building or contributing to scalable and maintainable test automation frameworks.
- Strong analytical and problem-solving skills with excellent attention to detail.
- Excellent communication and collaboration skills with the ability to work effectively across technical and business teams.
Nice to have
- Knowledge of BDD frameworks such as Behave is an advantage.
- Experience working with AWS or other cloud platforms is desirable.
- Familiarity with data-quality frameworks such as Great Expectations, Deequ, or similar custom solutions is an advantage.
Summarised by NextRaise from the employer’s description, which follows in full below.
Full description from employer
𝗧𝗵𝗶𝘀 𝗿𝗼𝗹𝗲 𝗶𝘀 𝗳𝗼𝗿 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗪𝗲𝗲𝗸𝗱𝗮𝘆'𝘀 𝗰𝗹𝗶𝗲𝗻𝘁𝘀
𝗦𝗮𝗹𝗮𝗿𝘆 𝗿𝗮𝗻𝗴𝗲: 𝗥𝘀 𝟭𝟬𝟬𝟬𝟬𝟬𝟬 - 𝗥𝘀 𝟭𝟲𝟬𝟬𝟬𝟬𝟬 (𝗶𝗲 𝗜𝗡𝗥 𝟭𝟬-𝟭𝟲 𝗟𝗣𝗔)
Experience: 5+ yrs
Location: Bengaluru, Karnataka, India
Job Type: Full-time
We are looking for an experienced Senior Data QA Engineer to ensure the accuracy, reliability, and quality of enterprise data pipelines, ETL processes, and data platforms. The role focuses on validating data transformations, business logic, KPIs, metrics, and data quality across modern platforms such as Snowflake, Databricks, and Hive.
The ideal candidate will combine strong SQL, Python, data engineering, and test automation skills with a deep understanding of data quality and validation. You will work closely with Data Engineers, Data Analysts, Product Managers, and Engineering teams to identify potential issues early and ensure reliable, production-ready data releases.
Requirements
Key Responsibilities
- Validate ETL pipelines, data transformations, business logic, and data quality across Snowflake, Databricks, Hive, and other data platforms.
- Design and execute comprehensive test scenarios for data pipelines, data warehouses, and analytical datasets.
- Translate business and technical requirements into effective test cases covering KPIs, metrics, calculations, and business rules.
- Use advanced SQL to validate large datasets, identify anomalies, reconcile data, and investigate data-quality issues.
- Partner with Data Engineers to identify potential failure points and proactively detect defects before production releases.
- Develop automated and reusable tests for data pipelines to improve coverage and reduce regression risk.
- Contribute to and enhance existing data test automation frameworks with a focus on scalability, reliability, and maintainability.
- Validate data accuracy, completeness, consistency, and integrity across source, transformation, and target systems.
- Perform regression testing and release validation for data platform changes.
- Collaborate with Data Analysts, Product Managers, Data Engineers, and Engineering teams to resolve data-quality issues.
- Support testing across batch and distributed data-processing environments.
- Use Python to develop automation scripts, validation utilities, and data-quality testing solutions.
- Integrate testing practices into CI/CD workflows to improve release quality and development velocity.
- Investigate production data issues, perform root-cause analysis, and help implement sustainable solutions.
- Maintain test documentation, validation standards, and reusable testing assets.
- Continuously improve data testing methodologies, automation coverage, and quality processes.
What Makes You a Great Fit
- 5+ years of experience in data quality, data QA, ETL testing, data engineering testing, or a closely related role.
- Strong hands-on experience validating data pipelines, ETL processes, and data warehouses in production environments.
- Expert-level SQL skills with experience working with very large datasets, including terabyte-scale data.
- Proven ability to identify data anomalies, inconsistencies, and quality issues through efficient SQL analysis.
- Strong experience with Snowflake, Databricks, Hive, or similar modern data platforms.
- Solid proficiency in Python and experience developing automated tests for data pipelines.
- Good understanding of Apache Spark, Airflow, and modern data-processing workflows.
- Strong understanding of data warehousing, ETL/ELT concepts, data transformations, and data validation.
- Familiarity with CI/CD principles and integrating automated testing into development and deployment workflows.
- Experience building or contributing to scalable and maintainable test automation frameworks.
- Knowledge of BDD frameworks such as Behave is an advantage.
- Experience working with AWS or other cloud platforms is desirable.
- Familiarity with data-quality frameworks such as Great Expectations, Deequ, or similar custom solutions is an advantage.
- Strong analytical and problem-solving skills with excellent attention to detail.
- Excellent communication and collaboration skills with the ability to work effectively across technical and business teams.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent professional experience is preferred.
- Strong ownership mindset with the ability to proactively identify quality risks and drive issues through resolution.
Company
Weekday AI
Bengaluru, India
Company facts come from this company's own listings. We only show what the postings themselves carry.