Functional AI Tester - GenAI
Sign up free to see how well your resume matches this role.
What you'll do
- Define risk-based test strategies and detailed test plans for GenAI features.
- Establish clear acceptance criteria with stakeholders for functional, safety, and data quality aspects.
- Build and maintain automated test suites using Python (e.g., PyTest, requests).
- Implement reusable utilities for prompt/response validation, dataset management, and result scoring.
- Create regression baselines and golden test sets to detect quality drift.
- Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc.
- Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements.
- Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks.
- Track quality metrics over time.
- Verify alignment between retrieved sources and generated answers.
- Verify adversarial tests.
- Measure retrieval relevance, precision/recall, grounding quality, and hallucination reduction.
What they're looking for
- 5+ years in software QA, including test strategy, automation, and defect management.
- 2+ years testing AI/ML or GenAI features, with hands-on evaluation design.
- 4+ years testing ETL/data pipelines and data quality.
- Python: Strong proficiency building automated tests and tooling (PyTest, requests, pydantic or similar).
- API testing: REST contract testing, schema validation, negative testing.
- GenAI evaluation: crafting prompt suites, golden datasets, rubric-based scoring, and automated evaluation pipelines.
- RAG testing: retrieval relevance, grounding validation, chunking/indexing verification, and embedding checks.
- ETL/data quality: schema and constraint validation, reconciliation, lineage awareness, data profiling.
- Understanding of LLM limitations and methods to detect/reduce hallucinations.
- Safety and compliance testing including PII handling and prompt-injection resilience.
- Strong analytical and debugging skills across services and data flows.
- Excellent written and verbal communication; ability to translate quality goals into measurable criteria.
Nice to have
- Experience with evaluation frameworks or tooling for LLMs and RAG quality measurement.
- Experience creating synthetic datasets to stress specific behaviors.
Summarised by NextRaise from the employer’s description, which follows in full below.
Full description from employer
- - - - - - - - - - - -
We are seeking a Quality Assurance (QA) Engineer focused on testing Generative AI (GenAI) applications with a strong emphasis on Python-based test automation, GenAI evaluation, and ETL/data quality validation. You will design and execute end-to-end test strategies that ensure our AI solutions are accurate, reliable, safe, and compliant.
About the Role
You will be involved in QA for GenAI features including Retrieval-Augmented Generation (RAG), conversational AI and Agentic evaluations. The role centers on:
Systematic GenAI evaluation (qualitative and quantitative metrics)
ETL and data quality testing for the data flows that feed AI systems
Python-driven automated testing
This position is hands-on and collaborative, partnering with AI engineers, data engineers, and product teams to define measurable acceptance criteria and ship high-quality AI features.
Key Responsibilities
Test strategy and planning
Define risk-based test strategies and detailed test plans for GenAI features.
Establish clear acceptance criteria with stakeholders for functional, safety, and data quality aspects.
Python test automation
Build and maintain automated test suites using Python (e.g., PyTest, requests).
Implement reusable utilities for prompt/response validation, dataset management, and result scoring.
Create regression baselines and golden test sets to detect quality drift.
GenAI evaluation
Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc.
Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements.
Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks.
Track quality metrics over time.
RAG and semantic retrieval testing
Verify alignment between retrieved sources and generated answers.
Verify adversarial tests.
Measure retrieval relevance, precision/recall, grounding quality, and hallucination reduction.
API and application testing
Test REST endpoints supporting GenAI features (request/response contracts, error handling, timeouts).
ETL and data quality validation
Test ingestion and transformation logic; validate schema, constraints, and field-level rules.
Implement data profiling, reconciliation between sources and targets, and lineage checks.
Verify data privacy controls, masking, and retention policies across pipelines.
Non-functional testing
Performance and load testing focused on latency, throughput, concurrency, and rate limits for LLM calls.
Cost-aware testing (token usage, caching effectiveness) and timeout/retry behavior validation.
Reliability and resilience checks including error recovery and fallback behavior.
Share results and insights; recommend remediation and preventive actions.
Required Qualifications
Experience
5+ years in software QA, including test strategy, automation, and defect management.
2+ years testing AI/ML or GenAI features, with hands-on evaluation design.
4+ years testing ETL/data pipelines and data quality.
Technical skills
Python: Strong proficiency building automated tests and tooling (PyTest, requests, pydantic or similar).
API testing: REST contract testing, schema validation, negative testing.
GenAI evaluation: crafting prompt suites, golden datasets, rubric-based scoring, and automated evaluation pipelines.
RAG testing: retrieval relevance, grounding validation, chunking/indexing verification, and embedding checks.
ETL/data quality: schema and constraint validation, reconciliation, lineage awareness, data profiling.
Quality and governance
Understanding of LLM limitations and methods to detect/reduce hallucinations.
Safety and compliance testing including PII handling and prompt-injection resilience.
Strong analytical and debugging skills across services and data flows.
Soft skills
Excellent written and verbal communication; ability to translate quality goals into measurable criteria.
Collaboration with AI engineers, data engineers, and product stakeholders.
Organized, detail-oriented, and outcomes-focused.
Nice to Have
Experience with evaluation frameworks or tooling for LLMs and RAG quality measurement.
Experience creating synthetic datasets to stress specific behaviors.
Company
Company facts come from this company's own listings. We only show what the postings themselves carry.