AI QA Engineer (GenAI & ML Systems)
Mercedes-Benz Research and Development India Private LimitedBangalore, India (Hybrid)
Job Description
We are hiring an AI QA Engineer to own the quality, reliability and safety of our GenAI and ML-driven engineering applications. You will define how we test non-deterministic systems, from prompts and RAG pipelines to agentic workflows, and set the release bar for AI features used across Powertrain Engineering.
Key Responsibilities
- Design evaluation frameworks for LLM, RAG and agentic systems, including accuracy, hallucination, consistency, latency and cost.
- Build golden datasets, regression suites and automated evaluation pipelines integrated into CI/CD.
- Run red-teaming and adversarial tests for prompt injection, jailbreaks, data leakage, bias and toxicity.
- Validate retrieval quality, tool-calling and multi-step agent behavior.
- Define quality gates and release criteria for AI features.
- Monitor production drift and failure modes, and lead root-cause analysis with AI engineers.
- Report quality metrics and risks to engineering leadership.
Required Qualifications
- 4-8 years in QA/SDET, including 1-2+ years testing ML or GenAI systems.
- Strong Python and test automation skills, including pytest and API testing.
- Hands-on experience with LLM evaluation tools such as Ragas, DeepEval, promptfoo or LangSmith.
- Working knowledge of LLM behavior, RAG architectures, embeddings and prompt engineering.
- Statistical thinking, including sampling, metric design, confidence and the limits of LLM-as-judge.
- CI/CD experience with GitHub Actions or Azure DevOps, Git and cloud fundamentals, preferably Azure.
Preferred Qualifications
- Experience with agent frameworks and MCP/tool integrations.
- Experience with observability tooling for tracing and drift monitoring.
- Exposure to engineering or automotive domains.
Who Qualifies
Skeptical by default, precise, comfortable with non-deterministic outputs, a clear writer, and a close collaborator to AI engineers.
Benefits
- Discounts for Employees
- Possible Health Benefits
- Mobile Phone for Employees
- Possible Meal-Discounts
- Company Retirement
- Hybrid Work Possible
- Mobility Offers
- Events for Employees
- Coaching
- Flextime Possible
- Near-Site Childcare
- Parking
- Canteen, Café
- Good Public Transport
- Barrier-Free Workplace
- Inhouse Doctor
Contact
Mercedes-Benz Research and Development India Private Limited
Plot No. 5-P, EPIP 1st Phase
560066 Bangalore
MBRDI Recruitment E-Mail: mbrdi_recruitment@mercedes-benz.com