AI Quality and LLM Evaluation Engineer | Senior SDET | RAG & Agentic Systems Testing

AI / LLM Testing
Expertise with DeepEval, RAGAS and Langfuse for LLM evaluation, including traces, prompt versioning, dataset management and evaluation runs. Proficiency in golden dataset regression, faithfulness, answer relevancy and hallucination detection, as well as RAG retrieval metrics such as hit rate, MRR, precision/recall@k and context relevancy. Agentic system testing covering multi-turn state, tool and function calling, and planning loops.

Test Frameworks and Automation Design
Hands-on experience with Playwright (UI, API, Component), Selenium, Cypress and Vitest. Framework and runner design, model based testing with xState, and application of POM and Screenplay patterns to build scalable, maintainable test infrastructure.

CI/CD, Infra and Observability
Practical knowledge of GitLab CI, GitHub Actions, Jenkins, Git, Docker and Kubernetes. Integration of monitoring and observability tools including Datadog, OpenTelemetry and Grafana, with experience in trace driven debugging and connecting evaluation harnesses into CI/CD pipelines as release gates.

Programming Languages
Primary development in Python and TypeScript, used for building production APIs, test frameworks, automation pipelines and evaluation harnesses.

API and Property Based Testing
Experience with Schemathesis, Hypothesis and fast-check for property based testing. Pydantic schema validation and REST, GraphQL, OpenAPI first approaches, with Postman for API exploration and testing.

Test Management and Process
Proficiency with Jira, Xray, TestRail, Qase.io and Allure for test management. Application of shift left and shift right strategies, testing trophy and diamond models, regression planning and release gate decision making.

Model Based Testing
Building model based test systems using xState, enabling generation of large numbers of test scenarios from a single state machine definition and automatic adaptation when product permissions change.

QA Process Design
Experience building QA processes from zero in multiple companies, owning quality strategy within feature squads, reviewing developer-written tests, advising on coverage gaps and test pyramid placement.