Lead QA automation for AI/LLM systems: define validation criteria for non-deterministic models, build automated and performance tests, validate RAG systems, perform API and integration testing, ensure release readiness and trace defects to data/models/requirements.
N-iX is looking for a Senior QA Automation Engineer to lead the validation and verification strategies for EDGE Group’s AI transformation. You will be responsible for defining "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards of the defense sector.
Qualifications
- 5+ years in QA Automation
- Experience testing data-driven systems, AI, or ML models
- Strong analytical skills for defining test criteria in non-deterministic systems
- Ability to collaborate with engineers, data scientists, and product teams
- Experience in regulated industries (defense, aerospace) will be a plus
- English level - at least Upper-Intermediate
Technical Requirements
- Strong experience in Python and test automation frameworks (Pytest, Selenium/Playwright, API testing)
- Experience with performance testing tools (JMeter, Locust, K6)
- Familiarity with LLM evaluation tools (e.g., DeepEval, TruLens)
- Knowledge of SQL and data validation tools
- Experience with CI/CD pipelines (GitLab) and version control (Git)
- Exposure to test reporting tools (Allure, TestRail)
Key Responsibilities1. AI & LLM Testing
- Build automated tests to evaluate AI outputs for accuracy, consistency, and reliability
- Validate Retrieval-Augmented Generation (RAG) systems using metrics like relevance and correctness
- Create regression tests to detect changes in AI behavior (prompt drift)
- Test integrations between AI systems and enterprise platforms (e.g., SAP)
- Perform API testing, including security and access control validation
- Design performance tests for latency, scalability, and system throughput
- Align test cases with system requirements and user needs
- Support release readiness through structured testing reviews
- Track and manage defects, ensuring traceability to data, models, or requirements
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Similar Jobs
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
The Client Legal Administrator manages corporate governance, compliance, and board meeting processes, ensuring accurate records and communication with stakeholders.
Top Skills:
Document Management SystemsExcelMicrosoft Office (WordOutlook)
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Lead end-to-end recruitment for Experienced Hires across Greece and Central Europe. Partner with hiring managers to define requirements, source and assess candidates, manage candidate experience, build talent pools, and drive continuous improvement with TA leads and coordinators.
Top Skills:
ExcelMS OfficePhenomWorkday
Information Technology
As the Director of Data Science, you'll mentor teams, oversee projects, drive data-driven decisions, and implement AI and ML solutions while enhancing our data pipeline.
Top Skills:
BigQueryClickhouseDruidMlNlpPythonRedshiftSQL
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
