About the company
Our client is an artificial intelligence company.
The role
Raydar is recruiting for this role on behalf of our client. Design and build test environments, evaluation harnesses and labeled datasets that measure how AI systems behave across multi-step tasks. The work combines reviewing system logs, spotting recurring errors and turning them into reusable test material. The role suits an engineer who wants to own problems from diagnosis through delivery.
What you'll do
- Construct simulated software environments that mimic the tools and workflows automated systems use.
- Build grading logic and evaluation systems that hold up when systems look for shortcuts.
- Create benchmarks whose scores track real-world results.
- Review interaction logs to find and measure previously unrecorded failure patterns.
- Turn findings into reusable datasets and test suites for internal improvement and external partners.
- Support data and training agreements alongside the founders while refining the evaluation tooling.
Requirements
What we're looking for
- One to seven years of relevant experience.
- Hands-on work with training data, post-training, benchmarks or evaluation for machine learning models.
- Judgment to tell which failures or data points are meaningful and valuable.
- A track record of building or contributing to benchmarks, datasets or environments used to train or test models.
- Skill in replicating common business software well enough that automated agents can interact with it reliably.
- Understanding of grader design, evaluation pitfalls and the ways systems game metrics.
- Prior work at a data vendor is relevant if it produced training material or model assessments, not expert labeling alone.
- A concrete example of building a benchmark, dataset, environment or evaluation system.
- Commercial awareness of what data and environments are worth.
- Ability to explain technical decisions and trade-offs in an interview.
Bonus points
- Found a real flaw in a widely used benchmark.
- Caught a model gaming a grader and fixed the grader.
- Built environments or benchmarks a leading AI lab used for training or evaluation.
- Startup founding experience or participation in a selective fellowship.
- High-level competitive achievement in mathematics, programming, debate or games.
- Understanding of how benchmarks and datasets support commercial model evaluation.
Benefits
Compensation and benefits
- Base salary: USD 150,000 to 300,000
- Equity
- Health, dental and vision coverage
- Retirement plan
- Flexible time off
Location and work model
- San Francisco, CA, United States
- On-site (in office)
- Full-time
Similar Jobs
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
