About the company
Our client is a fast-growing software company.
The role
Raydar is recruiting for this role on behalf of our client. Design and build the measurement systems used to judge the quality of machine learning outputs. You will define the scoring approach, assemble labeled reference data and run evaluation pipelines, then feed the findings back into model and product decisions.
What you'll do
- Define quality measures that capture what a good result looks like across a wide variety of inputs and edge cases.
- Build evaluation pipelines and harnesses that score output quality reliably and at scale.
- Create and curate labeled reference data so that evaluation results can be trusted.
- Connect evaluation findings back to model and product decisions.
- Present results in a way that gives the whole team clear, actionable answers.
Requirements
What we're looking for
- Proven experience building evaluations for LLM-based data extraction or data quality at scale.
- Experience creating the scoring approach and the evaluation system yourself, rather than only running an existing benchmark.
- Experience assembling human-labeled or verified reference data to compare model outputs against.
- Evaluation work that clearly influenced model or product decisions, visible in your work history.
- Strong Python skills.
- A solid grounding in statistics for choosing and interpreting quality measures.
- Confidence working on open-ended problems where there is no established way to measure success.
- A bias toward shipping, measuring and iterating quickly.
- Willingness to work on-site five days a week, now or after relocating before the start date.
Benefits
Compensation and benefits
- Base salary: USD 250,000 to 290,000 per year
- Equity
- Health, dental and vision coverage
- Life and disability insurance
- Retirement plan
- Paid time off
- Paid parental leave
Location and work model
- San Francisco, CA, United States or Toronto, ON, Canada
- On-site, five days a week
- Full-time
Similar Jobs
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
