About the company
Our client is a technology company.
The role
Raydar is recruiting for this role on behalf of our client. Design datasets, evaluation methods and measurement frameworks that shape how machine learning models are trained and assessed. Collaborate with external research partners, run fast experiments and convert findings into concrete specifications for large-scale training work.
What you'll do
- Design targeted data samples that reveal weaknesses in model behavior across a range of subject areas.
- Build and refine scoring rubrics and reward signals used in model training pipelines.
- Study how human annotators behave and run experiments to improve model capabilities.
- Develop quantitative methods for assessing dataset quality, diversity and downstream impact.
- Own the data pipelines that supply both naturally occurring and artificially generated training material.
- Work with research partners to turn their training goals into clear data and evaluation specifications.
Requirements
What we're looking for
- 1 to 4 years of experience in software or machine learning engineering.
- Strong Python skills and experience building dependable data pipelines, experimentation tooling or evaluation systems.
- Hands-on experience with language model evaluation, post-training, reward modeling or closely related ML infrastructure.
- Sound scientific judgment, including setting up well-controlled tests and drawing conclusions from imperfect results.
- Background in building or auditing training data and judging how well it supports model performance.
- Ability to learn unfamiliar domains quickly and reason across them.
Benefits
Compensation and benefits
- Base salary: USD 260,000 to 290,000 per year
- Equity
- Profit sharing
Location and work model
- San Francisco, CA, United States
- On-site
- Full-time
Similar Jobs
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
