AI Research Engineer

Dscout AI Training & Evals builds human data and evals that show whether AI works for real people. For ten years Dscout has captured how people behave in real context, with consent and scientific rigor. AI Training & Evals points that engine at AI: consented human behavior and judgment, turned into the training data, evals, and benchmarks AI teams trust.

Capability has outrun trust. Models ace benchmarks and still surprise people in the wild. A benchmark says how capable a model is. It does not say whether the model works for the person using it, across weeks, in the mess of real life. That gap is where we work.

We serve AI researchers, the product builders who ship alongside them, and the research teams who partner with both. We are a new unit inside an established company: live customers, a mature platform (React, Elixir, Python, Swift, Kotlin; SOC 2 and HITRUST certified), a small senior team, and room to build. We work AI-native. Agents write code, evals gate changes, and we aim to do in months what used to take years.

A note from our Chief AI Officer & GM. The questions that matter most about AI are not answered by a bigger leaderboard. Does a model work for a real person across three weeks of real use? Does a rubric measure what it claims? Does an AI judge agree with the people it stands for? We have a decade of methods for studying people in real context, and we will publish our methods and benchmarks so the field can check our work. Join us to do rigorous science that matters.

Why this seat

• Work on a problem the field has not solved: measuring whether AI works for real people, over real time.
• Study something most evals do not have: real participants, doing real tasks in their own context.
• Own the benchmark. You are founding it, not joining a team that has one.
• Publish. Methods and benchmarks go to the research community, with first authorship on the benchmark.
• Join a small senior team with direct access to the Head of AI Research and the GM.

This is a research-first role focused on questions where the right rubric, benchmark, or method does not yet exist. You will find the gaps, propose bold directions, and take projects from first hypothesis through experiments and benchmark construction to publication. You will also build: the science has to run inside live customer programs.

What you will do

Rubrics and QA science

• Design rubrics and test whether they measure what they claim: validity, reliability, and sensitivity.
• Measure how reliable human raters are, and when disagreement is noise versus real signal.
• Calibrate AI judges against real human judgment, and study when they drift away from it.
• Build quality signals that put evidence behind every data point we ship.

Long-horizon, real-user benchmarks

• Design Dscout’s first benchmark: real users, in real context, over long horizons. Multi-week tasks across tools, not one-shot prompts.
• Define the protocol, task design, scoring, and release. Work with our delivery team on fieldwork.
• Work out how to keep a living benchmark honest over time: drift, contamination, difficulty, and what counts as a meaningful change.
• Write the paper and own the public release.

Human judgment as training and eval signal

• Study when real-world human feedback improves a model and when it adds noise.
• Explore how preference and judgment data from representative people differ from expert data.
• Turn findings into methods our forward deployed engineers and delivery team can run in the field.

New research directions

• These areas are the core, not the whole list. You will have significant latitude to propose new programs, such as calibrating user simulations against real behavior, human-AI collaboration, and behavioral evaluation of agents.
• Co-author with outside labs and academics on studies that need real-user data. Help run our paper-reading group.

A good first year

• A QA and rubric framework running on live customer programs, with documented validity.
• Benchmark v0 released, with a paper submitted to a top venue.
• At least one external team using or citing our work.
• Delivery colleagues using your methods without needing you in the room.

How we will hire: a conversation with the Head of AI Research; a research talk on a project you led; a design exercise (how would you validate a rubric for a long-horizon task?); a conversation with the GM, forward deployed engineers, and delivery partners.

What we care about most. A strong publication record helps. We care most about whether you can find the question that matters, design a rigorous way to answer it, and move fast enough that the answer is still useful.

What you bring

• Published research, or strong preprints and workshop papers, in evals, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
• A PhD, a PhD in progress, or an MS with two or more strong research internships at labs or top industry groups.  If you do not have advanced degrees but have a history of creative machine learning research work, this is just as desirable.
• Strong applied statistics: inter-rater reliability, confidence intervals, power, and a nose for when a metric misleads.
• Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
• Strong Python, and the ability to build research prototypes and evaluation pipelines other people run. You use AI coding tools as a real part of your workflow.
• Experience running experiments on messy human data, not only curated datasets.
• Clear technical writing, and the ability to explain your work to a lab researcher and to a product manager.
• Based in the Bay Area.

Nice to have: long-horizon or agentic evaluation work; psychometrics, item response theory, or human-subjects research; open-source benchmark or dataset contributions.

Compensation: $250-400K OTE + Equity. Includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 holidays, 12 weeks of paid parental leave, 401k match, and an education stipend.

This is an ideal list. If the role excites you and you do not check every box, we would still love to hear from you.