Design and implement scientifically valid automated evaluations of frontier AI systems, focusing on impacts to user wellbeing, autonomy, mental health, and decision-making. Develop user simulators, LLM-as-a-judge pipelines, evaluation rubrics, and population-specific methods. Collaborate with research engineers, scientists, frontier AI labs, regulators, domain experts, and affected communities to improve evaluation realism, measurement, and oversight.
Salary range: $250,000 - $450,000/year + benefits
Description: Transluce is a fast-moving nonprofit research lab building the public tech stack for AI evaluation and oversight. We are pioneering research into the behaviors of AI chatbots and their effect on user wellbeing, and we’re improving outcomes for millions of sensitive AI interactions with vulnerable users.
About the role: As an AI Behavior Researcher, you will lead projects to design and develop automated evaluations of frontier AI systems that are technically sophisticated, scientifically valid, and concretely impactful. This includes expanding on our existing evaluation pipelines to conduct novel analyses of AI behaviors that affect the autonomy and wellbeing of specific user groups (e.g., children or users located in countries beyond the United States).
As an early member of a highly collaborative team, you will learn and grow quickly, and work directly with frontier labs to improve AI evaluations design, with regulators to improve independent oversight of AI, and with domain experts and affected populations to enhance the realism and relevance of our evaluations.
Core responsibilities:
- Develop novel, valid automated evaluations of AI’s impacts on users, including their mental health and decision making.
- Write code to implement and run automated evaluations, such as user simulators or LLM-as-a-judge pipelines.
- Design methods to improve the ecological validity and realism of automated evaluations for specific populations, such as customizing existing user simulation methods to capture the vocabulary used by children.
- Write and revise judge rubrics to evaluate model behaviors related to user wellbeing and decision making, systematizing abstract, socially situated concepts into clear measurement criteria.
- Collaborate with scientists and research engineers to productionize best practices in AI behavioral evaluation.
Minimum qualifications:
- Expertise on quantitative generative AI evaluation and measurement. Good intuition about how to systematize and operationalize complex social concepts.
- Relevant experience designing and validating automated AI evaluation methods, such as LLM-as-a-judge systems or multi-turn benchmarks.
- Proficiency in Python to implement analysis and evaluation tooling.
- Meticulous, good experimental design, epistemic self-awareness and transparency.
- Ability to balance between the needs of AI researchers and domain experts, as well as between researchers and senior decision makers.
- Strong communication skills, low ego, openness to giving and receiving feedback.
Preferred qualifications (not required):
- Experience running automated evaluations at scale or in a production context.
- Experience conducting controlled human subjects experiments to validate automated evaluation methods.
- Experience in customer-facing, consulting, or forward-deployed roles translating ambiguous stakeholder needs into concrete deliverables.
- Experience or training in human-centered design or HCI research methods, including working with domain experts or impacted communities.
- Experience or demonstrated interest in studying AI’s psychological or social impacts, such as for crisis support, manipulation or sycophancy, political persuasion, or displacing human relationships.
- Experience designing multilingual generative AI evaluations.
- Experience and comfort using AI coding agents at work.
We are hiring at all levels of experience and would encourage those enthusiastic about the role who do not meet all of the qualifications to apply. We are located in San Francisco and excited to work together in-person. We are open to sponsoring international visas.
Transluce San Francisco, California, USA Office
1301 Sansome St, San Francisco, California, United States, 94111 1122
Similar Jobs
.png)
Consumer Web • eCommerce • Marketing Tech • Retail • Software • Analytics • Generative AI
Own the positioning, messaging, launches, packaging, pricing, adoption, and revenue growth strategy for Klaviyo Personalization. Partner with Product, Sales, Customer Success, Enablement, and Marketing to translate AI and data science capabilities into compelling customer value. Shape the roadmap through market insight, develop enablement and thought-leadership materials, influence cross-functional teams, and measure launch and program performance.
Top Skills:
Ai/MlAnalyticsCustomer Data Platforms (Cdp)Marketing AutomationPredictive Models
Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3
Lead the technical vision, architecture, and roadmap for Circle’s identity and access platform. Manage and develop backend and frontend engineers, guide architectural changes, review designs and code, troubleshoot production issues, participate in incident response, and partner with Security, Product, Risk, and engineering teams. The role focuses on authentication, authorization, customer and agent identity, secure delegation, platform reliability, migrations, and operational readiness.
Top Skills:
Auth0CedarJavaKotlinMcpOauth 2.XOidcOktaOpen Policy Agent (Opa)SAMLSpiffeSpireTypescript
Cloud • Information Technology • Internet of Things • Machine Learning • Software • Cybersecurity • Infrastructure as a Service (IaaS)
Supports RF network performance and RAN optimization across 4G, 5G, 6G, and small-cell deployments. Responsibilities include developing generative AI automation tools, launching carriers and sites, conducting verification and drive testing, monitoring KPIs, troubleshooting radio and baseband issues, and delivering engineering projects through customer acceptance. The role requires telecommunications or engineering knowledge, automation familiarity, structured problem-solving, and collaboration with customers and cross-functional teams.
Top Skills:
4G5G6GGenerative AiPower BIPythonRanRf OptimizationSmall CellsTableau
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
