Top Tech Jobs & Startup Jobs in San Francisco Bay Area, CA

Design and author research-oriented, multi-step benchmark tasks for frontier AI models; create reference solutions and evaluation standards using Python and notebooks; review AI outputs to identify methodological and analytical flaws; collaborate with AI researchers to refine evaluation methodologies.

Validate and harden AI evaluation benchmarks by designing test cases, reviewing tasks and reference solutions, debugging Python evaluation scripts, identifying edge cases and exploits, developing repeatable QA processes, and collaborating with researchers to ensure reliable, reproducible benchmark results.

Assess finance tasks and workflows for professional realism, clarity, and completeness. Identify missing assumptions, ambiguities, edge cases, and impractical constraints. Validate workflow logic, evaluate whether tasks measure professional finance capabilities, and provide concise, structured written feedback and improvement recommendations while collaborating with other finance experts during a high-engagement remote sprint.

Create clear, structured rater guidelines, evaluation rubrics, and reviewer instructions for human evaluators across domains. Review and revise documentation to remove ambiguity, design scoring frameworks, handle edge cases, and collaborate with researchers and SMEs to ensure consistent, high-quality AI evaluations.

Evaluate and review AI-generated legal content for accuracy and quality, answer legal questions, recommend authoritative references, share real-world legal workflows and standards, and collaborate with a multidisciplinary team to improve AI performance in legal domains. Part-time, remote contractor role (~10 hours/week).

Perform structured evaluations of AI assistants using MCP and plugins: design realistic prompts, execute multi-step personal workflows (screen-recorded), assess outputs for usefulness and safety, identify failures, and write detailed reports and rubrics. Remote, US-based contractor role requiring 20+ hours/week and rapid task turnaround.

Design and run red-team evaluations to find failure modes and edge cases in frontier LLMs. Create reproducible, multi-step benchmark tasks, document technical findings, and collaborate with researchers to refine grading and improve model robustness. Commit ~35 hours/week as a remote independent contractor.

Evaluate and improve LLM financial reasoning by designing real-world finance tasks, creating reference solutions, scoring AI outputs with structured rubrics, refining evaluation frameworks, and collaborating with researchers to raise data quality and model performance. Must provide detailed feedback and identify gaps to enhance AI across finance use cases.

Design realistic chip architecture and RTL, build verification environments and testbenches, evaluate AI-generated hardware solutions, provide technical feedback, and collaborate to improve AI understanding of digital design, verification, and semiconductor workflows. Commit 20–40 hours/week remotely as an independent contractor.

Contribute financial expertise to train and evaluate AI language models: design finance tasks, produce reference solutions, evaluate AI outputs with rubrics, collaborate with SMEs, and identify knowledge gaps to improve AI reasoning in finance. Independent contractor role requiring at least 10 hours/week.