Judgment Labs Logo

Judgment Labs

Applied AI Engineer

Reposted One Month Ago
Be an Early Applicant
In-Office
San Francisco, CA, USA
Mid level
In-Office
San Francisco, CA, USA
Mid level
Build scalable systems to aggregate, index, and analyze large-scale agent interaction data; develop agent-based evaluation systems; design post-training and optimization workflows; build internal tools and infrastructure for experimentation, analysis, and training; ship research into production and collaborate cross-functionally to improve agent behavior.
The summary above was generated by AI
The Role:

We are looking for Applied AI Engineers to build AI systems that use agent interaction data to understand how agents behave, evaluate them at scale, and improve them through learning and feedback.

Your research will not live on a whiteboard. You’ll work directly with real-world agent data, apply frontier methods in production, and see your work ship into the product. By making agent behavior measurable and debuggable, your systems will support teams deploying agents across finance, legal, operations, and other high-stakes workflows. You will own projects end-to-end, with significant autonomy, and work closely with the team to build self-improving agent systems.

What You'll Do:
  • Build AI systems to aggregate, index, and analyze large-scale long-running agent interaction data in order to extract meaningful signals

  • Design and implement post-training and optimization workflows to improve agents, both internally and for customers

  • Build agent platform infrastructure, including orchestration, runtimes, and developer tools that help teams define, test, deploy, and iterate on complex agent workflows

  • Build internal tools and infrastructure that support rapid experimentation, analysis, and training

  • Work closely with product to integrate agents into customer-facing workflows

  • Collaborate with external companies and research partners on frontier AI research

What We're Looking For

Every hire clears three bars, no exceptions:

  • Agency. You are intellectually curious, self-directed, and stay up to date with the latest research, blogs, trends, and ideas.

  • Depth of thought. You can reason clearly about abstract systems, and ideally have experience working on agents, RL, or the infrastructure that supports them.

  • Ownership. You own outcomes, not just tasks. You use freedom to experiment responsibly, make business-driven decisions, and focus first on work that moves the company forward.

More specifically, you should bring strength in at least one of the following areas:

  • Data quality, evaluation, benchmarking, and hands-on work with messy production data

  • Agent systems built or evaluated in real-world or production settings

  • Reinforcement learning, post-training, agents, or machine learning fundamentals

  • Infrastructure and systems work across training, data pipelines, evaluation, or model serving

  • Translating research into product while balancing customer constraints, technical tradeoffs, and business impact

  • Turning ambiguous problems into clear, well-designed plans

HQ

Judgment Labs San Francisco, California, USA Office

425 Bush St, San Francisco, California, United States, 94108 3708

Similar Jobs

3 Days Ago
Hybrid
San Francisco, CA, USA
185K-210K Annually
Senior level
185K-210K Annually
Senior level
Artificial Intelligence • Marketing Tech • Automation
Lead the design and implementation of production agentic AI systems for enterprise compliance. Responsibilities include multi-agent orchestration, RAG and memory systems, stateful Temporal workflows, evaluation and safety frameworks, OCR and document understanding pipelines, and AWS infrastructure. The role requires hands-on Python engineering, LLM application development, distributed systems expertise, and end-to-end ownership in an in-office San Francisco startup environment.
Top Skills: Amazon BedrockAmazon S3Amazon SagemakerAWSAws LambdaCrewaiDocument ParsingEmbeddingsLanggraphLarge Language Models (Llms)OcrPythonRetrieval-Augmented Generation (Rag)Structured RetrievalTemporalTypescriptVector SearchVision-Language Models (Vlms)
3 Days Ago
Hybrid
San Francisco, CA, USA
220K-270K Annually
Senior level
220K-270K Annually
Senior level
Healthtech
Own the end-to-end design, development, deployment, evaluation, and monitoring of production AI systems for healthcare workflows. Build reliable agents and copilots using retrieval, rules, state machines, and human oversight. Partner cross-functionally with engineering, product, clinical, operations, and analytics teams to integrate AI into real-world workflows, improve measurable outcomes, shape architecture, mentor engineers, and ensure safe handling of uncertainty.
Top Skills: Agent ArchitecturesBackend ApisData ModelingDocument IntelligenceKnowledge GraphsLanguage ModelsModel ApisModel Fine-TuningMultimodal ModelsObservabilityOntologiesReinforcement LearningRetrieval SystemsState MachinesSynthetic Data GenerationWorkflow Engines
3 Days Ago
In-Office
Milpitas, CA, USA
Senior level
Senior level
Internet of Things
Own applied AI models powering Arlo’s smart vision products, including computer vision, vision-language, retrieval, and agent systems. Responsibilities span model training, fine-tuning, evaluation, data curation, embeddings, vector search, inference optimization, production deployment, and telemetry. The role requires hands-on technical leadership, cross-functional collaboration, mentoring, and end-to-end ownership of scalable AI features across edge and cloud environments.
Top Skills: AWSDockerFfmpegGpu InferenceLoraMlopsPythonPyTorchQloraRagTensorrt-LlmTritonVector SearchVllm

What you need to know about the San Francisco Tech Scene

San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.

Key Facts About San Francisco Tech

  • Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Google, Apple, Salesforce, Meta
  • Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
  • Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
  • Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account