Sully.ai Logo

Sully.ai

Applied Research Scientist

Posted Yesterday
Be an Early Applicant
Remote
Hiring Remotely in US
Senior level
Remote
Hiring Remotely in US
Senior level
Design and build clinical-grade automated evaluation pipelines (LLM-as-judge + human review), define benchmarks, run rigorous experiments, partner with engineering to implement CI gates for accuracy, factuality, and safety, and publish findings to improve model robustness and reduce hallucinations.
The summary above was generated by AI

About Us
At Sully.ai, We’re Building the Most Impactful Healthcare Company on Earth

We believe that access to a great doctor is a basic human right. Today, that’s not a reality. Delays, misdiagnoses, administrative chaos, and burnout plague the system.

Our Mission: One Human, One Doctor. We build AI teammates that augment clinicians — scribes, nurses, receptionists, translators — all powered by our own world-class models and deployed in real-world care.

Our Traction

  • 450+ organizations signed 16 months

  • AI agents cut admin by ~2.8 hours daily and reduce onboarding 85%.

  • 5M+ Clinical Tasks completed to date, serving 36+ specialties.

  • Raised $25M from YC, Eric Yuan, Amity, Semper Virens

  • Patented AI architecture (MedCon-1) outperforms GPT-4.5, Gemini, Claude on clinical reasoning tasks

Sully requires A-players capable of 4 months = 1 year output.

Key Responsibilities
  • Build and scale automated evaluation pipelines (LLM-as-judge + human review) with clinical-grade benchmarks.

Hard Requirements
  • Proven experience designing agentic processes and LLM evaluation/benchmarking frameworks.

  • Strong Python and ML background (PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex).

  • Demonstrated ability to design rigorous experiments and translate findings into production.

  • Track record of published research or deep applied work in LLMs and agent evaluation.

  • Strong communication and technical writing skills to articulate complex findings clearly.

  • First-Month Focus

  • Audit existing evaluation approaches for clinical and agentic tasks.

  • Define initial benchmarks and build early automated pipelines.

  • Partner with engineering to land first set of CI gates for accuracy, factuality, and safety.

  • 90 Days

  • Deliver a repeatable evaluation framework with automated pipelines in production.

  • Demonstrate measurable improvements in robustness, hallucination reduction, or safety.

  • Publish or present internal research findings that directly shape product reliability.

  • If you’ve ever said, “I want to do work that actually matters”, this is it. Let’s build something life-changing, together.

Key Results (First 90 Days)
  • Deliver a repeatable evaluation framework with automated pipelines in production.

  • Demonstrate measurable improvements in robustness, hallucination reduction, or safety.

  • Publish or present internal research findings that directly shape product reliability.

Who Thrives Here

  • Entrepreneurial to your core: You think in outcomes, thrive in chaos, and take ownership without limits

  • Mission-obsessed: You’re here to save lives, not just ship features — patients and doctors are your why.

  • Impact-driven & fast-moving: You sprint toward hard problems and ship with sharp judgment.

  • Elite teammate: You raise the bar through high standards, direct feedback, and craft excellence.

Why Join Sully.ai?
🔥 Revolutionizing the antiquated $800B+ Healthcare market

🧠 50%+ of us are ex-founders. We hire A-players, not passengers

⚡️ Speed matters - we operate with urgency, autonomy, and ownership

🧪 You’ll work on real, first-of-their-kind problems at the edge of AI and medicine

❤️ Your work helps doctors reclaim their time - and patients get better, faster care

Sully.ai is an equal opportunity employer. In addition to EEO being the law, it is a policy that is fully consistent with our principles. All qualified applicants will receive consideration for employment without regard to status as a protected veteran or a qualified individual with a disability, or other protected status such as race, religion, color, national origin, sex, sexual orientation, gender identity, genetic information, pregnancy or age. Sully.ai prohibits any form of workplace harassment. 

Top Skills

Python,Pytorch,Tensorflow,Hugging Face,Langchain,Llamaindex
HQ

Sully.ai Mountain View, California, USA Office

883 N Shoreline Blvd, Mountain View, California, United States, 94043

Similar Jobs

4 Days Ago
In-Office or Remote
San Francisco, CA, USA
Senior level
Senior level
Generative AI
Lead research and product development of document understanding and multimodal AI. Design and evaluate VLMs, RAG and retrieval systems; build benchmarks and datasets; ship production-ready components (table extraction, structure parsing, semantic indexing, multimodal QA); and work with engineering, product, and customers to deploy models and agentic workflows at scale.
Top Skills: Python,Pytorch,Vision-Language Models,Rag,Retrieval Frameworks,Embeddings,Lora,Quantization,Llm Orchestration,Agentic Ai,Serverless,Apis,Scalable Ml Tooling
3 Hours Ago
Easy Apply
Remote
US
Easy Apply
100K-130K Annually
Mid level
100K-130K Annually
Mid level
Information Technology • Cybersecurity
The Product Marketing Manager at Huntress will focus on positioning, messaging, content creation, sales enablement, and go-to-market strategy for Managed ESPM, requiring collaboration with Product, Marketing, and Sales teams.
3 Hours Ago
Easy Apply
Remote
31 Locations
Easy Apply
120K-130K Annually
Senior level
120K-130K Annually
Senior level
Artificial Intelligence • Consumer Web • Digital Media • Information Technology • Social Impact • Software
The Senior Quality Engineer will lead quality initiatives, drive test automation, and ensure high standards throughout the engineering process, collaborating with cross-functional teams and implementing best practices.
Top Skills: AppiumBuildkiteJavaScriptPlaywrightTypescriptWebdriverio

What you need to know about the San Francisco Tech Scene

San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.

Key Facts About San Francisco Tech

  • Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Google, Apple, Salesforce, Meta
  • Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
  • Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
  • Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account