Design, build, launch, and operate resilient AI infrastructure services, tools, and open-source projects. Develop Kubernetes- and Slurm-based systems for large-scale GPU training, including APIs, automated job recovery, checkpointing, telemetry, and failure diagnosis. Lead team planning, mentor engineers, review designs, and establish production support practices such as monitoring, alerting, CI/CD, testing, and on-call operations. Collaborate with customers and the open-source community to improve ML infrastructure.
NVIDIA’s DGX Cloud team builds the infrastructure behind large-scale AI training for NVIDIA’s own training workloads and for our customers. We build services, tools and open-source software that make ML workloads more efficient and resilient.
We’re looking for a Senior Software Engineer for the NVIDIA Recovery and Resilience team. You’ll own our AI infrastructure services, tools and open-source projects to support resilient and efficient training from design through production. You’ll also mentor engineers and decide how the team runs what it builds.
What You Will Be Doing:
- Drive the design, build, launch and operation of our services, tools and open-source projects.
- Plan and assign the team’s work, mentor junior engineers and review designs.
- Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA’s own training workloads and customer workloads. This includes supercomputers at major cloud providers and on-premises clusters.
- Build APIs that connect to NVIDIA’s resiliency systems: telemetry, break/fix automation, and checkpoint and execution systems.
- Build a job submission layer that hides infrastructure failures from users and restarts long-running training jobs automatically.
- Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures and recommending fixes.
- Drive our open-source work. Release components so customers and the community can use them, and work with them to improve those components.
- Set up how the team runs its services: support, monitoring, alerts, CI/CD, integration test gates and on-call plans.
What We Need To See:
- BS, MS or PhD in Computer Science or a related field (or equivalent experience), plus 12+ years of experience.
- Proven experience who has taken large systems from design through production.
- Strong experience with Kubernetes and/or Slurm at scale.
- Hands-on experience with PyTorch, TensorFlow, JAX or Ray.
- Strong understanding of NVIDIA GPUs and networking, including how they fail.
- Experience running production services, including CI/CD, monitoring, alerting and on-call.
Ways to Stand Out from the Crowd:
- Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
- Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
- Background in building external services or maintaining open-source projects in ML infrastructure.
224,000 USD – 356,500 USD for Level 5; and 272,000 USD – 431,250 USD for Level 6.
You will also be eligible for equity and benefits.
This posting is for an existing vacancy.
NVIDIA uses AI tools in its recruiting processes.
NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.NVIDIA Santa Clara, California, USA Office
2701 San Tomas Expressway, Santa Clara, CA, United States, Santa Clara
NVIDIA San Francisco, California, USA Office
San Francisco, United States
NVIDIA San Jose, California, USA Office
San Jose, United States
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Provides front-line patient access and practice support by verifying insurance eligibility, obtaining referrals and authorizations, documenting registration information, collecting balances, and maintaining admission and preregistration requirements. Trains staff on systems and processes, serves as a computer-system superuser, leads department improvement efforts, and supports special projects. Works 32 hours weekly across facilities in Walnut Creek, Pleasanton, and Brentwood with variable shifts.
Top Skills:
Electronic Health Records (Ehr)Electronic Insurance Verification SystemsMS Office
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Provides direct patient care in a clinical setting, including collecting and documenting patient information, assisting with injections, EKGs, phlebotomy, examinations, diagnostic procedures, care coordination, EMR documentation, and room preparation. Requires completion of a Medical Assistant program, BLS certification, and at least one year of Medical Assistant experience.
Top Skills:
Electrocardiogram (Ekg)Electronic Medical Records (Emr)
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Manages regulatory affairs operations for a managed healthcare organization. Responsibilities include responding to regulatory agencies, reviewing laws and regulations, coordinating project plans, preparing filings, tracking regulatory metrics, supporting communications, and implementing process improvements. The role requires healthcare or managed care experience, strong stakeholder communication, regulatory research skills, and advanced Microsoft Office proficiency.
Top Skills:
Microsoft 365 CopilotExcelMicrosoft PowerpointMicrosoft Word
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
