Design and optimize communication stacks and runtime for large-scale distributed training (including MoE and hybrid parallelism). Improve collective performance, reduce latency, ensure fault-tolerant execution, debug NCCL/UCX/RDMA issues, and co-design topology-aware orchestration for 1,000+ GPU clusters.
About the Institute of Foundation Models
The Institute of Foundation Models (IFM) designs and operates ultra-scale GPU supercomputing systems to train next-generation foundation models. We believe performance, fault tolerance, and scalability are co-designed across model architecture, communication systems, runtime, and hardware topology.
This role sits at the core of that effort — driving communication performance, distributed reliability, and cross-layer optimization for large-scale training workloads.
The Mission
We are looking for a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including hybrid parallelism and Mixture-of-Experts (MoE) workloads.
This is not a network operations role. This is a systems-level engineering position focused on performance engineering, distributed debugging, and communication-runtime co-design.
· Design and optimize expert-parallel and hybrid-parallel communication patterns
· Drive high-performance hierarchical collectives for MoE workloads
· Co-design runtime orchestration with communication topology awareness
· Reduce tail latency and improve determinism across thousands of GPUs
· Architect fault-tolerant distributed execution under real-world cluster failures
Core Technical Scope
· Communication-compute overlap and topology-aware collective optimization
· Deep debugging of NCCL, RDMA, and custom communication layers
· Hybrid expert parallel strategies in modern large-scale MoE systems
· Elastic and resilient distributed job orchestration concepts
· Congestion analysis and routing optimization across InfiniBand/RoCE fabrics
· Microbenchmarking and performance modeling for communication-heavy workloads
Expected Technical Depth
· Hybrid expert parallel communication for Mixture-of-Experts training
· Scaling behavior under network pressure
· Distributed orchestration for elastic, large-scale training
· Fault detection and recovery in distributed GPU workloads
· Cross-layer bottlenecks: GPU ↔ NIC ↔ PCIe ↔ NVSwitch ↔ Fabric ↔ Scheduler
Required Background
· Experience optimizing distributed training at 1,000+ GPU scale (or equivalent depth)
· Hands-on expertise with RDMA, InfiniBand, RoCE, and GPUDirect RDMA
· Deep familiarity with NCCL and/or UCX internals
· Strong systems programming ability (C/C++, Rust, or Go)
· Strong familiarity with modern model training frameworks such as PyTorch
· Ability to troubleshoot and profile training performance issues related to communication bottlenecks
· Ability to translate research ideas into production-grade optimizations
· Experience debugging distributed hangs, desynchronization, and performance regressions
What We Mean by "Hardcore"
· You can explain why an communication degrades at scale and how to fix it
· You have improved real cluster throughput via communication redesign
· You can trace a distributed hang across ranks and identify the root cause
· You are comfortable working at the boundary between hardware and runtime
Application Requirements
· Include a link to your GitHub (required)
· Provide links to relevant distributed systems, HPC, or large-scale training projects
· Include a list of publications and/or public technical reports (if applicable)
· Describe the hardest distributed debugging problem you solved
· Include measurable performance improvements you have delivered
Academic Qualifications
Master’s, or Bachelor’s + 1 year of relevant experience.
Visa Sponsorship
This position is eligible for visa sponsorship.
Benefits Include
*Comprehensive medical, dental, and vision benefits
*Bonus
*401K Plan
*Generous paid time off, sick leave and holidays
*Paid Parental Leave
*Employee Assistance Program
*Life insurance and disability
Similar Jobs
Artificial Intelligence • Hardware • Software • Quantum Computing
Owner of backend services for a network and security platform: build high-throughput telemetry ingestion, time-series storage, streaming pipelines, and APIs. Write production Go/Rust, ensure reliability/observability, deploy containerized services on Kubernetes across cloud providers, participate in on-call and incident response, and influence platform architecture and engineering standards.
Top Skills:
AWSAzureDockerGCPGnmiGoGrafanaGrpcInfluxdbKafkaKinesisKubernetesNetconfOpentelemetryPrometheusRestRustSnmpSQLTerraformTimescaledb
Fintech • Machine Learning • Payments • Software • Financial Services
Lead a team to develop distributed systems using Golang and Python on Kubernetes, collaborate on AI/ML solutions, and drive technological innovation.
Top Skills:
AWSAzureCSSDockerGCPGoHTMLJavaJavaScriptKubernetesNoSQLPythonRdbmsSQLTypescript
Fintech • Machine Learning • Payments • Software • Financial Services
Lead diverse technology projects in distributed microservices and full stack systems, mentoring team members and collaborating with product managers on cloud-based solutions.
Top Skills:
AWSCSSDockerGoGCPHTMLJavaJavaScriptKubernetesAzureNosql DatabasesOpen Source RdbmsPythonSQLTypescript
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
