Software Engineer - Distributed Training

We are looking for an experienced software engineer to help build, optimize, and maintain large-scale distributed training infrastructure based on the PyTorch ecosystem. This role focuses on production-grade training workflows involving multi-GPU and multi-node orchestration, high-performance communication layers, and advanced parallelism strategies.

You’ll work alongside infrastructure and machine learning teams to ensure training jobs are efficient, scalable, and resilient.

• Develop and support distributed PyTorch training jobs using torch.distributed / c10d
• Integrate and maintain frameworks like Megatron-LM, DeepSpeed, and related LLM training stacks
• Diagnose and resolve distributed training issues (e.g., NCCL hangs, OOM, checkpoint corruption)
• Optimize performance across communication, I/O, and memory bottlenecks
• Implement fault tolerance, checkpointing, and recovery mechanisms for long-running jobs
• Write tooling and scripts to streamline training workflows and experiment management
• Collaborate with ML engineers to ensure compatibility with orchestration and container environments (e.g., Slurm, Kubernetes)

• Deep experience with PyTorch and torch.distributed (c10d)
• Hands-on experience with at least one of: Megatron-LM, DeepSpeed, or FairScale
• Proficiency in Python and Linux shell scripting
• Experience with multi-node GPU clusters using Slurm, Kubernetes, or similar
• Strong understanding of NCCL, collective communication, and GPU topology
• Familiarity with debugging tools and techniques for distributed systems

Preferred Skills

• Experience scaling LLM training across 8+ GPUs and multiple nodes
• Knowledge of tensor, pipeline, and data parallelism
• Familiarity with containerized training environments (Docker, Singularity)
• Exposure to HPC environments or cloud GPU infrastructure
• Experience with training workload orchestration tools or custom job launchers
• Comfort with large-scale checkpointing, resume/restart logic, and model I/O

Bonus Skills

• Profiling tools: PyTorch Profiler, Nsight, nvprof, or equivalent
• Experience with performance tuning in distributed training environments
• Contributions to ML infrastructure open-source projects
• Familiarity with storage, networking, or RDMA/GPU Direct technologies
• Understanding of observability in ML pipelines (metrics, logs, dashboards)

• Challenging projects.
• A friendly and inclusive workplace culture.
• Competitive compensation.
• A great benefits package.
• Catered lunch