ML Platform Engineer, Distributed Training

What You Will Do:

Training platform
• Work with the client's infrastructure team on the GKE training platform (GPU node pools, job queueing and scheduling, quota and capacity) from the training workload's side.
• Diagnose and fix platform failures at the root (for example a controller that keeps running out of memory) rather than restarting them.
• Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for training throughput.
Distributed training
• Run, scale and debug multi-node distributed training with PyTorch (FSDP, DeepSpeed, Megatron or NeMo-style) and Ray Train.
• Profile jobs and remove bottlenecks across compute, memory, input pipelines and interconnect.
Benchmarking and evals
• Design and run performance benchmarks for the cluster and training stack: throughput, scaling efficiency, cost per run.
• Build and maintain model evaluation harnesses so changes to the platform or training recipes are measured, not assumed.
• Instrument training and the platform with telemetry and monitoring, and keep end-to-end pipeline tests running against what users actually run.
Ownership on the client team
• Treat the platform as your own: proactively find issues, plug gaps and propose solutions before anyone asks, keep the team informed of status and risks, and look after the scientists who use it.
• Explain and defend your design decisions to client engineers and scientists in detail.
• Stay effective while the client's priorities and team structure change.

Your Technical Toolkit:

• Core Languages: Mastery of Python and shell scripting

• AI/LLM Ecosystem: Extensive experience with Google Gemini, GPT-4, or LLaMA; deep knowledge of Prompt Engineering and Fine-tuning

• Data & Search: Expertise in Vector Databases (Vertex AI Vector Search, pgvector, etc.) and implementing Semantic Search

• Infrastructure: Hands-on experience with Google Cloud (Vertex AI) and building scalable software architectures

• Frameworks: Proficiency in LangChain, LlamaIndex, or similar orchestration layers

• Mindset: A strong software engineering foundation—you write clean, maintainable code and understand the full SDLC

Basic Qualifications:

5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training.

Strong Python software engineering: production-quality code, tests and code review. Coding is central to this role, not an afterthought.

GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity and quota management.

Distributed PyTorch and Ray Train: has personally run and debugged multi-node training jobs.

RDMA and GPU networking: GPUDirect/RDMA, NCCL tuning, multi-node GB200 (or H200) topology.

Benchmarking and evals: has designed performance benchmarks and model evaluations, and used them to make decisions.

Deep enough understanding to explain how and why a system works, without leaning on AI coding tools to carry the reasoning.

A track record of visible ownership: present, communicative, and accountable on a demanding team.

Personal Attributes:

Nice to have

Healthcare or life-sciences data (genomics, clinical, biomedical), and working under HIPAA.

Observability and telemetry for training platforms: metrics, tracing, alerting, end-to-end pipeline testing.