Lead and hands-on build Luma's inference serving platform: architect scheduling, routing, autoscaling, and observability across thousands of GPUs; operate and optimize heterogeneous fleets; ship new model architectures from research to production; own SLOs, cost, incidents, and team hiring/mentorship.
The Role
What You’ll Do
Representative Projects
Background
Bonus Points
About Luma
Inference is where all of Luma’s compute meets all of Luma’s users. The inference platform team owns the entire serving stack — from request routing, scheduling, and queueing to fleet-wide orchestration across thousands of GPUs spanning multiple clusters, clouds, and hardware vendors. The team has a dual mandate: maximize the efficiency, reliability, and unit economics of production inference for millions of users, and enable research to move fast — new model architectures should go from research checkpoint to production in days, and our serving stack increasingly powers training itself through online reinforcement learning.
We are hiring a Tech Lead Manager to lead this team through its next phase of growth. This is a hands-on leadership role, not a pure management position: we expect you to spend at least 50% of your time as an individual contributor — designing, building, and debugging in the serving stack — alongside hiring and growing the team, setting technical direction, and partnering across research, product, and infrastructure. You lead by shipping, and you set the technical bar for the team through your own work.
- Spend at least half your time hands-on in the serving stack: architect and build core platform components, own the hardest design decisions, and debug the toughest production incidents yourself
- Lead, grow, and develop the inference engineering team: own hiring, coaching, and career growth, and build the team’s operational culture — on-call, incident response, capacity planning, and postmortems
- Set the technical roadmap for the serving platform: model serving engines, request routing and scheduling, autoscaling, caching, observability, and deployment pipelines
- Own the platform’s SLOs and economics: latency and availability targets, GPU utilization, and cost per generation across every model we serve
- Partner closely with research to ship new model architectures into production on day zero, and to integrate serving into online RL and evaluation loops
- Manage and optimize inference workloads across heterogeneous fleets — multiple clusters, clouds, and GPU vendors — including capacity planning and hardware bring-up
- Build sophisticated scheduling and queueing systems that optimally leverage expensive GPU resources against live traffic patterns, cluster availability, and user priority
- Design intelligent routing and scheduling that optimizes request distribution across thousands of GPUs in multiple regions and clouds
- Stand up disaggregated prefill/decode serving with tiered KV-cache reuse across GPU memory, DRAM, NVMe, and network storage
- Autoscale and hot-swap models across the fleet to dynamically match GPU supply with live demand across production, research, and experimental workloads
- Take a new multimodal architecture from research checkpoint to a production deployment serving millions of users, including quantization, speculative decoding, and precision/regression validation across hardware platforms
- Build end-to-end tracing that follows any inference request through its full lifetime — queueing, routing, prefill, decode, and delivery
- Integrate the inference stack into an online reinforcement learning pipeline where serving throughput directly gates training progress
- 8+ years of engineering experience in large-scale distributed systems or ML infrastructure, with several years building and operating model-serving or inference platforms in production
- Experience running inference platforms at scale — you have operated fleets on the order of thousands of GPUs across multiple clusters or clouds, and you understand what breaks at that scale
- Technical leadership experience, including managing or leading engineers through periods of rapid growth — and a genuine desire to keep at least half your time in hands-on technical work rather than move into pure management
- Deep, practical expertise in LLM and foundation-model serving engines (vLLM, SGLang, TensorRT-LLM, or equivalent) — ideally you’ve modified engine internals, debugged edge cases under load, and contributed improvements back
- Strong command of the serving-performance toolkit: continuous batching, KV-cache management, quantization, speculative decoding, and parallelism strategies (TP/EP/pipeline)
- Strong Python and PyTorch; experience operating services on Kubernetes at scale
- Experience with queues, scheduling, traffic control, and fleet management at scale
- Experience serving diffusion, video, or other multimodal generative models (not just text), and with FFmpeg/multimedia processing
- Experience with modern networking stacks — RDMA (RoCE, InfiniBand), NVLink — including KV-cache transfer and multi-node serving topologies
- Experience across heterogeneous accelerator platforms (NVIDIA, AMD, TPU, Trainium) and the porting/validation work that comes with them
- Contributions to open-source serving infrastructure (vLLM, SGLang, Ray, Kubernetes ecosystem)
- Systems-language depth (Rust, C++, CUDA/HIP) for kernel- and runtime-level optimization
Luma’s mission is to build unified general intelligence that can generate, understand, and operate in the physical world.
We believe that multimodality is critical for intelligence. To go beyond language models and build more aware, capable and useful systems, the next step function change will come from vision. So, we are working on training and scaling up multimodal foundation models for systems that can see and understand, show and explain, and eventually interact with our world to effect change.
Luma AI San Francisco, California, USA Office
San Francisco, CA, United States
Similar Jobs
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
The Engagement Marketing Manager will focus on activation and habituation through experimentation, analytics, and collaboration across Product, Engineering, and Design teams to drive customer engagement and product decisions.
Top Skills:
Ai ToolsBrazeIterableModeSalesforce Marketing CloudSQLTableau
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Design, build, and operate distributed ledger, transaction, settlement, and treasury systems. Develop durable ledgering APIs, emit trusted data for finance, maintain high-reliability services to SLOs, lead complex cross-team projects, mentor engineers, and integrate AI tooling across the development lifecycle.
Top Skills:
AmpAuroraAWSClaudeCodexDatabricksDatadogGoGooseGraphQLJavaJooqKotlinKubernetesMcpMySQLProtocol BuffersPythonSnowflakeSQLTemporalTerraformVitess
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Own end-to-end reconciliation and reporting for payment settlements: design automated SQL-based reports, reconcile card-network and banking reports to internal ledgers, drive discrepancy root cause and resolution, define metrics and controls, partner with Engineering/Data to build pipelines, support month-end close and audits, mentor analysts, and lead process automation and control improvements.
Top Skills:
AchBigQueryExcelFednowLookerPythonRtpSnowflakeSQLSwiftTableau
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
