Own performance optimization for foundation-model training across thousands of accelerators. Diagnose and resolve bottlenecks across data paths, kernels, parallelism, communication, observability, and long-running stability. Partner with modeling, infrastructure, simulation, data-mining, and robotics teams; establish reusable performance improvements and technical direction. Build profiling infrastructure and agentic automation to reproduce jobs, verify numerical parity, detect regressions, and automate optimization workflows.
XPENG is a leading smart technology company at the forefront of innovation, integrating advanced AI and autonomous driving technologies into its vehicles, including electric vehicles (EVs), electric vertical take-off and landing (eVTOL) aircraft, and robotics. With a strong focus on intelligent mobility, XPENG is dedicated to reshaping the future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.
The Mission: We are building the next generation of end-to-end autonomous driving on physical AI foundation models — vision-language-action policies and world models — and the same training stack carries our robotics program. These training runs span thousands of accelerators and days of wall-clock — how fast we train them is how fast we learn. As a Senior Staff Machine Learning Performance Engineer, you will own that speed: working with the modeling teams on their hardest bottlenecks, setting the technical direction for performance, and turning one-off wins into defaults — including the agentic workflows that find the next round of them automatically.
Job Responsibilities:
- Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
- Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
- Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
- Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
- Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.
Minimum Skill Requirements:
- Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.
- Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
- A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
- In-depth understanding of computer architecture and high-performance computing.
- Strong C++ and Python, with a solid command of data structures and algorithms.
Preferred Skill Requirements:
- Hands-on experience writing and tuning CUDA or Triton kernels.
- Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout.
- Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
- Building profiling, tracing or benchmarking infrastructure that other teams depend on — CUPTI, Perfetto, or equivalent.
- Using LLM agents to automate parts of an engineering workflow.
- Autonomous driving, robotics, or another domain with hard latency, power and memory constraints.
- Publications or open-source contributions in ML systems or performance.
What do we provide:
- A fun, supportive and engaging environment.
- Infrastructures and computational resources to support your work.
- Opportunity to work on cutting edge technologies with the top talents in the field.
- Opportunity to make a significant impact on the transportation revolution by the means of advancing autonomous driving.
- Competitive compensation package.
- Snacks, lunches, dinners, and fun activities.
The base salary range for this full-time position is $244,140, - $413,160, in addition to bonus, equity and benefits. Our salary ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training.
We are an Equal Opportunity Employer. It is our policy to provide equal employment opportunities to all qualified persons without regard to race, age, color, sex, sexual orientation, religion, national origin, disability, veteran status or marital status or any other prescribed category set forth in federal or state regulations.
XPeng Motors Palo Alto, California, USA Office
Palo Alto, CA, United States, 94301
XPeng Motors San Jose, California, USA Office
San Jose, United States
Similar Jobs
Aerospace
Own full-cycle recruiting for engineering, technical, and business roles, including sourcing, candidate screening, interview coordination, offer support, pipeline reporting, and candidate communications. Partner with hiring managers to define requirements and hiring strategies, build passive-candidate pipelines, and improve recruiting processes. The role is fully onsite in Torrance, California and requires U.S. citizenship.
Top Skills:
Applicant Tracking Systems (Ats)GreenhouseLinkedin Recruiter
Aerospace
Lead end-to-end flight software development for autonomous spacecraft, including mission operations tools, ground services, embedded vehicle software, telemetry, commanding, testing, simulation, and flight readiness. Collaborate with avionics, propulsion, GNC, and systems teams to define interfaces and data models. Support CI/CD, software- and hardware-in-the-loop testing, environmental testing, and flight campaigns in a fast-moving, on-site aerospace startup environment.
Top Skills:
Ci/CdCloud-Based Ground SystemsCommandingDevOpsEmbedded SoftwareFault ToleranceHardware-In-The-Loop TestingMission Control SystemsMission Operations SoftwareReal-Time SystemsSafety-Critical SystemsSimulation EnvironmentsSoftware-In-The-Loop TestingTelemetryVisualization
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Manage enterprise customer implementations of Samsara’s IoT platform, from deployment planning through adoption and value realization. Build launch roadmaps, track implementation and account health, train executives and end users, resolve challenges, and coordinate with Sales, Support, Sales Engineering, Product, and other teams. Mentor Customer Success and Support colleagues while managing multiple customer engagements and strengthening long-term customer relationships.
Top Skills:
Enterprise SoftwareIotSaaS
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
