Senior Systems Engineer, Kernel Automation & Validation
CoreWeave is seeking a specialized Kernel Automation Engineer to join our HAVOCK Team. In this role, you will be the architect of our validation infrastructure. You won't just run tests; you will design the framework that ensures stability across the entire vertical stack—from Firmware to Kernel to NCCL to Slurm.
We need an engineer who understands that a kernel is only as good as the workloads it supports. You will build the CI/CD pipelines and hardware-in-the-loop test harnesses that validate our custom Linux builds against next-generation GPUs, high-speed interconnects, and AI schedulers.
Hardware - Acceleration - Virtualization - Operating Systems - Containerization - Kubelet
Our Team’s Stack:
- Languages: Python (expert level), Go, Bash, C
- CI/CD & Automation: GitHub Actions, Jenkins, Ansible, Terraform, ArgoCD
- Validation Tools: LAVA (Linaro), LabGrid, kselftest, Pytest, Syzkaller
- The Stack: BIOS/BMC Firmware, Linux Kernel, OFED/RDMA drivers, NCCL, Slurm
- Hardware: Bare metal provisioning (PXE, IPMI), NVIDIA GPU’s, ConnectX
Focus Areas:
- End-to-End Test Framework: building a unified test harness that triggers validation from low-level firmware up to high-level Slurm job scheduling.
- Hardware-in-the-Loop: Managing and automating a fleet of bare-metal DUTs (Device Under Test) for kernel regression testing.
- Performance Validation: Automating benchmarks for NCCL (GPU-to-GPU communication) and RDMA to catch latency regressions before production.
- Build Pipeline Evolution: Refactoring the "Havock" build system to allow any engineer to spin up a custom kernel and test it on real hardware via CLI.
- Firmware/Driver Interop: Automating matrix testing between different Kernel versions, NVIDIA Drivers, and NIC Firmware.
Responsibilities:
- Design and implement a "full-stack" validation pipeline that ensures a kernel change doesn't break NCCL rings or Slurm scheduling logic.
- Develop a CLI tool (Go/Python) that allows kernel engineers to launch ad-hoc test runs on specific hardware SKUs.
- Automate the provisioning of test nodes, including BIOS/BMC configuration and firmware flashing.
- Integrate standard kernel tests (LTP, kselftest, blktests) into a modern CI/CD workflow.
- Write "Glue" code to connect low-level kernel outputs (dmesg, serial logs) to high-level reporting dashboards.
Requirements:
- 5+ years of experience in Test Automation, Release Engineering, or Systems Development.
- Strong Python/Go skills: Ability to write complex test frameworks and CLI tools (beyond simple scripting).
- The "Vertical" Perspective: Experience debugging or testing interactions between Firmware, Kernel, and Userspace.
- CI/CD Expertise: Experience building pipelines that manage hardware resources (not just software builds).
- HPC Knowledge: Familiarity with Slurm (job scheduling), MPI, or NCCL (collective communications).
- Experience with bare-metal provisioning (PXE, iPXE, Kickstart).
Nice-to-haves:
- Experience testing RDMA/InfiniBand networks.
- Familiarity with NVIDIA GPU architecture and the CUDA ecosystem.
- Experience writing Reproducers for kernel bugs in C.
- Knowledge of PyTorch distributed training flows.
Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $153,000/year in our lowest geographic market up to $242,000/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience.
What We Offer
The range we’ve posted represents the typical compensation range for this role. To determine actual compensation, we review the market rate for each candidate which can include a variety of factors. These include qualifications, experience, interview performance, and location.
In addition to a competitive salary, we offer a variety of benefits to support your needs, including:
- Medical, dental, and vision insurance - 100% paid for by CoreWeave
- Company-paid Life Insurance
- Voluntary supplemental life insurance
- Short and long-term disability insurance
- Flexible Spending Account
- Health Savings Account
- Tuition Reimbursement
- Ability to Participate in Employee Stock Purchase Program (ESPP)
- Mental Wellness Benefits through Spring Health
- Family-Forming support provided by Carrot
- Paid Parental Leave
- Flexible, full-service childcare support with Kinside
- 401(k) with a generous employer match
- Flexible PTO
- Catered lunch each day in our office and data center locations
- A casual work environment
- A work culture focused on innovative disruption
Our Workplace
While we prioritize a hybrid work environment, remote work may be considered for candidates located more than 30 miles from an office, based on role requirements for specialized skill sets. New hires will be invited to attend onboarding at one of our hubs within their first month. Teams also gather quarterly to support collaboration
California Consumer Privacy Act - California applicants only
CoreWeave is an equal opportunity employer, committed to fostering an inclusive and supportive workplace. All qualified applicants and candidates will receive consideration for employment without regard to race, color, religion, sex, disability, age, sexual orientation, gender identity, national origin, veteran status, or genetic information.
As part of this commitment and consistent with the Americans with Disabilities Act (ADA), CoreWeave will ensure that qualified applicants and candidates with disabilities are provided reasonable accommodations for the hiring process, unless such accommodation would cause an undue hardship. If reasonable accommodation is needed, please contact: [email protected].
Export Control Compliance
This position requires access to export controlled information. To conform to U.S. Government export regulations applicable to that information, applicant must either be (A) a U.S. person, defined as a (i) U.S. citizen or national, (ii) U.S. lawful permanent resident (green card holder), (iii) refugee under 8 U.S.C. § 1157, or (iv) asylee under 8 U.S.C. § 1158, (B) eligible to access the export controlled information without a required export authorization, or (C) eligible and reasonably likely to obtain the required export authorization from the applicable U.S. government agency. CoreWeave may, for legitimate business reasons, decline to pursue any export licensing process.
Top Skills
CoreWeave Sunnyvale, California, USA Office
CoreWeave Sunnyvale, CA Office
Sunnyvale, California, United States
Similar Jobs at CoreWeave
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
