Provide overnight operational coverage for large-scale GPU clusters: monitor health and performance, triage incidents, support researchers, run recovery procedures, validate deployments and upgrades, track utilization, and build automation and monitoring tools.
About MBZUAI
The Institute for Foundation Models (IFM) operates some of the world's largest AI supercomputing environments.
Position Summary
This role provides operational coverage during Abu Dhabi overnight hours and serves as a primary point of contact for infrastructure monitoring, incident triage, researcher support, and production operations.
The Institute for Foundation Models (IFM) operates some of the world's largest AI supercomputing environments.
Position Summary
This role provides operational coverage during Abu Dhabi overnight hours and serves as a primary point of contact for infrastructure monitoring, incident triage, researcher support, and production operations.
Responsibilities
• Monitor health, performance, and availability of large-scale GPU clusters.
• Respond to incidents and perform first-level triage.
• Support researchers and troubleshoot job failures.
• Execute operational runbooks and recovery procedures.
• Validate cluster deployments, upgrades, and maintenance activities.
• Track infrastructure utilization and operational metrics.
• Develop automation and monitoring tools.
• Contribute to documentation and reporting.
• Respond to incidents and perform first-level triage.
• Support researchers and troubleshoot job failures.
• Execute operational runbooks and recovery procedures.
• Validate cluster deployments, upgrades, and maintenance activities.
• Track infrastructure utilization and operational metrics.
• Develop automation and monitoring tools.
• Contribute to documentation and reporting.
Education
Bachelor's degree in Computer Science, Computer Engineering, Software Engineering, Information Technology, Electrical Engineering, Mathematics, Physics, or related disciplines.
Experience
• 2+ years in Linux systems administration, SRE, DevOps, cloud operations, HPC, or infrastructure operations.
• Strong Linux troubleshooting skills.
• Experience with scripting using Python or Bash.
• Strong Linux troubleshooting skills.
• Experience with scripting using Python or Bash.
Preferred Qualifications
• Slurm.
• GPU infrastructure.
• AWS, Azure, or GCP.
• Grafana, Prometheus, Datadog, or similar tools.
• Containers and Kubernetes.
• AI/ML infrastructure exposure.
• Research computing environments.
• GPU infrastructure.
• AWS, Azure, or GCP.
• Grafana, Prometheus, Datadog, or similar tools.
• Containers and Kubernetes.
• AI/ML infrastructure exposure.
• Research computing environments.
Compensation
$150,000 – $300,000/year + discretionary performance bonus + benefits
The posted salary range represents the company’s good faith estimate of the base compensation for this position upon hire. The actual base compensation offered may vary within this range depending on individual qualifications, including but not limited to relevant skills, experience, education, certifications, and specific business needs. In addition to base salary, total compensation includes an annual discretionary performance bonus and comprehensive benefits.
Benefits Include
- Comprehensive medical, dental, and vision benefits
- Bonus
- 401K Plan
- Generous paid time off, sick leave and holidays
- Paid Parental Leave
- Employee Assistance Program
- Life insurance and disability
Similar Jobs
Artificial Intelligence • Computer Vision • Hardware • Robotics • Metaverse
Designs and scales observability and telemetry platforms for distributed AI and HPC environments. Builds high-performance backend services, OpenTelemetry components, metrics pipelines, and real-time or batch data systems. Improves reliability, performance, capacity, and cost efficiency while developing monitoring, alerting, and service reliability frameworks. Requires production distributed-systems expertise, programming skills, time-series data experience, Kubernetes knowledge, and operational debugging capabilities.
Top Skills:
FlinkGoJavaKafkaKubernetesOpentelemetryPrometheusPromqlPythonSpark
Hardware • Semiconductor • Manufacturing
Develop and optimize HPC software for computationally intensive workloads across CPU and GPU architectures. Responsibilities include parallel algorithm development, performance profiling, memory and scalability optimization, GPU enablement, compiler and systems-level improvements, and support for scientific applications such as physics, fluid dynamics, and molecular dynamics. The role also contributes to architecture, technical decisions, code reviews, research, feature development, and collaboration across distributed engineering teams.
Top Skills:
Assembly LanguageAvxCC++CmakeCpu ArchitecturesCudaGpu ArchitecturesJuliaLinuxLlvmOnetbbOpenmpPythonSimdSimtSveTbb
Hardware • Manufacturing
Design, develop, and optimize HPC software using NVIDIA GPUs, AMD CPUs, MPI, CUDA, and OpenMP. Scale and manage distributed HPC clusters, optimize parallel algorithms, profile applications, and benchmark performance across multi-node environments. Collaborate with scientists, engineers, and IT teams to deliver scalable research and production systems while documenting configurations, processes, and performance improvements.
Top Skills:
Amd CpusAmd UprofAnsibleCC++CudaCudnnDockerIntel VtuneMpiNcclNvidia GpusNvidia NsightOpenmpPackerPbsPyscfPythonPyTorchSingularitySlurmTensorFlowTensorrtTerraform
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
