Monitors and supports a 24x7 high-performance computing data center, triages system and facility alerts, develops monitoring and automation tools, maintains reliable software configurations, and coordinates maintenance and incident response. The role requires Linux command-line expertise, programming and scripting skills, network security knowledge, cross-team collaboration, and onsite physical and logical data center inspections. Familiarity with ServiceNow, ITSM, Kubernetes, observability platforms, and agentic AI automation is valuable.
The National Energy Research Scientific Computing Center (NERSC) is inviting applications for the position of Site Reliability Engineer. NERSC’s mission is to accelerate scientific discovery through high performance computing and data analysis for the DOE Office of Science programs. NERSC provides critical HPC and data systems and support for NERSC’s 11,000 plus users researching energy, physics, materials science, and chemistry and other DOE mission areas. As a Site Reliability Engineer in the Operations Technology Group, you will be a member of a 24x7 team that helps ensure NERSC is accessible, reliable, and secure for our scientific users. This position leverages advanced data collection and monitoring systems to proactively manage the health of our environment. Ultimately, your work ensures that NERSC’s computational power remains an uninterrupted catalyst supporting fundamental scientific research related to energy.
ESSENTIAL DUTIES & RESPONSIBILITIESDescribe the duties/functions essential to performing the job.
- Works an onsite 5-day weekly schedule consisting of Owl (midnight–8 am) shifts to monitor the NERSC HPC Facility.
- Review and respond to alerts from computer systems, storage, network, and other data center/facility-related systems by triaging or calling the appropriate on-call staff.
- Create appropriate solutions to improve processes, prevent issue recurrence, and automate responses to all routine service conditions.
- Identify issues and propose solutions that will improve monitoring capabilities or provide better automation for triage.
- Possess expertise in ServiceNow and its usage to develop and implement customized service management solutions.
- Respond to alerts from multiple systems to ensure that data collection continues 24/7, providing real-time information for diagnoses.
- Develop and maintain tools within the monitoring pipeline in collaboration with the Operations Team.
- Create new software to provide alerts and notifications from HPC system APIs into the monitoring pipeline.
- Builds and maintains application/tool configurations to ensure software runs reliably as data and user demands grow.
- Collaborate with other groups at NERSC to ensure that communication and workflows are clearly understood.
- Work closely with other NERSC groups to coordinate center-wide maintenance activities and manage diagnostic and notification software during maintenance periods.
- Perform regular physical and logical walkthroughs of the data center floor to monitor environmental health, power distribution units, and cooling infrastructure to ensure peak operational efficiency.
- Provide accurate information in the trouble ticketing system for outages, maintenance updates, and other incidents so that workflows and protocols can be appropriately tracked by others.
- Work on and resolve problems of diverse scope where data analysis requires the evaluation of identifiable factors.
- Demonstrate good judgment in selecting methods and techniques for obtaining solutions.
- Work on and resolve complex issues where the analysis of situations or data requires an in-depth evaluation of variable factors.
- Experience in or willingness to work within a 24/7 onsite team environment to support large-scale data centers or critical installations.
- Experience on Linux shell and working in a command-line (e.g. SSH) environment.
- Experience with developing tools using various programming languages such as C, C++, Perl, Java, or Python or a scripting language with knowledge of standard software development practices.
- Motivated, self-starter who can learn technologies that improve data center management in areas like Kubernetes, Prometheus/VictoriaMetrics, Alertmanager, building management software, evaporative
cooling, and power utilization.
- Experience with network security: configuring/maintaining ACLs, knowledge of firewalls
- Experience collaborating across technical teams to resolve operational bottlenecks and ensure system reliability and alignment with service-level objectives.
- Good to Have : Practical experience in developing and deploying Agentic AI or autonomous automation tools to streamline technical tasks.
- Experience with ServiceNow implementation is a plus
- Familiarity with ITSM best practices and an understanding of how to align service lifecycles with business goals is preferred.
Knowledge, Skills & Abilities
- Strong hands-on knowledge of the Linux shell and working in a command-line (e.g. SSH) environment.
- Strong hands-on knowledge on various programming languages such as C, C++, Perl, Java, or Python or a scripting language with knowledge of standard software development practices.
- Knowledge of and ability to work on large data communications networks/ Network Protocols and IT infrastructure supporting highly available systems and applications.
- Strong communication skills and ability to work effectively across multiple technical teams.
- Good to Have : Ability to build, and deploy Agentic AI solutions that utilize autonomous agents to automate decision-making, optimize complex workflows, and enhance proactive system monitoring.
Similar Jobs
Financial Services
Leads site reliability engineering for complex application and platform environments. Defines reliability requirements, observability strategies, service-level objectives, roadmaps, and production-readiness practices. Mentors engineers, debugs distributed systems, advances AI-assisted reliability workflows, and champions resilient, secure, auditable operations. Builds and oversees cloud-native infrastructure, APIs, event-driven systems, logging and monitoring pipelines, automation, and infrastructure-as-code solutions.
Top Skills:
Amazon SqsAutogenAWSAzureCassandraChaos MonkeyChromaCi/CdClaudeCrewaiDatadogDockerDynamoDBDynatraceFlinkFluentdGCPGithub CopilotGitopsGoGrafanaGremlinHadoopInfluxdbJavaKafkaKubernetesLangchainLanggraphLitmuschaosLogstashMcp ServersMongoDBNeo4JPineconePrometheusPythonPyTorchRabbitMQRestful ApisScikit-LearnSparkSplunkTensorFlowTerraformTigergraphTimescaledbVectorWeaviate
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Design, develop, deploy, monitor, and maintain AI-enabled enterprise and cloud security solutions. Build machine learning workflows, data pipelines, and scalable security capabilities using Python, Java, TensorFlow, and related libraries. Analyze security risks and data quality, translate technical findings into client recommendations, support security controls across cloud and infrastructure environments, review deliverables, and mentor junior team members.
Top Skills:
Artificial IntelligenceAWSCloud SecurityData PipelinesDatabricksEnterprise SecurityGCPJavaMachine LearningMachine Learning LibrariesAzurePythonSnowflakeTensorFlow
Cloud • Software
Lead the architecture, operation, and reliability of Cisco ThousandEyes’ FedRAMP-compliant AWS GovCloud platform. Build resilient cloud-native services using infrastructure as code, maintain high availability and security for high-volume data streams, automate compliance controls and auditing, and optimize multi-region infrastructure. Collaborate with engineering and security teams, lead technical initiatives, address operational risks, and establish engineering standards for regulated environments.
Top Skills:
AWSKubernetesPythonTerraform
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
