Principal Systems Administrator

ASRC Federal is searching for a Staff HPC Engineer to support InuTeq LLC out of NASA AMES, CA

ASRC Federal Data Solutions (AFDS) supports NASA's High Performance Computing Services program at the Ames Research Center in Mountain View, CA. Make a DIFFERENCE on a program that supports 4 on-site supercomputers, 18,000+ nodes, and 17+ combined petaflop supercomputer systems. Our program provides High Performance Computing services throughout the HPC lifecycle, including computational requirements, architecture, acquisition, and operations, for federal government customers. Additionally, our team supports the Human-Systems Integration (HSI) Division, which enables the development of complex aerospace systems through analysis and modeling of human performance and human-automation interactions.

Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement and assimilation of industry’s best practices. We have an immediate position for a Sr. Systems Administrator to join our Infrastructure team in Mountain View. This role supports the hundreds of infrastructure systems that support HPC operations.

Position Summary: The successful candidate will be an active supporting member of the ASRC Federal team reporting directly to the Manager of the HPC Computer Systems and Storage (CSS) group.

An individual at this skill level should have demonstrated Linux administration experience as well as working knowledge of HPC operations while contributing to the support of users of HPC resources on the various issues they might have getting jobs to run efficiently. This individual is expected to have deep knowledge of InfiniBand (HDR/EDR) network fabrics. This individual will be expected to participate in all aspects of HPC System Administration that includes such activities as installing, maintaining, and upgrading HPC systems. The individual, along with the entire HPC team, will be engaged in the day-to-day operations and support of the HPC resources. Activities may include system patching, OS upgrades, deploying new systems, writing scripts, and troubleshooting system issues on the HPC system. The ability to interact with users to determine symptoms and then reproduce their issues to isolate the causes is critical skills for this work. There may also be activities supporting testing, benchmarking, tool scripting, and analyzing trouble tickets to find patterns indicating system or user education issues.

Duties and Responsibilities:

• Maintains HPC clusters with over 8,000 nodes with InfiniBand, 100+ petabytes of data storage in production.
• InfiniBand fabric administration and operations
  • Administer and maintain EDR/HDR InfiniBand fabrics, including switches, Host Channel Adapters (HCAs), cables, transceivers, and associated components.
  • Monitor fabric health, link status, port errors, link speed, link width, congestion, and switch performance.
  • Manage Subnet Manager (SM) services, including OpenSM or vendor-provided fabric management solutions.
  • Maintain fabric topology, routing, addressing, partitioning, and configuration consistency.
  • Identify and resolve failed links, degraded ports, disconnected nodes, and fabric initialization issues
• Performance monitoring and optimization
  • Establish performance baselines for bandwidth, latency, throughput, and error rates.
  • Diagnose packet drops, symbol errors, link recovery events, congestion, and performance degradation.
  • Conduct bandwidth and latency testing using tools such as (I don’t know)
  • Optimize fabric routing, topology, and configuration to support MPI applications, parallel filesystems, and distributed HPC workloads.
• Linux, HCA, and software integration
  • Install, configure, and maintain InfiniBand drivers, firmware, OFED or inbox RDMA software stacks, and associated libraries.
  • Validate HCA compatibility with operating systems, kernels, switch firmware, and application software.
  • Troubleshoot RDMA, verbs, MPI, and connectivity issues between compute nodes and storage platforms.
  • Support integration with job schedulers, parallel filesystems, and HPC cluster provisioning tools.
  • Automate health checks, configuration validation, and routine maintenance using Bash or Python and, where appropriate, Ansible.
• Assists in developing and writing proposals and publications.
• Creates and provides clear documentation.
• After hours/weekend support as required
• Moderate Supercomputing System Administration that contributes to:
  • Day-to-day operations of the Linux HPC clusters and storage systems
  • Proactive monitoring, analyzing, and correct system issues
  • Development of scripts to automate repetitive tasks or tools to enhance support of the HPC systems
  • System performance analysis and tuning
  • Building, installing, and supporting user-requested software
  • Supporting evaluation and assessment of new HPC technology
  • Resolving user report issues and managing support tickets requests in Remedy

• Bachelor’s degree in computer science or related field
• Strong computer science background with in-depth systems-level knowledge in operating systems and networking
• A minimum of 7 years’ experience of administration of Linux systems
• Strong ability to analyze and troubleshoot network issue with a  strong understanding of HDR/EDR link speeds, HCAs, switch ports, LIDs, GUIDs, GIDs, subnet management, and fabric topology
• Strong ability to analyze, debug and maintain the integrity of an existing code base
• Demonstrated equivalence of 7+ years of Linux/UNIX user support experience and hands-on experience with administration of Linux systems
• Superior scripting skills and excellent attention to detail; proficiency in at least Python, Perl, or Bash
• Strong ability to interact with customers to understand needs, elicit requirements, and get feedback on prototype solutions
• Excellent communication and people skills; excellent time management and organizational skills
• Experience with system configuration management tools e.g. , ansible, puppet, chef
• Experience with revision control software e.g. CVS, SVN, Git
• Proficiency at technical writing

Preferred Qualifications:

• Proficiency with analysis and problem-solving skills for debugging and optimization of applications
• Familiarity/proficiency with OpenMP and Message Passing Interface (MPI) programming
• Experience with Lustre
• Experience with cloud technologies (AWS, Azure, GCP), OpenStack or Kubernetes is a plus