Job Req ID:  115961

HPC Sr. Systems Engineer

The Advanced Research Computing at Hopkins (ARCH) group is seeking a highly qualified and motivated HPC Sr. Systems Engineer to join the systems team. This system (ROCKFISH), with over 45,000 cores and several petabytes of storage, serves the HPC and data intensive science needs of researchers at Johns Hopkins University. The Sr Systems Engineer contributes to the strategic planning, design, testing, organization and implementation of cutting-edge technology projects for the facility. The systems team is responsible for the day-to-day administration of HPC clusters, High Performance storage systems, backups, networking, security and any other services related to the operation of a large HPC center. The successful candidate will have experience in similar roles in high performance computing (HPC) labs or university settings.


Specific Duties & Responsibilities


70% Systems Engineering, Administration, Security, and Oversight

  • Design, organize, plan, test and implement cutting-edge hardware designs for an HPC environment.
  • Extensively document systems processes so that users can easily find useful information and other IT staff can perform routine tasks and provide backup.
  • Provides stable solutions for HPC resources.
  • Maintain job scheduling and storage allocation systems and policies to ensure fair allocation of shared resources.
  • Maintain extensive monitoring systems to facilitate quick, proactive responses to routine failures, and to provide comprehensive performance data logging.
  • Provide general system administration backup and escalation for other staff.
  • Consult with and provide expertise to building engineers and other staff on new facilities to be under the control of MARCC.
  • Assist with facilities-related issues that directly affect MARCC.
  • Ensure resources meet the community’s needs and are highly available to the group with limited interruption.
  • Manage inventory of resources in coordination with respective vendors.
  • Automate user account creation, management, and purging.
  • Contribute to planning sessions on network and security issues for MARCC. Work closely with the central networking group.
  • Implement network configuration and security measures to assure effective utilization of resources.
  • Understand HPC technical needs. Work closely with the facility’s director and oversight groups to successfully implement policies and procedures.
  • Create and maintain a stable, secure operating system and software environment, which continues to meet users’ evolving research needs.
  • Implement and maintain secure measures to protect data subject to restrictions.
  • Manage data access restrictions on a per user and group basis.
  • Implement and maintain monitoring measures for data and system access.
  • Other Systems Tasks as assigned by supervisor.


20% Technological Research

  • Architect future clusters and plan the retirement of aging systems.
  • Offer technical advice on new projects that directly involve HPC computing at Hopkins.
  • Develop custom tools where necessary and contribute useful creations back to open-source development efforts where appropriate.
  • Research and implement new technologies that could be beneficial to HPC.
  • Test and vet new technology in support of HPC efforts.
  • Work with vendors to procure prototypes and demo units.


10% Training/Education

  • Continuously evaluate new tools and technologies for use in existing and future clusters.
  • Attending department and University-sponsored training to increase knowledge, improve skills, and learn new skills. May substitute University training for supervisor approved commercial job-related course offerings.


Additional Knowledge, Skills, & Abilities

  • Proven experience deploying large-complex scale projects.
  • Proven experience across multiple technologies with background in applications, databases, middleware, etc.
  • In-depth knowledge of the design and organization of cutting-edge technology in HPC environments.
  • In-depth understanding of HPC Cluster hardware and management software.
  • In-depth understanding of massive high performance parallel storage and methodologies.
  • Understanding and implementation of IT project management best practices
  • Expert knowledge of Unix/Linux systems administration, including all aspects of management, monitoring, performance analysis, and integration in potentially complex heterogeneous environments.
  • Expert Level Knowledge of networking, high speed interconnects, and network security principles in an HPC environment.
  • Expert knowledge of security measures necessary to protect the facility and its data (firewalls, ACLs, network monitoring).
  • Use of configuration management tools (e.g. Bright, xCAT, puppet, IPMI, ROCKS) to help maintain large-scale Linux clusters, supercomputers, storage systems, and smaller systems.
  • The ability to interact with peer institutions to support HPC directives effectively, furthering the goals of the MARCC facility.
  • Understand, implement, troubleshoot, and support job scheduling, resource management and workload management systems, including diagnosis of failed jobs, implementation of policies, and investigations of new features and services.
  • Plan for and deployment of patches and updates to the operating system and application software.
  • Understand and support hierarchical file system infrastructure, software and services, including high performance parallel storage, backup systems, and robotic tape libraries.
  • Develop reports and customize tools that automate the monitoring process of critical systems and alert team of issues automatically.
  • Evaluate, implement and manage appropriate high level complex software and hardware solutions by using best practices for the environment to ensure system integrity.
  • Install and configure infrastructure applications by following industry best practices to deliver effective solutions.
  • Maintain an effective schedule for systems backups and archive operations for mission critical systems.
  • Audit and maintain user access, authorization and authentication.
  • Generate periodic reports on resource utilization.
  • Maintain resource inventory using best practice applications.
  • Advanced knowledge of Linux, Apache, SQL, PHP/Python/Perl (LAMP) technology/toolkits.
  • Ability to handle high priority escalations whenever necessary.
  • Ability to multitask, while managing time and priorities.
  • Troubleshoot and solve difficult system issues as they arise.
  • Must be adaptable and able to meet conflicting deadlines.
  • Exceptional organizational skills.
  • Maintain effective and thorough documentation of all configuration and tasks performed.
  • Ability to automate systems administration tasks wherever possible.
  • Excellent oral and written interpersonal skills.
  • Ability to meet the physical requirements of the position.
  • Keep up to date on emerging technologies.
  • Excellent vendor relations skills.
  • Research, recommend, and implement new technologies based on their value to the research facility.
  • Ability to maintain confidentiality.
  • Excellent customer service skills.
  • Excellent communication skills.
  • Must demonstrate strong critical thinking and analytical reasoning.


Internal and External Contacts

  • This position will interact with an array of departmental and central administrative offices, faculty, staff, researchers, and students, and with numerous external constituents (i.e. other college administrators and faculty, private businesses, industry partners, officials of federal and local agencies and research foundations) for the purpose of accomplishing HPC technology goals. This includes providing instruction on protocol, regulations and guidelines pertinent to the agency and/or University. Works routinely with JHU and UMCP faculty, administrators, students, and researchers. Collaborates regularly with professional colleagues from the central IT@JH organization, and from other academic departments. Collaborates regularly with colleagues in industry and at other peer institutions.


Minimum Qualifications
  • Bachelor's Degree.
  • Six years related experience (Direct programming experience while pursuing education may count towards related experience. Significant undergraduate programming coursework or independent project can be considered as related experience.)
  • Additional education may substitute for required experience and additional related experience may substitute for required education, to the extent permitted by the JHU equivalency formula.


Preferred Qualifications
  • Seven years managing Linux servers, with direct experience managing HPC clusters.
  • Experience as a high-level Linux system administrator.
  • Experience managing mission critical services.
  • Familiarity with configuration of the HPC software stack, including MPI, OpenMP, Intel, and GNU compilers, Math libraries.
  • Experience with open-source software compilation
  • In-depth knowledge of TCP/IP networking and related protocols, InfiniBand, etc.
  • Experience with scientific application management packages like pymodules, modules.
  • Excellent scripting skills, python, perl, shell.
  • Programming skills in C, C++, or scientific language, desired but not required.
  • Experience with MySQL or Mariadb database programming, desired but not required.
  • Expert level knowledge of configuration management and monitoring tools (puppet, nagios, etc).
  • Experience configuring resource manager applications (like SLURM)
  • Experience with Apache administration.
  • Knowledge of scientific software applications in academic supercomputing environments.
  • Familiarity or experience with data subject to restrictions, desired but not required.

 


 

Classified Title: Sr. Systems Engineer 
Job Posting Title (Working Title): HPC Sr. Systems Engineer   
Role/Level/Range: ATP/04/PF  
Starting Salary Range: $85,500 - $149,800 Annually (Commensurate with experience) 
Employee group: Full Time 
Schedule: 37.5 hrs/wk, M-F 
Exempt Status: Exempt 
Location: Hybrid/Homewood Campus 
Department name: ​​​​​​​DSAI Institute 
Personnel area: Whiting School of Engineering 

 

 


Total Rewards
The referenced base salary range represents the low and high end of Johns Hopkins University’s salary range for this position. Not all candidates will be eligible for the upper end of the salary range. Exact salary will ultimately depend on multiple factors, which may include the successful candidate's geographic location, skills, work experience, market conditions, education/training and other qualifications. Johns Hopkins offers a total rewards package that supports our employees' health, life, career and retirement. More information can be found here: https://hr.jhu.edu/benefits-worklife/.

Education and Experience Equivalency
Please refer to the job description above to see which forms of equivalency are permitted for this position. If permitted, equivalencies will follow these guidelines: JHU Equivalency Formula: 30 undergraduate degree credits (semester hours) or 18 graduate degree credits may substitute for one year of experience. Additional related experience may substitute for required education on the same basis. For jobs where equivalency is permitted, up to two years of non-related college course work may be applied towards the total minimum education/experience required for the respective job.

Applicants Completing Studies
Applicants who do not meet the posted requirements but are completing their final academic semester/quarter will be considered eligible for employment and may be asked to provide additional information confirming their academic completion date.

Background Checks
The successful candidate(s) for this position will be subject to a pre-employment background check. Johns Hopkins is committed to hiring individuals with a justice-involved background, consistent with applicable policies and current practice. A prior criminal history does not automatically preclude candidates from employment at Johns Hopkins University. In accordance with applicable law, the university will review, on an individual basis, the date of a candidate's conviction, the nature of the conviction and how the conviction relates to an essential job-related qualification or function.

Diversity and Inclusion
The Johns Hopkins University values diversity, equity and inclusion and advances these through our key strategic framework, the JHU Roadmap on Diversity and Inclusion.

Equal Opportunity Employer
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

EEO is the Law
https://www.eeoc.gov/sites/default/files/2023-06/22-088_EEOC_KnowYourRights6.12ScreenRdr.pdf

Accommodation Information
If you are interested in applying for employment with The Johns Hopkins University and require special assistance or accommodation during any part of the pre-employment process, please contact the Talent Acquisition Office at jhurecruitment@jhu.edu. For TTY users, call via Maryland Relay or dial 711. For more information about workplace accommodations or accessibility at Johns Hopkins University, please visit https://accessibility.jhu.edu/.

Vaccine Requirements
Johns Hopkins University strongly encourages, but no longer requires, at least one dose of the COVID-19 vaccine. The COVID-19 vaccine does not apply to positions located in the State of Florida. We still require all faculty, staff, and students to receive the seasonal flu vaccine. Exceptions to the COVID and flu vaccine requirements may be provided to individuals for religious beliefs or medical reasons. Requests for an exception must be submitted to the JHU vaccination registry. This change does not apply to the School of Medicine (SOM). SOM hires must be fully vaccinated with an FDA COVID-19 vaccination and provide proof of vaccination status. For additional information, applicants for SOM positions should visit https://www.hopkinsmedicine.org/coronavirus/covid-19-vaccine/ and all other JHU applicants should visit https://covidinfo.jhu.edu/health-safety/covid-vaccination-information/.

The following additional provisions may apply, depending upon campus. Your recruiter will advise accordingly.
The pre-employment physical for positions in clinical areas, laboratories, working with research subjects, or involving community contact requires documentation of immune status against Rubella (German measles), Rubeola (Measles), Mumps, Varicella (chickenpox), Hepatitis B and documentation of having received the Tdap (Tetanus, diphtheria, pertussis) vaccination. This may include documentation of having two (2) MMR vaccines; two (2) Varicella vaccines; or antibody status to these diseases from laboratory testing. Blood tests for immunities to these diseases are ordinarily included in the pre-employment physical exam except for those employees who provide results of blood tests or immunization documentation from their own health care providers. Any vaccinations required for these diseases will be given at no cost in our Occupational Health office.

Hybrid: On-site 1-2 days a week