Digital LAB Expert

Description of Professional Duties for BR&TE

Laboratory Administrator

Introduction

The BR&TE Laboratory (or “lab”) consists of a server center located at the BR&TE premises in Madrid and designed to support project activities that involve collaboration between BR&TE, contractors, and other Boeing sites. In addition, the lab has been designed to support R&D activities that require a testing environment for software and hardware devices that do not comply with Boeing standards. The main features of the BR&TE Laboratory are the following:

• Support for approximately 60 Boeing users and 200 external users
• Remote access to the lab via internet
• Access policy and network segmentation to restrict access to lab services
• Non-standard HW/SW components (though adhering to Boeing standards where possible)
• Physical isolation from the Boeing Enterprise Network
• Multi-factor authentication for all lab users
• Alarm monitoring 5 days per week and 10 hours per day locally within the BR&TE site
• Operating hours from 9 AM to 7 PM CET, Monday through Friday (with the exception of official holidays in Madrid, Spain) with off shift support required on special occasions
• Normally available outside of operating hours but without operational support
• System maintenance performed on a scheduled basis, normally off shift
• Application maintenance performed on a scheduled basis
• Tolerance for downtime in case of disaster is that the system will be unavailable until new hardware, network and facility components are acquired
• System back-ups of production and infrastructure systems taken regularly and stored off-site

The lab administrator is responsible for ensuring that the BR&TE Laboratory is functioning as described. In the sequel we shall describe in greater detail specific duties and tasks required of the lab administrator.

Maintenance of Core Computing Infrastructure

By core computing infrastructure, we mean:

• Racked storage of computing equipment
• Power and HVAC units
• Layer-2 switching infrastructure
• Wireless networks
• Layer-7 firewalls
• Core layer-7 services such as DNS, NTP, syslog, LDAP, and authentication services
• Administrative tasks include:
• Racking devices in data room
• Network design, segmentation, VLAN distribution, cabling, etc.
• Configuring individual devices and services
• Ensuring the clear documentation is available
• Monitoring all services for interruptions or anomolous behavior
• Upgrading and patching software as necessary
• Debugging anomolous behavior
• Purchasing new equipment

Major technologies include: FortiGate, FortiAuthenticator, PaloAlto PanOS, Cisco Nexus optical switches, Cisco Catalyst switches, bind9, ntpd, rsyslog, Microsoft ActiveDirectory, Kerberos, RADIUS, APC switched PDUs, Liebert HPM, Liebert NXC 30, nagios, Cisco C9136I-E, Cisco C9800 APC, and APC racks.

Remote Connectivity

Must maintain remote connectivity solutions for the various remote collaboration cases:

• VPN connection by lab users
• Site-to-site network integration with remote collaborators
• IoT connections
• Basic data-sharing with collaborating institutions

As novel use-cases arise, the lab administrator must be able devise new solutions that are well suited to the particular needs of each specific case. Many of these functionalities also entail extensive debugging and technical support sessions with remote users. As such, advanced knowledge of all relevant communications protocols is necessary.

Major technologies include: ipsec, esp, ike, cryptographic protocols, PA Global- Connect, FortiClient, sftp, ownCloud, etc.

Development of Administrative Software

In order to automate routing administrative tasks of monitoring, configuring, and reporting, we have developed an in-house administrative web application using Django. Besides the usual software maintenance tasks associated with any development effort, the lab administrator must:

• Identify lab tasks that are susceptible to automation and develop new functionalities using the Django framework.
• Update configuration code as controlled infrastructure changes.
• Add features as requested by auditors for better compliance with their respective guidelines.

Much of the software controls computing infrastructure within the lab, so that a prerequisite for development tasks is a solid knowledge of how those devices work. In addition, major technologies include: python3, graphviz, LaTex, selenium, mysqldb, freeradius, cron, apache2, html/css, scheme, gimp, etc.

Technical Support for Lab Users

The lab administrator must be available to handle any technical support requests from the lab users. To this end we have a ticket system where issues are communicated and resolved in a timely manner.

ISO Compliance

We are regularly audited for varying levels of compliance with several ISO standards:

• ISO 9001 Quality Management Systems
• ISO 27001 Information Security Management
• ISO 14644 Air Cleanliness in Controlled Environments

The lab administrator must be present at these audits, providing all requested information to the auditors. Typically these audits yield a task list of actions to be done, which the lab administrator must also perform within a reasonable timeframe.

Since these standards tend to steer the work environment towards a repertoire of documented procedures, they also influence tools such as our Django web application, formalizing frequently-perfomed tasks into automated programs.

Information Security

Though the BR&TE Laboratory is a development environment, we try to enforce as much security policy as possible. While IT security is a very complex series of practices, we can highlight a few common security tasks for the lab administrator:

• Vulnerability scans
• Patching and updating of all software
• Malware scanning
• Intrusion detection systems
• Layer-7 firewalls
• Encrypted/trusted communication protocols
• Development/testing of disaster recovery plans
• Software backups
• etc.

Major technologies include: PaloAlto tools, Fortinet tools, ossec, clamav, Forti- Client, FortiEMS, nagios, openssl, nexpose, etc.

Hypervisors

Virtualization is an important service offered by the BR&TE Laboratory, and the lab administrator must maintain the hypervisor infrastructure, tuning vhosts to maximize performance and reliability. Of note:

• All linux VMs are deployed using a cloud deployment system developed inhouse
• All vhosts are incrementally replicated using software developed in-house
• Major technologies: qemu, kvm, cuda, numa, nvme2, Dell IDRAC, Kubernetes, btrfs, etc.

Backup

We have custom backup technologies based on HDFS and BTRFS. The lab administrator must maintain these tools, define backup policy, and ensure that it is enforced. Since this is a big data environment, many complex issues arise.

X.509 Infrastructure

We an internally managed CA for internal domains, while Boeing and DigiCert sign CSRs for our external domains. The lab administrator has the following tasks:

• Generate certificates for all computers, users, etc.
• Renew expiring certificates
• Revoke certificates no longer in use
• Maintain internal CA
• etc.

For maintenance of the internal CA, knowledge of openssl and X.509 certificates in general is necessary.

ADSB Network

The lab administrator must maintain our custom ADSB network:

• Deploying antenna devices as necessary
• Monitoring remote antennas for connectivity
• Sharing incoming data with interested projects

Procurement

All equipment purchased for the BR&TE Laboratory must be approved by the lab administrator in order to ensure standards are met for storage, networking, power, etc. The lab administrator typically provides consultation for the purchase of hardware and must have knowledge about appropriate solutions for Cloudera, hypervisor, storage, and networking equipment.

Delegation and Human Resources

The lab administrator typically works with assistants and engineers from other projects, with whom the lab administrator must assign tasks, review results, etc.

Team leadership and communication skills are the main requisites here. For hiring lab assistants, must draft job description and participate in hiring process.

Research & Development

The BR&TE Laboratory is a research & development environment, and ultimately the lab administrator must participate in research & development projects within the context of adminstering the lab. Much of the backup technology, ADSB network, etc. currently in place comes from such initiatives. The ability to handle such projects with minimal supervision and provide stable solutions is necessary.

If you are a professional who wants to make a difference, you are just what we are looking for.



Technologies and Tools

• Hypervisors: qemu, kvm, cuda, numa, nvme2, Dell IDRAC, Kubernetes, btrfs, etc.
• Information Security: PaloAlto tools, Fortinet tools, ossec, clamav, Forti- Client, FortiEMS, nagios, openssl, nexpose, etc.
• Development of Administrative Software: python3, graphviz, LaTex, selenium, mysqldb, freeradius, cron, apache2, html/css, scheme, gimp, etc.
• Remote Connectivity: ipsec, esp, ike, cryptographic protocols, PA Global- Connect, FortiClient, sftp, ownCloud, etc.
• Maintenance of Core Computing Infrastructure: FortiGate, FortiAuthenticator, PaloAlto PanOS, Cisco Nexus optical switches, Cisco Catalyst switches, bind9, ntpd, rsyslog, Microsoft ActiveDirectory, Kerberos, RADIUS, APC switched PDUs, Liebert HPM, Liebert NXC 30, nagios, Cisco C9136I-E, Cisco C9800 APC, and APC racks.

Additional Valued Skills

• Expereince development of a computing cluster for studying commercial air traffic and optimizing routes

ISO standards

• ISO 9001 Quality Management Systems
• ISO 27001 Information Security Management
• ISO 14644 Air Cleanliness in Controlled Environments