Red Arch Solutions

Intelligence systems and software engineering solutions

Senior Apache NiFi/Data Integration Engineer

Data EngineeringConfirmed live in the last 24 hours

Full-timeReston, VA, USAIn-person

About the Job

Requirements

At least 12 years of experience working with users to define database use, data analytics, and the data user experience.
At least 12 years of data integration, extract-transform-load, or enterprise middleware experience.
At least 8 years of hands-on Apache NiFi development experience.
Experience designing and supporting large-scale enterprise data ingestion architectures.
Experience onboarding new data sources and integrating complex systems.
Demonstrated expertise in NiFi flow design, process groups, remote process groups, site-to-site communications, controller services, parameter contexts, flow versioning, NiFi Registry, clustered deployments, provenance tracking, back-pressure management, flow performance tuning, and secure flow design.
Experience developing Java-based NiFi processors, controller services, reporting tasks, custom APIs, schema validation components, metadata extraction processors, content parsing processors, security enforcement processors, and data quality validation processors.
Strong experience with Java, Spring Framework, JSON, XML, Avro, Parquet, REST APIs, SQL, PostgreSQL, Elasticsearch, Kafka, Git, Docker, Kubernetes, and Linux.
Understanding of Zero Trust Architecture, data-centric security, public key infrastructure, Transport Layer Security, identity and access management, data governance, metadata management, and enterprise architecture principles.
Active TS/SCI clearance is required.

Responsibilities

Design, develop, test, deploy, and maintain complex Apache NiFi data flows supporting enterprise data onboarding and integration requirements.
Engineer scalable and fault-tolerant ingestion pipelines capable of processing structured, semi-structured, and unstructured data.
Develop reusable flow frameworks and integration patterns to accelerate onboarding activities.
Optimize flow performance, throughput, latency, and resource utilization across clustered environments.
Troubleshoot and resolve complex data movement, transformation, and processing issues.
Develop custom Apache NiFi processors, controller services, reporting tasks, and extensions using Java.
Create specialized processors for metadata extraction, schema validation, data tagging, attribute enrichment, security classification enforcement, data quality validation, API integrations, and workflow automation.
Extend NiFi capabilities to support mission-specific onboarding requirements.
Develop shared processor libraries and reusable components.
Integrate enterprise systems using REST APIs, SOAP services, JDBC/ODBC connections, message queues, Kafka, file-based transfers, and cloud storage services.
Design data ingestion and routing strategies across multiple environments.
Support onboarding of mission and enterprise data assets into approved repositories and data services.
Implement metadata collection, tagging, and enrichment capabilities within data onboarding workflows.
Support enterprise data catalog and governance initiatives.
Ensure onboarding workflows comply with metadata standards, taxonomy requirements, and stewardship processes.
Collaborate with Data Stewards and Data Owners to improve data quality and discoverability.
Implement secure data movement patterns aligned with Zero Trust principles.
Integrate authentication, authorization, encryption, and auditing capabilities into data pipelines.
Coordinate with cybersecurity and Zero Trust teams to ensure compliance with enterprise security requirements.
Support Attribute-Based Access Control and policy-driven data access models.

Desired Qualifications

Experience supporting DIA, NSA, NGA, or other Intelligence Community customers.
Experience integrating with data catalog platforms and metadata repositories.
Experience with ontology, taxonomy, and semantic tagging implementations.
Familiarity with Cloudera DataFlow, Hortonworks DataFlow, or large-scale NiFi deployments.
Experience developing automated onboarding workflows.
Knowledge of enterprise data governance and stewardship processes.
Experience operating in highly regulated or classified environments.

Apache NiFi Engineer / Data Integration Engineer

Position Overview
Red Arch Solutions is seeking a highly skilled Apache NiFi Engineer to support enterprise data onboarding, integration, and transformation initiatives within a Zero Trust environment. The successful candidate will design, develop, deploy, and maintain complex Apache NiFi data flows that enable secure, scalable, and reliable movement of mission and enterprise data across disparate systems.
This role requires deep expertise in Apache NiFi architecture, custom processor development, enterprise integration patterns, metadata management, data transformation, and workflow automation. The engineer will work closely with Data Stewards, Enterprise Architects, Zero Trust Core Services teams, Data Catalog personnel, system owners, and mission stakeholders to onboard new data sources and ensure compliance with enterprise data standards and security requirements.

KEY RESPONSIBILITIES
Apache NiFi Development & Engineering

Design, develop, test, deploy, and maintain complex Apache NiFi data flows supporting enterprise data onboarding and integration requirements.
Engineer scalable and fault-tolerant ingestion pipelines capable of processing structured, semi-structured, and unstructured data.
Develop reusable flow frameworks and integration patterns to accelerate onboarding activities.
Optimize flow performance, throughput, latency, and resource utilization across clustered environments.
Troubleshoot and resolve complex data movement, transformation, and processing issues.

Advanced Processor Development

Develop custom Apache NiFi processors, controller services, reporting tasks, and extensions using Java.
Create specialized processors supporting:
- Metadata extraction
- Schema validation
- Data tagging
- Attribute enrichment
- Security classification enforcement
- Data quality validation
- API integrations
- Workflow automation
Extend NiFi capabilities to support mission-specific onboarding requirements.
Develop shared processor libraries and reusable components.

Enterprise Data Integration

Integrate enterprise systems using:
- REST APIs
- SOAP Services
- JDBC/ODBC Connections
- Message Queues
- Kafka
- File-Based Transfers
- Cloud Storage Services
Design data ingestion and routing strategies across multiple environments.
Support onboarding of mission and enterprise data assets into approved repositories and data services.

Metadata & Data Governance

Implement metadata collection, tagging, and enrichment capabilities within data onboarding workflows.
Support enterprise data catalog and governance initiatives.
Ensure onboarding workflows comply with metadata standards, taxonomy requirements, and stewardship processes.
Collaborate with Data Stewards and Data Owners to improve data quality and discoverability.

Zero Trust & Security Integration

Implement secure data movement patterns aligned with Zero Trust principles.
Integrate authentication, authorization, encryption, and auditing capabilities into data pipelines.
Coordinate with cybersecurity and Zero Trust teams to ensure compliance with enterprise security requirements.
Support Attribute-Based Access Control (ABAC) and policy-driven data access models.

REQUIRED QUALIFICATIONS
Experience

12+ years experience. Experience working with users to help define their database use, data analytics, and the data user experience.
12+ years of data integration, ETL, or enterprise middleware experience.
8+ years of hands-on Apache NiFi development experience.
Experience designing and supporting large-scale enterprise data ingestion architectures.
Experience onboarding new data sources and integrating complex systems.

Advanced Apache NiFi Skills
Demonstrated expertise with:

NiFi Flow Design
Process Groups
Remote Process Groups
Site-to-Site Communications
Controller Services
Parameter Contexts
Flow Versioning
NiFi Registry
Clustered Deployments
Provenance Tracking
Back-Pressure Management
Flow Performance Tuning
Secure Flow Design

Exceptional Processor Development Skills
Experience developing custom:

Java-based NiFi Processors
Controller Services
Reporting Tasks
Custom APIs
Schema Validation Components
Metadata Extraction Processors
Content Parsing Processors
Security Enforcement Processors
Data Quality Validation Processors

Technical Expertise
Strong experience with:

Java
Spring Framework
JSON
XML
Avro
Parquet
REST APIs
SQL
PostgreSQL
Elasticsearch
Kafka
Git
Docker
Kubernetes
Linux

Security & Architecture
Understanding of:

Zero Trust Architecture
Data-Centric Security
PKI
TLS
Identity and Access Management
Data Governance
Metadata Management
Enterprise Architecture Principles

Clearance

Active TS/SCI clearance required.

Education (Desired)

Bachelor's degree in Computer Science, Engineering, Information Systems, or related technical discipline.

DESIRED QUALIFICATIONS

Experience supporting DIA, NSA, NGA, or other Intelligence Community customers.
Experience integrating with Data Catalog platforms and metadata repositories.
Experience with ontology, taxonomy, and semantic tagging implementations.
Familiarity with Cloudera DataFlow, Hortonworks DataFlow, or large-scale NiFi deployments.
Experience developing automated onboarding workflows.
Knowledge of enterprise data governance and stewardship processes.
Experience operating in highly regulated or classified environments.

Ideal Candidate
The ideal candidate is not simply a NiFi user, but a NiFi architect and developer who can build custom processors from scratch, solve complex integration challenges, optimize enterprise-scale data flows, and lead onboarding efforts across multiple systems and organizations. This individual should be the technical expert that other engineers rely upon for advanced flow design, custom processor development, and enterprise data integration solution

Salary Range for this opening $185-215K #CJ

The Red Arch Solutions pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Determination of official compensation or salary relies on several factors including, but not limited to, level of position, job responsibilities, geographic location, scope of relevant work experience, educational background, certifications, contract-specific affordability, organizational requirements, alignment with local internal equity as well as alignment with market data.

Red Arch Solutions Benefits Snapshot:

100% paid employee healthcare premiums. CareFirst Advantage Best in Class Benefits.
HaloScripts Concierge prescription medication service
Generous PTO.
10 % 401K 6% match vested day one and up to 4% profit sharing contribution.
Annualized bonus compensation.
Spot bonuses for hard work.
Tuition reimbursement.
529 College Saving Plan.
College Loan Pay Back Program.
11 Paid Federal Holidays
Generous Referral Bonuses.

Red Arch Solutions provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

Have more questions about Red Arch? Email us at [email protected]

About the Company

Red Arch Solutions

Intelligence systems and software engineering solutions

Red Arch Solutions provides specialized intelligence and engineering services for the U.S. national security community. It offers state-of-the-art tactical and strategic intelligence, systems, and software engineering solutions to address unique challenges faced by the intelligence community. Its product approach combines highly skilled personnel with analytical and engineering capabilities to collect, analyze, create, and engineer solutions that support critical missions. The company differentiates itself through a strong people-first culture, emphasizing career mobility, work-life balance, and individualized career paths, aiming to retain highly trained professionals as its core asset. The goal is to deliver value-added, synergistic support to customers by leveraging deep expertise across the intelligence community while maintaining commitment to equal opportunity and affirmative action.

Company size: 11-50
Company stage: Not listed
Total funding: Not listed
Headquarters: Columbia, SC
Founded: 2004

Recently Posted Jobs

Explore more opportunities at Red Arch Solutions

See more jobs