Systems Architect 2 - Data Streaming Platforms

Job Summary:

The Systems Architect II – Data Streaming Platforms is a senior technical leader responsible for designing and guiding the evolution of streaming and integration solutions across CSX.

This role serves as a design authority for data streaming platforms, defining architecture patterns, integration approaches, and system designs that enable reliable, scalable data movement across applications, platforms, and infrastructure.

The Systems Architect II leads the design of complex, cross-domain streaming solutions leveraging platforms such as Kafka, Palantir Foundry, and Microsoft Fabric Real-Time Intelligence, ensuring alignment between platform capabilities, engineering delivery, and business outcomes.

The role owns architectural direction, trade-off decisions, and total cost of ownership (TCO) for streaming and integration solutions while advancing AI-enabled patterns across streaming and integration workflows.

Primary Responsibilities:

Streaming Architecture & Design Leadership
Design and guide end-to-end data streaming architectures across applications, platforms, and infrastructure

• Define and evolve event-driven, messaging, and integration patterns, including: 
  • Kafka-based architectures
  • Event pipelines into Palantir Foundry
  • Real-time ingestion and processing patterns within Microsoft Fabric Real-Time Intelligence
• Lead architectural decisions for complex, high-scale streaming use cases across multiple domains
• Ensure designs meet requirements for scalability, resiliency, performance, and security

Platform Alignment & Integration Strategy

• Align solution designs with data streaming platform capabilities and constraints across: 
  • Kafka (event backbone)
  • Palantir Foundry (data modeling and operational analytics)
  • Microsoft Fabric Real-Time Intelligence (real-time reporting and analytics)
• Guide integration between: 
  • Data producers and consumers
  • Streaming platforms and downstream systems
  • Operational and analytical platforms
• Guide consistent adoption of approved platform patterns across engineering teams

Standards, Patterns, and Reusability

• Define standard patterns for streaming integration, including: 
  • Topic design and partitioning strategies
  • Schema management and data contracts
  • Event design for cross-platform interoperability
• Establish reusable reference architectures and implementation patterns for engineering teams
• Promote platform-first design over one-off custom solutions

Infrastructure as Code & Platform Automation

• Define and standardize Infrastructure as Code patterns for provisioning and managing:
  • Streaming infrastructure (Kafka clusters, topics, and supporting services)
  • Platform integrations with Palantir Foundry and Microsoft Fabric
• Guide the use of Terraform and ArgoCD (or equivalent tooling) for consistent, declarative, and repeatable deployments
• Ensure infrastructure is: 
  • Version-controlled (source of truth in Git)
  • Environment-consistent (dev/test/prod)
  • Auditable and reproducible
• Promote automated environment provisioning over manual configuration

CI/CD & Deployment Architecture

• Define CI/CD and deployment patterns for streaming and platform solutions, including:
  • Release orchestration, validation, and rollback
  • Code deployment (applications, stream processing logic)
  • Infrastructure deployment (IaC pipelines managed via Git workflows)
  • Configuration and schema promotion across environments
• Establish standards for: 
  • Build, test, and release automation
  • Environment promotion strategies
  • Release governance, validation, and rollback strategies
• Ensure integration of CI/CD pipelines with platform workflows across: 
  • Kafka ecosystems
  • Foundry data pipelines
  • Fabric real-time analytics components
• Drive adoption of automated testing, validation, and continuous delivery practices

AI Architecture & Team Enablement

• Define and promote AI-enabled patterns across data streaming platforms that create measurable value in architecture and operations, including:
  • Real-time data enrichment and intelligent processing of event streams
  • Integration of streaming data with AI/ML and agent-based workflows
  • Event-driven triggers for AI-driven decisioning and automation
• Guide integration of streaming platforms with AI capabilities within Foundry and Fabric, establishing scalable, reusable, and cost-conscious architecture patterns
• Identify opportunities to leverage AI to improve data quality, anomaly detection, operational automation, and real-time decision support
• Enable engineering teams to apply AI effectively in daily work through practical guidance in two areas:
  • AI in the architecture — designing systems that are AI-ready, extensible, and aligned to performance, cost, and operational constraints
  • AI in the team's practices — applying AI-assisted development, analysis, and operational tooling in practical, responsible ways

Financial & Operational Accountability

• Own architectural trade-offs including: 
  • Performance vs cost
  • Build vs buy
  • Platform utilization efficiency
• Evaluate run cost, compute usage, and platform economics, including impacts from streaming volume, processing patterns, and AI usage 
• Ensure solutions are operationally sustainable and cost-effective

Technical Leadership & Influence

• Provide architectural guidance to engineers, platform teams, and product partners
• Lead design and architecture reviews for streaming and integration solutions
• Mentor engineers in event-driven architecture, platform usage (Kafka, Foundry, Fabric), Infrastructure as Code, CI/CD, and practical use of AI tools in engineering and system design
• Influence direction through technical expertise and clarity of design

Minimum Qualifications

• Bachelor’s degree from an accredited institution required in Computer Science, Information Systems/Technology or related major field of study.
• 7 or more years of experience required in engineering/administration.

Equivalent Minimum Qualifications

• High School diploma / GED.
• 9 or more years of experience in Information Systems.

Preferred Qualifications

• Master's degree from an accredited institution required in Computer Science, Information Technology, or related major field of study.
• 5 or more years of experience in systems engineering/administration.
• 10 or more years of experience in Transportation/Intermodal Operations, Information Systems, or Logistics

Knowledge and Skills

• Advanced knowledge of event-driven architecture, data streaming patterns, message broker design, and real-time integration concepts.
• Strong understanding of Kafka-based architectures, including topic design, partitioning strategies, schema management, producers, consumers, and operational considerations.
• Experience designing integrations across enterprise platforms, including Palantir Foundry, Microsoft Fabric Real-Time Intelligence, cloud services, application systems, and analytical environments.
• Knowledge of Infrastructure as Code, GitOps, CI/CD, and automated deployment practices using tools such as Terraform, ArgoCD, Azure DevOps, GitHub, or equivalent technologies.
• Ability to design scalable, secure, resilient, and cost-effective streaming solutions aligned to enterprise architecture standards and operational requirements.
• Working knowledge of AI-enabled development, intelligent automation, real-time enrichment, anomaly detection, and AI/ML integration patterns within streaming and data platforms.
• Strong analytical and architectural decision-making skills, including the ability to evaluate trade-offs across performance, cost, complexity, reliability, and time to delivery.