C

Caterpillar

Makes heavy machinery, engines and gas turbines

Senior Data Platform & Streaming Architect

Data EngineeringConfirmed live in the last 24 hours
Full-timeNo salary listedIn-person

About the Job

Requirements

  • A bachelor's degree or higher in computer science, software engineering, data engineering, distributed systems, automation, or a related field is required.
  • At least 8 years of experience in data platforms, stream processing, distributed systems, or large-scale backend platform development, including architecture and delivery of production-grade real-time data systems.
  • Deep proficiency in Kafka, Pulsar, or similar distributed messaging systems, including partitions, replicas, consumer groups, message ordering, reliable delivery, and capacity planning.
  • Proficiency in Apache Flink, Spark Structured Streaming, or similar stream-processing technologies, including state management, windowing, checkpoints, fault tolerance, backpressure, and performance optimization.
  • Familiarity with CDC, real-time ETL/ELT, event-driven architecture, and data synchronization approaches.
  • Familiarity with ClickHouse, time-series databases, PostgreSQL, object storage, or other large-scale analytical data stores.
  • Strong understanding of distributed-systems fundamentals, including consistency, availability, idempotency, distributed state, failure recovery, and data replication.
  • Production-level engineering ability with Java, Scala, Python, Go, or a similar language.
  • Familiarity with Kubernetes, containerized deployment, and cloud-native data infrastructure.
  • Ability to analyze production-system performance, plan capacity, diagnose failures, and improve reliability.
  • Ability to abstract data models, event models, data contracts, and service interfaces from complex business scenarios and drive implementation across teams.

Responsibilities

  • Design real-time data architecture for autonomous-driving backend and onsite platforms, spanning data ingestion, message transport, real-time computation, online storage, query services, and historical archiving.
  • Build and evolve streaming-data infrastructure using technologies such as Kafka, Pulsar, Flink, and CDC to process high-frequency vehicle, task, alert, and production-operations data.
  • Design unified event models, data contracts, topic plans, schema management, and version-compatibility mechanisms to reduce coupling between products and services.
  • Build real-time processing capabilities for filtering, transformation, joins, window calculations, streaming aggregation, state calculations, anomaly detection, and real-time metrics.
  • Design high-throughput, low-latency processing architectures and resolve production issues involving message backlogs, backpressure, duplicate consumption, out-of-order events, latency, data loss, and large messages.
  • Establish reliable data consistency and delivery mechanisms using appropriate strategies such as idempotency, retries, checkpoints, at-least-once processing, or exactly-once processing.
  • Plan online and historical storage architectures using ClickHouse, time-series databases, OLAP, and object storage, balancing query performance, data scale, retention periods, and infrastructure costs.
  • Build a real-time data service layer that provides unified data-access interfaces to operations centers, digital twins, scheduling and collaboration systems, onsite clients, and agent applications.
  • Establish data quality, lineage, metadata, and governance mechanisms for core vehicle, equipment, task, alert, and production data, and maintain consistent data definitions.
  • Support data synchronization and replication across cloud, mine-site local, and edge environments, addressing weak or interrupted connectivity, reconnection, bandwidth limits, backfill, and isolation across multiple mine sites.
  • Establish observability and capacity-management systems for the data platform, continuously monitoring throughput, end-to-end latency, consumer backlogs, failure rates, storage growth, and key data-quality measures.
  • Collaborate with backend-platform, digital-twin, connectivity-services, operational-intelligence, scheduling-algorithm, artificial-intelligence-platform, and infrastructure teams to define data interfaces and ownership boundaries.

Desired Qualifications

  • Experience with autonomous driving, industrial IoT, smart mining, construction equipment, robotics, or large-scale equipment-data platforms.
  • Hands-on experience building systems handling tens of thousands to hundreds of thousands of events per second, hundreds to thousands of concurrent devices, or real-time platforms across multiple sites.
  • Production architecture experience with Kafka, Flink, and ClickHouse or Iceberg.
  • Familiarity with MQTT, IoT gateways, device telemetry, and cloud-edge data synchronization.
  • Experience building event-sourcing, data-replay, real-time digital-twin, or time-series data platforms.
  • Familiarity with Schema Registry, data contracts, data lineage, data catalogs, and data-quality management.
  • Experience with multi-cluster, multi-region, or hybrid-cloud/on-premises deployment architectures.
  • Familiarity with weak connectivity, offline operation, data backfill, and bandwidth optimization in industrial environments.

About the Company

Caterpillar

Makes heavy machinery, engines and gas turbines

Caterpillar makes construction and mining machines, diesel and natural gas engines, industrial gas turbines and locomotives for contractors, mines, energy companies and data center builders. It is the world's largest maker of construction and mining equipment, and its yellow Cat bulldozers and excavators work in nearly every country. Customers buy or rent machines through independent Cat dealers, while its engines and turbines generate electricity and push natural gas through pipelines. Caterpillar earns money from equipment sales and from the parts and repairs that dealers supply over a machine's working life. It was formed in 1925 when California's Holt and Best tractor companies merged, and it is based in Irving, Texas.

Company size
10,001+
Company stage
IPO
Headquarters
Irving, TX
Founded
1925

Recently Posted Jobs

Explore more opportunities at Caterpillar

See more jobs