K

Koch Industries

Private conglomerate in energy and materials

Senior Data Engineer - Databricks

Data EngineeringConfirmed live in the last 24 hours
Full-timeAtlanta, GA, USANo salary listed

About the Job

Requirements

  • Experience designing, building, and supporting production-grade Databricks, Delta Lake, and Apache Spark pipelines for batch and streaming workloads.
  • Experience developing Databricks Jobs, including Lakeflow Declarative Pipelines and Structured Streaming, and multi-task workflows with scheduling, parameters, dependencies, retries, notifications, and operational monitoring.
  • Experience implementing Databricks infrastructure and environment-aware deployment automation using Terraform, Databricks Asset Bundles, or other Infrastructure as Code practices, Git, and CI/CD workflows.
  • Experience implementing data governance and access controls with Unity Catalog, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions.
  • Experience using Python and Spark diagnostic tools, including job logs, Spark UI, query plans, partitioning, join optimization, and Adaptive Query Execution, to troubleshoot and optimize production data workloads.
  • Must have legal authorization to work permanently in the United States for any employer without requiring a visa transfer or visa sponsorship.

Responsibilities

  • Design, build, deploy, and support Databricks-native data engineering solutions, including Delta Lake pipelines, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader ingestion pipelines, Unity Catalog resources, and CI/CD automation.
  • Develop reliable, governed data platforms and pipelines that improve the accuracy, consistency, accessibility, and trustworthiness of enterprise data assets.
  • Create automated tools, reusable templates, and self-service patterns that accelerate data development, ingestion, feature engineering, and consumption.
  • Operate and improve the platform through performance monitoring, incident resolution, migrations, upgrades, observability, and cost optimization aligned with long-term scalability and resilience goals.
  • Partner with product owners, operations teams, architects, and delivery teams to identify common use cases, prioritize platform enhancements, and expand adoption.
  • Integrate Databricks pipelines with enterprise governance, metadata, lineage, classification, certification, ownership, and data-quality capabilities.
  • Develop and curate governed data assets for AI and advanced analytics use cases, including semantic models, APIs, Genie Spaces, and AI agents.
  • Implement validation rules, monitoring, and remediation processes that improve the fitness and reliability of critical data assets.

Desired Qualifications

  • Experience building production ingestion pipelines with Databricks Auto Loader.
  • Advanced Python experience applied to data engineering, automation, reusable libraries, and platform capabilities.
  • Experience with open table formats such as Delta Lake, Apache Hudi, or Apache Iceberg.
  • Experience developing governed data assets for AI, semantic modeling, APIs, advanced analytics, Genie Spaces, or AI agents.
  • Experience with AWS, Microsoft Azure, or Google Cloud and native storage services such as Amazon S3, Azure Data Lake Storage Gen2, OneLake, or Google Cloud Storage.

About the Company

Koch Industries

Private conglomerate in energy and materials

Koch Industries is a large, private, diversified conglomerate that operates in energy, chemical processing, and manufacturing. It started from an innovation that turned heavy crude oil into gasoline and grew through long‑term investments rather than short‑term market moves. The company generates value by refining fuels and producing chemical products, and it operates Georgia-Pacific, a major maker of paper and building products. Its business model relies on reinvesting most earnings back into the company to fund steady, long‑term growth rather than paying out frequent dividends or seeking quick profits. Koch differentiates itself from many competitors by staying privately owned, which reduces pressure from public markets and enables longer‑range planning. The company’s goal is to build scale and durability across its holdings by sustainable, multi‑decade growth rather than rapid, episodic gains.

Company size
5,001-10,000
Company stage
Growth Equity (Non-Venture Capital)
Total funding
$985.2M
Headquarters
Wichita, KS
Founded
1940

Recently Posted Jobs

Explore more opportunities at Koch Industries

See more jobs