Principal Data Engineer
No salary listed
Expert
Bachelor's
Remote in USA+1 moreMore locations: Charlotte, NC, USA
RemoteUnited States (remote): Applicants must currently be authorized to work in the United States; visa sponsorship is not available.
Data & AnalyticsData Engineering
No H1B Sponsorship
About the job
Requirements
- A bachelor's degree in Computer Science, Engineering, or a related field and at least 10 years of data engineering experience in a high-availability, business-critical environment are required.
- Hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL, and cluster and job optimization, is required.
- Experience migrating legacy Azure SQL Server or another relational database management system to a Databricks Lakehouse, including schema translation, data validation, and cutover strategies, is required.
- Strong proficiency in Apache Kafka for streaming pipelines, including producers and consumers, topic design, partitioning strategy, and Kafka Connect, is required.
- Expert-level PySpark and/or Scala Spark skills, including performance tuning, broadcasting, partitioning, and caching, are required.
- A deep understanding of data architecture patterns, including medallion (bronze/silver/gold), Lambda/Kappa, event sourcing, and streaming-first designs, is required.
- Hands-on experience with Azure cloud services, including ADLS Gen2, Azure Event Hubs, ADF, Azure Key Vault, and Azure Monitor, is required.
- Strong knowledge of infrastructure components, including networking, cloud storage (Delta/Parquet), and cloud cost optimization, is required.
- Applicants must currently be authorized to work in the United States on a full-time basis; visa sponsorship is not available.
Responsibilities
- Lead the design and implementation of scalable, cloud-native data architectures on Databricks, including Delta Lake, Unity Catalog, and Lakehouse patterns.
- Own and execute migration from legacy Azure SQL Server to Databricks, including schema translation, ETL/ELT re-platforming, data validation, and cutover planning.
- Define data modeling standards, including medallion architecture and star/snowflake schemas, and ensure consistency across pipelines and domains.
- Evaluate and recommend tools, frameworks, and platforms to support long-term data strategy and organizational goals.
- Collaborate with Solution and Enterprise Architects to review and approve data architecture designs.
- Architect and implement Kafka-based streaming pipelines for real-time data ingestion, transformation, and delivery.
- Design event-driven architectures and streaming topologies using Kafka Streams, ksqlDB, or Spark Structured Streaming on Databricks.
- Establish patterns for schema management with Confluent Schema Registry, consumer group strategy, offset management, and dead-letter queuing.
- Ensure streaming pipelines meet latency, throughput, and fault-tolerance service-level requirements.
- Debug and optimize Spark jobs, Delta Lake tables, and SQL workloads for performance, cost efficiency, and maintainability.
- Lead code reviews for senior engineers to enforce standards, best practices, and technical quality.
- Manage pipeline quality, data models, and continuous integration and continuous delivery workflows; guide teams on continuous improvement.
- Promote data management practices for data quality, lineage, observability, and governance.
- Identify opportunities to improve service delivery methods, processes, and resource utilization.
- Mentor data engineers at all levels, with particular emphasis on developing senior talent.
- Establish and evolve data engineering standards, best practices, and technical-debt management.
- Stay current on data platform trends, including Databricks releases, the Kafka ecosystem, and open table formats, and contribute to the long-term architectural vision.
- Develop plans for data security, disaster recovery, backup, business continuity, and archiving across the Lakehouse.
- Design and enable AI and agentic capabilities on Databricks, including Databricks Genie for natural-language data exploration and self-service analytics.
- Architect clean, governed, well-documented Delta table foundations for Genie spaces, AI/BI dashboards, and LLM-driven data agents.
- Collaborate with ML and AI teams to build and maintain feature stores, vector stores, and retrieval-augmented generation pipelines on Databricks.
- Evaluate and integrate agentic frameworks, including LangChain and Mosaic AI Agent Framework, to automate data workflows and enable intelligent data products.
- Define governance and observability standards for AI-driven data pipelines to ensure reliability, auditability, and responsible AI practices.
- Partner with project managers and business leaders on enterprise data initiatives.
- Collaborate across teams to influence and strengthen data engineering practices organization-wide.
- Work with Analytics, ML, and product engineers to design and deliver end-to-end data solutions that meet business needs.
Desired Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with ksqlDB, Kafka Streams, or Spark Structured Streaming for stateful stream processing.
- Confluent Platform experience, including Schema Registry, Kafka Connect connectors, role-based access control, and cluster management.
- Advanced Azure SQL Server expertise (2018+/Azure SQL MI), including stored procedures, indexing strategies, and query plan analysis, for migration contexts.
- Experience with dbt (data build tool) for SQL-based transformation layers on Databricks.
- Proficiency with Git and continuous integration and continuous delivery pipelines for data engineering, such as Azure DevOps and GitHub Actions.
- Experience working in Agile environments, including Scrum or Kanban.
- Familiarity with data governance frameworks, data cataloging (Unity Catalog, Microsoft Purview), and data quality tools (Great Expectations, Monte Carlo).
- Familiarity with secure coding practices, including OWASP Top 10 and secrets management.
- Experience implementing DataOps practices, including automated testing, data observability, pipeline continuous integration and continuous delivery, data contracts, and service-level monitoring across the Lakehouse.
- Hands-on MLOps experience, including model versioning (MLflow), experiment tracking, model serving, and integrating ML pipelines with production data workflows on Databricks.
- Familiarity with Databricks Mosaic AI (formerly MLflow + Model Serving) for end-to-end MLOps lifecycle management.
- Experience with real-time ML feature stores or Lakehouse-based ML pipelines.
About the company
AvidXchange provides a SaaS platform that automates accounts payable and vendor payments. It includes invoice management, bill payment, purchase order, and check printing to digitize and speed payment workflows. Clients subscribe and may pay transaction fees; the system digitizes invoices, automates approvals, sends e-payments, and can print checks, cutting processing time from about 10 days to 3. It differentiates by offering a comprehensive cloud-based suite across many industries with a focus on speed, security, and cash-flow improvement; goal is to simplify and accelerate financial operations, reduce manual tasks, and lower costs.
Company Size
1,001-5,000
Company Stage
IPO
Headquarters
Charlotte, North Carolina
Founded
2000
Get referred to AvidXchange
See people who can refer or advise you
Simplify's Take
What believers are saying
- June 2026 Workday Marketplace launch expands reach into enterprise finance buyers.
- June 2026 ParishSOFT and Centerbase deals deepen embedded-payments distribution.
- Corpay closed the acquisition October 15, 2025, backing growth with payments expertise.
What critics are saying
- Corpay owns 34% and can buy the rest in 2028, limiting independence.
- AvidXchange remains unlisted since October 15, 2025, reducing hiring and investor visibility.
- Tipalti, BILL, and ERPs attack embedded AP, compressing pricing before 2027 renewals.
What makes AvidXchange unique
- AvidXchange embeds AP automation directly inside Workday, Ministry Brands, and Centerbase workflows.
- Its AvidPay Network reaches 1.5 million suppliers, reducing vendor onboarding friction.
- The take-private by TPG and Corpay gives AvidXchange capital for faster product integration.
Help us improve and share your feedback! Did you find this helpful?
Benefits
- Health Insurance
- Dental Insurance
- Life Insurance
- Disability Insurance
- 401(k) Company Match
- Parental Leave
- Unlimited Paid Time Off
- Paid Vacation
- Hybrid Work Options
- Wellness Program
- Tuition Reimbursement