Senior Backline Engineer
Spark
Updated on 2/1/2024
Databricks

5,001-10,000 employees

Unified, open platform for enterprise data
Company Overview
Databricks is on a mission to simplify and democratize data and AI, helping data teams solve the world’s toughest problems. As the world’s first and only lakehouse platform in the cloud, Databricks combines the best of data warehouses and data lakes to offer an open and unified platform for data and AI.
Data & Analytics

Company Stage

Series I

Total Funding

$4.2B

Founded

2013

Headquarters

San Francisco, California

Growth & Insights
Headcount

6 month growth

4%

1 year growth

30%

2 year growth

115%
Locations
United States
Experience Level
Entry
Junior
Mid
Senior
Expert
Desired Skills
Microsoft Azure
Python
Apache Spark
SQL
Apache Kafka
Java
AWS
Elasticsearch
Scala
Hadoop
CategoriesNew
Software Engineering
Requirements
  • Minimum 6 years' experience developing, testing, and sustaining Python or Java or Scala-based applications
  • Comfortable with compiling, building and navigating the Apache Spark source code
  • Experience in Big Data/Hadoop/Spark/Kafka/Elasticsearch data pipelines
  • Good understanding of AI & ML using python ex:sklearn
  • Proficiency in GQL (GraphQL) and its applications in AI platforms
  • Hands-on experience with YAML for configuration and data representation
  • Hands-on experience with SQL-based database systems
  • Experience in JVM, GC, Thread dump-based troubleshooting
  • Experience with AWS or Azure related services
  • Bachelor's degree in Computer Science or a related field is required
Responsibilities
  • Troubleshoot, resolve and suggest deep code-level analysis of Spark to address complex customer issues related to Spark core internals, Spark SQL, Structured Streaming and Databricks Delta
  • Provide best practices guidance around Spark runtime performance and usage of Spark core libraries and APIs for custom-built solutions developed by Databricks customers
  • Help the support team with detailed troubleshooting guides and runbooks
  • Contribute to automation and tooling programs to make daily troubleshooting efficient
  • Work with the Spark Engineering Team and spread awareness of upcoming features and releases
  • Identify Spark bugs and suggest possible workarounds
  • Demonstrate ownership and coordinate with engineering and escalation teams to achieve resolution of customer issues and requests
  • Participate in weekend and weekday on call rotation
Desired Qualifications
  • Understanding of general GenAI topics like LLMs and prompt engineering