P

Prevalent AI

Cybersecurity risk management platform & analytics

Senior Data Engineer

Data EngineeringPosted on Nov 17
Full-timeKochi, Kerala, IndiaNo salary listedIn-person

About the Job

Requirements

  • The candidate must have 5–7 years of relevant software or data engineering experience, including substantial hands-on development.
  • The candidate must have strong Python programming skills and practical experience developing, debugging, and optimizing production-grade PySpark applications; Spark development must be a core engineering skill, not merely exposure through prebuilt pipelines or orchestration tools.
  • The candidate must understand distributed processing, Spark execution internals, shuffle behavior, partitioning, and parallelism.
  • The candidate must have practical knowledge of data skew and mitigation, adaptive query execution, broadcast hash and sort-merge joins, shuffle optimization, and bottleneck diagnosis using Spark UI, Spark History Server, or execution plans.
  • The candidate must independently solve data-processing problems involving joins, aggregations, window functions, and deduplication using DataFrame APIs and Spark SQL.
  • The candidate must have a strong command of Python data structures, functions, exception handling, debugging, and maintainable code.
  • The candidate must have strong SQL skills, including joins, aggregations, window functions, and query-optimization fundamentals.

Responsibilities

  • Design, develop, test, and maintain production-grade PySpark applications using DataFrame APIs and Spark SQL.
  • Implement complex transformations, joins, aggregations, window functions, deduplication, and incremental processing.
  • Translate data requirements into efficient, scalable solutions that account for data characteristics, edge cases, and execution costs.
  • Apply understanding of Spark execution, including lazy evaluation, driver and executor roles, directed acyclic graphs, jobs, stages, tasks, partitions, and shuffle boundaries.
  • Diagnose performance bottlenecks using Spark UI, Spark History Server, execution plans, and application logs.
  • Identify and resolve data skew, excessive shuffling, spills, inefficient joins, poorly sized partitions, long-running tasks, and resource-utilization issues.
  • Apply adaptive query execution, appropriate join strategies, partitioning, repartitioning, coalescing, partition pruning, and predicate pushdown to improve performance.
  • Investigate production issues, establish root causes, and validate fixes using measurable outcomes.
  • Build and maintain reliable batch and incremental processing pipelines with appropriate error handling, logging, data validation, and reconciliation.
  • Troubleshoot job failures and performance regressions; contribute to testing, code reviews, documentation, and reusable engineering practices.
  • Monitor workload performance and improve runtime, resource efficiency, and operational stability.
  • Own assigned engineering problems from analysis through implementation, testing, and deployment.
  • Evaluate alternative solutions, explain technical trade-offs, and collaborate with engineers, architects, and other stakeholders.

Desired Qualifications

  • Experience with Spark Structured Streaming and incremental or event-driven processing.
  • Experience with Kafka or similar streaming platforms.
  • Experience with Delta Lake, Apache Iceberg, or other lakehouse technologies.
  • Experience with Spark environments such as Databricks or Amazon EMR.

About the Company

Prevalent AI

Cybersecurity risk management platform & analytics

Prevalent AI provides tailored cybersecurity risk management solutions that unify security data on a single platform. It uses security data science and the NIST Cybersecurity Framework to guide activities in identifying, protecting, and detecting threats. The platform consolidates data and applies analytics to help clients understand and control their risk, delivering scalable and cost-effective capabilities. Differences from competitors come from serving a selective set of organizations seeking proactive risk management with customized solutions, clear guidance, and direct control over their security posture. The company aims to help organizations stay ahead of the evolving threat landscape by improving their cybersecurity posture through precise risk management and effective incident response.

Company size
51-200
Company stage
Growth Equity (Venture Capital)
Total funding
$22M
Headquarters
London, United Kingdom
Founded
2017

Recently Posted Jobs

Explore more opportunities at Prevalent AI

See more jobs