C

Cortea AI

AI quality layer for audit firms

Senior/Staff AI Engineer - Quality & Evals

AI Application EngineeringConfirmed live in the last 24 hours
Full-timeBerlin, Germany

About the Job

Requirements

  • Have strong Python and/or backend engineering experience.
  • Have a solid understanding of evaluation methods for large language model and agent systems, including deterministic checks, ground truth, large language model-as-judge, human review, and quality metrics, and be able to assess when each approach is appropriate.
  • Have deployed and operated systems in the cloud, ideally on Google Cloud Platform.
  • Have hands-on experience building end-to-end retrieval or machine learning pipeline evaluation systems and using large language model observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
  • Be comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.
  • Understand system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.
  • Bring senior-level engineering judgment, make architectural decisions, communicate trade-offs, and build systems that other engineers can extend.
  • Be comfortable with ambiguity, reason from first principles, and be excited to build infrastructure for artificial intelligence systems actively used in production.

Responsibilities

  • Build online and offline evaluation systems for large language model agents, including pipelines using golden datasets, ground-truth data, human review workflows, and experiment results.
  • Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
  • Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.
  • Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
  • Manage observability tools for tracing, monitoring, debugging, and experiment management of audit agents.
  • Work with backend engineers to improve the speed and reliability of retrieval and reasoning agents.

Desired Qualifications

  • Design data pipelines, ETL/ELT workflows, event-processing systems, or feedback loops for production data.
  • Build infrastructure around large language model-based products or agentic systems, including optimizing large language model usage, context windows, reasoning tokens, or model selection.
  • Work with production traces from complex distributed systems.
  • Build internal platforms for engineers, domain experts, or operations teams.
  • Use workflow orchestration systems such as Temporal or similar.
  • Have familiarity with audit, finance, compliance, or other high-accuracy domains.
  • Have experience in an early-stage startup or fast-moving engineering environment.

About the Company

Cortea AI

AI quality layer for audit firms

Cortea AI builds an AI quality layer for audit firms that sits alongside existing systems to improve audit reliability. Its Audit Quality Agents automatically review audit reports, financial statements, disclosure notes, and workpapers, then cross-checks figures and supporting documents to flag inconsistencies and filing issues. Auditors stay in control and use Cortea’s checks to confirm sign-off, rather than replacing human judgment. The platform integrates with current audit workflows to boost quality and capacity, supporting expansion in the UK, Germany, and the US.

Company size
11-50
Company stage
Seed
Total funding
$16.6M
Headquarters
Berlin, Germany
Founded
2024

Recently Posted Jobs

Explore more opportunities at Cortea AI

See more jobs