Remote in USA+2 moreMore locations: Dallas, TX, USA | Columbus, OH, USA
Remote
Software EngineeringAI Application Engineering
About the job
Requirements
Have shipped production large language model systems where correctness matters, such as extraction pipelines, agentic workflows, or document-understanding systems.
Build and maintain evaluation datasets and detect regressions before they reach users.
Exercise judgment about when to use a model versus deterministic functions, favoring deterministic logic when financial or compliance consequences are involved.
Own a system end-to-end, ship it, and operate it in production.
Apply engineering hygiene as a standard practice.
Responsibilities
Build and scale the audit workbench, a pipeline-based harness taking an engagement from raw evidence to reviewable workpaper.
Help set technical direction to ensure the audit workbench is usable and scalable from the first 100 clients to the next 10,000.
Develop ingestion and extraction, recomputation and exception detection, workpaper generation, evaluation infrastructure, and a frontend review surface.
Establish schema design, deterministic rule logic, validation, and structured human review for a reliable audit system.
Work on the boundary between deterministic and stochastic processes, evaluation harnesses, and traceability, as well as prompts and orchestration.
Build exact code for mathematical and regulatory logic while using models for extraction and narrative drafting.
Partner closely with partner CPAs on domain and review standards.
Desired Qualifications
Familiarity with accounting, audit, ERISA, 401(k) administration, or financial document processing; audit experience is not required.
Experience with layout-aware or OCR extraction, retrieval systems, or structured-output generation.
Experience building internal tools or review-and-approval user interfaces.
About Pepper Auditors
Pepper's mission is to protect the integrity of every American's employee benefit plan. We are a fast-growing team of technologists and CPAs working to build a modern audit firm that can serve as the fulfillment engine for federally mandated compliance work streams.
The Role
We’re looking for an experienced software engineer to help build and scale our audit workbench; a pipeline-based harness — not a generic agent loop — that takes an engagement from raw evidence to reviewable workpaper. As of today that looks roughly like:
Ingestion and extraction
Recomputation and exception detection
Workpaper generation
Eval infrastrcuture
Frontend review surface
As one of the first engineering hires, you will help set the technical direction to ensure our audit wokrbench is usable and scalable from our our first 100 clients to our next 10,000.
The hard part
Most "build an AI agent" roles optimize for demo velocity. This one optimizes for a number a CPA stakes their license on. The reliability of this system comes from schema design, deterministic rule logic, validation, and structured human review — not from hoping the model is good. You'll spend as much time on the deterministic-versus-stochastic boundary, eval harnesses, and traceability as on prompts and orchestration. The model does extraction and narrative drafting; the math and the regulatory logic are exact code. If "build something auditable, reproducible, and provably correct in a regulated domain" is more interesting to you than "ship a chatbot," you'll like it here.
You're a strong fit if you
Have shipped production LLM systems where correctness mattered — extraction pipelines, agentic workflows, or document-understanding systems — ideally in a domain with real consequences (fintech, legal, healthcare, regtech).
Build evals as a reflex. You've maintained golden datasets and caught regressions before they reached users, and you treat "we can't measure whether this got better" as an unacceptable state.
Have strong judgment about where to use a model and where to write a deterministic function — with a hard bias toward the latter when dollars or compliance are on the line.
Are a genuine 0→1 builder — you can own a system end-to-end, ship it, and operate it in production. You're energized, not unsettled, by being the second engineer in the building.
Treat engineering hygiene as default, not overhead.
Nice to haves
Familiarity with accounting, audit, ERISA, 401(k) administration, or financial document processing. You do not need audit experience. Our partner CPAs own the domain end-to-end, and the regulatory logic lives in versioned, deterministic code you'll help build.
Layout-aware / OCR extraction, retrieval systems, or structured-output generation.
Experience building internal tools or review-and-approval UIs.
Logistics
Compensation: Estimated $180,000–$230,000 base + equity
Location: Remote-first (U.S.).¹ Columbus, Ohio is a plus but not required.
Reporting: Directly to the founding team, partnering closely with our partner CPAs on domain and review standards.
We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic.