S

Steampunk

Federal software solutions delivering user-focused outcomes

AI Evaluation Scientist

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeMcLean, VA, USA$105k - $145k /yr

About the Job

Requirements

  • Ability to hold a position of public trust with the U.S. government.
  • Bachelor’s or Master’s degree in Computer Science, Statistics, Machine Learning, Cognitive Science, Human-Computer Interaction, Data Science, or a related field.
  • At least 2 years of experience evaluating machine learning models, natural language processing systems, or generative AI models; experience with large language models is preferred.
  • Familiarity with evaluation metrics, statistical testing, dataset creation, and experimental design for AI systems.
  • Proficiency in Python and relevant libraries such as PyTorch, Hugging Face, scikit-learn, and LangChain.
  • Proficiency in AI evaluation frameworks such as Ragas or DeepEval.
  • Proficiency with AI traceability and observability tools based on the OpenTelemetry protocol.
  • Experience analyzing structured and unstructured data, including text, documents, and embeddings.
  • Understanding of large language model behavior, prompt evaluation, retrieval pipelines, or retrieval-augmented generation architectures.
  • Exposure to responsible AI concepts and governance-aligned evaluation criteria, such as fairness, transparency, and reliability.
  • Familiarity with the OWASP Top 10 Risks for Large Language Models.
  • Expected to be on camera during interviews and assessments; identity verification may include taking the applicant’s picture.

Responsibilities

  • Implement evaluation frameworks for AI models, including accuracy, robustness, relevance, bias, hallucination-rate, and safety metrics.
  • Build and maintain automated evaluation scripts, tests, and pipelines to assess AI model outputs and detect performance drift over time.
  • Develop benchmark datasets, challenge sets, and scenario-based test cases tailored to mission and user needs.
  • Perform structured error analysis and behavioral audits of large language models, retrieval-augmented generation systems, and predictive models, documenting findings and improvement recommendations.
  • Collaborate with AI developers, LLMOps engineers, and data scientists to support iterative experimentation, model hardening, and quality improvements.
  • Contribute to the design of human-in-the-loop evaluation workflows, integrating qualitative and quantitative insights into evaluation reports.
  • Help map evaluation outcomes to responsible AI principles such as fairness, transparency, reliability, and safety.
  • Partner with AI governance analysts to ensure evaluation outputs support compliance, documentation, and risk assessments.
  • Document evaluation processes, criteria, and results for technical and non-technical audiences.

Desired Qualifications

  • Experience working in agile or iterative development environments.
  • Local to the Washington, DC metro area preferred.
  • Relevant certifications are helpful but not required, including NIST AI RMF (AISIC), INFORMS CAP, and AWS, Azure, or Google machine learning certifications.

About the Company

Steampunk

Federal software solutions delivering user-focused outcomes

Steampunk helps U.S. federal government agencies deliver digital programs and services. The company partners with government clients, using a startup-style, customer-centered delivery approach to design, develop, and deploy software and services that support mission goals. Its products and work are executed in close collaboration with clients, emphasizing rapid delivery and high-quality user experiences. Steampunk differentiates itself by focusing specifically on federal customers and by embedding client input at every step of the process, from planning through deployment, so outcomes meet user needs and mission requirements at speed. The overall goal is to produce reliable, usable solutions that achieve clear mission impacts for government programs.

Company size
501-1,000
Company stage
Not listed
Total funding
Not listed
Headquarters
Winston-Salem, NC
Founded
2002

Recently Posted Jobs

Explore more opportunities at Steampunk

See more jobs