S

Simile

Enterprise AI platform simulating human behavior

Member of Technical Staff, Model Evaluations

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeIn-person

About the Job

Requirements

  • You can show how you turned an ambiguous quality question into a validated measurement that informed a decision, investigate surprising results by inspecting examples and assumptions, test whether an evaluation is misleading, compare against meaningful baselines, explain measurement limitations, and revise methods or conclusions when evidence warrants it.
  • You can work with large language model outputs, understand modeling team needs, and assess whether model changes improved relevant behavior. Prior specialization in language model training or post-training is not required.
  • You are comfortable reasoning about noisy data, uncertainty, sampling, distributions, calibration, confidence intervals, measurement validity, bias, variance, and the difference between an observed result and the underlying population quantity it estimates.
  • You can quickly build internal tools, scripts, dashboards, labeling workflows, analyses, or automated evaluation pipelines. You are comfortable with Python, SQL, R, notebooks, language model application programming interfaces, and agentic coding tools such as Codex, Claude Code, Cursor, or equivalent systems, while validating outputs and planning for long-term reliability.
  • You drive workstreams while building prototypes, inspecting data, debugging workflows, and improving methods; make progress when data or infrastructure are incomplete; work with partners to resolve dependencies; communicate findings and limitations clearly; make pragmatic scope decisions; and follow through until results are useful.

Responsibilities

  • Design evaluations, metrics, rubrics, datasets, dashboards, and workflows to measure whether the company’s models accurately predict human behavior across customer use cases, populations, question types, and decision contexts.
  • Evaluate new model versions, diagnose regressions, identify priorities for model-improvement cycles, and maintain stable evaluation suites representing capabilities customers care about.
  • Build evaluations for qualitative responses, retrieval, survey generation, AI-generated research reports, customer-facing outputs, and other product surfaces; translate subjective quality concerns into rubrics, labeled data, automated graders, release criteria, and model-improvement signals.
  • Develop rigorous methods to compare simulated responses against human data, customer studies, company-collected ground truth, and behavioral datasets; support reasoning about sampling error, uncertainty, calibration, margin of error, representativeness, and ground truth for noisy human behavior.
  • Use modern agentic coding tools to build internal tools, inspect model outputs, create labeling workflows, validate evaluations, and turn ambiguous evaluation questions into working systems.
  • Develop repeatable benchmarks for behavioral predictions using transaction and product-interaction data, intervention effects, individual trajectories, and emerging multi-agent settings.

Desired Qualifications

  • Experience building regression suites, benchmark datasets, model-comparison dashboards, automated graders, release criteria, or evaluations for customer-facing research products.
  • Experience with sampling, weighting, power analysis, uncertainty quantification, Bayesian modeling, psychometrics, polling, or measurement theory.

About the Company

Simile

Enterprise AI platform simulating human behavior

Simile builds AI models that simulate human behavior to forecast how individuals and groups will respond to scenarios using generative agents grounded in real data. The platform runs simulations of events like product launches, policy changes, or earnings calls to support decision-making. It differentiates itself by grounding agents in hundreds of interviews and behavioral science, combining Stanford-backed research with enterprise-focused decision intelligence. Its goal is to help organizations test decisions and forecast outcomes before acting, reducing uncertainty in strategic planning.

Company size
51-200
Company stage
Late Stage VC
Total funding
$300M
Headquarters
Palo Alto, CA
Founded
2025

Recently Posted Jobs

Explore more opportunities at Simile

See more jobs