B

Bank of America

Runs a US consumer bank, brokerage and investment bank

Senior Engineer - AI Monitoring & Control Capabilities

AI Application Engineering +1Confirmed live in the last 24 hours
Full-time

About the Job

Requirements

  • A Bachelor's or Master's degree in Computer Science, Data Science, MIS, a related field, or equivalent practical experience is required.
  • At least 10 years of software engineering experience designing, building, and scaling production applications is required.
  • Strong proficiency in Python, including common data and machine learning libraries, is required.
  • Hands-on experience deploying and operating containerized workloads on OpenShift or Kubernetes is required.
  • Proven experience taking large language model- or retrieval-augmented generation-based applications into production and maintaining them at scale is required.
  • Experience building APIs and integrating third-party platforms is required.
  • Working experience developing AI agents, including OpenAI APIs and LangChain, is required.
  • Practical understanding of inference performance optimization is required.
  • Experience applying guardrails, observability, and model-based evaluation to AI applications is required.
  • Experience delivering within Agile and Agile-at-scale environments against firm timelines is required.
  • The ability to influence outcomes and meet delivery commitments in a matrixed organization without direct authority is required.
  • Strong problem-solving, communication, and stakeholder-management skills are required.

Responsibilities

  • Own delivery of critical vendor applications from build through production support in a fully on-premises OpenShift environment, meeting committed milestones and quality expectations.
  • Build and maintain APIs and integrations connecting vendor guardrail and observability platforms to enterprise AI agent solutions.
  • Establish and manage workflows that monitor performance, safety, latency, throughput, quality, and guardrail effectiveness in support of AI risk and control objectives.
  • Implement and maintain model evaluation frameworks using LLM-as-a-Judge and specialized language models.
  • Develop observability dashboards and alerting using Grafana or similar tools, producing audit-friendly evidence and traceability.
  • Troubleshoot production issues and improve scalability, resilience, and operational efficiency to sustain availability for critical business services.
  • Contribute to evaluation methodologies and guardrail controls for agentic and multi-step AI systems.
  • Lead and mentor engineers, establish development standards, and drive process improvements across cross-functional delivery.
  • Provide input into financial planning, budget tracking, forecasting, and management of supported applications and services.

Desired Qualifications

  • Experience delivering technology solutions within a large, complex, highly regulated enterprise, preferably in financial services.
  • Experience with annotation pipelines, feedback loops, fine-tuning, and model alignment techniques.
  • Familiarity with prompt versioning and lifecycle management.
  • Experience with vendor-provided guardrail and observability platforms such as Galileo, LangSmith, Arize, WhyLabs, Braintrust, Agenta, Fiddler, MLflow, Weights & Biases, LangFuse, and RAGAS.
  • Working knowledge of additional AI agent frameworks such as LlamaIndex or Haystack.
  • Familiarity with evaluation and guardrails for agentic, multi-step AI workflows.

About the Company

Bank of America

Runs a US consumer bank, brokerage and investment bank

Bank of America runs one of the largest US banks, serving consumers, small businesses, wealthy families, corporations and institutional investors. Alongside checking accounts, credit cards, mortgages and business loans, it owns the Merrill brokerage, a private bank for wealthy clients and a global investment bank that advises on deals and trades stocks and bonds. Customers reach it through thousands of branches and ATMs and its mobile banking app. It earns money mainly from the gap between what it charges on loans and pays on deposits, plus fees from advice, asset management, deal work and trading. It grew from a San Francisco bank founded in 1904 and is headquartered in Charlotte, North Carolina.

Company size
10,001+
Company stage
IPO
Headquarters
Charlotte, NC
Founded
1904

Recently Posted Jobs

Explore more opportunities at Bank of America

See more jobs