BMC Software

IT management software for mainframes

Senior Data Scientist

Natural Language Processing & LLMsPosted on Nov 9

Full-timeRemote in UKNo salary listed

About the Job

Requirements

At least 8 years of relevant Data Science, Machine Learning, or Applied AI experience.
A bachelor's degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
Advanced Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
Strong statistics, experimental design, hypothesis testing, ML validation, and applied classification, regression, ranking, clustering, forecasting, and anomaly detection knowledge.
Hands-on LLM prompting, embeddings, model comparison, and RAG design and optimization, including chunking, retrieval, and reranking.
Practical instruction fine-tuning or parameter-efficient model adaptation experience, with measurable baseline comparisons.
Experience designing evaluation datasets, automated quality metrics, and statistical model, LLM, and RAG experiments.
Hands-on MLOps and LLMOps experience, including experiment tracking, model, dataset, and prompt versioning, reproducible pipelines, registries and artifacts, evaluation automation, quality gates, and monitoring.
Working knowledge of agent planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
Understanding of inference latency, throughput, concurrency, token cost, and on-premises versus provider-hosted model tradeoffs.
Familiarity with AWS and/or Kubernetes or Red Hat OpenShift, and experience collaborating with platform engineering.
Experience independently delivering complex AI/ML solutions, making sound technical decisions within assigned initiatives, and mentoring other Data Scientists.
Experience contributing to reusable modeling, evaluation, and MLOps/LLMOps practices within project teams.

Responsibilities

Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
Develop and evaluate intent classification, semantic matching, agent, skill, and tool ranking, and model-routing techniques; compare learned approaches with rules, embeddings, and LLM-based baselines.
Use statistical modeling to estimate execution latency, token demand, inference costs, workload trends, and resource requirements.
Apply clustering and anomaly detection to identify recurring intents, capability gaps, unusual agent behavior, and execution regressions.
Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments.
Evaluate commercial API and self-hosted open-weight models for accuracy, reasoning, instruction adherence, structured outputs, latency, throughput, and total cost.
Design and optimize system instructions, prompt templates, few-shot examples, tool-calling prompts, and context assembly strategies.
Benchmark model alternatives for specific enterprise workloads and document quality, cost, and latency tradeoffs.
Collaborate with AI Engineering and Platform Engineering on inference performance, quantization tradeoffs, and GPU/VRAM capacity assumptions.
Design and optimize document parsing, chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
Evaluate vector, keyword/BM25, hybrid, and semantic retrieval, including access-aware retrieval where relevant.
Diagnose retrieval versus generation failures; measure Recall@K, nDCG, groundedness, faithfulness, and end-to-end answer quality.
Compare embedding and reranking models using representative enterprise datasets and reproducible benchmarks.
Assess when prompting, RAG, supervised instruction fine-tuning, or parameter-efficient adaptation is appropriate.
Curate instruction datasets and holdout evaluations; apply SFT, LoRA, QLoRA, or comparable methods.
Measure gains over base-model and prompting/RAG baselines, including generalization, regressions, data leakage, and operating costs.
Collaborate on packaging and validating fine-tuned model artifacts and adapters.
Apply modeling and evaluation expertise to intent routing, agent and skill discovery, MCP tool selection, workflow selection, and planning effectiveness.
Evaluate task completion, tool-call correctness, argument extraction, delegation quality, and recovery from failures.
Work with AI Engineering on agentic frameworks and understand MCP tool schemas and A2A Agent Cards, skills, and task delegation.
Contribute decision models and quality signals without owning agent runtime, MCP server, or A2A protocol implementation.
Design golden datasets, holdout sets, human review protocols, LLM-as-judge calibration, and statistical comparison methods.
Automate evaluations and regression checks for prompts, foundation models, RAG, fine-tuned models, routing, and agents.
Define measurable quality thresholds for accuracy, relevance, groundedness, instruction following, task completion, and cost/performance.
Investigate regressions and use evidence to guide model, prompt, retrieval, and routing decisions.
Implement and maintain reproducible experimentation, model evaluation, and lifecycle workflows for assigned AI/ML initiatives, including automated validation and production-readiness checks.
Apply reproducible experiment tracking across datasets, code, model versions, hyperparameters, prompts, retrieval configurations, metrics, and artifacts.
Develop and maintain automated training, fine-tuning, benchmarking, and evaluation pipelines, with model registry and artifact traceability.
Define and integrate quality gates, regression tests, and model, prompt, and RAG release-readiness criteria with engineering teams.
Analyze production quality and drift signals, agent routing accuracy, latency, throughput, and token usage; recommend remediation.
Collaborate with engineering on packaging, deployment handoffs, observability, rollback criteria, and ongoing model lifecycle management.
Independently lead complex Data Science initiatives from problem formulation through model development, evaluation, and production handoff.
Make technical decisions within assigned initiatives, balancing model quality, performance, cost, and delivery requirements.
Improve reusable modeling, benchmarking, experimentation, and MLOps/LLMOps practices within assigned initiatives.
Collaborate with AI Engineering, AI Quality, Platform Engineering, Product Management, and domain experts.
Translate business requirements into measurable AI and ML objectives.
Mentor junior and mid-level Data Scientists and contribute to technical reviews.
Communicate complex AI and ML concepts to technical and non-technical stakeholders.

Desired Qualifications

Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
Hands-on use of LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or other agentic frameworks.
Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or comparable MLOps/LLMOps and evaluation tooling.
Experience with vLLM, NVIDIA Triton, quantization, GPU memory sizing, or distributed inference.
Experience with Python backend APIs such as FastAPI or Flask, and OpenTelemetry and AI observability.
Experience with AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
Experience with enterprise code understanding, legacy modernization, or domain-specific model adaptation.
Familiarity with Agile practices and Atlassian Jira/Confluence.
A master's degree in a related field.

CareerArc Code

CA-JK

Hybrid: #LI-Hybrid

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.

We're looking for a Senior Data Scientist to join our AMI group. This hands-on role combines Machine Learning, statistical modeling, Generative AI, RAG, model evaluation, and fine-tuning for enterprise AI applications. AMI supports commercial AI providers and on-premises models, with agents, MCP tools, and A2A interoperability delivered in partnership with AI Engineering.

About the Role:

You will independently design, develop, evaluate, and operationalize complex AI/ML solutions. You will contribute hands-on technical expertise, apply rigorous experimentation and MLOps/LLMOps practices, improve modeling and evaluation approaches within assigned initiatives, and mentor other Data Scientists.

You will be a part of the AI Migration team, responsible for building tools that help customers transition from competitor products to BMC solutions. You'll work at the intersection of software engineering and AI, combining code-based conversion with AI-assisted processing to tackle migration challenges. Your work will help reduce manual effort and enable services to deliver accurate, timely customer migrations

Key Responsibilities:

Machine Learning & Applied Data Science

Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques; compare learned approaches with rules, embeddings, and LLM-based baselines.
Use statistical modeling to estimate execution latency, token demand, inference costs, workload trends, and resource requirements.
Apply clustering and anomaly detection to identify recurring intents, capability gaps, unusual agent behavior, and execution regressions.
Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments.

Foundation Models, Prompting & Context Engineering

Evaluate commercial API and self-hosted open-weight models for accuracy, reasoning, instruction adherence, structured outputs, latency, throughput, and total cost.
Design and optimize system instructions, prompt templates, few-shot examples, tool-calling prompts, and context assembly strategies.
Benchmark model alternatives for specific enterprise workloads and document quality/cost/latency tradeoffs.
Collaborate with AI Engineering and Platform Engineering on inference performance, quantization tradeoffs, and GPU/VRAM capacity assumptions.

Retrieval-Augmented Generation (RAG)

Design and optimize document parsing, chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
Evaluate vector, keyword/BM25, hybrid, and semantic retrieval, including access-aware retrieval where relevant.
Diagnose retrieval versus generation failures; measure Recall@K, nDCG, groundedness, faithfulness, and end-to-end answer quality.
Compare embedding and reranking models using representative enterprise datasets and reproducible benchmarks.

Model Fine-Tuning & Adaptation

Assess when prompting, RAG, supervised instruction fine-tuning, or parameter-efficient adaptation is appropriate.
Curate instruction datasets and holdout evaluations; apply SFT, LoRA, QLoRA, or comparable methods.
Measure gains over base-model and prompting/RAG baselines, including generalization, regressions, data leakage, and operating costs.
Collaborate on the packaging and validation of fine-tuned model artifacts and adapters.

Agentic AI, MCP & A2A

Apply modeling and evaluation expertise to intent routing, agent/skill discovery, MCP tool selection, workflow selection, and planning effectiveness.
Evaluate task completion, tool-call correctness, argument extraction, delegation quality, and recovery from failures.
Work with AI Engineering on agentic frameworks and understand MCP tool schemas and A2A Agent Cards, skills, and task delegation.
Contribute decision models and quality signals without owning agent runtime, MCP server, or A2A protocol implementation.

AI Evaluation & Experimentation

Design golden datasets, holdout sets, human review protocols, LLM-as-judge calibration, and statistical comparison methods.
Automate evaluations and regression checks for prompts, foundation models, RAG, fine-tuned models, routing, and agents.
Define measurable quality thresholds for accuracy, relevance, groundedness, instruction following, task completion, and cost/performance.
Investigate regressions and use evidence to guide model, prompt, retrieval, and routing decisions.
Implement and maintain reproducible experimentation, model evaluation, and lifecycle workflows for assigned AI/ML initiatives, including automated validation and production readiness checks.

MLOps, LLMOps & Model Lifecycle

Apply reproducible experiment tracking across datasets, code, model versions, hyperparameters, prompts, retrieval configurations, metrics, and artifacts.
Develop and maintain automated training, fine-tuning, benchmarking, and evaluation pipelines, with model registry and artifact traceability.
Define and integrate quality gates, regression tests, and model/prompt/RAG release-readiness criteria with engineering teams.
Analyze production quality and drift signals, agent routing accuracy, latency, throughput, and token usage; recommend remediation.
Collaborate with engineering on packaging, deployment handoffs, observability, rollback criteria, and ongoing model lifecycle management.

Technical Leadership & Collaboration

Independently lead complex Data Science initiatives from problem formulation through model development, evaluation, and production handoff.
Make technical decisions within assigned initiatives, balancing model quality, performance, cost, and delivery requirements.
Improve reusable modeling, benchmarking, experimentation, and MLOps/LLMOps practices within assigned initiatives.
Collaborate with AI Engineering, AI Quality, Platform Engineering, Product Management, and domain experts.
Translate business requirements into measurable AI and ML objectives.
Mentor junior and mid-level Data Scientists and contribute to technical reviews.
Communicate complex AI and ML concepts clearly to technical and non-technical stakeholders.

Must-Have Skills & Experience

8+ years of relevant Data Science, Machine Learning, or Applied AI experience.
Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
Advanced Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
Strong statistics, experimental design, hypothesis testing, ML validation, and applied classification, regression, ranking, clustering, forecasting, and anomaly detection knowledge.
Hands-on LLM prompting, embeddings, model comparison, and RAG design/optimization, including chunking, retrieval, and reranking.
Practical instruction fine-tuning or parameter-efficient model adaptation experience, with measurable baseline comparisons.
Experience designing evaluation datasets, automated quality metrics, and statistical model/LLM/RAG experiments.
Hands-on MLOps and LLMOps: experiment tracking, model/dataset/prompt versioning, reproducible pipelines, registries/artifacts, evaluation automation, quality gates, and monitoring.
Working knowledge of agent planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
Understanding of inference latency, throughput, concurrency, token cost, and on-prem versus provider-hosted model tradeoffs.
Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), and collaboration with platform engineering.
Clear communication, independent problem solving, cross-functional collaboration, and mentoring.
Demonstrated ability to independently deliver complex AI/ML solutions, make sound technical decisions within assigned initiatives, and mentor other Data Scientists.
Experience contributing to reusable modeling, evaluation, and MLOps/LLMOps practices within project teams.
Evidence We Will Look for in Hiring
Delivered measurable improvements in model quality, RAG retrieval, routing accuracy, fine-tuning outcomes, or inference cost against documented baselines.
Built reproducible ML/LLM experiments, automated evaluation pipelines, tracked artifacts, and model quality gates.
Demonstrated independent delivery of complex AI/ML solutions, rigorous model validation, collaboration on production monitoring, mentoring of other Data Scientists, and improvements to technical practices within assigned initiatives.
Demonstrated improvements in model quality, reliability, performance, or operational efficiency within specific projects or applications.

Nice-to-Have Skills

Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
Hands-on use of LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or other agentic frameworks.
Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or comparable MLOps/LLMOps and evaluation tooling.
Experience with vLLM, NVIDIA Triton, quantization, GPU memory sizing, or distributed inference.
Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
Enterprise code understanding, legacy modernization, or domain-specific model adaptation.
Agile practices and Atlassian Jira/Confluence.
Master’s degree in a related field.

#LI-Remote

Our commitment to you!

BMC’s culture is built around its people. We have 6000+ brilliant minds working together across the globe. You won’t be known just by your employee number, but for your true authentic self. BMC lets you be YOU!

If after reading the above, You’re unsure if you meet the qualifications of this role but are deeply excited about BMC and this team, we still encourage you to apply! We want to attract talents from diverse backgrounds and experience to ensure we face the world together with the best ideas!

BMC is committed to equal opportunity employment regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, disabled veteran or status as a protected veteran. If you need a reasonable accommodation for any part of the application and hiring process, visit the accommodation request page.

BMC Software maintains a strict policy of not requesting any form of payment in exchange for employment opportunities, upholding a fair and ethical hiring process.

About the Company

BMC Software

IT management software for mainframes

What BMC Software does: It provides software to manage and optimize large enterprise IT environments, from IBM mainframes to modern cloud systems. How its product works: Its tools monitor, manage, and automate IT infrastructure across mainframes, distributed servers, and cloud platforms, helping teams keep systems running, secure, and efficient. How it stands out from competitors: It has a long track record starting with mainframe software and has grown through targeted acquisitions to offer a broad suite for end-to-end IT operations, spanning on‑premises and cloud, with strategic ownership changes that gave it room to adapt. What its goal is: to help enterprises operate complex IT environments more reliably and efficiently by providing integrated software for mainframes, distributed systems, and cloud management.

Company size: 5,001-10,000
Company stage: IPO
Headquarters: Houston, TX
Founded: 1980

Recently Posted Jobs

Explore more opportunities at BMC Software

See more jobs