NCS

ICT consulting, digital transformation, cybersecurity services

AI Engineer

AI Application Engineering +1Confirmed live in the last 24 hours

Full-timeSingaporeNo salary listedHybrid

About the Job

Requirements

At least 6 years of software engineering experience building production services, including at least 1 year leading small projects.
Strong proficiency in one systems language—Python, TypeScript, Go, or Java—and comfort with a second.
Hands-on experience with containers, Kubernetes, continuous integration and continuous delivery, infrastructure as code using Terraform, and a cloud platform such as Azure, AWS, or GCP.
Practical experience building retrieval-augmented generation or agent applications, designing prompts, using tools, and implementing safety patterns.
Experience designing schemas, batch and stream pipelines, and search indexes; proficiency with SQL and one vector database.
Experience with unit and integration testing, load testing, and golden datasets for large-language-model evaluations.
Knowledge of secrets management, policy enforcement, security scanning, and least-privilege identity and access management.

Responsibilities

Design, build, and operate production-grade agentic and generative AI systems end to end, delivering services, robust APIs, reusable components, and secure pipelines connecting large language models, tools, knowledge, and enterprise systems.
Design multi-agent systems with planning, tool use, and delegation, and expose them through REST or gRPC APIs.
Implement tool adapters for SQL, search, document stores, web calls, and code execution with strict type contracts and safe sandboxes.
Build model gateway integrations with routing, rate limits, retries, and fallback chains.
Build retrieval-augmented generation services with chunking, enrichment, embeddings, indexing, and hybrid or vector search.
Implement ingestion pipelines for documents, tickets, chat, and ERP or CRM data, including personally identifiable information redaction and metadata governance.
Optimize retrieval quality using chunking strategies, rerankers, and query rewriting, with offline and online evaluation and A/B tests.
Version, diff, and test prompts and graphs, including unit tests for prompts and tools, golden sets, and regression suites.
Build evaluation harnesses for latency, cost, accuracy, toxicity, hallucination, and guardrail hit rates, and integrate them into continuous integration.
Add drift detection for conversational systems and implement safe shutdown and automatic rollback.
Package services as containers and deploy them to Kubernetes with Helm and Argo CD; configure autoscaling, horizontal and vertical pod autoscaling, and resource quotas.
Implement policy chains and guardrails, including policy enforcement, personally identifiable information protections, and image scanning.
Instrument tracing, metrics, logs, and cost measurement per request and model.
Manage secrets, signed images, software bills of materials, and least-privilege identity and access management.
Build tenant isolation and data residency controls, and implement red-team and blue-team prompts and jailbreak defenses.
Ship connectors and events for SAP, CRM, and ITSM systems and Kafka topics; design idempotent, retry-safe processors.
Automate business workflows with pro-code services first, exposing low-code surfaces only where appropriate.
Partner with Product, Data, and Platform teams to define service-level agreements, service-level objectives, and success metrics.
Mentor engineers on AI-as-software practices, and run design reviews and postmortems.

Desired Qualifications

Production experience with agent orchestration such as LangGraph or Semantic Kernel and distributed computing such as Ray.
Experience tuning search and retrieval, including BM25 and vector hybrid search, reranking, and query planning.
Experience with large-scale observability using OpenTelemetry and cost and performance optimization across model and router layers.
Experience in regulated or high-throughput domains such as telecommunications, finance, or healthcare, including multi-tenant and data-residency patterns.
Experience with domain integrations such as SAP, CRM, or ITSM; event-driven architectures such as Kafka or Debezium; and policy enforcement such as OPA or Gatekeeper.
Familiarity with TM Forum APIs or business support system and operations support system patterns, particularly in a telecommunications context.

Job Description

What will you do:

Design, build, and operate production-grade agentic and GenAI systems—end to end. You’ll ship services (not just notebooks): robust APIs, reusable components, and secure pipelines that connect LLMs, tools, knowledge, and enterprise systems. You’ll pair strong software engineering with modern AI practices (RAG, agent orchestration, policy chains, evals) to deliver measurable business outcomes at scale.

1. Agent & Application Engineering

Design multi-agent systems (MAS) with planning, tool-use, and delegation (e.g., LangGraph/Semantic Kernel); expose them via REST/gRPC APIs (FastAPI/Express/Java/Go).
Implement tool adapters (SQL, search, document stores, web calls, code exec) with strict type contracts and safe sandboxes.
Build model gateway integrations (OpenAI/Azure OpenAI/Bedrock/Vertex; self-hosted vLLM/TGI) with routing, rate-limits, retries, and fallback chains.

2. Retrieval, Data & Knowledge

Stand up RAG services: chunking, enrichment, embeddings, indexing, hybrid/vector search (pgvector/Pinecone/Weaviate; OpenSearch/Azure AI Search).
Implement ingestion pipelines (Airflow/Prefect/Celery/Ray) for docs, tickets, chat, and ERP/CRM data; handle PII redaction and metadata governance.
Optimize retrieval quality (chunking strategies, re-rankers, query rewriting) with offline/online evaluation and A/B tests.

3. Quality, Testing & Evaluation

Treat prompts and graphs as code: version, diff, and test them (unit tests for prompts/tools; golden sets; regression suites).
Build evaluation harnesses (latency, cost, accuracy, toxicity, hallucination, guardrail hit-rates); wire into CI.
Add drift detection for conversational systems; implement safe shutdown and auto-rollback.

4. Platform & Operations

Package services as containers; deploy to Kubernetes with Helm/Argo CD; configure autoscaling, HPA/VPA, and resource quotas.
Implement policy chains and guardrails (OPA/Gatekeeper for policy, Presidio for PII, Trivy for image scanning).
Instrument deep observability: tracing (OpenTelemetry), metrics (Prometheus), logs (ELK/OPENSEARCH), cost meters per request/model.

5. Security & Compliance

Manage secrets (HashiCorp Vault/KMS), signed images, SBOMs; enforce least-privilege IAM.
Build tenant isolation and data residency controls; implement red/blue team prompts and jailbreak defenses.

6. Integration & Enterprise Workflows

Ship connectors and events for SAP/CRM/ITSM and Kafka topics; design idempotent, retry-safe processors.
Automate business workflows with pro-code services first; expose low-code surfaces only where appropriate.

7. Collaboration & Leadership

Partner with Product, Data, and Platform teams to define SLAs/SLOs and success metrics.
Mentor engineers on “AI as software” practices; run design reviews and postmortems.

Qualifications

The ideal candidate should possess:

6+ years in software engineering (prod services, not just prototypes), including 1+ year leading small projects.
Strong in one systems language (Python/TypeScript/Go/Java) and comfortable in a second.
Hands-on with containers, Kubernetes, CI/CD (GitHub Actions/GitLab/Jenkins), IaC (Terraform), and cloud (Azure/AWS/GCP).
Practical LLM experience: building RAG/agent apps, prompt design, tool-use, and safety patterns.
Data skills: designing schemas, batch/stream pipelines, and search indexes; proficiency with SQL and one vector DB.
Testing mindset: unit/integration tests, load tests, golden datasets for LLM evals.
Security basics: secrets, policies, scanning, and least-privilege IAM.

Preferred Qualifications

Agent orchestration (LangGraph, Semantic Kernel) and distributed compute (Ray) in production.
Search/retrieval tuning (BM25 + vector hybrid, re-ranking, query planning).
Observability at scale with OpenTelemetry; cost/perf optimization across model/router layers.
Experience in regulated or high-throughput domains (e.g., telco, finance, healthcare); multi-tenant and data-residency patterns.
Domain integrations (SAP/CRM/ITSM), event-driven architectures (Kafka/Debezium), and policy enforcement (OPA/Gatekeeper).
Familiarity with TM Forum APIs / BSS-OSS patterns is a plus (if in telco context).

Tech Stack (Illustrative)

Languages: Python, TypeScript/Node.js (plus Go/Java bonus)
Frameworks: FastAPI/Express, LangGraph/Semantic Kernel, Ray/Celery, Airflow/Prefect
Storage/Search: Postgres, Redis, S3/Blob; pgvector/Pinecone/Weaviate; OpenSearch/Azure AI Search
LLM Runtime: OpenAI/Azure OpenAI/Bedrock/Vertex; vLLM/TGI; inference routers/gateways
Platform: Docker, Kubernetes, Helm, Argo CD, Terraform, Vault, Istio
Observability: OpenTelemetry, Prometheus/Grafana, ELK/OpenSearch
Quality & Safety: pytest/Jest, prompt/unit test harnesses, guardrails, Presidio, Trivy

Additional Information

Why Join NCS

Lead high-impact Data & AI advisory programs for major enterprises and public sector clients.

Shape enterprise strategies and governance frameworks that drive real transformation.

Work with a talented, multidisciplinary team in a collaborative environment.

Competitive compensation and strong professional development support.

We are driven by our AEIOU beliefs—Adventure, Excellence, Integrity, Ownership, and Unity—and we seek individuals who embody these values in both their professional and personal lives. We are committed to our Impact: Valuing our clients, Growing our people, and Creating our future.

Together, we make the extraordinary happen.

Learn more about us at ncs.co and visit our LinkedIn career site.

Scam Alert

We are aware of fraudulent job offers and impersonations of NCS recruiters. Phishing emails using convincing-looking but fake addresses are also commonly used to trick you into thinking that they come from official NCS sources.

Please note that all official communications from NCS Group will only be sent from verified corporate email addresses. Always check that the sender’s email address ends with the genuine NCS domain, @ncs.com.sg and beware of extra letters, symbols or misspellings. When in doubt, verify the sender’s identity by contacting us at [email protected].

NCS