Service Desk AI Quality Analyst

Daily Activities:

• Monitor AI-assisted Service Desk performance, including deflection, resolution, escalation, fallback, and containment rates.

• Review AI interactions, failed intents, fallback cases, and escalated conversations to identify quality issues and recurring patterns.

• Tag, classify, and analyze AI failures and user support scenarios.

• Identify gaps in knowledge articles, intents, automation coverage, and AI responses.

• Analyze Service Desk tickets, trends, user feedback, and operational data to identify opportunities for improvement.

• Prepare dashboards, reports, and insights on AI and Service Desk performance using tools such as Excel and Power BI.

• Provide structured recommendations and feedback to Knowledge Management, Automation, AI, and Service Desk teams.

• Track improvement actions and assess whether implemented changes are delivering the expected outcomes.

• Support incident, request, and user inquiry management as part of normal Service Desk operations.

• Participate in Service Desk queue management and ensure appropriate handling and escalation of support cases.

• Collaborate with technical and business stakeholders to understand recurring user pain points and service experience gaps.

• Contribute to continual service improvement initiatives and the optimization of AI-assisted support processes.

Business Value: 

• The Service Desk AI Quality Analyst helps maximize the business value generated from AI-enabled IT support by ensuring that AI capabilities are continuously measured, improved, and aligned with user needs. The role delivers value by:

• Improving AI effectiveness: Identifying and addressing failed intents, inaccurate responses, and ineffective automation.

• Increasing automation value: Identifying opportunities to increase AI deflection and reduce unnecessary human intervention

• Improving user experience: Using user feedback, ticket data, and interaction analysis to identify and resolve recurring experience gaps.

• Reducing support effort: Helping reduce repetitive Service Desk workload through improved knowledge, automation, and AI capabilities.

• Improving service quality: Ensuring AI-assisted support delivers consistent and reliable outcomes.

• Supporting data-driven decisions: Providing measurable insights into AI performance, Service Desk trends, and improvement opportunities.

• Driving continual improvement: Connecting operational Service Desk insights with AI, automation, and Knowledge Management improvement activities.

• Protecting business value from AI investment: Ensuring AI capabilities continue to deliver measurable improvements in efficiency, quality, and customer satisfaction. 

The Role:

AI Quality & Performance Management

• Monitor and report on key AI performance indicators, including deflection, resolution, escalation, fallback, and containment rates.
• Review AI interactions and identify quality issues, failed intents, and inappropriate or incomplete responses.
• Categorize and analyze AI failure scenarios to identify root causes and recurring patterns.
• Identify opportunities to improve AI intents, conversational flows, knowledge sources, and automation.
• Validate the effectiveness of AI improvements and monitor performance following changes.
• Maintain accurate records of AI quality issues, trends, and improvement opportunities.

Data Analysis & Reporting

• Analyze Service Desk ticket data, AI interaction data, and user feedback to identify trends and service gaps.
• Develop regular AI quality and Service Desk performance reports.
• Create and maintain dashboards and analytical views using Excel, Power BI, or equivalent tools.
• Translate data into clear insights, recommendations, and actionable improvement initiatives.
• Track key performance indicators and identify areas requiring further investigation.
• Knowledge & Automation Improvement
• Identify knowledge gaps that contribute to AI failures or poor user outcomes.
• Recommend improvements to knowledge articles, categorization, intents, and automation coverage.
• Provide structured feedback to Knowledge Management, Automation, AI, and Service Desk teams.
• Support testing and validation of new or enhanced AI and automation capabilities.
• Help ensure knowledge and automation solutions reflect actual user needs and Service Desk demand patterns.

Service Desk Operations

• Handle incidents, service requests, and user support inquiries as required.
• Participate in Service Desk queue management and operational activities.
• Apply standard ITSM processes when handling incidents and requests.
• Escalate issues appropriately and maintain accurate ticket information.
• Use operational experience to identify recurring issues and opportunities for AI or process improvement.

Continual Service Improvement

• Identify recurring Service Desk and AI-related issues and recommend improvement actions.
• Participate in service improvement initiatives and working groups.
• Monitor the outcomes of improvement initiatives and provide evidence of their effectiveness.
• Promote a culture of data-driven, user-focused continual improvement.
• Share insights and trends with relevant stakeholders to support informed decision-making.

The Requirements

Education & Experience

• At least 2 years’ experience working in an IT Service Desk, IT operations, or ITSM environment.
• Practical understanding of incident and service request management.
• Experience with ServiceNow or a comparable ITSM platform is preferred.
• Experience analyzing operational data, ticket trends, or service performance is preferred.
• Experience working with AI-assisted support, chatbots, automation, or conversational AI is advantageous.

Technical & Analytical Skills

• Strong analytical and problem-solving capabilities.
• Good working knowledge of Microsoft Excel, including data analysis and reporting.
• Experience with Power BI or similar business intelligence and visualization tools.
• Ability to interpret data, identify patterns, and translate findings into actionable recommendations.
• Basic understanding of conversational AI concepts, including intents, fallback scenarios, escalation, automation, and knowledge-driven responses.
• Understanding of ITSM principles and Service Desk workflows.

Service & Business Skills

• Strong customer and user service orientation.
• Excellent attention to detail and ability to identify recurring patterns and anomalies.
• Ability to understand both technical and business perspectives.
• Strong written and verbal communication skills.
• Ability to present analytical findings clearly to both technical and non-technical stakeholders.
• Strong collaboration skills and ability to work across Service Desk, AI, Automation, Knowledge Management, and other technology teams.
• Ability to manage multiple priorities and work effectively in an operational environment.

WTW is an Equal Opportunity Employer

-