A step by step guide to AI maturity in IT operations

Stage 1: Use of AI to understand normal behaviour

Analysis of historical data can help understand many interesting behavioural aspects of the IT operations. Understanding the normal behaviour of workload, performance, availability, capacity, and other such metrics lays the foundation for doing any advanced analysis. We divide this stage into two sub-stages.

Stage 1(a): Profiling of the individual behaviour of entities.

This stage is about understanding the normal behaviour of individual entities such as applications, middleware, servers, network devices, and so on. Such analysis helps to create a baseline of the behaviour of different components. An accurate understanding of this baseline is crucial as it forms the foundation for sophisticated analysis such as noise suppression, incident triage, business KPI predictions, change-impact analysis, and so on. Furthermore, this baseline itself points to several insights to help better understand the changes, anomalies, trends, and patterns present. Below are some questions to assess your maturity in this stage:

• Do you collect metrics and events data and perform basic statistical analysis on it?
• Do you derive changes, trends, patterns, and outliers in metrics such as CPU utilization of server, response time of URL, reads/writes of disk, and so on?
• Do you derive normal behaviour thresholds in these metrics?
• Do you derive anomalies in these metrics?
• Do you detect frequent, recent, and persistent events such as server not accessible, or URL failure, or service not responding, etc?
• Do you derive temporal patterns in these events?

This stage of analysis can leverage classic statistical techniques of data distribution analysis and probability analysis. Various data mining and machine learning solutions also prove extremely useful such as pattern mining, anomaly detection, trend, and seasonality analysis, clustering, and classification.

Stage 1(b): Profiling of the collective behaviour of entities

The previous stage analyzed each entity in isolation. However, many interesting insights present themselves by doing a collective analysis of multiple entities. This stage focusses on analyzing how multiple entities impact each other. Such analysis helps in mining multivariate anomalies which otherwise go unnoticed by looking at each metric in isolation. Saptio-temporal analysis helps in mining problem signatures that cut across tech stack. These signatures are very useful for various use cases such as problem management, root-cause analysis, alerts aggregation and prediction, change impact analysis. Below are some questions to assess your maturity in this stage:

• Do you detect events that co-occur? For example, servers that go down together, or filesystems that fill up together.
• Do you detect cause-effect signatures in events across the tech stack? For example, high server CPU utilization leading to high URL response time.
• Do you analyze multiple metrics to detect complex anomalies? For example, detecting anomalies by combined analysis across workload and performance metrics.

Graph mining solutions in addition to correlations and association rule mining offer powerful levers to mine such signatures. Statistical and AI/ML levers such as auto encoders, clustering algorithms, Bayesian networks, and Markov models help in developing creative solutions for this analysis.

Stage 2: Use of AI to derive systemic insights

After establishing a baseline of behaviour profile, the next level of maturity is to derive systemic insights. These insights are derived with a specific objective such as capacity planning, risk assessment, cost optimization, and so on. Based on the objectives, this stage can involve multiple substages. Below we present some examples of these insights.

Stage 2(a): Capacity analysis

Insights derived in this stage can help prevent capacity bottlenecks and capacity-induced risks. They also help in preventing capacity wastage and reducing spend waste. By regularly assessing cloud capacity and promptly detecting spend anomalies, businesses can ensure efficient resource utilization, control costs, and maintain a healthy balance between performance and expenses in their cloud infrastructure. Below are some questions to assess maturity in this stage.

• Do you identify candidates for resource augmentation?
• Do you identify candidates for resource rationalization?
• Do you assess cloud capacity and detect spend anomalies?
• Do you recommend opportunities for cloud spend optimization?

Analytics levers of clustering, classification, and anomaly detection can help detect capacity and spend anomalies. Forecasting, simulation modelling, genetic algorithms, and multi-objective optimization solutions prove useful to derive capacity optimization recommendations.

Stage 2(b): Risk assessment

These insights assess risk across IT operations and identify opportunities for problem management and prevention. Identifying high-risk entities helps organizations proactively mitigate potential risks and improve the overall resilience. Risk mitigation is accelerated by mining problem signatures, identification of the root cause of recurring problems, and enabling targeted solutions for long-term resolution. Recommendations for corrective actions for recurring issues empower enterprises to address problems swiftly and minimize their impact. Below are some questions to assess your maturity in this stage:

• Do you identify entities at risk? These are entities that are observing frequent problems and that are impacting critical business functions. For example, high filesystem utilization leading to database performance problems and application unavailability.
• Do you detect recurring problems and mine their problem signatures?
• Do you derive root-cause of recurring problems?
• Do you recommend corrective actions to eliminate recurring problems?

Anomaly detection, fault propagation models, event correlation, Markov chains offer effective solutions to mine problem signatures and assess risks. AI/ML techniques such as causal inference, neural networks, decision trees, SVMs, Bayesian networks can be employed to perform root-cause analysis, while multi-objective optimization, change-impact analysis, simulation modelling, as well as augmented intelligence offer effective levers to recommend corrective actions.

Stage 2(c): Change analysis

Analysis of change events and behavioural changes lead to many interesting insights to assess effectiveness of change management. Change-impact signatures help in mining insights into what caused the change and what was the effect of change. Detection of change-induced risks helps proactively address issues to maintain system stability and prevent any inadvertent effect of changes. Another important aspect is that the organizations need to find the optimal time to implement changes in their estate to minimize disruption and ensure efficient change management. Planning changes often require capturing various constraints and preferences to recommend a plan. Below are some questions to assess your maturity in this stage:

• Do you detect behavioural changes in entities and correlate them with reported change events?
• Do you derive change-impact signatures across entities (application, middleware, database, server, storage, network)?
• Do you detect change-induced risks?
• Do you recommend right time to plan changes?

AI/ML techniques such as fault propagation models, simulation modelling, graph mining, hidden Markov models, correlations and regressions can aid in detecting change-induced risks. Techniques such as linear programming, genetic algorithms, and multi-objective optimization best help in recommend plans for introducing changes.

Conversational intelligence

Unless the AI-driven insights are made explainable and accessible to the end users, the adoption of these insights will always face challenges. Hence, only deriving insights is not enough. It is essential to make them easy to access, understand, and consume. Hence, conversational intelligence is an important aspect of this stage. It focuses on simplifying the consumption of AI-driven insights for the operations and business users. Below are some questions to assess your maturity of conversational intelligence:

• Are you addressing insight fatigue and helping the users easily find their insights of interest?
• Are you making the insights explainable? Are you providing textual and visual evidence to explain the insight and the underlying reasoning process?

Data storytelling can provide effective levers to enable conversational intelligence. Identifying insights of interest based on user preferences and persona, creating chains of insights to form the bigger picture, creating objective-driven reports and dashboards prove to be some powerful ways to make insights accessible to the end-user. Advances in generative AI have also simplified the creation of conversational experiences for the end-users.

Stage 3: Use of AI to operationalize insights

So far, we have been using AI/ML levers to mine insights from historical data. The next stage makes use of these insights and operationalizes them in practice. Based on the use cases of interest, this stage can involve multiple substages. Below we present some examples of these insights

Stage 3(a) Event management

These insights help to improve event management by identifying opportunities to suppress false alerts, group related alerts, and prioritize alerts. Failing to suppress irrelevant alerts or aggregate related alerts can cause alert overload, leading to fatigue among operators. This can result in missed critical alerts, delayed response to genuine issues, reduced efficiency, and increased risk of overlooking important system problems within the flood of unnecessary notifications. Neglecting alert prioritization in enterprises leads to delayed response to critical threats, and increased vulnerability to security breaches. Below are some questions to assess your maturity in this stage:

• How do you suppress false alerts? How are the thresholds defined to generate alerts? Are the thresholds manually set or dynamically computed based on behavioural patterns present in the data?
• How do you filter irrelevant alerts? How are the filtering rules defined? Are these rules manually set or dynamically computed based on the data?
• How do you aggregate related alerts? Are these rules manually defined? Are these rules computed based on spatio-temporal correlations present in the data?
• How do you prioritize alerts? Are these rules manually defined? Are priorities automatically derived based on the impact of alerts?

Relying solely on human-defined rules is often not effective because of limitations in capturing intricate patterns, scalability challenges, and inability to adapt to changes. AI and ML techniques such as anomaly detection, pattern mining, machine learning classifiers, fault impact analysis, and so on. can provide effective tools to suppress, aggregate, and prioritize alerts and thus reduce alert fatigue.

Stage 3(b) Incident management

These insights help to reduce incident resolution time by assisting in triage and resolution. Diagnosing and resolution of incidents is crucial for organizations to swiftly identify, address, and rectify issues, minimizing service disruptions, maintaining user satisfaction, and safeguarding overall system reliability. Below are some questions to assess your maturity in this stage:

• How do you diagnose an incident? Do you rely primarily on pre-coded diagnostic procedures, or are the diagnosis procedures auto derived by learning the context of the incident?
• How do you resolve an incident? Do you rely primarily on pre-coded resolution procedures, or are the resolution procedures auto derived by learning the context of the incident?
• In case of manual resolution, how do you identify the resolver to assign the resolution to?

Relying solely on pre-coded diagnosis procedures may not cover all possible scenarios or adapt to evolving issues. Similarly, pre-coded resolution procedures lack flexibility, struggle with novel issues, rely on historical knowledge, and hinder innovative problem-solving, making them inferior for addressing diverse or evolving incidents.

Real-time insights and adaptive analysis using AI and ML techniques are essential for effective diagnosis, as they enable a comprehensive and accurate understanding of incidents, thereby improving the efficiency and thoroughness of the troubleshooting process.

Collaborative learning

The effectiveness of this stage can be significantly boosted by collaborative learning which focusses on leverage the intelligence of domain experts to review, validate, and enhance the knowledge of the AI engines. Below are some questions to assess your maturity in collaborative learning:

· Are you bringing human-in-the-loop to learn from human experience to understand new, unknown, and exception scenarios?

· Are you bringing human-in-the-loop to review and validate AI-driven decisions?

· Are you separating generic factual knowledge from customized situational knowledge?

· Are you modelling the information provided by human experts to create wider reusable knowledge?

Knowledge modelling is an essential component to enable collaborative learning. Feedback-based learning and reinforcement learning can help to effectively capture the user-feedback. RAG and fine-tuning of large language models, both aid in adapting pre-trained LLMs for specific tasks and domains.

Stage 4: Use of AI to derive transformation

In the previous stages, AI/ML techniques are used to derive insights by analyzing the past and present. The next stage is to use AI to derive transformation plans. This is where the predictive and prescriptive capabilities of AI/ML are dominantly used.

Stage 4(a) Predict future behaviours

These insights help predict future behaviour and generate early warning signals to take preventive actions. Not predicting future events can harm an enterprise by leading to unpreparedness for upcoming challenges and missed opportunities. It results in being reactive instead of proactive in decision-making, and thus impacting competitiveness and growth. Similarly, neglecting prevention for predicted events increases risks and misses opportunities for proactive mitigation. Below are some questions to assess your maturity in this stage:

• How do you predict future events? What type of events can you predict? How early can you predict these events?
• How do you recommend prevention of a predicted event? Do you consider cost, benefits, risks, and side effects while generating the recommendations? Do you adapt the recommendations based on user feedback?

AI/ML levers such as pattern mining, classification trees, hidden Markov models, and neural networks aid in predicting future events for enterprises. Techniques such as multi-objective optimization, linear programming, and genetic algorithms provide effective ways to recommend prevention for predicted events.

Stage 4(b) Plan for growth and change

Enterprises need to assess the impact of business or technology changes to foresee potential disruptions, optimize decision-making, and ensure a smooth transition. The insights in this stage help to plan for such technology or business changes. It offers information to derisk the change management process. Below are some questions to assess your maturity in this stage:

• Can you assess the impact of technology or business change?
• Do you recommend prevention of any adverse effects of change?
• Do you recommend when and how to execute the change?

AI/ML techniques, including change scenario analysis and predictive modelling, can be employed for what-if analysis, providing insights into potential outcomes, and helping organizations make informed decisions regarding changes.

Conclusion

The long-term impact of AIOps is going to be transformative for the IT operations. However, implementing AIOps requires a systematic strategy to realize the value of AI in IT operations. Each organization has its own AIOps journey. However, a step-by-step journey towards AI maturity helps organizations better understand what AI has to offer to IT operations, and how to best implement AIOps to achieve the desired level of AI maturity.