Improving business resiliency with closed-loop IT event management

Automotive companies have substantial design and research divisions that are heavily dependent on their IT infrastructure. And nowadays, manufacturing also depends on IT support for its precision, automated machinery. So a high-tech manufacturer cannot afford to have any IT disruptions that could affect its operations.

The customer has a vast IT infrastructure landscape to support its various manufacturing and back-office functions. To manage the landscape effectively, the IT command center team had installed multiple monitoring tools to alert them of any events (unexpected changes) or problems. This system was expected to keep all technology components running smoothly and maintain high uptime.

However, “eyes on screen monitoring” of all the events in multiple dashboards from individual monitoring tools was overwhelming for the command center team, with the monitoring tools generating approximately 50,000 alerts every month. Each alert required generating a follow-up ticket, a manual process that took 15-20 minutes. Even worse, only 20% of the alerts met the criteria for “incidents” (actual problems).

The team had to spend so much time tackling false positives that they missed some incidents. This caused substantial downtime and loss of service efficiencies. To manage the barrage of alerts, the company had to expand the command center team, increasing its overhead costs.

First, ignio™ AIOps created a comprehensive IT landscape blueprint for the customer. This provided the customer more visibility over its IT landscape and how it was connected and configured. ignio was then deployed to alleviate the alert load from the command center team so they could focus on actual incidents. It seamlessly integrates with the customer’s existing monitoring tools to provide an enterprise dashboard with a single-pane view of all alerts in real time, for zero-touch operation. ignio now manages at least 83% of the customer’s IT alerts.

ignio streamlines event management in several stages, starting with rule-based suppression of false alerts. Then it draws on AI-driven pattern-matching capabilities to spot duplicates of legitimate alerts and delete them. As a result, it has significantly reduced the alert noise through suppressing more than three-quarters of the false positives and intelligent alert correlation and aggregation, helping the command center team focus on the right issues.

To speed up handling of the remaining, genuine alerts, ignio automatically creates the trouble tickets in the customer’s third-party IT service management tool and intelligently routes them to the appropriate resolver group. It enriches the tickets with contextual information that will aid faster resolution.

ignio even resolves some incidents autonomously, analyzing the issues to determine their root cause and leveraging its pre-built knowledge library to find an appropriate solution (such as restarting a down server or redistributing data processing loads to reduce CPU utilization). Over time, the share of autonomous resolutions has increased as the customer has seen how successfully ignio can fix problems by itself.

ignio AIOps has shrunk the mean time to resolve an incident (MTTR) by up to 95 percent. Overall, the command center team was able to save more than 50 man-hours of work a day, for a total of about 18,300 hours saved a year. This freed up the equivalent of at least six full-time staffers for other duties.

• 76% of false alerts suppressed
• 35% true alerts resolved
• 95% reduction in MTTR
• 18,300 hours of command center effort (equal to six FTE) reduced in one year