Improve system agility, security, and operational efficiency with noise suppression, quicker incident resolution, and automation

Retail chains with a heterogeneous business model and a large number of locations rely on their IT backbones to ensure information is shared seamlessly throughout the whole system. Any issues in the IT infrastructure can result in downtime, which interferes with real-time transactions and in turn the customer experience. Moreover, retailers serving leisure needs face stiff competition in the post-COVID environment for consumers’ discretionary spending, which increases the pressure to reduce their operational costs and delays. This includes time spent onboarding and off-boarding employees, a sizable chore for the retail sector, which has frequent turnovers in its workforce.

The customer’s back-office operations rely on a wide range of applications, including order processing, inventory records, human resources, and password management. The entire IT estate is managed by a command center operations team that would collect alerts system-wide from a variety of monitoring tools, then manually assign them to respective stakeholders to resolve. These manual processes created delays in problem detection and resolution. This in turn led to operational challenges such as inaccurate inventory or downtime for online shopping portals.

Adding to the IT command center’s workload were thousands of requests a year for routine services such as JML (joining, moving, leaving) processes, where system credentials were issued, updated, or canceled, or helping employees who forgot their system passwords. The customer wanted a solution that would ensure maximum system uptime, while also reducing the IT administrative burden through easier issue management and resolution.

THE PROBLEM – Manual processes generating hundreds of IT requests a day

The customer’s monitoring tools generate over 10,000 alerts per month. These alerts flow to the IT command center team, who must check each one and rule out false positives or duplicates. If they find that an alert is genuine, they need to create an incident ticket and send it to the right stakeholder for resolution, which they need to track to ensure timely closure. Moreover, they must continually follow up to ensure the most critical alerts are prioritized, as well as monitor the website’s performance. In addition, a substantial share of the IT command center team’s workload stemmed from routine, time consuming tasks that didn’t require as much judgment.

First, the team had to manually reconcile discrepancies between a point-of-sale application tracking merchandise levels at stores and the centralized SAP ERP system, which the client uses to keep overall inventory records. This involved tracing individual product SKUs in the sales database and manually updating corresponding records in the SAP system. Second, they received about 300 requests a month for JML services. These entailed manually providing, updating, or removing access to email, timecards, and important business applications. JML requests had to be handled swiftly to minimize the waiting period for new employees or the risk of unauthorized system access by outgoing staff. Each request took the IT team at least 30 minutes, sometimes as much as four hours.

In addition, the team received over 2,000 routine help calls per month from the staff. Almost half of those calls were simple requests to reset system passwords; doing so represented nearly 20 percent of the team’s workload.

These repeated, mundane tasks pulled the command center team’s time and attention away from more pressing matters and caused fatigue.

ignio AIOps quickly integrated itself with all the monitoring tools to analyze the retailer’s historical operations data and gain an “understanding” of what valid operations look like. Whenever alerts are generated, ignio uses its machine learning capabilities to rule out false positives so only the genuine alerts proceed.

ignio’s event management capacity creates incident logs of the alerts in the system and ensures no duplicate incidents are created. Then it routes the alert to the right expert on the team for a resolution via email. So far, the ignio AIOps solution has reduced noise from false or duplicated alerts by 60 percent. It has shortened Mean Time to Respond (MTTR) by 80 percent, from about 20 minutes to under 5.

Moreover, ignio tracks unique order failure and status of critical jobs, and it monitors failed jobs and retriggers them to resolve the issue. ignio also monitors the customer website to ensure it is functioning properly, thus reducing the command center team’s operational load.

ignio automation seamlessly took over the staff onboarding and offboarding process; this alone reduced the team’s workload by 13%. It can now execute access provision and revocation requests within minutes. The customer was able to mobilize its workforce more efficiently, with less fatigue, especially during pandemic-induced operational shifts. Moreover, ignio now automatically closes departing employees’ accounts by midnight of their last day, rather than the next workday. This reduces the client’s security risk exposure due to ex-employees.

To streamline password resets, the ignio team created a chatbot-based, self-service solution. This eliminates at least 70% of the reset requests that used to flow to the IT command center. Plus, the new system is more secure. Previously, resets were validated only by confirming the requester’s name in a phone call. The new chatbot automatically sends a one-time password to the requester’s mobile phone, which the user must authenticate before getting the requested reset.

To address the inventory problem, ignio now connects to the customer’s store database and automatically extracts current inventory records to be updated in the customer’s ERP application. The team no longer has to handle it. Now, more than 40% of the IT processes that the command center manages are automated by ignio. The retailer estimates that ignio has saved the team more than 90 person-hours of work a month on mundane, repetitive tasks. On top of this, all employees with JML requests or other routine IT needs have greatly reduced their unproductive waiting time, giving the business more agility.

• 80% improved Mean Time to Respond(MTTR)
• 50% alerts deduplicated
• 60% reduction in overall noise
• 90hrs of manual effort saved per month