Serving customers over digital channels has been a growing priority for financial institutions for many years, and the disruption of the COVID-19 pandemic and social distancing measures has only increased the importance of online services via website and mobile applications. For financial institutions, it’s no exaggeration to say that ensuring excellent customer and employee digital experiences has never been more important.
At a recent ThousandEyes EMEA financial forum, we heard from Peter Grobler, Head of Managed Network Services, Old Mutual Limited, about how Old Mutual transitioned to remote working at scale. He also outlined what his team is doing to ensure their IT operations are well positioned to continue supporting the needs of the business in a post COVID-19 world. In this blog, we’ll share a summary of his presentation.
Old Mutual Limited is a premium African financial services group that offers a broad spectrum of financial solutions to retail and corporate customers across key markets in 14 countries. The company employs 30,000 people and operates in 14 countries across Africa and Asia. Peter commented in his presentation, “Because we operate across many countries, it brings about many challenges in how people consume applications. My team is responsible for Managed Network Services. From a service delivery assurance point of view, we are accountable for connectivity services – for branch, campus and cloud on ramp – as people consume apps or services via SaaS – both internally and externally. Our aim is to provide information to our business and support teams.”
COVID-19 Changes the IT Paradigm
Pre-COVID-19, Peter and his team monitored and reported on IT assets that they controlled. Old Mutual’s IT monitoring and tooling was focused on “inside out” with the biggest focus being on driving a predictable, improved user experience. The infrastructure team had control of “what” was provisioned, “who” could use assets, as well as “how” these resources were managed. The arrival of COVID-19 would disrupt the status quo.
Peter explained, “The arrival of COVID-19 turned our world upside down—we realized we weren’t in control. We started to realize that we had no visibility into staff members’ home networks, the ISP(s) and devices they use. And because staff decide on what they use and how they provision … and for connectivity mediums we were seeing 3G, LTE, ADSL, Fiber—our inability to diagnose issues became a real challenge.”
Peter continued, “Before COVID-19 arrived, Old Mutual had circa 2,000 employees working from home/remotely, but this leapt to 8,000+ employees overnight. Our use of Office 365 and Microsoft Teams was around 2,000 employees pre-COVID-19. We have recently recorded 14,000+ employees actively using it now throughout the day for their collaboration needs.”
Because of social distancing restrictions, Old Mutual also had to move all call center employees to a working-from-home model. This presented Peter’s team with another challenge. Peter recalled, “Staff members working from home started to get very frustrated. They would say, ‘I am working from home and I can’t log into the remote desktop,’ ‘I am unable to authenticate,’ and ‘I am not receiving emails,’ or ‘my Microsoft Teams calls have erratic performance.’ Some employees wanted to know if Microsoft was having an issue, while others wondered if their Internet was the problem. This led to a lot of speculation about Internet connections—whether they were slow or fast. People were experiencing extremely erratic calls, while some experienced VPN issues. Some of our staff in the call centers complained that they couldn’t hear our customers. This all impacts our ability to service our customers.”
Peter predicted that even in a future post-COVID-19 environment these issues will still be around: “We will still have a big reliance on the Internet. Our Cloud and SaaS solutions need to connect with service branches and home-based users. This also includes brokers and the customers’ ability to interact with Old Mutual applications consumed via Internet and hosted environments such as our colocation data centers, or our AWS and Azure services.
He believes that it is critical his team can accurately pin-point performance and usability issues: “This is business critical and a key enabler to ensure proactive measures can be taken by the infrastructure management, end user support and AppDev teams. They need to use the info available and make informed decisions, ideally before users complain.”
The need for this visibility was why Peter turned to ThousandEyes.
ThousandEyes Drives Visibility Across Old Mutual Limited
Peter explained the team’s approach to deploying ThousandEyes capabilities: “We use the ThousandEyes Cloud Agent to do availability tests on VPN environments. We’ve also started using cloud tests for AWS in our African locations. We’ve also been doing page load tests for our key web applications. We’ve started using synthetics tests starting for our O365 applications—sending and receiving emails and popping messages into Teams. This is good insight into how the environment operates.
We’ve also set up tests for our end user ISPs—plus the key ISPs Old Mutual uses for last mile Internet. We’ve replicated many of these tests across the organization. And we are also using testing metrics that we can gather on our internal networks. This is helpful for our branches where and when issues pop up.
We’ve set up and deployed 1,600 Endpoint Agents to date, and these are mostly in the call center space. These help us to understand a home network infrastructure’s ability to work—what the performance of the device looks like, and how the local ISP impacts the agent’s ability to do their work. We use ThousandEyes agents as a collective to give us an impression of how Old Mutual’s last mile ISP providers operate.”
The deployment of the endpoints was a quick process—all 1,600 agents were deployed within a couple of days. The team opted for a zero-touch approach, wherever possible. Most of the rollout was done in an automated fashion with software being installed on users’ machines outside of business hours. Only a select few staff members were encouraged to download their agents manually.
To address the issues facing contact center staff, the operations team ran a contact center device remediation exercise. Peter explained ThousandEyes role in the initiative: “ThousandEyes enabled us to identify issues on laptops where resources were under the minimum requirement of CPU and RAM. In relation to RAM performance, we identified that out of 287 devices, 232 were remediated based on ThousandEyes data. For CPUs, we found that 259 devices had sub-par processor performance. We set up dashboards that show us which users have issues, and we also set up dashboards for specific business segments.”
This breadth and depth of data provided by ThousandEyes helped the team isolate critical issues.
Peter then walked through the team’s remediation process, “Once we have a view of physical device health, then we look at local gateway loss. In some cases, we still had some complaints about apps not working. A user reporting issues might initially show good throughput, so we’d test the ISP network and use an FTP server to download a file. Then we look at average throughput. Here we can see that this staff member is seeing 20-27 megs on average, but it’s dropping to 500 kb. This was cause for concern, as through a wired connection at their house, using their ISP, we should expect sustained high throughput and low latency.”
Peter adds, “When we stepped through the timeline to unpack where things went wrong, we found that the staff member was on a wireless connection via their iPhone.”
This new depth of visibility has been a game changer for Peter’s team, “The nice thing with this is that we can start to use this type of information in our action centers. Now, when someone has an issue, we can have a real-time conversation with them. We have insight we never previously had, whereas before we had to just speculate about what the issue might be.”
From Insight to Resolution
For call center teams, the quality of voice on their applications is important. The infrastructure team started getting a lot of complaints that call center employees were experiencing erratic behavior from their voice applications. Peter explained the team’s approach to monitoring voice, “This is always a challenge, particularly through VPN. We set up smart tests from Endpoint Agents to Old Mutual’s edge via the handoff to our ISP backhaul. Through this, we can have a view of sustained performance over time. Our voice was starting to degrade, and we could see an event via ThousandEyes where our loss went up to 100%, and this was happening across 1,000 agents. We used this information to engage our ISP and they confirmed that this ISP had an issue on their backhaul.”
Having this level of rapid clarity has allowed the business to move forward, and reduced stress for the IT team. Peter concluded, “This is brilliant for us. Previously in our action centers you would start a conversation, get the partners together, and the issue disappears. Then you have to close the incident without a route cause, it causes a lot of frustration.”
At the conclusion of his presentation, Peter highlighted several organizational benefits the company has experienced due to ThousandEyes capabilities. ThousandEyes has become one of the “go to” tools in Old Mutual’s action centers. Peter explained how it helped the infrastructure team to “short-circuit” the previous long times it took to find the root cause of issues, “we can pull and walk through live data for specific user issues and our infrastructure teams have become proactive—they now have the ability to engage with users to unpack real-time issues. Our support teams are also empowered because they can leverage focused dashboards that are relevant to their areas of accountability.”
Improving Microsoft Teams Performance
Peter has also used the solution to investigate issues he’s experienced directly: “I was on multiple Teams calls, and I was consistently experiencing degraded Teams performance. Across the agents we use, there was a degradation issue. With path visualization we were able to see that this was a Microsoft issue, not an ISP issue. This brought us insight we never had before, and I was able to communicate the cause to our executive.”
Resolving Service Issues at the Service Desk
The team also resolved issues that were impacting the performance of its Service Desk, which was hosted out of the UK. Users were experiencing issues connecting via the URL and the desktop. ThousandEyes data identified that the issue was caused by a node in the UK. Armed with this information, the service provider was then able to have the right conversation internally to resolve the problem. Peter summarized, “Being able to visually show the issue brings a level of clarity that we did not have before. This helped us to redirect the issue, and we wouldn’t have been able to do that previously. Resolution time would have been much longer.”
Solving ISP Transit Issues Impacting AWS Performance
Another issue that was quickly averted was AWS service degradation between Europe and Africa. ThousandEyes Cloud Agent tests showed loss and routing loop issues between Ireland and Malawi. Peter stated, “This data enabled the team in Malawi to have the right level of insight. They then engaged with their local ISP and flagged that the issue was being caused by the IP transit provider they were using. Having visibility down to this level is invaluable, previously we would have speculated on the cause. Now we don’t have to speculate.”
Looking Ahead: Expanding Visibility Even Further
Peter concluded with a view of how Old Mutual Limited plans to leverage ThousandEyes in additional areas, “Our journey will continue. We have a strong drive to move more workloads to AWS. We will also set up SIP and RTP monitoring via Enterprise Agents for our contact center. The ThousandEyes Synthetic tests showed considerable value for key OML applications, and in the near future, we are looking to monitor additional activity, not just page loads. We also have a need for enhanced reporting so the ThousandEyes professional services team is helping us integrate Power BI reporting. A key focus for us is to get that all-encompassing view for all our SA ISPs and how our staff use them.”
If you would like to find out how ThousandEyes can help you ensure your customers and employees receive a smooth digital experience, we’d be happy to arrange a demo.