“The pandemic uncovered flaws in our original DR plan. With the support and financial investment from senior management, combined with the hard work and dedication of our front-line employees, CSG was able to meet the needs of our customers. The mail got out.” — Tommy York, CSG

Before the pandemic many print-mail service providers had what they thought were solid disaster recovery (DR) plans. These plans included contracts with national and local firms to meet client SLAs in the event of a disaster. Files were transmitted and paper stocks were held in inventory in the event a facility could not produce client statements and invoices. These plans were tested on an annual basis. Clients received reports of successful DR testing.

Across the industry, companies prepared for disasters as localized events. DR plans defined operational procedures when major storms would take out power to a facility or blocked roads would prevent staff from getting to work. DR plan activation would mean centralized data files would move to a non-impacted facility and production would continue from a different location. Similar plans with regional and national suppliers have worked for 30+ years to support print mail service providers meeting contractual obligations to get the mail out. Early in the pandemic I suggested it was time to reevaluate business contingency and DR plans. I wrote, How’s your Company's Contingency Plan Working?

The Mail Must Go Out

CSG is a global service provider. Tommy York, VP of Strategic Business for the Output Solutions group, shared that CSG had a well-defined and tested DR plan. Their plan included both internal recovery plans within their three facilities, supplemented by a contract with a national DR supplier. When the pandemic hit, they informed their DR supplier that they wanted to activate the plan and move some work as a precaution.

Then CSG had a production employee test positive and closed a facility briefly for deep cleaning. The DR supplier was overwhelmed by simultaneous requests from other clients. This meant that the DR supplier could only accept less than 1% of the daily volume they had contracted to accept from CSG. The need to move large volumes of work far exceeded any plans for DR support.

CSG’s operations team quickly moved into action to develop alternate plans. They knew they had to expand their internal DR capacity and mitigate risk for their clients. They bought additional inserters and planned changes to the production facilities to accommodate additional work and contingency planning.

Lessons Learned:

Disasters are regional. Pandemics are global.

The industry model for outsourced DR vendors for business continuity plans could not support multiple clients and the large-scale shutdowns that occurred during a pandemic. No matter how well-written and how detailed the contracts service obligations, third-party vendors could not support multiple clients simultaneously.

Communicate, Communicate, Communicate

Communicate often to employees and customers. Business needs, health and safety concerns, and protocols were changing rapidly last year. Ongoing and consistent communication about practices and procedures keep employees, customers and key suppliers informed. Straight forward communication provides peace of mind and creates alignment within teams. Ask for feedback from all interested parties. CSG found employees and customers provided excellent feedback as the business situation was changing.

Be Flexible

CSG defined a plan to pre-stage capacity and have dedicated DR equipment at each production facility. They made significant investments to support an internal DR plan. Good planning takes time. Ordering, installing, and configuring new equipment took several months. Planning for inventory and defining DR workflows for client specific applications had to be tested and documented.

Employees are your most valuable asset

Take care of your employees. Do everything you can to make sure they are safe and healthy. In addition to ongoing communications, CSG provided multiple incentives to employees during the last 18 months. The leadership team acknowledged how hard they were working in demanding situations. CSG provided multiple financial incentives including a one-time increase in hourly wages as well as other incentives to employees.

Define expectations

CSG had a variety of SLA obligations, including DR terms and conditions across their customer contracts. After expanding their internal DR capabilities to cover all clients, CSG is in the process of updating customer contracts with consistent expectations for DR.

DR used to be a boring operational topic that was dictated by contract language and tiresome annual testing. COVID was the ultimate test for all DR planning. It revealed the many flaws in a third-party service model for contingency planning. I hope your DR plans are shiny and new. And include all the lessons learned in the last two years. Need help in defining DR plans that work? Please reach out at: lritarossi@highrockstrategies.com