NCIS goes live this weekend (part 1 of 3)

This coming weekend, while we are enjoying the Labor Day holiday, Seattle City Light (SCL) and Seattle Public Utilities (SPU)  plan to finally launch their late, over budget NCIS billing and customer service system.

This is the first installment of a 3-part series looking at NCIS, the new customer service and billing system for Seattle Public Utilities and Seattle City Light. This post looks at what will happen during the rollout, what the utilities’ customers will see, and what happened over the last six months to get to this point.

So here’s how this goes down:

  • Wednesday evening (August 31), the utilities take a final backup and snapshot of the old billing system.
  • Thursday and Friday, the IT staff build the new NCIS system and convert over as much of the old system’s data as can be done automatically. While this is happening, SPU and SCL staff are manually processing the things they would ordinarily do in the billing system.
  • Saturday the remaining data that couldn’t be converted automatically will be taken care of manually. Saturday night, the first “batch run” of bills, reports, and other overnight processing will take place on the new system.
  • Sunday and Monday, an army of SPU and SCL staff will come into the office and enter into the new system everything they did manually on Thursday and Friday. While that is happening, the IT staff will put the finishing touches on the new NCIS system and make final checks.
  • Tuesday morning, as SPU and SCL employees return to work, the NCIS system goes live.  The first bills from the new system also get mailed out that day to be delivered to customers later in the week.

There will be a series of checkpoints over the course of the week-long effort, where a “no-go” decision can be made and everything can be rolled back if something goes wrong.

Now if everything goes right, customers will notice little change. Our bills will be almost identical; the only difference will be that we all get new account numbers. If you pay your bill by cash, check, or using the autopay system that SCL and SPU offer, nothing changes. However, if you pay your bill online using your bank’s bill pay system, then you will need to update the account numbers you gave the bank. Your old account number will continue to work for 130 days, though, so you have some time to change it. Your next bill will have an insert reminding you of all this.

SPU and SCL have spent the last six months working their butts off to ensure that this goes smoothly. Their motivation comes from the nightmare that happened in Los Angeles when it rolled out its new billing system three years ago.

The Los Angeles Department of Water and Power launched its new billing system in 2013; originally projected to cost $87 million, by the time it launched that number had risen to $181 million. It suffered from many of the same problems that Seattle’s NCIS project faced: over-ambition, poor software development practices, and a lack of transparency on ongoing status. But for some unfathomable reason, when the Los Angeles project got to the stage that Seattle was at last winter (a lot of bugs, much more testing to do, and an under-trained staff) their senior management decided to launch anyway. And it was exactly the disaster you would expect. The system performed so poorly that in the month following launch the utility was forced to issue 137,000 “estimated bills” because accurate ones could not be produced on time. Three months later, it was issuing $100 million in delayed bills per month. Then, as the number of past-due accounts accumulated, it made the decision to limit the number of accounts sent to collection so as not to overwhelm its call center (customers sent to collections are known to call at a very high rate). By September 2014, one year after the launch, the LA water and power utility had over $700 million in bills awaiting payment, of which $205 million was more than more than 90 days past due (in July of 2013, before the switchover, there was $436 million total, with $42 million of that 90+ days past due).

LA bills awaiting payment

Since then, the utility has regularly fiddled with its rules for sending past-due accounts to collection to try to draw down the total amount without drowning its call center.  It isn’t working.

LA call center queue

You can read the full California State Auditor’s Report on the Los Angeles system here.

So while Seattle City Light and Seattle Public Utilities dug themselves a deep hole through the implementation phase of the NCIS project, just as Los Angeles did with theirs,  they are attempting to redeem themselves by forging a different path for the validation and launch phase. They feel confident that they have fixed the known bugs, that they have tested the system under the load that is expected when it is in production, and their staff have been well trained and had time to practice.

They have a bit of an advantage in their load testing compared to a service that Amazon, Microsoft or any other web company might produce because their load comes from two main sources:  the highly consistent batch processing that runs every night to create 1/60 of the utility’s bills (bills are sent out once very two months, and customers’ bill dates are spread evenly across the period); and the users of the system, who are mainly the employees of the utilities themselves. Both of those loads are relatively easy to simulate: they run batch processing tests every night with a load larger than a typical night’s on the production system, and they have run load tests with real users on the test system. So their confidence is appropriately high.

They have also run practice simulations of the cutover to the new system, ensuring both that their staff know what to do and that they have identified and ironed out any anomalies that might arise. Of course, there will still be anomalies when they cut over and launch the next system next Tuesday; you can’t make a large change like this without some things going wrong. But they have used the last six months to prepare for that: they will have “floorwalkers” throughout the utility to assist employees encountering problems, and to quickly escalate any problems that can’t be solved there. They are also adding temporary employees to the call center to be ready for an increase in customer calls, and they have contingency plans in place for unexpected customer service issues.

One of the biggest issues with the system, almost from the first day of the project, has been the component that prints bills. In an interview last week, Tim Almond, the QA consultant on the project, noted that you can’t test all the scenarios for a complex system like bill printing (he’s right), but “the secret is to keep yourself out of the newspaper.”  Bad bills will get printed; they do today as well. The trick is to have the right manual checks in place so that a bill so inaccurate that it would get headlines in the newspaper will get caught and corrected before it’s mailed out. To that end, they have put in place a set of rules in the NCIS system to flag bills that should be double-checked by a human. All the bills for the utilities’ largest customers will be checked by hand, as well as a representative sample from the rest of the bills. In addition, if a residential bill is 150% of the average it will also be flagged for manual review. With those rules in place, the NCIS team feels comfortable that the risk of a major billing snafu is low.

With all of their hard work over the last several months, and the people and measures in place to catch and deal with any problems that arise, the team feels that they are ready as they can be for the big launch this weekend. I hope they are right, and I sincerely wish them good luck.

Part 2 tomorrow: what comes next.