NCIS went live today, with a couple of problems

The new NCIS billing and customer information system for Seattle City Light and Seattle Public Utilities went live this morning — mostly. According to the team, they had two problems with one of the 40 external applications that NCIS connects to. One of the problems is just slightly annoying; the other one is a small privacy breach that was caught and plugged quickly.

The problematic component is the one that allows customers to view their bill and make payments online, an online service run by a company called Kubra. NCIS electronically sends Kubra a big package of data for each daily billing run, which contains for each customer the account information, the amount due, and a PDF of the bill that the customer would otherwise have received in the mail. Kubra emails a notification to the customer that the bill is ready to be viewed with a link that takes them to a login screen and then to the PDF bill. Once logged in, the customer can also make payments or arrange for automatic payments.

This morning around 3:30am the NCIS team sent the first daily batch of “live” data over to Kubra, containing 1/60th of SPU’s and SCL’s customers (since the utilities bill every two months and spread the customer billing dates over that time). The first thing that went wrong was that Kubra sent out multiple email notifications to customers; 2306 signed up for e-billing got six copies of the email, and 735 who are also signed up for automatic payments got twelve. That’s annoying, but ultimately harmless.

The second thing that went wrong was more serious: the PDFs got mixed up, so if you clicked on  the link in the email and logged in to your SCL/SPU account on Kubra, you saw your own account, billing, and payment information, but other customers’ PDF bills in addition to their own.

According to SPU and SCL, the PDF bill contains a customer’s name, address, account number, consumption data (electricity for SCL; water, sewer,  and garbage/recycling/yard waste for SPU), any solar production credits, and any Utility Discount Program discount if applicable. It does NOT contain data on method of payment, so bank account and credit card data were not exposed.

The NCIS team was notified of the problem this morning around 10am. They took the Kubra system offline at 10:30am, and it remains offline. The time period between when the email notifications went out and when the PDFs were no longer available was seven hours –and fortunately most of that was in the wee hours of the morning on a national holiday.  Yet the utilities have confirmed that during that period some customers did log in and were in fact shown the wrong PDF bill; they are still trying to determine how many, and whose bills were wrongly shared.  Luckily, Kubra doesn’t email out the PDF bills directly, so shutting it down mid-morning instantly closed the privacy hole; the only data that inadvertently got out was when someone actually clicked through to Kubra, logged in, and tried to retrieve their PDF bill this morning before 10:30. Since only 3000 customers received email links this morning, and it was Labor Day, the actual number of wrong bills served up was probably fairly small.

Still, it’s a privacy hole and the NCIS team need to be accountable for it: not only finding and fixing the bug that caused it, but informing the people whose bill was shared with someone else. A spokesperson for the utilities told me this evening that they are still gathering that data, and that they will have a news release in the morning with more information.

As a former software developer, I can attest that these two bugs don’t represent major design flaws; they are the sort that are easily made with a typo in single line of code, and in fact may be harder to track down than to fix.  What is more surprising is that they weren’t caught during testing since they are so obvious. Through the spring the NCIS team’s QA consultant reports show a small but persistent set of bugs in the interface to Kubra, so it was clearly being tested and improved. Either there was a major oversight in the testing protocol, or there is something different in the official live environment than in the system they were testing with. I expect there will be much burning of the midnight oil tonight as they sort that out. And they do need to sort it out quickly, because Kubra remains shut down and they can’t have their online bill presentment and payment system offline for more than a couple of days before the backlog becomes significant and the hit to the revenue stream becomes noticeable.  As of 7:30pm this evening, the NCIS team has already restored Kubra access to those customers with a bill date prior to last night, so it only affects the customers in this morning’s batch of bills and those in daily batches moving forward until the problem is resolved.

With a system as big and complicated as NCIS, something was expected to go wrong today — and it did.  The PDF mix-up is serious, but if that’s the worst thing to go wrong, and if they can fix it quickly, then they are in pretty good shape overall. The team is still doing validation work on other parts of the system, so they can — and probably will — find other issues that they missed during testing. But tomorrow, when SCL and SPU staff come back from the long weekend and get to work, the real test begins.

Update 9/6/16 7:45am: The utilities  clarified this morning that they had backed out yesterday’s batch of bills sent to Kubra and restored online access to all customers.

Update 9/6/16  6:00pm:  here is a link to the utilities’ FAQ on the privacy issue.

If you enjoyed reading this, please support my work by making a contribution on Patreon!