City still covering up last week’s privacy leak

Eight days ago, city employees discovered a second vulnerability in the new NCIS billing system for Seattle Public Utilities and Seattle City Light, which exposed customers’ private information. The city still has not publicly disclosed this incident, in violation of its own policy.

The vulnerability is nearly identical to the one that occurred the week before on the launch day of the new billing system: some customers who tried to view their bill online were shown other customers’ bills as well as their own. Both times it was discovered the same way, when they were notified by a customer who had noticed the problem. Both times the city’s technical staff responded the same way: they shut down the electronic bill presentment component, removed the daily batches of bills that had the issue, re-enabled the system, and suspended sending new batches of e-bills until the bug could be fixed. That was the right thing to do, because it immediately closed the vulnerability: there was no longer a way for customers to see other customers’ bills.

But before that vulnerability was closed, customers’ information was exposed, and as of the last communication I received from city staff, they were still trying to dig through the system’s logs for both incidents to discover precisely which customers’ bills were viewed by others. But since the city knows which daily batches of bills had the problem, it does have a list of customers who may have been affected, and those customers have a right to know now what may have happened to their private information — and that they should be extra alert to identity fraud.

This is dictated by a policy called “responsible disclosure.” Microsoft is one of the leaders in this space, and its policy — generally considered to be conservative — argues for waiting for the vulnerability to be closed before disclosing it in order not to give the bad guys an opportunity to exploit it. Other organizations argue for faster disclosure regardless of the consequences. But in the case of the NCIS privacy vulnerabilities, the vulnerability hole is now closed and there is no justification for not disclosing it. Michael Mattmiller, the CTO of the City of Seattle and the head of the “Seattle IT” team that now supports and maintains NCIS, knows this all too well: he used to work for Microsoft as a Senior Strategist for Enterprise Cloud Privacy.

The City of Seattle’s privacy policy says:

 In the event that any information under our control is compromised as a result of a breach of security, the City will take reasonable steps to investigate the situation and where appropriate, notify those individuals whose information may have been compromised and take other steps, in accordance with any applicable laws and regulations.

That’s pretty weak in that it doesn’t explicitly specify timely notification of those affected, but it is clear that notification must happen.

When the first incident happened on September 5th, Seattle City Light and Seattle Public Utilities did the right thing and disclosed it within hours of closing the hole: they contacted reporters and they posted a notice on their web sites. They also sent an email to the City Council briefing them on the issue, with talking points they could use to explain the incident to their constituents the first morning after the problem was disclosed.

When the second incident was discovered on September 12th, they buried it. There was no outreach to reporters, and no notice to customers. I received an anonymous tip on the afternoon of the 13th, and my inquiries went unanswered until the afternoon of the 14th.

The morning of the 14th, I asked the staff of Council member Herbold, whose committee provides oversight of Seattle Public Utilities, whether they had been briefed on the issue. At noon that day, Mattmiller called one of the Council’s Central Staff members — but not one who works on utility issues — to discuss the incident. That same afternoon I asked Council member Kshama Sawant, whose committee provides oversight of Seattle City Light, whether she had been briefed. She told me that she had not, but (rightly) needed to check with her staff to see if they had.  Two hours later, the following email was sent from Megan Coppersmith, Public Information Officer for Seattle IT, to Council member Sawant:

From: Coppersmith, Megan Sent: Wednesday, September 14, 2016 3:57 PM To: Sawant, Kshama <Kshama.Sawant@seattle.gov> Cc: Virdone, Ted <Ted.Virdone@seattle.gov>; Kilduff, Tony <Tony.Kilduff@seattle.gov>; Chow, Calvin <Calvin.Chow@seattle.gov>; Lindsay, Peter <Peter.Lindsay@seattle.gov>; Mattmiller, Michael <Michael.Mattmiller@seattle.gov>; Hara, Mami <Mami.Hara@seattle.gov>; Thung, Melina <Melina.Thung@seattle.gov>; Brueger, Maura <Maura.Brueger@seattle.gov>; Hennessey, Bob <Bob.Hennessey@seattle.gov> Subject: NCIS update

Councilmember Sawant,

We wanted to give you an update on the implementation of the new customer information system.

Last week’s issue has been addressed, and we are now working on further coding that relates to electronic bill generation and online display through the e-billing system. 

Earlier this week, certain customers were again potentially given access to bills that did not belong to them. 

The City took the online e-billing system offline Monday night around 7 p.m. The service came back online around noon today and is now available to customers.

We are still investigating what customers were affected by this situation. 

We continue to look at ways to enhance this billing service and third-party coding review processes.

If you are free today at 4:30 p.m., I would like to set up a call to answer any additional questions you might have. If this time does not work for you, I can work with your staff to find an alternate time.

Please do not hesitate to reach out to me with additional questions.

Thank you,

Megan




Megan Coppersmith

Public Information Officer

SEATTLE INFORMATION TECHNOLOGY

T: 206.233.8736 | M: 206.430.0374 | megan.coppersmith@seattle.gov


TECHNOLOGY SOLUTIONS FOR THE CITY AND PUBLIC WE SERVE

Today, eight days after the vulnerability hole was discovered and closed, the city has still not made any public disclosure that they had a second incident, let alone inform affected customers that their private information has been exposed.

Wednesday evening, I asked Coppersmith a couple of simple questions to understand what their plan is for disclosure and why they are following a different plan form the nearly identical incident the previous week. Thursday evening, Coppersmith told me she was still waiting for “final approval” on the answers to my questions. Friday morning, this was the response I received (that took 36 hours to compose):

What is the plan once you have a list of the customers affected?

We intend to notify those individuals whose information may have been accessed or viewed by other customers, consistent with our privacy policy (http://www.seattle.gov/tech/initiatives/privacy).

Why was a different process followed this week?

We were alerted to the issue by a customer and immediately began investigation the issue. This issue involved a defect that we could not pinpoint quickly, so we took the system down as a precautionary measure.  We will be notifying those individuals who’s information may have been exposed.

(you’d think with 36 hours to write, review, and run this through an approval chain, they would have found and corrected their two typos)

The conclusion is clear: the city doesn’t want to talk publicly about this. Nevertheless, Friday morning I asked Coppersmith three follow-up questions:

1. Why did you take a very different approach to disclosure of a privacy incident this week, compared to last week’s nearly identical event?
2. Who made the decision to take a different approach?
3. Why should I (and your customers) interpret this week’s response as anything other than a deliberate attempt to keep this week’s repeat incident out of the press and deprive SCL’s and SPU’s customers of their right to know that their private information might have been compromised?

As of this writing (Tuesday evening, three business days later), I have yet to receive any response. I don’t expect one.

Over the weekend I asked (in email) Council members Herbold, Sawant and Harrell (who oversees Seattle IT) whether they were concerned with the city’s refusal to disclose last week’s privacy incident. I’m still waiting for a response from them as well.

Last week was very busy in City Hall, with major activity on several high-profile issues including the North Precinct, secure scheduling, move-in fees, the heroin addiction crisis, and homeless encampments. I can understand why the Mayor would not want to add in the noise of a repeat privacy vulnerability, nor the bad PR from another bug in the very-late, very-overbudget NCIS billing system.  But it isn’t his choice, or the choice of any of the other city leaders. This is a serious privacy leak, and it must be disclosed in a timely manner. The city has both moral and legal responsibilities to do so.

The city departments involved (Seattle City Light, Seattle Public Utilities, and Seattle IT) need to do a full disclosure now, and the City Council needs to dig into why they decided not to disclose it — and who made that decision. Given that the decision involves three different departments, and given Mayor Murray’s well-documented penchant for micromanaging communications, it wouldn’t be shocking to discover that this cover-up is being orchestrated in the Mayor’s office.