This afternoon the City Council conducted a hearing on the NCIS billing and customer service IT system being co-developed by Seattle Public Utilities, Seattle City Light, and SeattleIT, the city’s newly-consolidated IT department. It was neither the bloodbath I expected, nor was it a font of revelations as to why a $66 million project went $43 million over budget. But if you listened carefully, arranged the puzzle pieces just so, and squinted a little bit, you could make out roughly what happened — and the rough waters ahead.
Apart from the fact that the project is 65% over budget and a year late, the Council’s chief complaint is that the leadership of SCL and SPU made no effort to alert the Council that the project had gone so far off track. Despite that, the Council members present today were very polite, listened a lot, and asked thoughtful questions. This was due at least in part to their recognition that the people presenting were not the ones responsible for the lack of communication; that responsibility falls to the directors of SPU and SCL and the Mayor. While they did occasionally poke around the question of why no one felt the need to brief the Council, they spent most of their time trying to dig into the details of the project itself and the process that has been followed to develop it. They heard from the NCIS Executive Steering Committee (ESC), made up of Susan Sanchez and Melina Thung from Seattle Public Utilities, Kelly Enright from Seattle City Light, and Tom Nolan from Seattle IT. They separately heard from Tim Almond of TMG Consulting, who heading the quasi-independent quality assurance consultant team on the NCIS project. Almond and the ESC were remarkably in synch in their conclusions about what went wrong, if you knew how to translate between their languages.
Here is a summary of what the Council learned from them today:
The project was under-resourced. Almond apparently flagged this as an issue early on and throughout the project. The ECS members’ spin on it was that they were competing for IT resources with other high-priority projects (though they didn’t specify whether those projects were within SPU and SCL or in other city departments). Nevertheless, it speaks to an inability to effectively allocate available resources to projects, or conversely an appetite for projects that extends beyond available resources.
The project changed dramatically in scope over the course of the design phase. At the start of the design phase in January of 2014, the project team had identified 3000 business requirements for the NCIS system; based on that, they set a “go live” date of October 2015 and a budget of $66 million with 100 internal staff assigned to the project and Price Waterhouse Coopers (PWC) hired as a system integration consultant. When they had finished the design phase a year later, 25% of their 3000 business requirements had changed. Much of that change was “non-discretionary” — external systems that NCIS needed to integrate with had changed, cyber security and privacy best practices had evolved, and they had learned lessons by studying other utility CIS system implementations. But the reality was that the project looked very different, and a new forecast pegged the budget at $85 million and the “go live” date six months later in April 2016. The lesson here is simple: they never should have set a total budget and go live date in January of 2014, before they did any of the design work. You don’t do that with buildings, nor with large software projects. It was silly to think it could possibly be accurate, particularly with so many aspects of the project beyond their control. They should have first asked for the funds to pay for the design phase, then come back after that and asked for the funds for the full project development once they had the design in hand.
The team didn’t have the necessary technical experience. SPU’s and SCL’s current billing system is 15 years old; the new one is intended to use state-of-the-art technologies and be the foundation for their IT systems for at least the next decade. The ESC admitted that that their team didn’t have experience with this kind of large, integrative system nor many of the technologies it would utilize (they called it “missing skill sets”). The NCIS team also didn’t have experience with testing; their IT staff was also doing the testing, but they lacked the required knowledge and experience to do it well. Training their staff added further delay. In the end, the ESC was forced to expand its agreement with PWC to bring in key expertise and additional staff to test the system.
The switchover from the old system to the new one is problematic. Switching everything over takes three and a half days. They have done three “dress rehearsals” of this to-date, and the most recent one did not go well; that was the impetus for pushing the new “go live” date to September 2016. They are justified in wanting to get that process right; they can point to the recent example of Los Angeles, whose switch to a new system went badly and cost $242 million in lost revenues. But the problems run deeper: training the users of the new system has hit snags as well, as the initial training was too focused on the product and not on how the staff do their jobs. The utilities have also had issues with understaffing their customer service centers while their agents are in training on the new system, resulting in long queues, high call abandonment, and low customer satisfaction. They have a lot to sort out, and not much time in which to do it.
The plan for what happens after the “go live” date is still a work in progress. This is scary: the date is four months away, and they haven’t figured out who will continue to maintain the system on an ongoing basis, who will provide technical support for users having trouble with the new system, and whether they need “go to” people in each department to help internal staff. They claim they have stopgap measures in place.
The NCIS team was using the city’s regular budget process to communicate the dramatic budget overruns up the management chain. The ESC claimed that it was their standard practice to use the regular budget process to communicate changes, because NCIS was part of the base budget and not a “strategic initiative.” They had no experience or context for making reports to the City Council, and no one asked them to.
Council member Herbold listed three items for further follow-up from today’s meeting.
- She wants a continued discussion of the risks going forward and how the NCIS team plans to address them;
- She wants to hear more about the communications plan to let the utilities’ customers know about the new system and the changes they will see (like new account numbers);
- In Herbold’s words, “We want to have a continued conversation with all of the departments about Council expectations, and whether or not we want to create a policy or new structures to identify expectations for communication and consultation as scopes change and budgets change.”
The third item is really the heart of the matter, and Herbold is right that it is much broader than SPU, SCL and the NCIS system. Under Mayor Murray, the city’s departments have shown a reluctance to divulge information. We’ve seen the same pattern of behavior with SDOT: with the Pronto bike-share system (a chronic problem there) and the 23rd Avenue project. We’ve seen it in the Human Services Department as they respond to the homeless crisis. We’ve seen SDOT and the Parks Department fail to disclose the illegal tree-cutting in West Seattle.
And now we’re seeing a City Council that is rapidly losing patience with an opaque executive branch. Council member Gonzalez also raised the issue this afternoon, saying that when there is a change that will require a large budget impact that has been known for several months, “there is a conversation to be had about the responsibility of the departments to provide to the appropriate committee chairs what those impacts will be so we can, as a city, be as fiscally responsible as we can and anticipate those budget changes.”
Unfortunately, several of the people you would ideally want in the room were missing this afternoon: Ray Hoffman, Director of SPU; Larry Weis, Director of SCL; Mayor Murray (or one of his deputy mayors); and on the City Council side, Council President Harrell and Budget Committee Chair Burgess. The Council’s main weapon in this battle is the city budget: if the Council members can make their expectations known to the Mayor and his department heads, and back that up with credible threats to withhold funding when information is not forthcoming, they might just loosen the executive branch’s tongue. But the Council’s track record on this is spotty: when SDOT showed up and demanded money to bail out Pronto without the required business plan and after repeated revelations about hidden problems in the bike-share’s finances, the Council blinked and gave them the money anyway. On the other hand, when the Mayor asked for money to hire more police officers despite not delivering a long-overdue staffing report, the Council held its ground until the report eventually materialized. In the face of urgent problems such as homelessness and crime, whether the Council can maintain the discipline to use the budget as carrot and stick in its efforts to force transparency is yet to be seen.
In the meantime, despite the NCIS ESC’s assurances that they have high confidence in the September “go live” date, the project remains high risk. I am sure we will see them in front of the Council at least once more before the new system is up and running.