This morning the City Auditor’s office presented its report on the NCIS billing system that Seattle City Light and Seattle Public Utilities launched last September after running $34 million over budget and well over schedule.
For background, here’s my analysis from last June on what went wrong with the NCIS project. The City Auditor’s office had an advantage I didn’t enjoy: it had open access to everyone on the project and all their documents. However, the Auditors don’t have expertise in IT projects, so they brought in a consultant — Gartner — to provide support, expertise and recommendations as they completed their audit. The Auditor’s report confirms many of the issues I raised, and identifies some others.
The City Council asked them to answer three questions:
- Why did the project cost more and take longer than anticipated?
- Why weren’t Council members informed of cost and schedule changes?
- Was the Quality Assurance expert used effectively?
Question 1 has two parts: the cost and the schedule. But the answer for “why did it cost more?” is simple: it cost more because it took longer. The NCIS team had both a group of fulltime city employees, and a group of PriceWaterhouseCoopers (PWC) consultants on staff. The budget was based on the estimated number of months they were expected to take to build the product, and for every month longer it took, all those folks needed to be paid. In the end, internal labor costs were $20 million over budget, and PWC charges were $10 million over. There were overruns in other places as well, which ate up the projects contingency budget plus some, but the internal and external labor more than anything else accounts for the overspend.
But that begs the underlying question: why did the project turn out to be so much more work than originally anticipated? The Auditors boiled it down to two factors. The first is that there were “changes in scope and schedule” along the way, what IT professionals call “feature creep:” the tendency to add new features and functionality along the way. Of course, as I laid out in my analysis last year, that’s not really a fair way of looking at it. The underlying problem was that the NCIS team was forced to commit to a budget and schedule before any of the design work was done on the project, i.e. before they knew what they would need to build. When the design work was done, it turned out to be much more work to develop it than the original guess.
The second factor identified by the Auditors was “the project leaders intentionally prioritized quality.” This is a talking point that the NCIS leadership used throughout 2016, implying that they could have released it sooner, but given the disaster that the Los Angeles public utility had rolling out their new billing system they wanted to be absolutely sure that it was ready to go. And that talking point is a lie. As of February 2016, when the NCIS team officially decided to abandon their April 2016 “go live” date, they still had over 100 serious bugs to fix and were finding an additional 11 per day. Their most recent “day in the life” testing exercise had been a failure, and not all partner organizations were yet participating in their dress rehearsals for cutting over to the new system. Many of their test scripts had never been run. On top of all that, they still hadn’t finished implementing one component: the “Customer Self Service” portal. NCIS wasn’t even close to ready to launch. I have huge respect for the City Auditors, who do amazing work across numerous fields, but they completely missed the boat on this and got taken in by the NCIS team’s spin. The project wasn’t delayed out of an abundance of caution; it was delayed because it simply wasn’t finished. Certainly the NCIS team deserves some credit for not pushing it out the door early, but they really had no choice in delaying it.
Answering question 2 — why weren’t the City Council members informed — is much more in the Auditors’ wheelhouse. They concluded that at the time there was no existing formal mechanism for reporting the status of IT projects to the City Council. It wasn’t clear whose job that was, nor what the standard was for timely reporting (not to mention what meets the threshold for requiring an update). That’s all a bit of an excuse, since there was nothing preventing the Executive Branch from notifying the Council; it simply wasn’t required beyond the information in the annual capital budgeting process. The Auditors recommended that the CTO be responsible for reporting on the status of city IT projects. Michael Mattmiller, the current CTO and head of Seattle IT, has accepted that recommendation and has apparently already begun making monthly written status reports to the Council’s Affordable Housing, Neighborhoods and Finance Committee — though it could be argued that the reports should be distributed more broadly, and at the very least to the Education, Equity and Governance Committee which oversees IT technologies for the city.
The Auditors also found that the uncertainty behind cost estimates was not communicated clearly, and recommended that the CTO develop a new method for discussing budget estimates in the early phases of IT projects. Further, they found that financial reporting for large, complex IT projects needs improvement and recommended that the executive sponsors of such projects should assign a dedicated finance analyst to the project to provide regular reporting. Mattmiller also accepted this recommendation, though in this morning’s discussion it was unclear what the threshold should be for when the rule applies.
As for Question 3 — was the QA expert used effectively — the Auditors noted that project managers didn’t consistently raise the QA expert’s concerns to the Executive Steering Committee (ESC), and the ESC was often slow to respond in addressing those concerns. Eight of the risks that the QA expert raised were open for an average of 16 months. They recommended that project managers specifically be responsible for monitoring and tracking QA risks and presenting the ESC with options for resolving them, and in turn that the ESC should be held accountable for addressing those risks.
That’s fine to a certain point, but it misses the forest for the trees. The real problem was that you can’t run a large, complex IT project by a committee of executives, most of whom have no experience running IT projects. In the NCIS project there was the ESC (which varied over time between 5 and 8 people) and immediately below them a team of project managers organized into a “Project Managers Office.” The ESC held all the real power over the things that mattered: budget; schedule; approving design, scope and changes; and staffing resources. But they collectively didn’t know how to run a large IT project, and even if they did, they didn’t meet frequently enough to manage the day-to-day decision making.
At the request of the Auditors, Gartner produced a document that outlines best practices for government IT projects. In it, they present an organization chart that fills in the missing layer: a Project Director who is “responsible for providing an executive leadership presence for the project team and is the chief liaison with Executive Leadership.” And more to the point: this is someone who has significant experience managing IT projects, can see problems coming, and can make key decisions.
NCIS never had someone in that role, and it would have made all the difference in the world if it had.
The Council members at the hearing this morning (Herbold, O’Brien and Sawant) had a few key take-aways from the briefing. The biggest one was that they needed to work with the CTO’s office to clearly define their expectations for the content and timeliness of reports. Herbold also told me this afternoon that she and her colleagues are working on proposals for reforming the capital project budgeting process.
Now, lest you think that the NCIS project is an anomaly, it turns out that there is another big municipal IT project underway right now in Seattle that is suffering from many of the same issues. That is the “Summit reimplementation” project, an effort to replace the ancient and creaky financial management system used across the City of Seattle’s many departments. The current Summit system is widely reviled for limiting the extent to which departments can understand their own finances, and it has been customized so extensively by individual departments that it now gets in the way of aligning efforts across the city. After many years of hand-wringing, in 2011 the City Council passed a resolution to launch the “FinMAP” financial Management and Accountability Program that would build a new replacement system. From 2013 through 2015 the city developed the scope and cost estimates for the project, and in 2015 and 2016 it began building out the new system. This year is intended for final testing and working with individual departments to adapt their current business practices to the new system.
Like NCIS, Summit is built on top of an “off the shelf” system developed by a major software vendor (in the case of Summit, it’s PeopleSoft version 9.2). But whereas NCIS required substantial customization and a great deal of coding to connect it to more than 40 legacy applications used by Seattle City Light and Seattle Public Utilities, Summit requires far less hard-coded customization: most of the work is in specifying the city’s data and its structure, the forms used for entering and editing it, reports to be generated, and “business rules” for the kinds of operations that are allowed, who has permission to perform those operations, and how data is moved around and calculated. On the other hand, Summit is used by every single city department so the scope of the change is huge: in a typical year Summit generates 100,000 vouchers, 80,000 payments, 74,000 accounting journal entries, and 8,000 purchase orders; and it is used by over 800 city employees. That means that the business rules for every department need to be re-written (and thoroughly tested) for the new Summit system.
Over all the years that the old Summit system has been in place, business practices have diverged significantly across departments, and as mentioned before that has become an impediment to cross-department collaboration. So as part of the move to the new system, the Mayor and his department heads have pushed for limiting the amount of department-level customization allowed so that there is far more consistency. That sounds easy, but it isn’t: to do this means redefining the business practices in almost every department — and retraining staff to adopt those new business practices. If you are imagining there might be some resistance to that among city departments, you’re right. That’s a big issue, but not the only issue, that the Summit team is struggling with.
To the Summit leadership team’s credit, last fall they had enough self-awareness to recognize that they were in over their heads, and brought in an outside consultant to evaluate the project status and its major risks, and make some recommendations on how to move forward. They issued their report in January, and it echoed many of the issues that the NCIS team struggled with. They also gave the Summit leadership team credit for recognizing they had issues and trying to address them.
Nevertheless, they didn’t hold back on listing the major problems they found:
Within the Summit team itself, they found issues with project leadership, organizational capacity, project management and lack of technical architecture. But equally if not more serious, they found major issues with communication out to the city departments and with the overall “change management” effort to get those departments ready to move to the new system, to the point where there are significant risks that major city departments will not be ready to switch over when the new system launches. Here’s a sample:
The consultant made five recommendations for new initiatives to “right the ship:”
- Change the governance structure of the Summit project team.
- Improve and restructure the project team.
- Establish a formal change management program to drive the work that needs to happen with all the departments.
- Perform a comprehensive re-planning effort.
- Perform a “deep dive” review of the work being done by the Systems Integrator contractor to ensure that it is meeting contractual requirements and quality standards.
You can read through all the details of their recommendations here. The QA consultant for the Summit project reviewed these recommendations and largely agreed with them. The one point I would highlight here is that the Summit project team, like NCIS, also didn’t have a Project Director; it too was being managed directly by the Executive Steering Committee. The consultant highlighted that as one of its recommendations for change.
These recommendations are a big deal, and a lot of work. When the Summit leadership briefed the Council back in February, it admitted that it had brought in an outside consultant to do an assessment and they had some issues to address, but it never spoke to the breadth, depth and seriousness of those issues. (you can watch the video here)
The Summit leadership team has chosen to address several of the issues by “outsourcing” much of the leadership and management work to another outside consultant: Accenture.
It’s good that the city is moving to address the big problems, but it needs to face the reality that it won’t meet its target launch date of January 1, 2018. There is simply too much work left to do, many departments are not yet on board (including some big ones), and the project team is changing its leadership, its project management, its plan and its approach to “change management” while simultaneously doing a major technical review of its systems integrator. It will take months for all of that to settle out. As of last week’s risk management report, Accenture is already delaying delivery for 13 of its 15 deliverables in its contract, and the critical “change management” effort is still stalled. The Summit leadership team will decide next week whether to delay launch beyond next January, and the schedule for everything else will shift once that decision is made.
In the meantime, this has become a very expensive project: $130 million in total. The core Summit system has cost $84 million to develop, and there is another $46 million spread across city departments over the next two years for change management efforts to get each department ready to cut over to the new system.
The reorganized, centralized Seattle IT department celebrates its first birthday this month. Under CTO Mattmiller, it has been moving toward standardizing technologies, procurement and development processes. But it clearly has a long way to go before the city finds proficiency at developing its IT infrastructure.