Big bugs: State websites crash during IT project management hearing

[This was first published in the July 10, 2009 PIM Weekly Report.]
It’s been interesting in tech news lately, with the pilfered Goldman Sachs program trading code and the widespread attacks on U.S. and South Korean websites dominating the news. Minnesota had its own tech shortcomings come to light via a legislative audit yesterday; Session Daily has a summary of the hearing. Ironically, as various state government websites (those under .state.mn.us) sputtered and repeatedly went offline, Rep. Phyllis Kahn’s (DFL-Minneapolis) State Government Finance Division heard from the Office of the Legislative Auditor (OLA) about the management of state Information Technology (IT) projects, including a stinging report on the Minnesota State Retirement System (MSRS) IT security.

MSRS is not part of the executive branch, which leaves it in a bit of a twilight zone for IT oversight, but they brought in help about two years ago from the Office of Enterprise Technology to help get cleaned up anyhow. MSRS hired a full-time computer security specialist, which had helped improve the substandard IT situation that auditors discovered, but the OLA audit, led by Audit Manager Eric Wion and Auditor-in-Charge Aimee Martin, made eight troubling findings, including: poor firewall and wireless security vulnerabilities, a lack of account and password controls, deficient network design, and a lack of prompt security patching. In response, MSRS Executive Director David Bergstrom said "we believe that we’ve made good progress, especially within the past year," to enhance security, but emphasized "our sensitive member data and mission-critical applications reside securely" on the OET’s mainframe computers.

The Auditor’s more far-reaching report on IT project oversight (PDF) spawned a discussion on OET’s statutory oversight of all major state IT projects, and more broadly, governance over IT projects, as a surprisingly full hearing room listened intently. Reps. Keith Downey (R-Edina), Jeremy Kalin (DFL-North Branch), and Ryan Winkler (DFL-Golden Valley) all had interesting questions about the relevant statutes: the OET has some responsibilities to approve and, with projects larger than $10 million, to annually perform audits of all state-level IT projects.

Under statute 16E.0465, any projects must first be reviewed by the OET under certain criteria before the commissioner of finance may authorize the encumbrances or expenditures. Kalin suggested that the OET and Finance hadn’t fully complied with the statute, cautiously comparing it (half in jest) to the statutory unallotment battle at the Legislative Advisory Commission. Downey asked about the governance of IT project development; the auditors hadn’t focused on governance specifically, since it is a dimension of "portfolio management," which is OET’s strategic approach to handling all the projects.

If there are mistakes in the statute, they need to be brought forward and rectified at the Legislature, Kalin said. Winkler said that the Pawlenty Administration needs to back up OET and force agencies to do "what they don’t want to bother doing." The $8 million HealthMatch project was cited by many as a worst-case example of IT project development. The state’s top information security officer, Chris Buse, pointed out that consolidating data centers would greatly help with standardizing security and creating solid monitoring systems.

Legislative Auditor Jim Nobles concluded with sharp words: the Committee shouldn’t forget about tiny government entities, all of whom should be subject to enforced standards. Sometimes systems that don’t meet standards should be shut down so that people "get the message," Nobles suggested. Kahn joked that she likes to end things on a positive note; to great laughter Nobles responded, "Sorry, Madam Chair."

We got a bit of PIM crowdsourcing going during the hearing: many of PIM’s Twitter followers (which grew beyond 3800 this week!) quickly helped us verify the state of the state’s websites. From them, we learned of the self-explanatory and helpful DownForEveryoneOrJustMe.com, which indicates whether sites are offline. While OET staff testified, their site was down.