Office of Enterprise Technology alarmed over security risks via spending data APIs

PIM previously covered Rep. Paul Gardner’s (DFL-Shoreview) proposal (HF625) to implement Application Program Interfaces (APIs) in state accounting and procurement databases, an increasingly popular notion among open government advocates nationally. This PIM writer suggested to Minnesota Management and Budget (MMB) that they might want to look at making APIs available because their “TAP” Web-based application had some accessibility problems for people not using it via the ever-popular Microsoft Windows/Internet Explorer setup.

Gardner’s bill would set up APIs connected to state accounting systems, but it seems that the Office of Enterprise Technology (OET) isn’t too happy about the prospect. PIM obtained a letter from OET’s John Lally to Curt Yoakum at MMB. It seems OET doesn’t like the idea: they think about three or four percent of the “state’s user community” would be addressed by having APIs available, though the main point of the proposal — the public’s interest in gaining more direct access to spending data — isn’t quite addressed head-on.

Other points: the APIs should likely get directed to a server outside of the “firewalled zones,” and this server would have properly sanitized or “extract” data available updated at regular intervals. Point number three, saying external queries could cause corruption, kind of suggests that the servers don’t handle read requests properly (generally in database design, simply reading data fields shouldn’t cause corruption).

These all seem to be logical concerns for the people that would have to implement the measures, but it begs the question of whether the program interface that the current TAP system uses is going into these same sensitive production systems: after all, if the TAP web application has unlimited query powers, then it could also theoretically cause similar problems to those OET is worried about here. (Can TAP cause the same corruption via queries?)

Overall it seems OET would be put on the hook to work out the data practices rules for APIs, but whether or not these rules are already in the works seems the imminent question. We have just put in an inquiry with Yoakum to see if MMB has a followup, and will report further if it does.

[Letter styled for clarity, processed via OCR. Download the scanned PDF here].

Subject: Observations about the Searchable Database provisions

Without a detailed technical analysis of the implications of the proposed language, here are some recommendations about implementation. Do not, under any circumstances, give people outside the approved user community direct access to production systems. Queries against the system can result in:

1. Degradation of performance - poorly structured or database-wide ad hoc inquiries and report requests can easily consume system resources to the point of shutting the production system down. It takes a great deal of effort and sophisticated programming to frne tune queries against a relational database. Although this request comes with the API requirement, it is not clear where the development of API would reside, another potential complication.

2. Data integrity loss - the likelihood of external queries and internal transactions going against the same production records Can produce record or field contention, or conflict of access. This can lead to contamination of the records, interruption of system functions and loss of transactional integrity.

3. Security loss - allowing unidentified and unauthenticated users access to production systems, especially with the use of programming tools in place of structured inquiries, is tantamount to an invitation to hack into the data base, corrupting or destroying data and potentially destroying program operation and data privacy.

4. Requiring the system to meet a defInition of “Searchable” implies a separate and appropriate structuring of the database or at minimum, adding a ’searchable’ or presentation layer to the relational structure for the most simplistic inquiries.

5. API implementation only addresses a small group of the state’s user community. This on average would cover approximately 3 to 4%. “Searchable” without limits for the other 96% would not be covered unless there were pre-built APIs. The responsible solution would be to give access to a replicated database or extract, produced at specified intervals, that will serve as the database for public inquiries. In addition, you should insist that this database and any access software be installed outside our ftrewalled zones. Preparation of the data extract or replica should be preceded by appropriate restructuring and screening for data practices. Chris Buse’s staff can help with this area.

The costs for this duplicate database will be significant, and will fall into three categories: hardware costs for application, web and data servers; access and data preparation, including connectivity; database sanitation (for protected data) and replications; and operating costs for computing cycles to process inquiries, depending on the chosen technology. Some of the cost variables will include (a) storage (based on database size, configuration, version availability and other capacity issues; (b) speed of access and recovery; (c) volume of inquiries. Note that we did not make assumptions about data refresh rates, limits on the number and scope of inquiries, or fees for cost recovery by users. All of these will impact the cost and complexity of the reporting system.