By Charlie Hayward
Gov. Martin O’Malley talked to reporters about his delivery goals in December.
StateStat is a substantial part of Maryland state government’s transparency enhancements relating to performance measurement.
The program, an outgrowth of the CityStat initiative Martin O’Malley began as mayor, was much touted in the early days of the O’Malley-Brown administration, but now chugs along in obscurity. While StateStat conveys much information, it can be substantially improved so that it comes close to living up to the high expectations inherent in the governor’s StateStat messaging.
O’Malley promotes StateStat as a tool for managing and reporting the state’s performance related to 16 wide-ranging, high-priority goals. Those goals are reported within four categories: Opportunity, Security, Sustainability, and Health. The Governor’s Delivery Unit (GDU) works closely with StateStat and Maryland agencies to monitor and track progress or deterioration for each goal.
O’Malley talked about the progress toward the goals at a long meeting with reporters in December.
Enhancing transparency
StateStat serves to enhance transparency in several ways. For example, it conveys to the public much of the same extensive data reported by the state’s departments and agencies to the delivery unit for managing performance, and it contains considerable meeting minutes and other materials documenting interaction among the governor, GDU, and agency managers.
StateStat’s website evidences large-scale and proactive work towards achieving goals by O’Malley, his GDU teammates, and state agencies.
StateStat’s goals seem reasonable, unbiased and ambitious. And there is good documentation of regular, active engagement between the governor and state agencies, including good information sharing, course modifications, and direction setting. Moreover, the site is regularly updated to convey the latest executive direction and progress towards reaching goals. Indeed, as of March 2013 most goals’ delivery plans were completely updated, including more ambitious goals and new timelines.
Improvements could be made
However, there are many improvements the governor and his delivery unit team may want to consider. (On June 27, the head of StateStat, Matthew Power, responded to this analysis in the comments below.)
1. Goals have long-term deadlines, from 2014 to 2025, so mismatches abound with the governor’s remaining 19 months in office, and between the extensive short-term activities reported in StateStat, vis-à-vis deadlines set far into the future.
2. High-level goals are supported by two lower levels of statistics and data. Users can drill down into secondary statistics and below them into detailed datasets. However, the audit trails are weak among the three tiers. Navigating from the 1st to 2nd and 2nd to 3rd tiers is not “user friendly” because the more detailed data mostly don’t agree with and don’t support data in the higher tier. Weak audit trails could confuse users and marginalize the value of the lower-level data.
3. StateStat’s Reports tab leads users to numerous meeting minutes, reporting templates, and graphs. This information is uncategorized; users who want to evaluate reports for a particular goal are out of luck unless they want to reconstruct from the content of reports and meetings they think may be relevant, forward to the relevant goals.
Some delivery plans missing; no audits done
4. StateStat has no delivery plans for two of 16 of its goals Greenhouse Gases and Health IT.
5. State statutes governing StateStat provide that the Office of Legislative Audits should periodically audit the data. However, StateStat has never been audited and the most recent audit work on performance measures was July 2011. The auditor’s scope might include testing for perverse incentives that may be created by pressures on management to achieve numerical goals.
6. Based on a text-search of all laws passed in this year, there is only one reference to StateStat. This suggests StateStat is not being used by the legislature. That begs the question about who uses StateStat, whether the legislature is a stakeholder, and what is StateStat’s true value proposition compared to its cost. The governor may want to get legislators involved by soliciting their inputs on which goals to measure and what deadlines should be set.
7. One goal, Health IT, appears to be inactive since 2012.
Caveats needed
8. StateStat’s website should contain a section devoted to caveats about performance measures. First, many factors outside the governor’s control can have a direct and material effect on performance results. Second, performance measurement via statistics, while useful, is one of the least accurate ways of evaluating success or failure of government initiatives.
For instance, StateStat claims 132,800 jobs recovered since the great recession’s bottom (February 2010), or 91% of the goal. However, there is no way to know the true correlation between StateStat activity and jobs recovered and it’s conceivable the state’s other actions including last year’s tax hikes on the wealthy actually impaired job growth.
Performance auditing or program evaluations are expensive, but often are far more effective for evaluating how government is doing. Third, users should know that StateStat has no information about the cost-effectiveness of using taxpayer dollars for any goals being tracked.
9. The state, by law, reports extensive performance measurement data outside those available in StateStat. These “outside” data are reported annually in the state’s budget books. The StateStat website should link users to that additional measurement information and explain fully the relationships between the two measurements.
Conclusion
StateStat captures plenty of detail indicating the governor is focused on priorities embodied in his 16 goals. However, the site’s owners should recognize StateStat could be improved in many areas described above. Also, for those interested to see an alternative way to convey state performance, please compare StateStat against its counterpart in the State of Washington.
Charlie Hayward recently retired after 30 years’ experience with performance, IT, and financial auditing of a wide variety of government programs and activities. He is an avid reader of MarylandReporter.com. He can be reached at hungrypirana@verizon.net.
Here’s a response to this story from the governor’s office.
Maryland Reporter’s recent analysis of StateStat misses the
mark.
StateStat is not about a machine analyzing numbers that tell
you how you’re doing. It’s about a human process of investigation, analysis and
accountability.
Yes, StateStat produces objective metrics. But its real
value is in the relentless human interaction that allows us to identify areas
for improvement and makes possible creative problem-solving that ultimately
impacts the bottom line.
Thanks to this open and collaborative system, Maryland has
been able to achieve a 24 percent reduction in violent crime, an 11 percent
reduction in energy use, and a 16 percent reduction in infant mortality while
recovering 133,000 jobs that were lost during the Great Recession. StateStat is
helping the people of Maryland make progress even in the toughest of times. It
does this by helping us set goals, assign deadlines, measure performance, and
hold people accountable. In the end, it’s all about better choices and better
results.
–Matthew Power, Director of StateStat, Office of Gov. Martin
O’Malley
Not to mention the “datasets”on data.maryland.gov. There are 210 or so, so far. They appear to be randomly grabbed and the search function is poor.
I particularly like this dataset: https://data.maryland.gov/Public-Safety/Percent-of-Structurally-Deficient-Bridges/ycg8-2z84
2 rows; four columns; no usable information
I think this article tried to refer to the major problems in this StateStat data website, but didn’t come out to say what all analysts who try to gather information find…..the data is always skewed and with absolutely no oversight of the data that is collected it is found to be faulty and is often unreliable. This goes for CityStat as well. If you go to NYC from where O’Malley copied this system and having been around long enough to have a history (not to mention investigative reporting holding powers accountable) you will see the same complaints of lack of relevance and reliable data as we see here in Maryland. Let’s be clear….CityStat was designed to be a development tool for what in the early 1990s the 1% knew was a push for urban development and they wanted figures to target communities with money and associated programs. It was as well used as a crime fighting tool. As we know with any data collecting process, if there is no oversight as to the data being entered, then the goal becomes to make the data say what you want. THAT IS HOW THE DATA IS USED TODAY. IT GIVES GOVERNMENT FIGURES TO BACK POLICY THAT IT WANTS TO SEE IN PLACE.
As a researcher who is constantly accessing data I can tell you that regardless of the category…..crime, health care, poverty, education……all of the data is skewed. The state or city wants to show improvement in education then the data sets are produced to show an increased in achievement but has no basis in actual gains. It is all statistics. We have very little gains in education achievement but we have changed the way we calculate achievement to make it look like gains. It isn’t that funding education does not improve education, it is that most of education funding went to building the infrastructure for corporate advanced education platforms. College attendance up? Well, career ‘colleges’ or job training is now done at community colleges on the taxpayer dime. Same with crime stats where we heard quite publicly the police were skewing what they entered on incident reports to show a drop in crime. In reality much of Baltimore’s drop in crime had to do with the police simply not responding to calls especially in underserved communities where much crime occurs. Health care funding in underserved communities is up? Well, actually all of the funding is going to Enterprise Zones slated to gentrify a poor community moving to affluent so funds are not being used to help the poor, they are being used to build health infrastructure for what is expected to be an affluent neighborhood. People receiving social services? Well, social workers have caseloads that far exceed any capacity of actually providing anything more than a check-in by the client….if that.
All of the data on these StateStat and CityStats are just like this and the NYC model from where it came……even worse. It is easy to conclude that these data are not meant to give an accurate picture or else they would have a solid audit and oversight system in place where there is none. Rather, they are simply used to provide data that support whatever the politicians are selling. We are talking hundreds of millions of dollars invested in these programs at a time that Baltimore communities were going to blight and revenue being sent to programs that are not serving the desired purpose.
When Maryland public policy groups try to sell the public on the fact that Maryland is not bad as far as transparency I want to say……who paid you to say that? Just think of the non-existent Open Meetings where the public is left sitting while the meeting happens behind closed doors for goodness sake!