Regulations.gov, the federal government’s centralized regulatory docketing system that I look at in my new transparency paper, recently won an award from Government Computer News for “combining vision and IT innovations with an attention to detail and a willingness to collaborate.” The result of that award-winning combination, however, is not impressing everyone. A few days later the Congressional Research Service issued a report that catalogs the site’s shortcomings.1 (Another great dissection of Regulations.gov was performed by BNA and reported that “Cornell students studying human-computer interaction, when asked to evaluate the E-Rulemaking Web site’s public interface in early 2006, rated it ‘absolutely horrific[.]'”)
What’s striking to me is how what many believe is an unsatisfactory product is hailed as a success. Despite the hard work that many civil servants no doubt expended trying to make Regulations.gov a useful site, one has to admit it is confusing and difficult to use. Increased traffic is often cited by OMB in reports to Congress (PDF) as a measure of success. Increased web traffic was also mentioned in the GCN story about the award.
Looking at traffic, however, is tallying output, not outcomes; measuring activity, not results. One could conceivably build a website so unnavigable that it results in the number of “web hits” quadrupling because users have such a hard time finding what they need or because they have to click through many links before getting to what they want. Also, a total traffic number is difficult to judge. Are 150 million “hits” a good thing? Relative to what? Who knows.
Instead, what I’d like to know is whether Regulations.gov is making it easier for citizens to find and comment on regulatory proceedings. I see from the site’s “What’s New” section (I’d link to it but I can’t because the site uses 1990s-style frames technology2) that they conduct a regular “customer satisfaction survey.” I’d like to see those results published on the web. That sounds to me like a much better measurement of the site’s effectiveness.
-
From the CRS report:
To test the system, CRS attempted to locate information on a proposed rule issued by the U.S. Agency of International Development (USAID) in July 2007 that would exempt a new “Partner Vetting” system of records from the Privacy Act. The results differed depending on how the search was conducted. Using the “Search for Dockets” banner heading, identifying USAID as the issuing agency, searching for documents after July 1, 2007, and putting “partner vetting” in the keyword box yielded “no results.” However, using the “Search Documents” function, identifying USAID, identifying the “Subject” category, and putting “partner vetting” in the associated box yielded two dockets, one of which contained the subject rule and related documents and comments. Two other items were of note: (1) the list of agencies in the “Search Documents” function varied from one search to another (sometimes USAID was listed and sometimes it was not), and (2) the “Back” button did not allow the user to return to the previous page.
CRS also attempted to locate information on an EPA rule changing the emission standards for mercury. Using the “Search Documents” function, identifying EPA, using the “Subject” category, and putting “mercury” in the associated box led to a list of possible dockets, the first of which was for a January 2004 proposed rule. The docket contained a total of 6,902 documents across 277 pages of material. No index was provided, and the contents were not organized by type of document (e.g., agency- generated documents versus public comments) or chronologically.↑
- I tried opening the frame in a new window but I get an error message. It seems like the contents of the frame are dynamically generated. For an enlightening explanation of frames, as well as their pros and cons, see this article by Roger Johansson. The fact that this site, built in the 21st Century, uses frames is an indication of the technological problems within.↑
The Technology Liberation Front is the tech policy blog dedicated to keeping politicians' hands off the 'net and everything else related to technology.