Department of Energy

M-13-13 Milestone 7 - May 31st 2015

OMB Review In Progress: OMB is currently reviewing the agency for this milestone. This review status indicator will change once the review is complete.

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status in-progress
Reviewer Rebecca Williams
Last Updated August 24, 2015, 8:55 am EDT by Rebecca Williams

Assessment Summary

Public EDI is in the wrong format, with restrictive licensing and does not have an access level column:

PDL: Only 1 API is listed. 2% growth. Links are for the majority accessible.

Public Engagement:

Privacy & Security:

Use & Impact

Public Dataset Status

Dataset Link Quality

Status Indicator Automated Metrics
Overall Progress this Milestone
Inventory Updated this Quarter
1062 Number of Datasets
1 Number of APIs
Schedule Delivered Crawl details
5 Bureaus represented
26 Programs represented
? Number of public datasets
? Number of restricted public datasets
? Number of non-public datasets
Inventory > Public listing
+2% Percentage growth in records since last quarter
154,932 + Spot Check - datasets listed by search engine
Agency provides a public Enterprise Data Inventory on
License specified Crawl details
Status Indicator Automated Metrics
Overall Progress this Milestone
1062 Number of Datasets Crawl details
Number of Collections Crawl details
786 Number of Public Datasets with File Downloads Crawl details
1 Number of APIs Crawl details
2418 Total number of access and download links Crawl details
Quality Check: Links are sufficiently working Crawl details
2338 Quality Check: Accessible links Crawl details
43 Quality Check: Redirected links Crawl details
Quality Check: Error links Crawl details
20 Quality Check: Broken links Crawl details
+2% Percentage growth in records since last quarter
100% Valid Metadata Crawl details
/data exists Crawl details
/data.json Crawl details
Harvested by
Views on for the quarter
Status Indicator Automated Metrics
Overall Progress this Milestone
Description of feedback mechanism delivered Crawl details
Data release is prioritized through public engagement
Feedback loop is closed, 2 way communication
See below Link to or description of Feedback Mechanism
Status Indicator Automated Metrics
Overall Progress this Milestone
Data Publication Process Delivered Crawl details
Information that should not to be made public is documented with agency's OGC
Status Indicator Automated Metrics
Overall Progress this Milestone
See below Open Data Primary Point of Contact
POCs identified for required responsibilities
Status Indicator Automated Metrics
Overall Progress this Milestone
Identified 5 data improvements for this quarter
See below Primary Uses
The U.S. Energy Information Administration (EIA) is the principal federal statistical agency for energy and, in this capacity, collects, analyzes, and disseminates independent and impartial energy information to promote sound policymaking, efficient markets, and public understanding of energy and its interaction with the economy and the environment. EIA is the nation's premier source of unbiased energy data and conducts a comprehensive data collection and dissemination program that covers the full spectrum of energy sources, end uses, and energy flows. EIA’s data is used by the Administration, Congress, business and industry, financial markets, students, teachers, researchers, consultants, nonprofits, libraries, consumers, and other stakeholders for critical energy decisions, analysis, projections, news stories, investments, and academic work. Every year EIA conducts a web customer satisfaction survey, which in 2013 received 13,780 responses in seven days. When asked what task respondents were performing when they came to our website, the top responses were: Researching a topic Making a business decision Writing a report or article Educating myself about energy Making a forecast about energy Other: Overwhelmingly the #1 ‘other’ task was getting weekly gasoline and diesel prices to determine/calculate the fuel surcharge rate for drivers, invoicing, freight bill, etc.
Value or impact of data
See below Primary data discovery channels
EIA holds an Energy Conference held annually in Washington, DC, which brings together energy thought leaders and data users from industry, government, and academia to discuss current and future challenges facing energy markets and policymakers; and attracts over 900 U.S. and international attendees. EIA regularly posts Federal Register Notices when it expands or otherwise changes a data collection effort. EIA participated in the first and second “ Energy Datapalozza ” events, sponsored by the White House. EIA hosts a “beta” website that encourages external users to test and comment on the agency’s latest product and data enhancements and review other users’ comments. This approach allows EIA to “crowd test” innovations before they are introduced on the agency’s official website. EIA regularly disseminates and promotes awareness of its data through a comprehensive social media presence (Twitter, Facebook, LinkedIn, Flickr and YouTube) and a radio news service that regularly distributes audio releases of key agency data and information. Every year, EIA partners with organizations such as the National Association of State Energy Officials, Johns Hopkins School of Advanced International Studies, and the Center for Strategic and International Studies to have them host several EIA press conferences to promote the agency’s major information and data releases (Winter Fuel Outlook, Summer Fuel Outlook, International Energy Outlook, and the Annual Energy Outlook). · EIA Information Center. EIA responds to an average of 6,000 customer inquiries per year through its Information Center. Customers are provided with customized responses to information and data requests, assistance with explanation and interpretation of data and conversions, referrals to EIA experts for in-depth data and information requirements, and referrals to other DOE and federal agencies, state agencies, and trade associations as appropriate. These services are available via email at, with an average response time of one business day. · External presentations. Key EIA staff regularly make presentations to outside groups. Last year, Office of Web Management Director Mark Elbert, for example, delivered presentations that covered the agency’s recent enhancements to its data program at DOE’s Data Analytics Summit and the annual Wolfram’s Data Summit. · Outreach to external organizations. Through many different outreach avenues, EIA works with external stakeholders such as Congress, federal and state government, industry, associations and non-profits, and the media. Many EIA offices have regularly scheduled meetings with outside organizations such as the International Energy Agency, the National Association of State Energy Officials, the International Association for Energy Economics, the World Energy Congress, the National Petroleum Council, the American Council for an Energy-Efficient Economy, and the National Association of Regulatory Utility Commissioners. · The American Statistical Association Advisory Committee. The American Statistical Association's Committee on Energy Statistics provides technical advice to EIA at semi-annual meetings. Held in the spring and fall in Washington, DC, these meetings are open to the public. Sessions are a combination of EIA briefings and discussions intended to solicit technical advice from the Committee members, who are primarily mathematical statisticians, survey statisticians, and energy analysts. When requested, Committee members also provide advice to EIA throughout the year. · Performing independent expert reviews. Since 1980, EIA has used its Independent Expert Review Program to seek technical reviews of its data and analytical work from outside subject matter experts as a way to guarantee high-quality products for our customers and data users. The reviews provide rigorous critiques and advice to EIA on: current and proposed data systems; energy forecasting models and modeling systems; energy forecasts; proposed analyses; and current and proposed publications. The reviewers provide written reports on their recommendations and meet with EIA staff to present, discuss, and defend their findings and proposed solutions. EIA engaged the National Academy of Science's (NAS) Committee on National Statistics (CNSTAT) to convene a panel to conduct a comprehensive 30-month study of the Commercial Buildings Energy Consumption Survey (CBECS) and the Residential Energy Consumption Survey (RECS). The panel considered how EIA may improve data quality, geographic coverage, timeliness of data releases, and relevance of the data, as well as reviewed the survey design, frequency, scope, survey practice, and operations. A preliminary report was issued in spring 2010 that considered design and data collection options for the next CBECS. The final report, published in February 2012, made recommendations for the design, conduct, and dissemination of CBECS and RECS information for the next decade and beyond, including the level of resources likely to be required. CNSTAT consulted with the National Academies' Board on Energy and Environmental Systems in preparing its report. In addition to its public workshops, which were arranged to gather insight from stakeholders and data users as well as from statistical experts, the panel commissioned research papers to inform its deliberations.
See below User suggestions on improving data usability
Make EIA’s data more accessible by developing additional interactive, online tools like the electricity and coal data browsers and interactive maps that integrate EIA’s energy data with data from other federal agencies and outside sources. EIA has launched a new state energy portal for the agency’s public website that features instant state comparisons, interactive state rankings, an advanced state data finder to key statistics and analysis, and interactive mapping. Building on this infrastructure, EIA releases an interactive energy disruption map that combines real-time data feeds from the National Hurricane Center with more than 20 map layers from EIA’s surveys, the U.S. Bureau of Ocean Energy Management, the U.S. Department of Transportation, and; a flood vulnerability and assessment map that incorporates flood hazard information from the Federal Emergency Management Agency; and a full-page national map, available by theme, which EIA now promotes on its website as the U.S. Energy Mapping System. Add more EIA data to the agency’s Application Programming Interface and develop a bulk download capability . EIA's data Application Programming Interface (API) currently allows programmers to access to the agency’s petroleum, natural gas, electricity, coal, state energy, Annual Energy Outlook, and Short Term Energy Outlook data via a single web service. EIA’s newly added bulk download facility provides access to the same datasets via a short manifest linking to large downloadable files. The bulk download facility allows organizations needing the latest EIA data to download and efficiently process updates as indicated by the manifest in minutes rather than continuously trolling EIA's website. What users always want is to get the data faster . EIA has worked hard to trim the time between when we collect the data and when we make it available. It’s a trade-off, because the agency wants the data to be of high quality, so it doesn’t want to rush through the review and data editing process just to get some numbers out the door. · Conducting usability testing. When developing a new data product, EIA often conducts one-on-one usability testing, which offers web product developers a chance to see real users moving through content and applications while articulating the choices they are making. These user experiences are then used to refine our architectural choices (placement of labels and navigation, for example) and content choices (what to leave in, what to take out and how to better target information to achieve tasks). EIA has solicited potential participants for usability research and other types of product evaluations through the agency's email newsletters, an advertisement placed on, as well as other channels. · Developing user personas. EIA is among a growing number of federal agencies that have created
See below User suggestions on additional data releases
As a federal statistical agency, EIA releases pretty much everything it collects, with the exceptions of confidential data (the confidentiality has been promised to the respondents) and suppressed data (which is withheld to protect confidentiality). Users generally want more data on a gazillion other things. Although EIA has a limited budget, it tries to move as fast as resources will allow to cover the changing energy industry. As a result of the rapid growth and shifting location of U.S. natural gas and crude oil production, for example, EIA is expanding the geographic coverage of its current monthly natural gas production survey (otherwise known as the “914”) and adding collection of state-level data on crude oil and lease condensate production. But there are limits to what EIA can collect and provide.
Digital Analytics Program on /data

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

Expected Data.json URL (From Directory)
Resolved Data.json URL
Number of Redirects 1 redirects
HTTP Status 200
Content Type application/json
Valid JSON Invalid Check a JSON Validator
Detected Data.json Schema federal-v1.1
Datasets with Valid Metadata 100%(1062 of 1062) - The JSON file is invalid and can't be parsed without special processing
Valid Schema Valid
Datasets 1062
Datasets with Distribution URLs 96.6% (1026 of 1062)
Datasets with Download URLs 74.0% (786 of 1062)
Total Distribution URLs 2418
Total Download URLs 1136
Total APIs 1
Public Datasets 1027
Restricted Public Datasets 35
Non-public Datasets 0
Bureaus Represented 5
Programs Represented 26
License Specified 100% (1062 of 1062)
Datasets with Redactions 0.0% (0 of 1062)
Redactions without explanation (rights field) 0.0% (0 of 1062)
File Size 3.33MB
Last modified Tuesday, 26-May-2015 11:27:08 EDT
Last crawl Sunday, 31-May-2015 00:01:39 EDT
Analyze archive copies Analyze archive from 2015-05-31
Nearby Daily Crawls
/data page
Expected /data URL (From Directory)
Resolved /data URL
HTTP Status 0
Content Type
Last crawl Sunday, 31-May-2015 00:00:51 EDT
Expected /digitalstrategy.json URL (From Directory)
Resolved /digitalstrategy.json URL
HTTP Status 0
Content Type
Valid JSON Invalid Check a JSON Validator
Last crawl Sunday, 31-May-2015 00:00:59 EDT