Department of Energy

http://www.energy.gov/

Milestone 6 - February 28th 2015

OMB Review Complete: OMB has completed the agency review for this milestone. Agencies should contact their OMB desk officer if anything looks incorrect.

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status complete
Reviewer Jamie Berryhill
Last Updated April 17, 2015, 1:18 pm EDT by Jamie Berryhill

Assessment Summary

EDI: Excellent growth in datasets in the last quarter. Energy also appears to be making more data public.

Use and Impact: Energy provided detailed qualitative responses for all questions that show proof of continuous engagement with the user community and provide actionable recommendations for improvement. Energy also uses DAP on its /data page to track user metrics.

Note: The doughnut charts for Inventory Composition, Public Dataset Status, and Dataset Link Quality are new for this milestone, and they only appear is data are available. For detailed documentation on these charts, please see http://labs.data.gov/dashboard/docs#doughnut_charts. The "Other" category includes URLs that did not provide a response within 5 seconds of the automated crawl, which may indicate a server issue and not necessarily a completely non-working link.

Inventory Composition

Public Dataset Status

Dataset Link Quality

Status Indicator Automated Metrics
Overall Progress this Milestone
Inventory Updated this Quarter
1,057 Number of Datasets
23 Number of APIs
Schedule Delivered Crawl details
5 Bureaus represented
26 Programs represented
1006 Number of public datasets
34 Number of restricted public datasets
17 Number of non-public datasets
Inventory > Public listing
195.3% Percentage growth in records since last quarter
2,520 Spot Check - datasets listed by search engine
Agency provides a public Enterprise Data Inventory on Data.gov
License specified Crawl details
Status Indicator Automated Metrics
Overall Progress this Milestone
1040 Number of Datasets Crawl details
Number of Collections Crawl details
1004 Number of Public Datasets with File Downloads Crawl details
Number of APIs Crawl details
2307 Total number of access and download links Crawl details
Quality Check: Links are sufficiently working Crawl details
2236 Quality Check: Accessible links Crawl details
1.8 Quality Check: Redirected links Crawl details
0.7 Quality Check: Error links Crawl details
Quality Check: Broken links Crawl details
251 Percentage growth in records since last quarter
100 Valid Metadata Crawl details
/data exists Crawl details
/data.json Crawl details
Harvested by data.gov
4,574 Views on data.gov for the quarter
Status Indicator Automated Metrics
Overall Progress this Milestone
Description of feedback mechanism delivered Crawl details
Data release is prioritized through public engagement
Feedback loop is closed, 2 way communication
See below Link to or description of Feedback Mechanism
atiq.warraich@hq.doe.gov (although Energy still uses an email address, it has updated the address to go to a real person who actually works at the agency, and they are actively in the process developing two-way feedback. While not great, Energy is showing movement in the right direction)
Status Indicator Automated Metrics
Overall Progress this Milestone
Data Publication Process Delivered Crawl details
Information that should not to be made public is documented with agency's OGC
Status Indicator Automated Metrics
Overall Progress this Milestone
See below Open Data Primary Point of Contact
atiq.warraich@hq.doe.gov
POCs identified for required responsibilities

Best Practice: Department of Energy has been highlighted for demonstrating a best practice on the Use & Impact indicator

Status Indicator Automated Metrics
Overall Progress this Milestone
Identified 5 data improvements for this quarter
See below Primary Uses
The U.S. Energy Information Administration (EIA) is the principal federal statistical agency for energy and, in this capacity, collects, analyzes, and disseminates independent and impartial energy information to promote sound policymaking, efficient markets, and public understanding of energy and its interaction with the economy and the environment. EIA is the nation's premier source of unbiased energy data and conducts a comprehensive data collection and dissemination program that covers the full spectrum of energy sources, end uses, and energy flows. EIA’s data is used by the Administration, Congress, business and industry, financial markets, students, teachers, researchers, consultants, nonprofits, libraries, consumers, and other stakeholders for critical energy decisions, analysis, projections, news stories, investments, and academic work. Every year EIA conducts a web customer satisfaction survey, which in 2013 received 13,780 responses in seven days. When asked what task respondents were performing when they came to our website, the top responses were: Researching a topic Making a business decision Writing a report or article Educating myself about energy Making a forecast about energy Other: Overwhelmingly the #1 ‘other’ task was getting weekly gasoline and diesel prices to determine/calculate the fuel surcharge rate for drivers, invoicing, freight bill, etc.
See below Value or impact of data
The Weekly Natural Gas Storage Report—the amount of natural gas in underground storage (data collected by EIA) released by EIA each Thursday at 10:30 moves financial markets and has been designated by OMB as a Principal Economic Indicator (PEI). Thegasoline and diesel prices (data collected by EIA) released by EIA each Monday afternoon are used by businesses around the nation to calculate the fuel surcharge for their businesses. The special analyses EIA conducts at Congressional request use EIA data, models, and energy expertise to answer important policy questions. EIA has developed new interactive mapping tools with many illustrative layers that show potential impacts of hurricanes and floods on specific energy infrastructure. These maps are designed to be especially useful to emergency responders and state governments during crisis times or in planning situations. EIA data (both up-to-the-minute and historical time series covering a wide variety of fuel types and prices) allow a wide range of users to understand the energy situation in the United States and the world. It’s an unbiased, official starting point for all kinds of analysis and decision-making. The media use EIA data as an impartial source to write analysis pieces that reach wide audiences and have impacts far beyond the EIA website.
See below Primary data discovery channels
EIA holds an Energy Conference held annually in Washington, DC, which brings together energy thought leaders and data users from industry, government, and academia to discuss current and future challenges facing energy markets and policymakers; and attracts over 900 U.S. and international attendees. EIA regularly posts Federal Register Notices when it expands or otherwise changes a data collection effort. EIA participated in the first and second “Energy Datapalozza” events, sponsored by the White House. EIA hosts a “beta” website that encourages external users to test and comment on the agency’s latest product and data enhancements and review other users’ comments. This approach allows EIA to “crowd test” innovations before they are introduced on the agency’s official website. EIA regularly disseminates and promotes awareness of its data through a comprehensive social media presence (Twitter, Facebook, LinkedIn, Flickr and YouTube) and a radio news service that regularly distributes audio releases of key agency data and information. Every year, EIA partners with organizations such as the National Association of State Energy Officials, Johns Hopkins School of Advanced International Studies, and the Center for Strategic and International Studies to have them host several EIA press conferences to promote the agency’s major information and data releases (Winter Fuel Outlook, Summer Fuel Outlook, International Energy Outlook, and the Annual Energy Outlook). · EIA Information Center. EIA responds to an average of 6,000 customer inquiries per year through its Information Center. Customers are provided with customized responses to information and data requests, assistance with explanation and interpretation of data and conversions, referrals to EIA experts for in-depth data and information requirements, and referrals to other DOE and federal agencies, state agencies, and trade associations as appropriate. These services are available via email at InfoCtr@eia.gov, with an average response time of one business day. · External presentations. Key EIA staff regularly make presentations to outside groups. Last year, Office of Web Management Director Mark Elbert, for example, delivered presentations that covered the agency’s recent enhancements to its data program at DOE’s Data Analytics Summit and the annual Wolfram’s Data Summit. · Outreach to external organizations. Through many different outreach avenues, EIA works with external stakeholders such as Congress, federal and state government, industry, associations and non-profits, and the media. Many EIA offices have regularly scheduled meetings with outside organizations such as the International Energy Agency, the National Association of State Energy Officials, the International Association for Energy Economics, the World Energy Congress, the National Petroleum Council, the American Council for an Energy-Efficient Economy, and the National Association of Regulatory Utility Commissioners. · The American Statistical Association Advisory Committee. The American Statistical Association's Committee on Energy Statistics provides technical advice to EIA at semi-annual meetings. Held in the spring and fall in Washington, DC, these meetings are open to the public. Sessions are a combination of EIA briefings and discussions intended to solicit technical advice from the Committee members, who are primarily mathematical statisticians, survey statisticians, and energy analysts. When requested, Committee members also provide advice to EIA throughout the year. · Performing independent expert reviews. Since 1980, EIA has used its Independent Expert Review Program to seek technical reviews of its data and analytical work from outside subject matter experts as a way to guarantee high-quality products for our customers and data users. The reviews provide rigorous critiques and advice to EIA on: current and proposed data systems; energy forecasting models and modeling systems; energy forecasts; proposed analyses; and current and proposed publications. The reviewers provide written reports on their recommendations and meet with EIA staff to present, discuss, and defend their findings and proposed solutions. EIA engaged the National Academy of Science's (NAS) Committee on National Statistics (CNSTAT) to convene a panel to conduct a comprehensive 30-month study of the Commercial Buildings Energy Consumption Survey (CBECS) and the Residential Energy Consumption Survey (RECS). The panel considered how EIA may improve data quality, geographic coverage, timeliness of data releases, and relevance of the data, as well as reviewed the survey design, frequency, scope, survey practice, and operations. A preliminary report was issued in spring 2010 that considered design and data collection options for the next CBECS. The final report, published in February 2012, made recommendations for the design, conduct, and dissemination of CBECS and RECS information for the next decade and beyond, including the level of resources likely to be required. CNSTAT consulted with the National Academies' Board on Energy and Environmental Systems in preparing its report. In addition to its public workshops, which were arranged to gather insight from stakeholders and data users as well as from statistical experts, the panel commissioned research papers to inform its deliberations.
See below User suggestions on improving data usability
Make EIA’s data more accessible by developing additional interactive, online tools like the electricity and coal data browsers and interactive maps that integrate EIA’s energy data with data from other federal agencies and outside sources. EIA has launched a new state energy portal for the agency’s public website that features instant state comparisons, interactive state rankings, an advanced state data finder to key statistics and analysis, and interactive mapping. Building on this infrastructure, EIA releases an interactive energy disruption map that combines real-time data feeds from the National Hurricane Center with more than 20 map layers from EIA’s surveys, the U.S. Bureau of Ocean Energy Management, the U.S. Department of Transportation, and Ventyx.com; a flood vulnerability and assessment map that incorporates flood hazard information from the Federal Emergency Management Agency; and a full-page national map, available by theme, which EIA now promotes on its website as the U.S. Energy Mapping System. Add more EIA data to the agency’s Application Programming Interface and develop a bulk download capability. EIA's data Application Programming Interface (API) currently allows programmers to access to the agency’s petroleum, natural gas, electricity, coal, state energy, Annual Energy Outlook, and Short Term Energy Outlook data via a single web service. EIA’s newly added bulk download facility provides access to the same datasets via a short manifest linking to large downloadable files. The bulk download facility allows organizations needing the latest EIA data to download and efficiently process updates as indicated by the manifest in minutes rather than continuously trolling EIA's website. What users always want is to get the data faster. EIA has worked hard to trim the time between when we collect the data and when we make it available. It’s a trade-off, because the agency wants the data to be of high quality, so it doesn’t want to rush through the review and data editing process just to get some numbers out the door. · Conducting usability testing. When developing a new data product, EIA often conducts one-on-one usability testing, which offers web product developers a chance to see real users moving through content and applications while articulating the choices they are making. These user experiences are then used to refine our architectural choices (placement of labels and navigation, for example) and content choices (what to leave in, what to take out and how to better target information to achieve tasks). EIA has solicited potential participants for usability research and other types of product evaluations through the agency's email newsletters, an advertisement placed on EIA.gov, as well as other channels. · Developing user personas. EIA is among a growing number of federal agencies that have created
See below User suggestions on additional data releases
As a federal statistical agency, EIA releases pretty much everything it collects, with the exceptions of confidential data (the confidentiality has been promised to the respondents) and suppressed data (which is withheld to protect confidentiality). Users generally want more data on a gazillion other things. Although EIA has a limited budget, it tries to move as fast as resources will allow to cover the changing energy industry. As a result of the rapid growth and shifting location of U.S. natural gas and crude oil production, for example, EIA is expanding the geographic coverage of its current monthly natural gas production survey (otherwise known as the “914”) and adding collection of state-level data on crude oil and lease condensate production. But there are limits to what EIA can collect and provide.
Digital Analytics Program on /data

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

data.json
Expected Data.json URL http://www.energy.gov/data.json (From USA.gov Directory)
Resolved Data.json URL http://www.energy.gov/sites/prod/files/2015/02/f20/PDL-2-23-2015.json
Number of Redirects 1 redirects
HTTP Status 200
Content Type application/json
Valid JSON Valid
Detected Data.json Schema federal-v1.1
Datasets with Valid Metadata 100%(1026 of 1026)
Valid Schema Valid
Datasets 1026
Datasets with Distribution URLs 0.0% (0 of 1026)
Total Distribution URLs 0
Public Datasets 998
Restricted Public Datasets 28
Non-public Datasets 0
Bureaus Represented 5
Programs Represented 26
File Size 2.68MB
Last modified Monday, 23-Feb-2015 14:09:08 EST
Last crawl Wednesday, 25-Feb-2015 23:00:55 EST
Analyze archive copies Analyze archive from 2015-02-28
/data page
Expected /data URL http://www.energy.gov/data (From USA.gov Directory)
Resolved /data URL http://www.energy.gov/data/open-energy-data
Redirects 1 redirects
HTTP Status 200
Content Type text/html; charset=utf-8
Last modified Wednesday, 25-Feb-2015 23:00:51 EST
Last crawl Wednesday, 25-Feb-2015 23:00:55 EST
/digitalstrategy.json
Expected /digitalstrategy.json URL http://www.energy.gov/digitalstrategy.json (From USA.gov Directory)
Resolved /digitalstrategy.json URL http://www.energy.gov/sites/prod/files/2013/11/f5/digitalstrategy.json
Redirects 1 redirects
HTTP Status 200
Content Type application/json
Valid JSON Invalid Check a JSON Validator
Last modified Thursday, 21-Nov-2013 17:11:46 EST
Last crawl Wednesday, 25-Feb-2015 23:00:56 EST