Department of Energy

http://www.energy.gov/

Enterprise Data Inventory - Volume and composition over time

M-13-13 Milestone 11 - May 31st 2016

OMB Review Complete: OMB has completed the agency review for this milestone. Agencies should contact their OMB desk officer if anything looks incorrect.

Selected: M-13-13 Milestone 11 - May 31st 2016

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status	complete
Reviewer	Bryant Renaud
Last Updated	August 17, 2016, 4:08 pm EDT by Bryant Renaud

Assessment Summary

Other: No Chief Data Officer specified.

Lasted edited on July 22, 2016, 12:10 pm EDT by Bryant Renaud

Inventory Composition

Public Dataset Status

Dataset Link Quality

	Status	Indicator
	Overall Progress this Milestone
	Inventory Updated this Quarter
1483	Number of Datasets
158	Number of APIs
5	Bureaus represented
100%	Percentage of bureaus represented
21	Programs represented
39%	Percentage of programs represented
1291	Number of public datasets
187	Number of restricted public datasets
5	Number of non-public datasets
16.8%	Percentage growth in records since last quarter
To a great extent (50-75%)	To what extent is your agency’s Enterprise Data Inventory (EDI) complete?
See below	What steps have you taken to ensure your Enterprise Data Inventory is complete
Each Program and Staff Office has been given
	Agency provides a public Enterprise Data Inventory on Data.gov
	Agency provided updated Enterprise Data Inventory to OMB
100%	License specified	Crawl details
2	Number of datasets with redactions
0.1	Percent of datasets with redactions

Best Practice: Department of Energy has been highlighted for demonstrating a best practice on the Public Data Listing indicator

	Status	Indicator
	Overall Progress this Milestone
1420	Number of Datasets	Crawl details
19	Number of Collections	Crawl details
1205	Number of datasets not contained in a collection	Crawl details
1366	Number of Public Datasets with File Downloads	Crawl details
158	Number of APIs	Crawl details
4	Number of public APIs	Crawl details
154	Number of restricted public APIs	Crawl details
	Number of non-public APIs	Crawl details
3988	Total number of access and download links	Crawl details
	Quality Check: Links are sufficiently working	Crawl details
1394	Quality Check: Accessible links	Crawl details
2569	Quality Check: Redirected links	Crawl details
2	Quality Check: Error links	Crawl details
3	Quality Check: Broken links	Crawl details
95.9	Quality Check: Percentage of download links in correct format as specified in metadata	Crawl details
12.6	Quality Check: Percentage of download links in HTML	Crawl details
4.2	Quality Check: Percentage of download links in PDF	Crawl details
13.4	Percentage growth in records since last quarter
100%	Valid Metadata	Crawl details
	/data exists	Crawl details
	Provides datasets in human-readable form on /data
	/data.json	Crawl details
	Harvested by data.gov
1234	Number of public datasets	Crawl details
181	Number of restricted public datasets	Crawl details
5	Number of non-public datasets	Crawl details
14.9	Percent growth of public datasets
1.7	Percent growth of restricted public datasets
	Percent growth of non-public datasets
18.5	Percent datasets licensed as U.S. Public Domain
76.7	Percent datasets licensed as Creative Commons Zero
4.9	Percent datasets with other licenses
	Percent datasets with no license

	Status	Indicator
	Overall Progress this Milestone
	Description of feedback mechanism delivered	Crawl details
	Data release is prioritized through public engagement
	Provided narrative evidence of data improvements based on public feedback this quarter
	Feedback loop is closed, 2 way communication
See below	Link to or description of Feedback Mechanism
http://energy.gov/data/public_engagement
	Provides valid contact point information for all datasets

	Status	Indicator
	Overall Progress this Milestone
	Data Publication Process Delivered	Crawl details
	Information that should not to be made public is documented with agency's OGC
See below	Describe the agency's data publication process
http://www.energy.gov/data/procedures-public-release-data

	Status	Indicator	Automated Metrics
	Overall Progress this Milestone
See below	Open Data Primary Point of Contact
Robbie.Green@hq.doe.gov
	POCs identified for required responsibilities
	Chief Data Officer (if applicable)

	Status	Indicator	Automated Metrics
	Overall Progress this Milestone
	Provided narrative evidence of open data impacts for this quarter
	Digital Analytics Program on /data
1584	Views on data.gov for this quarter
3.8%	Percentage growth in views on data.gov for this quarter
4699	Views on agency /data page for this quarter

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

data.json

Expected Data.json URL	http://www.energy.gov/data.json (From USA.gov Directory)
Resolved Data.json URL	http://www.energy.gov/sites/prod/files/2016/05/f32/doe-pdl-5-26-2016.json
Number of Redirects	1 redirects
HTTP Status	200
Content Type	application/json
Valid JSON	Valid
Detected Data.json Schema	federal-v1.1
Datasets with Valid Metadata	100%(1420 of 1420)
Valid Schema	Valid
Datasets	1420
Number of Collections	19
Number of datasets not in a collection	1205
Datasets with Distribution URLs	96.2% (1366 of 1420)
Datasets with Download URLs	48.9% (694 of 1420)
Total Distribution URLs	3988
Total Download URLs	998
Total APIs	158
Public APIs	4
Restricted Public APIs	154
Non-public APIs	0
Public Datasets	1234
Restricted Public Datasets	181
Non-public Datasets	5
Normally there would be a set of quality assurance fields here to verify that the download links included within the metadata are functioning properly, but the results of those tests are not currently available.
Bureaus Represented	5
Programs Represented	18
License Specified	100% (1420 of 1420)
Datasets with Redactions	0.1% (2 of 1420)
Redactions without explanation (rights field)	0.1% (1 of 1420)
File Size	4.58MB
Last modified	Thursday, 26-May-2016 10:59:01 EDT
Last crawl	Monday, 30-May-2016 00:02:30 EDT
Analyze archive copies	Analyze archive from 2016-05-31
Nearby Daily Crawls	Wednesday, 25-May-2016 04:13:12 EDT Thursday, 26-May-2016 04:12:42 EDT Friday, 27-May-2016 04:12:38 EDT Saturday, 28-May-2016 04:12:49 EDT Sunday, 29-May-2016 04:00:02 EDT Tuesday, 31-May-2016 04:01:47 EDT

/data page

Expected /data URL	http://www.energy.gov/data (From USA.gov Directory)
Resolved /data URL	http://www.energy.gov/data
Redirects
HTTP Status	0
Content Type
Last crawl	Monday, 30-May-2016 00:02:06 EDT

/digitalstrategy.json

Expected /digitalstrategy.json URL	http://www.energy.gov/digitalstrategy.json (From USA.gov Directory)
Resolved /digitalstrategy.json URL	http://www.energy.gov/sites/prod/files/2016/05/f31/digitalstrategy-5-23-2016-v2_0.json
Redirects	1 redirects
HTTP Status	200
Content Type	application/json
Valid JSON	Valid
Last modified	Monday, 23-May-2016 11:51:44 EDT
Last crawl	Monday, 30-May-2016 00:02:10 EDT

Digital Strategy

Date specified: Thursday, 21-Nov-2013 13:31:01 EST

Date of digitalstrategy.json file: Monday, 23-May-2016 11:51:44 EDT

1.2.4 Develop Data Inventory Schedule - Summary

Summarize the Inventory Schedule

The attached schedule provides a rough outline of the activities the Department of Energy will undertake from Nov 2013 - Nov 2014 to enrich, expand and open the Department's data inventory.

1.2.5 Develop Data Inventory Schedule - Milestones

Title	Develop Initial Enterprise Data Inventory (EDI)
Description	Create Enterprise Data Inventory and post JSON File to OMB MAX site
Milestone Date	2013-11-29
Description of how this milestone expands the Inventory	Develop a robust, enterprise-wide data inventory
Description of how this milestone enriches the Inventory	Develop a single repository to document and categorize the Department's enterprise data assets
Description of how this milestone opens the Inventory	Validate publically-available datasets and enhance metadata to improve public access to information

Title	Post Initial Public Data Listings
Description	Review Enterprise Data Inventory and post non-sensitive datasets on DOE's website
Milestone Date	2013-11-29
Description of how this milestone expands the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone enriches the Inventory	Validate publically-available datasets and enhance metadata to improve public access to information
Description of how this milestone opens the Inventory	Expand the number of datasets available to the public

Title	Update the Enterprise Data Inventory on a Quarterly Basis
Description	Collaborate with Program and Staff Offices to ensure all data assets have been identified and accounted for in the inventory
Milestone Date	2014-11-01
Description of how this milestone expands the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone enriches the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone opens the Inventory

Title	Quarterly Update to EDI
Description	Work with 3 to 5 Program/Staff Offices to collect data assets and update the inventory.
Milestone Date	2014-02-28
Description of how this milestone expands the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone enriches the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone opens the Inventory

Title	Quarterly Update to EDI
Description	Work with 3 to 5 Program/Staff Offices to collect data assets and update the inventory.
Milestone Date	2014-05-31
Description of how this milestone expands the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone enriches the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone opens the Inventory

Title	Quarterly Update to EDI
Description	Work with 3 to 5 Program/Staff Offices to collect data assets and update the inventory.
Milestone Date	2014-08-31
Description of how this milestone expands the Inventory	Expand the number of datasets available to the public
Description of how this milestone enriches the Inventory	Add new datasets and make progress in developing the EDI
Description of how this milestone opens the Inventory

Title	Quarterly Update to EDI
Description	Work with 3 to 5 Program/Staff Offices to collect data assets and update the inventory.
Milestone Date	2014-11-01
Description of how this milestone expands the Inventory
Description of how this milestone enriches the Inventory
Description of how this milestone opens the Inventory

Title	Provide Monthly Updates to Public Data Listings
Description	Post new datasets to Energy's web site on a monthly basis
Milestone Date	Monthly
Description of how this milestone expands the Inventory
Description of how this milestone enriches the Inventory
Description of how this milestone opens the Inventory

1.2.6 Develop Customer Feedback Process

Describe the agency's process to engage with customers

The U.S. Department of Energy will use the energy.gov platform and 3rd party social media to engage with customers and collect input on data. Specifically, energy.gov/data and energy.gov/developer will contain articles and blog posts that may be used to directly seek information from the public in compliance with the Memorandum on Social Media, Web-Based Interactive Technologies, and the Paperwork Reduction Act. In addition, the DOE Web Council is made up from stakeholders across all branches of the complex and the National Labs. This council has expertise that will be invaluable for determining which projects should be prioritized for transparency and impact. We will solicit their feedback via email and / or in person meetings.  Program offices may use additional means to best engage their specific stakeholders.  Metrics may be collected in a variety of ways, including web analytics.

1.2.7 Develop Data Publication Process

Describe the agency's data publication process

The Department of Energy's data release procedures are available at http://energy.gov/node/771991