Department of the Interior

http://www.doi.gov/
Enterprise Data Inventory - Volume and composition over time

M-13-13 Milestone 11 - May 31st 2016

OMB Review Complete: OMB has completed the agency review for this milestone. Agencies should contact their OMB desk officer if anything looks incorrect.

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status complete
Reviewer Bryant Renaud
Last Updated August 17, 2016, 4:10 pm EDT by Bryant Renaud

Assessment Summary

Fails to document non-public, and restricted public datasets in EDI. EDI is public but not the same file as PDL. Due to technical issues some automated metrics are not available for this agency for this quarter.

Other: DAP tracker not properly installed on /data page. Errors in json file see duplicate "accessLevel" fields.

Fails to have 100% valid data.json

Lasted edited on July 22, 2016, 12:48 pm EDT by Justin Grimes

Inventory Composition

Public Dataset Status

Status Indicator Automated Metrics
Overall Progress this Milestone
Inventory Updated this Quarter
95850 Number of Datasets
376 Number of APIs
6 Bureaus represented
22% Percentage of bureaus represented
4 Programs represented
3% Percentage of programs represented
40938 Number of public datasets
54912 Number of restricted public datasets
Number of non-public datasets
Percentage growth in records since last quarter
To some extent (25-50%) To what extent is your agency’s Enterprise Data Inventory (EDI) complete?
See below What steps have you taken to ensure your Enterprise Data Inventory is complete
The volume of data that DOI has is more than huge. In order to ensure data quality, data security, and data reliability, increase interoperablity, DOI has updated DOI Data Resource Management Departmental Manuals, and four data related policies. They are Data Governance, DRM Strategy, Metadata Policy and Open Data Policy. DOI Bureaus and Offices have been working on establishing data governance and data stewardship groups working with Programs SMEs, Data Professionals and IT Specialists. These efforts will root data resource management into many mission program areas, connect and support information management processes, acquisitions, project management. These efforts will build effectiveness and efficiency to create, manage, and share data for internal and external data users. DOI continuously working on Data Governance, to make sure that data are reviewed, checked with LRM before data are released. Completing EDI is a long process. Building the useful, meaningful EDI will take even more efforts.
Agency provides a public Enterprise Data Inventory on Data.gov
Agency provided updated Enterprise Data Inventory to OMB
100% License specified Crawl details
Number of datasets with redactions
Percent of datasets with redactions
Status Indicator Automated Metrics
Overall Progress this Milestone
40935 Number of Datasets Crawl details
Number of Collections Crawl details
40916 Number of datasets not contained in a collection Crawl details
40462 Number of Public Datasets with File Downloads Crawl details
376 Number of APIs Crawl details
376 Number of public APIs Crawl details
Number of restricted public APIs Crawl details
Number of non-public APIs Crawl details
65883 Total number of access and download links Crawl details
Quality Check: Links are sufficiently working Crawl details
Quality Check: Accessible links Crawl details
Quality Check: Redirected links Crawl details
Quality Check: Error links Crawl details
Quality Check: Broken links Crawl details
Quality Check: Percentage of download links in correct format as specified in metadata Crawl details
Quality Check: Percentage of download links in HTML Crawl details
Quality Check: Percentage of download links in PDF Crawl details
Percentage growth in records since last quarter
95.8% Valid Metadata Crawl details
/data exists Crawl details
Provides datasets in human-readable form on /data
/data.json Crawl details
Harvested by data.gov
Number of public datasets Crawl details
Number of restricted public datasets Crawl details
Number of non-public datasets Crawl details
-38.7 Percent growth of public datasets
Percent growth of restricted public datasets
Percent growth of non-public datasets
Percent datasets licensed as U.S. Public Domain
Percent datasets licensed as Creative Commons Zero
Percent datasets with other licenses
Percent datasets with no license
Status Indicator Automated Metrics
Overall Progress this Milestone
Description of feedback mechanism delivered Crawl details
Data release is prioritized through public engagement
Provided narrative evidence of data improvements based on public feedback this quarter
Feedback loop is closed, 2 way communication
See below Link to or description of Feedback Mechanism
http://usinterior.ideascale.com/
Provides valid contact point information for all datasets
Status Indicator Automated Metrics
Overall Progress this Milestone
Data Publication Process Delivered Crawl details
Information that should not to be made public is documented with agency's OGC
See below Describe the agency's data publication process
DOI will develop a method for discovering unidentified datasets and linking existing datasets to new uses in a methodical way to support business goals. In collaboration with DOI security and PII professionals, the Data Service Team will develop a process for data release of those datasets that are not already public, and a method for determining when data are of a sensitive nature and should not be made public. The majority of DOI mission data is already available publically.

Best Practice: Department of the Interior has been highlighted for demonstrating a best practice on the Human Capital indicator

Status Indicator Automated Metrics
Overall Progress this Milestone
See below Open Data Primary Point of Contact
lin_zhang@ios.doi.gov
POCs identified for required responsibilities
See below Chief Data Officer (if applicable)
jerry_johnston@ios.doi.gov
Status Indicator Automated Metrics
Overall Progress this Milestone
Provided narrative evidence of open data impacts for this quarter
Digital Analytics Program on /data
12566 Views on data.gov for this quarter
2.4% Percentage growth in views on data.gov for this quarter
Views on agency /data page for this quarter

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

data.json
Expected Data.json URL http://www.doi.gov/data.json (From USA.gov Directory)
Resolved Data.json URL https://data.doi.gov/data.json
Number of Redirects 2 redirects
HTTP Status 200
Content Type text/plain; charset=UTF-8
Valid JSON Valid
Datasets with Valid Metadata 95.8%(39214 of 40916)
Valid Schema Invalid
For more complete and readable validation results, see the full schema validator results
Schema Errors There are validation errors on 1702 records

Only showing errors from the first 10 records:

Errors on record 0:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 1:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 2:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 3:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 4:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 5:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 6:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 7:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 8:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 9:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Datasets 40916
Number of Collections 0
Number of datasets not in a collection 40916
Datasets with Distribution URLs 98.9% (40462 of 40916)
Datasets with Download URLs 98.9% (40462 of 40916)
Total Distribution URLs 65883
Total Download URLs 65883
Total APIs 0
Public APIs 0
Restricted Public APIs 0
Non-public APIs 0
Public Datasets 40916
Restricted Public Datasets 0
Non-public Datasets 0
Bureaus Represented 4
Programs Represented 6
License Specified 100% (40916 of 40916)
Datasets with Redactions 0.0% (0 of 40916)
Redactions without explanation (rights field) 0.0% (0 of 40916)
File Size 114.67MB
Last modified Sunday, 22-May-2016 01:56:04 EDT
Last crawl Saturday, 28-May-2016 00:10:48 EDT
Analyze archive copies Analyze archive from 2016-05-31
Nearby Daily Crawls
/data page
Expected /data URL http://www.doi.gov/data (From USA.gov Directory)
Resolved /data URL https://www.doi.gov/data
Redirects 1 redirects
HTTP Status 200
Content Type text/html; charset=utf-8
Last modified Friday, 27-May-2016 00:00:36 EDT
Last crawl Saturday, 28-May-2016 00:00:48 EDT
/digitalstrategy.json
Expected /digitalstrategy.json URL http://www.doi.gov/digitalstrategy.json (From USA.gov Directory)
Resolved /digitalstrategy.json URL https://www.doi.gov/digitalstrategy.json
Redirects 1 redirects
HTTP Status 404
Content Type text/html; charset=utf-8
Valid JSON Invalid Check a JSON Validator
Last modified Friday, 27-May-2016 20:48:13 EDT
Last crawl Saturday, 28-May-2016 00:00:48 EDT