Department of the Interior

http://www.doi.gov/
Enterprise Data Inventory - Volume and composition over time

M-13-13 Milestone 8 - August 31st 2015

OMB Review Complete: OMB has completed the agency review for this milestone. Agencies should contact their OMB desk officer if anything looks incorrect.

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status complete
Reviewer Justin Grimes
Last Updated October 6, 2015, 10:16 am EDT by Justin Grimes

Assessment Summary

Steps for improving:

1) Improve metadata quality and formatting issues in PDL 2) Increase number of published datasets (e.g. add download links) 3) Provide responses to "Use and Impact" questions

Lasted edited on October 6, 2015, 10:01 am EDT by Justin Grimes

Inventory Composition

Public Dataset Status

Dataset Link Quality

Status Indicator Automated Metrics
Overall Progress this Milestone
Inventory Updated this Quarter
88164 Number of Datasets
122 Number of APIs
Schedule Delivered Crawl details
3 Bureaus represented
5 Programs represented
87260 Number of public datasets
904 Number of restricted public datasets
Number of non-public datasets
Inventory > Public listing
4.5% Percentage growth in records since last quarter
Spot Check - datasets listed by search engine
Agency provides a public Enterprise Data Inventory on Data.gov
100% License specified Crawl details
Status Indicator Automated Metrics
Overall Progress this Milestone
83515 Number of Datasets Crawl details
Number of Collections Crawl details
54207 Number of Public Datasets with File Downloads Crawl details
Number of APIs Crawl details
75639 Total number of access and download links Crawl details
Quality Check: Links are sufficiently working Crawl details
69014 Quality Check: Accessible links Crawl details
5089 Quality Check: Redirected links Crawl details
77 Quality Check: Error links Crawl details
1271 Quality Check: Broken links Crawl details
-1% Percentage growth in records since last quarter
98.5% Valid Metadata Crawl details
/data exists Crawl details
/data.json Crawl details
Harvested by data.gov
11450 Views on data.gov for the quarter
Status Indicator Automated Metrics
Overall Progress this Milestone
Description of feedback mechanism delivered Crawl details
Data release is prioritized through public engagement
Feedback loop is closed, 2 way communication
See below Link to or description of Feedback Mechanism
https://www.doi.gov/digitalstrategy/
Status Indicator Automated Metrics
Overall Progress this Milestone
Data Publication Process Delivered Crawl details
Information that should not to be made public is documented with agency's OGC
Status Indicator Automated Metrics
Overall Progress this Milestone
See below Open Data Primary Point of Contact
lin_zhang@ios.doi.gov
POCs identified for required responsibilities
Status Indicator Automated Metrics
Overall Progress this Milestone
Identified 5 data improvements for this quarter
See below Primary Uses
These are examples of the primary users of U.S. Department of the Interior (DOI). The USGS Water Use Information Program investigates the occurrence, quantity, quality, distribution, and movement of surface and underground waters and disseminates the data to the public, State and local governments, public and private utilities, emergency responders, private sector (Kayakers, Fishing Industry, etc.) and other Federal agencies involved with managing our water resources. USGS data such as this is critical to both planning for and responding to hazards such as tsunamis, and for long-term planning addressing climate change, coastal erosion and ongoing coastal development. Private companies use DOI data in developing new services and products to meet public needs in many areas. While it looks as though these questions may be for internal purposes only, Azavea has endeavored to answer them as they apply to our use of Department of Interior datasets. We hope the information is helpful. Azavea uses the data in a number of contexts, including software development, GIS mapping, and data analytics. From: Parsons Brinckerhoff Parsons Brinckerhoff utilizes these datasets to create 3D models for project pursuits and project management. Uses range from master planning to roadway design. They are highly used resources for data reconnaissance during pursuits as well as the foundation for design on various projects. From USGS: USGS provides data to a very wide range of users, including scientists and academic institutions in dozens of disciplines, disaster management and planning authorities, mapping and geospatial companies, and Federal, state and local land management agencies From ONRR: The Office of Natural Resources Revenue (ONRR) initiated project in 2004 to implement new technology and processes to provide statistical information on the web and to support a centralized data request function. This function is managed by the Data Services & Statistical Reporting Office (DSSR) utilizing new technology and software, the Information Request and Query Management (IRQM) tool, to compile program data for either web publishing activities or data requests. The established process and procedures allow for consistent, verifiable, and reproducible collection and analysis of data in response to data requests from both internal and external entities. These requests include routine data collection for numerous federal agencies (i.e. BLM, BIA, OMB, DOJ, EIA, GAO, OIG, etc.), States, and American Indian tribes, and non-routine requests from the ONRR Data Requests Mailbox. The DSSR responses and provides data sets to approximately 500 data requests annually. In response to our customers, the ONRR Statistical Website was updated in 2012 to provide better and easier navigation throughout the site, downloadable data sets, and better explanation of our data.
Value or impact of data
See below Primary data discovery channels
Through the DOI data catalog, DOI websites, and data.gov. In addition, through the USGS.gov websites, state Water Offices, state authorities, universities, Press Releases, Fact Sheets, Publications, and Research shared through peer-reviewed literature. This is an ongoing, close relationship with key partners where data is shared and made available in the course of the work. In addition, Azavea subscribes to e-mails, listservs, Twitter and other alerts that keep us apprised of new releases or updates. Further, we regularly sponsor and attend hackathons focused on open data to both share and learn more about available data sources. From: Parsons Brinckerhoff: Datasets are primarily learned by attending professional societies such as; the Transportation Research Board (TRB), International Highway Engineering Exchange Program (IHEEP), Geospatial Organizations; Esri Users Conference, Coalition of Geospatial Organizations (COGO) and the American Society of Civil Engineers (ASCE). Vendor consultation and basic research on the internet has also led to discovery. From USGS In addition to the challenge outlined above, USGS has participated in two other challenges (addressing climate and biodiversity), and routinely corresponds with citizens on a range of scientific topics by email, phone and through social media From ONRR: The primary use of our data is for revenue stream analysis and support for budget activities The primary channels for the users to learn about our data is through onrr.gov, useiti.doi.gov, and data.gov
See below User suggestions on improving data usability
Suggestions for improving access and usability of this data has focused around providing additional services for USGS data, including: stream site locations, gauge information, regional local view/subset and various data formats. From: Parsons Brinckerhoff: Standardization, interoperability between datasets From ONRR We have received the following suggestion on improving the usability of our data: o Ability to query variable date ranges (i.e. FY, CY, specific monthly date ranges) Suggestions on additional data resources to release o Allocated production volumes o County-level reported royalty revenues and production volumes
See below User suggestions on additional data releases
From: Parsons Brinckerhoff: Continued refinement on the fidelity/accuracy of the data. From USGS: Users have requested that USGS data be made available in more aggregated sets, and linked with complementary or similar data from other agencies. These are central goals of the Open Water Data Initiative (http://acwi.gov/spatial/owdi/)
Digital Analytics Program on /data

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

data.json
Expected Data.json URL http://www.doi.gov/data.json (From USA.gov Directory)
Resolved Data.json URL https://data.doi.gov/data.json
Number of Redirects 2 redirects
HTTP Status 200
Content Type text/plain; charset=UTF-8
Valid JSON Valid
Datasets with Valid Metadata 98.5%(82242 of 83515)
Valid Schema Invalid
For more complete and readable validation results, see the full schema validator results
Schema Errors There are validation errors on 1273 records

Only showing errors from the first 10 records:

Errors on record 0:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 1:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 2:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 3:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 4:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 5:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 6:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 7:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 8:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 9:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Datasets 83515
Datasets with Distribution URLs 64.9% (54207 of 83515)
Datasets with Download URLs 64.9% (54207 of 83515)
Total Distribution URLs 75639
Total Download URLs 75639
Total APIs 0
Public Datasets 83515
Restricted Public Datasets 0
Non-public Datasets 0
Bureaus Represented 3
Programs Represented 5
License Specified 100% (83515 of 83515)
Datasets with Redactions 0.0% (0 of 83515)
Redactions without explanation (rights field) 0.0% (0 of 83515)
File Size 194.17MB
Last modified Thursday, 13-Aug-2015 09:56:05 EDT
Last crawl Friday, 21-Aug-2015 00:20:24 EDT
Analyze archive copies Analyze archive from 2015-08-31
Nearby Daily Crawls
/data page
Expected /data URL http://www.doi.gov/data (From USA.gov Directory)
Resolved /data URL http://www.doi.gov//data/index.cfm
Redirects 2 redirects
HTTP Status 200
Content Type text/html;charset=UTF-8
Last crawl Friday, 21-Aug-2015 00:02:04 EDT
/digitalstrategy.json
Expected /digitalstrategy.json URL http://www.doi.gov/digitalstrategy.json (From USA.gov Directory)
Resolved /digitalstrategy.json URL http://www.doi.gov/digitalstrategy.json
Redirects
HTTP Status 200
Content Type text/plain
Valid JSON Invalid Check a JSON Validator
Last modified Monday, 02-Dec-2013 10:38:22 EST
Last crawl Friday, 21-Aug-2015 00:02:05 EDT