Department of the Interior

http://www.doi.gov/

Milestone 9 - November 30th 2015

OMB Review Complete: OMB has completed the agency review for this milestone. Agencies should contact their OMB desk officer if anything looks incorrect.

Leading Indicators

These indicators are reviewed by the Office of Management and Budget

Review Status complete
Reviewer Justin Grimes
Last Updated January 11, 2016, 2:44 pm EST by Justin Grimes

Assessment Summary

Department of the Interior needs to address the following issues: 1) "Fails to document all non-public and restricted datasets, redactions, restrictive licenses and provides an explanation for non-disclosure (in the rights field)"; 2) "Fails to document non-public, and restricted public datasets"; 3) "Fails to organize data assets in to Collections"; 4) "Fails to have 100% valid data.json". For public engagement, the ideascale site does not show active use since January 2014; Department of the Interior could perhaps have in person public engagement to maker users aware of this functionality and feedback opportunity.

Inventory Composition

Public Dataset Status

Dataset Link Quality

Status Indicator Automated Metrics
Overall Progress this Milestone
Inventory Updated this Quarter
174449 Number of Datasets
152 Number of APIs
Schedule Delivered Crawl details
4 Bureaus represented
6 Programs represented
64625 Number of public datasets
109824 Number of restricted public datasets
Number of non-public datasets
Inventory > Public listing
Percentage growth in records since last quarter
Spot Check - datasets listed by search engine
Agency provides a public Enterprise Data Inventory on Data.gov
100% License specified Crawl details
Status Indicator Automated Metrics
Overall Progress this Milestone
37100 Number of Datasets Crawl details
Number of Collections Crawl details
36664 Number of Public Datasets with File Downloads Crawl details
152 Number of APIs Crawl details
54568 Total number of access and download links Crawl details
Quality Check: Links are sufficiently working Crawl details
48549 Quality Check: Accessible links Crawl details
5035 Quality Check: Redirected links Crawl details
82 Quality Check: Error links Crawl details
664 Quality Check: Broken links Crawl details
-55.8% Percentage growth in records since last quarter
96.3% Valid Metadata Crawl details
/data exists Crawl details
/data.json Crawl details
Harvested by data.gov
46,098 Views on data.gov for the quarter
Status Indicator Automated Metrics
Overall Progress this Milestone
Description of feedback mechanism delivered Crawl details
Data release is prioritized through public engagement
Feedback loop is closed, 2 way communication
See below Link to or description of Feedback Mechanism
http://usinterior.ideascale.com/
Status Indicator Automated Metrics
Overall Progress this Milestone
Data Publication Process Delivered Crawl details
Information that should not to be made public is documented with agency's OGC
Status Indicator Automated Metrics
Overall Progress this Milestone
See below Open Data Primary Point of Contact
lin_zhang@ios.doi.gov
POCs identified for required responsibilities
Status Indicator Automated Metrics
Overall Progress this Milestone
Identified 5 data improvements for this quarter
See below Primary Uses
These are examples of the primary users of U.S. Department of the Interior (DOI). The USGS Water Use Information Program investigates the occurrence, quantity, quality, distribution, and movement of surface and underground waters and disseminates the data to the public, State and local governments, public and private utilities, emergency responders, private sector (Kayakers, Fishing Industry, etc.) and other Federal agencies involved with managing our water resources. USGS data such as this is critical to both planning for and responding to hazards such as tsunamis, and for long-term planning addressing climate change, coastal erosion and ongoing coastal development. Private companies use DOI data in developing new services and products to meet public needs in many areas. While it looks as though these questions may be for internal purposes only, Azavea has endeavored to answer them as they apply to our use of Department of Interior datasets. We hope the information is helpful. Azavea uses the data in a number of contexts, including software development, GIS mapping, and data analytics. From: Parsons Brinckerhoff Parsons Brinckerhoff utilizes these datasets to create 3D models for project pursuits and project management. Uses range from master planning to roadway design. They are highly used resources for data reconnaissance during pursuits as well as the foundation for design on various projects. From USGS: USGS provides data to a very wide range of users, including scientists and academic institutions in dozens of disciplines, disaster management and planning authorities, mapping and geospatial companies, and Federal, state and local land management agencies From ONRR: The Office of Natural Resources Revenue (ONRR) initiated project in 2004 to implement new technology and processes to provide statistical information on the web and to support a centralized data request function. This function is managed by the Data Services & Statistical Reporting Office (DSSR) utilizing new technology and software, the Information Request and Query Management (IRQM) tool, to compile program data for either web publishing activities or data requests. The established process and procedures allow for consistent, verifiable, and reproducible collection and analysis of data in response to data requests from both internal and external entities. These requests include routine data collection for numerous federal agencies (i.e. BLM, BIA, OMB, DOJ, EIA, GAO, OIG, etc.), States, and American Indian tribes, and non-routine requests from the ONRR Data Requests Mailbox. The DSSR responses and provides data sets to approximately 500 data requests annually. In response to our customers, the ONRR Statistical Website was updated in 2012 to provide better and easier navigation throughout the site, downloadable data sets, and better explanation of our data. Nov 2015, From BIA: The tribal leader directory is published annually because Tribal elections and other changes in tribal leadership occur throughout the year. Since the current feedback form on the internet is new channel to engage with public, we still receive the feedback from mails, fax and emails daily. We currently have staffs in 12 regional offices to engages with tribes and update the data on quarterly basis and staffs in the central office to validate the accuracy. Nov 2015, From BOR: These data are used by universities attempting to provide regional and broad-scale information about western water as a means to assist policy-level decision-makers. In addition, all levels of government are able to provide streamflow information to their constituents for planning purposes. The primary channels for users to learn about data are Reclamation’s news bulletins and its web pages. Web pages can show real-time information about stream flow that is helpful particularly during the Spring run-off. The suggestions coming primarily from academia are for a more consistently-formatted data deliverable. The bureau is working on this, along with meeting the desire for more reservoir level data in the future. Nov 2015, From BSEE Well Log Data Images are used in decision support of GOM leasing, production activity. BSEE/BOEM provides this data to a very wide range of users, including oil and gas operators, vendors, academic institutions and individuals along with other Federal and State agencies. Nov 2015, From USGS USGS provides data to a very wide range of users, including scientists and academic institutions in dozens of disciplines, disaster management and planning authorities, mapping and geospatial companies, and Federal, state and local land management agencies.
Value or impact of data
See below Primary data discovery channels
Through the DOI data catalog, DOI websites, and data.gov. In addition, through the USGS.gov websites, state Water Offices, state authorities, universities, Press Releases, Fact Sheets, Publications, and Research shared through peer-reviewed literature. This is an ongoing, close relationship with key partners where data is shared and made available in the course of the work. In addition, Azavea subscribes to e-mails, listservs, Twitter and other alerts that keep us apprised of new releases or updates. Further, we regularly sponsor and attend hackathons focused on open data to both share and learn more about available data sources. From: Parsons Brinckerhoff: Datasets are primarily learned by attending professional societies such as; the Transportation Research Board (TRB), International Highway Engineering Exchange Program (IHEEP), Geospatial Organizations; Esri Users Conference, Coalition of Geospatial Organizations (COGO) and the American Society of Civil Engineers (ASCE). Vendor consultation and basic research on the internet has also led to discovery. From USGS In addition to the challenge outlined above, USGS has participated in two other challenges (addressing climate and biodiversity), and routinely corresponds with citizens on a range of scientific topics by email, phone and through social media From ONRR: The primary use of our data is for revenue stream analysis and support for budget activities The primary channels for the users to learn about our data is through onrr.gov, useiti.doi.gov, and data.gov From USGS: USGS provides data through a broad range of vehicles, including the USGS Science Data Catalog, the DOI data catalog, USGS.gov and USGS Science Center websites, and data.gov, as well as state Water Offices, state authorities, universities, Press Releases, Fact Sheets, Publications, and research shared through peer-reviewed literature.
See below User suggestions on improving data usability
Suggestions for improving access and usability of this data has focused around providing additional services for USGS data, including: stream site locations, gauge information, regional local view/subset and various data formats. From: Parsons Brinckerhoff: Standardization, interoperability between datasets From ONRR We have received the following suggestion on improving the usability of our data: o Ability to query variable date ranges (i.e. FY, CY, specific monthly date ranges) Suggestions on additional data resources to release o Allocated production volumes o County-level reported royalty revenues and production volumes From USGS Users have requested that USGS data be made available in more aggregated sets, and linked with complementary or similar data from other agencies. These are central goals of this partnership with NASA and AWS, as well as the Climate Data Initiative.
See below User suggestions on additional data releases
From: Parsons Brinckerhoff: Continued refinement on the fidelity/accuracy of the data. From USGS: Users have requested that USGS data be made available in more aggregated sets, and linked with complementary or similar data from other agencies. These are central goals of the Open Water Data Initiative (http://acwi.gov/spatial/owdi/)
Digital Analytics Program on /data

Automated Metrics

These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot

data.json
Expected Data.json URL http://www.doi.gov/data.json (From USA.gov Directory)
Resolved Data.json URL https://data.doi.gov/data.json
Number of Redirects 2 redirects
HTTP Status 200
Content Type text/plain; charset=UTF-8
Valid JSON Valid
Datasets with Valid Metadata 96.3%(35738 of 37100)
Valid Schema Invalid
For more complete and readable validation results, see the full schema validator results
Schema Errors There are validation errors on 1362 records

Only showing errors from the first 10 records:

Errors on record 0:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 1:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 2:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 3:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 4:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 5:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 6:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 7:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 8:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Errors on record 9:
distribution
  • array value found, but a null is required
  • array value found, but a string is required
  • failed to match at least one schema
Datasets 37100
Number of Collections 0
Datasets with Distribution URLs 98.8% (36664 of 37100)
Datasets with Download URLs 98.8% (36664 of 37100)
Total Distribution URLs 54639
Total Download URLs 54639
Total APIs 0
Public Datasets 37100
Restricted Public Datasets 0
Non-public Datasets 0
Bureaus Represented 4
Programs Represented 6
License Specified 100% (37100 of 37100)
Datasets with Redactions 0.0% (0 of 37100)
Redactions without explanation (rights field) 0.0% (0 of 37100)
File Size 99.34MB
Last modified Sunday, 29-Nov-2015 03:09:47 EST
Last crawl Monday, 30-Nov-2015 23:10:01 EST
Analyze archive copies Analyze archive from 2015-11-30
Nearby Daily Crawls
/data page
Expected /data URL http://www.doi.gov/data (From USA.gov Directory)
Resolved /data URL https://www.doi.gov/data
Redirects 1 redirects
HTTP Status 200
Content Type text/html; charset=utf-8
Last modified Monday, 30-Nov-2015 16:39:07 EST
Last crawl Monday, 30-Nov-2015 23:01:43 EST
/digitalstrategy.json
Expected /digitalstrategy.json URL http://www.doi.gov/digitalstrategy.json (From USA.gov Directory)
Resolved /digitalstrategy.json URL https://www.doi.gov/digitalstrategy.json
Redirects 1 redirects
HTTP Status 404
Content Type text/html; charset=utf-8
Valid JSON Invalid Check a JSON Validator
Last modified Monday, 30-Nov-2015 16:38:43 EST
Last crawl Monday, 30-Nov-2015 23:01:43 EST