U.S. Agency for International Development
Enterprise Data Inventory - Volume and composition over time
Milestone 24 - August 31st 2019
Previous Milestone: The milestone selected is the most recently complete one. The Automated Metrics are a snapshot from the milestone date.
OMB Review Has Not Begun: OMB has not begun reviewing the agency for this milestone. The review will begin after the milestone date.
Leading Indicators
These indicators are reviewed by the Office of Management and Budget
| Review Status | |
|---|---|
| Reviewer | |
| Last Updated |
Assessment Summary
| Status | Indicator | Automated Metrics |
|---|---|---|
| Overall Progress this Milestone | ||
| Inventory Updated this Quarter | ||
| Number of Datasets | ||
| Number of APIs | ||
| Bureaus represented | ||
| Percentage of bureaus represented | ||
| Programs represented | ||
| Percentage of programs represented | ||
| Number of public datasets | ||
| Number of restricted public datasets | ||
| Number of non-public datasets | ||
| Percentage growth in records since last quarter | ||
| To what extent is your agency’s Enterprise Data Inventory (EDI) complete? | ||
| What steps have you taken to ensure your Enterprise Data Inventory is complete | ||
| Agency provides a public Enterprise Data Inventory on Data.gov | ||
| Agency provided updated Enterprise Data Inventory to OMB | ||
| License specified | Crawl details | |
| Number of datasets with redactions | ||
| Percent of datasets with redactions |
| Status | Indicator | Automated Metrics |
|---|---|---|
| Overall Progress this Milestone | ||
| Number of Datasets | Crawl details | |
| Number of Collections | Crawl details | |
| Number of datasets not contained in a collection | Crawl details | |
| Number of Public Datasets with File Downloads | Crawl details | |
| Number of APIs | Crawl details | |
| Number of public APIs | Crawl details | |
| Number of restricted public APIs | Crawl details | |
| Number of non-public APIs | Crawl details | |
| Total number of access and download links | Crawl details | |
| Quality Check: Links are sufficiently working | Crawl details | |
| Quality Check: Accessible links | Crawl details | |
| Quality Check: Redirected links | Crawl details | |
| Quality Check: Error links | Crawl details | |
| Quality Check: Broken links | Crawl details | |
| Quality Check: Percentage of download links in correct format as specified in metadata | Crawl details | |
| Quality Check: Percentage of download links in HTML | Crawl details | |
| Quality Check: Percentage of download links in PDF | Crawl details | |
| Percentage growth in records since last quarter | ||
| Valid Metadata | Crawl details | |
| /data exists | Crawl details | |
| Provides datasets in human-readable form on /data | ||
| /data.json | Crawl details | |
| Harvested by data.gov | ||
| Number of public datasets | Crawl details | |
| Number of restricted public datasets | Crawl details | |
| Number of non-public datasets | Crawl details | |
| Percent growth of public datasets | ||
| Percent growth of restricted public datasets | ||
| Percent growth of non-public datasets | ||
| Percent datasets licensed as U.S. Public Domain | ||
| Percent datasets licensed as Creative Commons Zero | ||
| Percent datasets with other licenses | ||
| Percent datasets with no license |
| Status | Indicator | Automated Metrics |
|---|---|---|
| Overall Progress this Milestone | ||
| Description of feedback mechanism delivered | Crawl details | |
| Data release is prioritized through public engagement | ||
| Provided narrative evidence of data improvements based on public feedback this quarter | ||
| Feedback loop is closed, 2 way communication | ||
| Link to or description of Feedback Mechanism | ||
| Provides valid contact point information for all datasets |
| Status | Indicator | Automated Metrics |
|---|---|---|
| Overall Progress this Milestone | ||
| Data Publication Process Delivered | Crawl details | |
| Information that should not to be made public is documented with agency's OGC | ||
| Describe the agency's data publication process |
| Status | Indicator | Automated Metrics |
|---|---|---|
| Overall Progress this Milestone | ||
| Open Data Primary Point of Contact | ||
| POCs identified for required responsibilities | ||
| Chief Data Officer (if applicable) |
Automated Metrics
These metrics are generated by an automated analysis that runs every 24 hours until the end of the quarter at which point they become a historical snapshot
data.json
| Expected Data.json URL | http://www.usaid.gov/data.json (From USA.gov Directory) |
|---|---|
| Resolved Data.json URL | https://www.usaid.gov/data.json |
| Number of Redirects | 1 redirects |
| HTTP Status | 200 |
| Content Type | application/json |
| Valid JSON | Invalid Check a JSON Validator |
| Detected Data.json Schema | federal-v1.1 |
| Datasets with Valid Metadata | 100%(316 of 316) - The JSON file is invalid and can't be parsed without special processing |
| Valid Schema | Valid |
| Datasets | 316 |
| Number of Collections | 6 |
| Number of datasets not in a collection | 201 |
| Datasets with Distribution URLs | 52.8% (167 of 316) |
| Datasets with Download URLs | 51.6% (163 of 316) |
| Total Distribution URLs | 370 (but only 60 accessible) |
| Total Download URLs | 365 |
| Total APIs | 4 |
| Public APIs | 4 |
| Restricted Public APIs | 0 |
| Non-public APIs | 0 |
| Public Datasets | 187 |
| Restricted Public Datasets | 68 |
| Non-public Datasets | 61 |
|
The fields below serve as quality assurance to verify that the download links included within the metadata are functioning properly |
|
| Server Not Found | 0.8% (3 of 370) |
| Working links (HTTP 2xx) | 16.2% (60 of 370) |
| Broken links (HTTP 4xx) | 0.8% (3 of 370) |
| Error Links (HTTP 5xx) | 0.0% (0 of 370) |
| Redirected Links (HTTP 3xx) | 82.2% (304 of 370) |
| Correct format | 48.3% (29 of 60) |
| PDF for raw data | 0.0% (0 of 60) |
| HTML for raw data | 35.0% (21 of 60) |
| Bureaus Represented | 1 |
| Programs Represented | 39 |
| License Specified | 100% (316 of 316) |
| Datasets with Redactions | 0.0% (0 of 316) |
| Redactions without explanation (rights field) | 0.0% (0 of 316) |
| File Size | 789.91KB |
| Last modified | Friday, 05-Oct-2018 13:00:33 EDT |
| Last crawl | Sunday, 02-Jun-2019 02:36:56 EDT |
| Analyze archive copies | Analyze archive from 2019-08-31 |
| Nearby Daily Crawls | |
| Expected /data URL | http://www.usaid.gov/data (From USA.gov Directory) |
|---|---|
| Resolved /data URL | https://data.usaid.gov/ |
| Redirects | 2 redirects |
| HTTP Status | 200 |
| Content Type | text/html; charset=utf-8 |
| Last crawl | Sunday, 02-Jun-2019 02:36:35 EDT |
| Expected /digitalstrategy.json URL | http://www.usaid.gov/digitalstrategy.json (From USA.gov Directory) |
|---|---|
| Resolved /digitalstrategy.json URL | https://www.usaid.gov/digitalstrategy.json |
| Redirects | 1 redirects |
| HTTP Status | 200 |
| Content Type | application/json |
| Valid JSON | Valid |
| Last modified | Friday, 03-May-2019 14:17:17 EDT |
| Last crawl | Sunday, 02-Jun-2019 02:36:35 EDT |
Date specified: Wednesday, 07-Aug-2019 09:50:00 EDT
Date of digitalstrategy.json file: Friday, 03-May-2019 14:17:17 EDT1.2.4 Develop Data Inventory Schedule - Summary
Summarize the Inventory Schedule
USAID will coordinate with Washington and overseas operating units to expand the number of data assets included in its Enterprise Data Inventory and in the Public Data Listing available at www.usaid.gov/data. USAID will enrich its data with tags including, but not limited to USAID’s Federal Program Inventory, operating unit designation, Presidential Initiative or focus country where applicable. USAID will assign senior staff to ensure that data are open and host trainings to build awareness and institutional capacity in using machine‐readable and open formats.
1.2.5 Develop Data Inventory Schedule - Milestones
| Title | FY2014 Quarter 1 release |
|---|---|
| Description | Initial release of USAID's public data listing. |
| Milestone Date | 28 February 2014 |
| Description of how this milestone expands the Inventory | Identify existing data assets; Develop enterprise data inventory; Develop public data listing |
| Description of how this milestone enriches the Inventory | Tag data assets with OMB mandated core metadata |
| Description of how this milestone opens the Inventory | Establish information governance structure and assign responsibilities; Develop data publication process; Develop inventory schedule; Develop customer feedback process; Release approved data |
| Title | FY2014 Quarter 2 release |
|---|---|
| Description | FY2014 Quarter 2 update to USAID’s public data listing |
| Milestone Date | 31 May 2014 |
| Description of how this milestone expands the Inventory | Prioritize data for release |
| Description of how this milestone enriches the Inventory | Develop Agency‐specific metadata tags |
| Description of how this milestone opens the Inventory | Apply data publication process; Release approved data; Respond to customer inquiries |
| Title | FY2014 Quarter 3 release |
|---|---|
| Description | FY2014 Quarter 3 update to USAID’s public data listing |
| Milestone Date | 30 August 2014 |
| Description of how this milestone expands the Inventory | Draft of Agency-wide Development Data policy completed, socialized for Agency-wide review, and submitted for senior leadership clearance; Prioritize data for release |
| Description of how this milestone enriches the Inventory | Refine and apply Agency‐specific metadata tags |
| Description of how this milestone opens the Inventory | Apply data publication process; Release approved data; Respond to customer inquiries |
| Title | FY2014 Quarter 4 release |
|---|---|
| Description | FY2014 Quarter 4 update to USAID’s public data listing |
| Milestone Date | 30 November 2014 |
| Description of how this milestone expands the Inventory | Create portal to allow implementing partners to submit data directly to USAID; Upgrade www.usaid.gov/data with Agency-wide data management platform; Prioritize data for release |
| Description of how this milestone enriches the Inventory | Apply Agency-specific metadata tags |
| Description of how this milestone opens the Inventory | Agency-wide policy and procedure training and socialization; Release approved data; Respond to customer inquiries |
1.2.6 Develop Customer Feedback Process
Describe the agency's process to engage with customers
USAID has three mechanisms for customers to provide feedback on its data management practices: 1) GitHub - USAID recommends clicking on the USAID GitHub link to report bugs or computer-related issues with its public data listing. 2) StackExchange - USAID recommends clicking on the StackExchange link to pose general questions about its public data listing.
1.2.7 Develop Data Publication Process
Describe the agency's data publication process
1) USAID operating unit identifies a data asset for potential release; 2) USAID makes a determination as to whether release is permitted by law, subject to all privacy, confidentiality, security, and other valid requirements, including: a) Security: USAID reviews the data asset for security restrictions related to the release of classified information; b) Privacy: In accordance with the Privacy Act of 1974 and other legal authorities, USAID reviews the data asset for the presence of Personally Identifiable Information (PII); c) Exceptions: USAID reviews the data asset for information excepted from release by OMB Bulletin 12-01 and the Freedom of Information Act, where applicable (for example, information that would jeopardize the safety of U.S. personnel); 3) Operating unit funding the data a) completes redactions; b) seeks clearance of related operating units; and c) proposes a data access level; 4) Office of the General Counsel ensures redactions and data access levels are consistent with legal requirements; and 5) USAID publishes the data asset to its public data listing at www.usaid.gov/data.