If any publications or research results are derived in full or in part from the C3.ai COVID-19 Data Lake, please make sure to credit the C3.ai COVID-19 Data Lake by referencing the case study at https://c3.ai/customers/covid-19-data-lake/.

Get started using the C3.ai COVID-19 Data Lake with R and Python notebooks. Use the online hosted notebooks to easily try out the C3.ai COVID-19 Data Lake APIs in the cloud without any downloads or local environment setup. Use the downloadable notebooks to edit the code and save your results locally.

The R Quickstart notebook shows simple API calls and the breadth of data available in the C3.ai COVID-19 Data Lake.

Online hosted R Notebook

To try out the C3.ai COVID-19 Data Lake without any downloads or local environment setup, run the R Notebook in your browser using Binder:

Downloadable R Notebook

To edit the notebook and save your results locally, follow the following steps.

Local environment setup

Ensure that the following are installed on your computer:

• R from CRAN, and
• R Studio

Download R Notebooks

• Download zip file containing R Notebooks and library functions

Troubleshooting

While opening the R notebook (.Rmd file), if you see the error:

Unable to locate R binary by scanning standard locations.

then you probably did not install R from CRAN. Make sure you install both R Studio and R from CRAN.

The Python Quickstart notebook shows simple API calls and the breadth of data available in the C3.ai COVID-19 Data Lake.

Online hosted Python Jupyter Notebook

To try out the C3.ai COVID-19 Data Lake without any downloads or local environment setup, run the Python Jupyter Notebook in your browser using Binder:

Downloadable Python Jupyter Notebook

To edit the notebook and save your results locally, follow the following steps.

Local environment setup

Ensure that the following are installed on your computer:

• Python 3, and
• Jupyter Notebook

Download Python Jupyter Notebooks

• Download zip file containing Python Jupyter Notebooks and library functions

Troubleshooting

• Ensure that you have Python 3 and not Python 2.7.

• While opening the Jupyter Notebook, if you see the error:

  "Error loading notebook: An unknown error occurred while loading this notebook. This version can load notebook formats or earlier. See the server log for details.

  then you can probably resolve this by installing Python from Anaconda.

• If you see error messages regarding pandas functions such as json_normalize or explode, make sure that you are using a pandas version of at least 1.0.0. See the pandas installation guide for installation instructions.

The R Deep Dive notebook explores datasets in mobility and case counts and provides a starting point for detailed analysis into the relationships between these datasets.

Online hosted R Notebook

Run the Deep Dive notebook in your browser using Binder:

Downloadable R Notebook

To edit the Deep Dive notebook and save your results locally, download the zip file containing R Notebooks and library functions. For additional details, see the R Quickstart section.

Release Notes for 4.0 (June 23, 2020)

What's new:

• Added six new datasets (see C3.ai APIs for COVID-19 Unified Data):
  • The New York Times: All-Cause Mortality,
  • University of Oxford: Coronavirus Government Response Tracker,
  • US Census Bureau: International Census,
  • The World Bank: Finance Related Policy Responses to COVID-19,
  • PlaceIQ Exposure Indices,
  • IBM: Weather Company Data.
• Added two new C3.ai Types: LocationExposure and PolicyDetail.
• Added Deep Dive notebook exploring mobility data and case counts.
• Revised Quickstart notebooks to include new C3.ai Types and APIs.
• Renamed Policy Type to LocationPolicySummary for clarity. Query via Policy Type is deprecated but still supported.
• Revised document for clarity.
  
  

Release Notes for 3.0 (June 2, 2020)

What's new:

• Added four new datasets (see C3.ai APIs for COVID-19 Unified Data):
  • Corona Data Scraper: COVID-19 Coronavirus Case Data
  • Centers for Disease Control and Prevention (CDC): VaxView
  • Cytel: Global Coronavirus COVID-19 Clinical Trial Tracker
  • Google: COVID-19 Community Mobility Reports
• Added two new C3.ai Types: VaccineCoverage and ClinicalTrial
• Added one new API: GetProjectionHistory
• Revised Quickstart notebooks to include new C3.ai Types and APIs
• Added online hosting for Quickstart notebooks
• Revised document for clarity
  
  

Release Notes for 2.0 (May 15, 2020)

What's new:

• Added eleven new datasets: see C3.ai APIs for COVID-19 Unified Data
• Added six new C3.ai Types: PopulationData, PatientRoute, Hospital, Diagnosis, DiagnosisDetail, and LocationPolicySummary
• Added two new APIs: GetImageURLs and AllVersionsForPolicy
• Categorized C3.ai Types in navigation bar for ease of navigation
• Revised Quickstart notebooks to include new C3.ai Types and APIs
• Revised and restructured document for clarity
  
  

Release Notes for 1.0 (April 22, 2020)

What's new:

• GetArticleMetadata now provides metadata for multiple article using "ids" rather than "id"
• Added "Interpolated" JHU metrics for OutbreakLocation EvalMetrics API
• Added additional clarifications and wording adjustments
  
  
  

Release Notes for 0.1

Using these APIs, you can pull together data from multiple COVID-19 data sources with a single API call. This is made possible by using C3.ai Types.

If you are new to the concept of a C3.ai Type, then it is easier to think of a C3.ai Type as an entity that holds the data. Using C3.ai Types makes it possible to programmatically interact with a unified, federated image of COVID-19 data.

On this page, entries such as OutbreakLocation, LineListRecord are the names of C3.ai Types. Each C3.ai Type holds data of a certain kind. For example:

• OutbreakLocation stores location data such as countries, provinces, cities, where COVID-19 outbeaks are recorded, and
• LineListRecord stores individual-level information such as symptoms, travel history, reported onset, and discharge status from laboratory-confirmed COVID-19 patients.

While each such C3.ai Type holds the data of a particular kind, you can use these APIs to connect up the data from multiple C3.ai Types. For example, you can join the data from two C3.ai Types, BiologicalAsset and Sequence. This can be accomplished by using the include option in the fetch API call.

The following is an example entity relationship diagram showing how C3.ai Types are connected. Not all fields are shown in the below diagram. Refer to the Fields table for a C3.ai Type for a full listing of the fields for that C3.ai Type. See, for example, LineListRecord.

The include parameter is a powerful way to fetch data from multiple C3.ai Types. This parameter can also be used to fetch specific fields from a single C3.ai Type.

When you want to join data from two C3.ai Types, you make a fetch API call to one C3.ai Type, and use include in your request body to refer to the second C3.ai Type. The returned objects will contain fields and data from both the C3.ai Types.

See the section Using Include for detailed examples showing how to use include to combine data from multiple C3.ai Types.

The following APIs are currently suppported:

• fetch (for all C3.ai Types presented here)
• evalmetrics (for OutbreakLocation)
• getprojectionhistory (for OutbreakLocation)
• getarticlemetadata (for BiblioEntry)
• getimageurls (for Diagnosis)
• allversionsforpolicy (for LocationPolicySummary)

Use POST requests to access these APIs.

NOTE: If you are new to the concept of RESTful API, this Postman Learning Center is a good place to start. All APIs described in this documentation can be verified using the Postman client.

The following table shows the APIs available for specific data sources (more data sources are being added):

All C3.ai APIs described in this documentation must be accessed using POST requests.

If you receive the following error:

{
  "message": "Missing Authentication Token"
}

then you are probably using GET or another request, and you should instead use POST. No authentication token is required to access the APIs.

All C3.ai APIs described in this documentation must be used with the following header settings:

The request JSON for the fetch API should be used with the filter key. This filter key can be used in the fetch call to select any combination of the fields in the data. A few examples follow:

IMPORTANT: For a list of fields available for a C3.ai Type, refer to the fields section of that C3.ai Type in this document.

To fetch the data that match the specific values of the id field of the data:

  {
    "spec" : {"filter": 'id == "Afghanistan"'}
  }

A few other examples:

  {
    "spec" : {"filter": 'id == "Afghanistan" && age == 45'}
  }

or,

  {  // See BiologicalAsset
    "spec": {
        "filter": "isolationSource == 'feces' && location == 'Japan'",
        "limit": -1
    }  
  }

or, using a "contains(field, "string")" format:

  { // See LineListRecord
    "spec": {
        "filter": "gender == 'male' && lineListSource == 'OPEN' && age <= 20 && contains(relevantTravelHistoryLocation,'Wuhan')"
   }
  }

The fetch API returns two main kinds of information in its response:

• The data from the C3.ai Type, fetched as an array of the objects.
• Metadata, or data describing data, such as:
  • The number of objects fetched.
  • Number of rows of information.
  • An indicator if more data exists in the C3.ai Type that was not returned.
• See the section Limits for the number of entries returned per fetch API call.

For full details on request and response JSON, see the several examples provided in the fetch API for all the C3.ai Types in this documentation.

In a C3.ai Type, the data type of a field can be a C3.ai Type. For example, the field links in TherapeuticAsset is of the type ExternalLink. This is how these two C3.ai Types are connected.

For example, to join data from these two connected C3.ai Types, TherapeuticAsset and ExternalLink, use the include parameter as follows:

• Make a fetch API call to TherapeuticAsset.
• The field links in TherapeuticAsset is of ExternalLink Type. Using the dot notation on the links field, you can access any field in the ExternalLink. For example, specifying links.url will resolve into ExternalLink.url, which will obtain the url field data from the ExternalLink.
• Notice that we have not issued a fetch call to ExternalLink. See the full fetch example, including the response objects, below.

Example 1: Join data from TherapeuticAsset and ExternalLink (click arrow to open)

  {
    "spec": {
      "include": "productType, description, origin, links.url",
      "filter": "origin =='Milken'",
      "limit" : 3
    }
  }

  { 
    "objs": [
      {
          "productType": "TAK-888, antibodies from recovered COVID-19 patients",
          "origin": "Milken",
          "links": [
              {
                  "url": "https://www.wsj.com/articles/drugmaker-takeda-is-working-on-coronavirus-drug-11583301660?mod=article_inline",
                  "therapeuticAsset": {
                      "id": "milkentreatment_001"
                  },
                  "id": "1919184b-460e-4725-8c2c-0ab225a58c1c",
                  "meta": {
                      "fetchInclude": "[id,url,therapeuticAsset,version]",
                      "fetchType": "ExternalLink"
                  },
                  "version": 1
              },
              {
                  "url": "https://phrma.org/coronavirus",
                  "therapeuticAsset": {
                      "id": "milkentreatment_001"
                  },
                  "id": "fdc646d7-eada-47af-b782-89216996b7ec",
                  "meta": {
                      "fetchInclude": "[id,url,therapeuticAsset,version]",
                      "fetchType": "ExternalLink"
                  },
                  "version": 1
              }
            ],
            "id": "milkentreatment_001",
            "meta": {
                "fetchInclude": "[productType,description,origin,{links:[id,url]},id,version]",
                "fetchType": "TherapeuticAsset"
            },
            "version": 1
        },
        {
            "productType": "Antibodies from mice, REGN3048-3051, against the spike protein",
            "origin": "Milken",
            "links": [
                {
                    "url": "https://www.statnews.com/2020/03/19/an-updated-guide-to-the-coronavirus-drugs-and-vaccines-in-development/",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "154217eb-08fc-4ba8-aeb2-6ac8d93710a1",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                },
                {
                    "url": "https://www.bnnbloomberg.ca/gilead-s-drug-leads-global-race-to-find-coronavirus-treatment-1.1395231",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "2984da95-4e96-4e27-9365-0b17ee04c091",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                },
                {
                    "url": "https://uk.reuters.com/article/uk-china-health-treatments-factbox/factbox-global-efforts-to-develop-vaccines-drugs-to-fight-the-coronavirus-idUKKBN20D2MD?rpc=401&",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "61d19389-fe0b-4e7a-84dc-07dbca2d20c3",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                },
                {
                    "url": "https://www.marketwatch.com/story/these-nine-companies-are-working-on-coronavirus-treatments-or-vaccines-heres-where-things-stand-2020-03-06",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "812e1849-5403-4940-8d46-0d372dc94bbd",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                },
                {
                    "url": "https://www.fiercebiotech.com/research/fast-moving-regeneron-eyes-summer-clinical-trial-for-covid-19-antibody-cocktail-therapy",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "b583ddb9-68e3-4896-b8b8-bb4d283c5b79",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                },
                {
                    "url": "https://www.fiercepharma.com/pharma/regeneron-s-r-d-war-room-sleepless-nights-and-esprit-de-corps-hunt-for-covid-19-therapy?mkt_tok=eyJpIjoiTUROaFpUYzJOVE14TjJNMSIsInQiOiJRT0YxZVBYWHJQQVlNYzFcL3JUSUdac1Zqa0VXMGk2MHpQS3kxZXdmaWhyQlJ2MWFuR1wvb0FGb1pDWExGQ0pYQzYrN1dGMW9BRlFMUlo2WE1XOVZQN2pxaE1MUUFQSHFlYlVCc0xKdmJoTm1xdHdcL0hPeXJKS0llODJTTGR5aHZjQ2Z3ZE1KU1BFZ013R0JyeU9qTkxSZmc9PSJ9&mrkid=72869502",
                    "therapeuticAsset": {
                        "id": "milkentreatment_002"
                    },
                    "id": "f09ff2a3-2ca0-48cd-b373-6bd0a396d221",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                }
            ],
            "id": "milkentreatment_002",
            "meta": {
                "fetchInclude": "[productType,description,origin,{links:[id,url]},id,version]",
                "fetchType": "TherapeuticAsset"
            },
            "version": 1
        },
        {
            "productType": "Antibodies from recovered COVID-19 patients",
            "origin": "Milken",
            "links": [
                {
                    "url": "http://www.koreaherald.com/view.php?ud=20200312000885",
                    "therapeuticAsset": {
                        "id": "milkentreatment_003"
                    },
                    "id": "d2073283-ec4f-4f5f-821a-8cb6019ed80e",
                    "meta": {
                        "fetchInclude": "[id,url,therapeuticAsset,version]",
                        "fetchType": "ExternalLink"
                    },
                    "version": 1
                }
            ],
            "id": "milkentreatment_003",
            "meta": {
                "fetchInclude": "[productType,description,origin,{links:[id,url]},id,version]",
                "fetchType": "TherapeuticAsset"
            },
            "version": 1
        }
    ],
    "count": 3,
    "hasMore": true
  }

Example 2: Join data from BiologicalAsset and Sequence (click arrow to open)

In the following example, the include parameter is used with this keyword, which obtains all the fields from BiologicalAsset and the field sequence from Sequence.

  {
    "spec": {
      "include": "this, sequence.sequence",
       "filter": "exists(sequence.sequence)",
       "limit" : 3
    }
  }

  { 
    "objs": [
      {
        "sequence": {
            "sequence": "SGFRKMAFPSGKVEGCMVQVTCGTTTLNGLWLDDVVYCPRHVICTSEDMLNPNYEDLLIRKSNHNFLVQAGNVQLRVIGHSMQNCVLKLKVDTANPKTPKYKFVRIQPGQTFSVLACYNGSPSGVYQCAMRPNFTIKGSFLNGSCGSVGFNIDYDCVSFCYMHHMELPTGVHAGTDLEGNFYGPFVDRQTAQAAGTDTTITVNVLAWLYAAVINGDRWFLNRFTTTLNDFNLVAMKYNYEPLTQDHVDILGPLSAQTGIAVLDMCASLKELLQNGMNGRTILGSALLEDEFTPFDVVRQCSGVTFQ",
            "id": "5R7Y_A"
        },
        "assetType": "protein sequence",
        "species": "Severe acute respiratory syndrome-related coronavirus",
        "genus": "Betacoronavirus",
        "family": "Coronaviridae",
        "authors": "Fearon,D., Powell,A.J., Douangamath,A., Owen,C.D., Wild,C., Krojer,T., Lukacik,P., Strain-Damerell,C.M., Walsh,M.A., von Delft,F.",
        "genBankTitle": "Chain A, main protease",
        "id": "5R7Y_A",
        "meta": {
            "tenantTagId": 4,
            "tenant": "covid",
            "tag": "prod",
            "created": "2020-04-02T03:39:25Z",
            "createdBy": "dataloader",
            "updated": "2020-04-02T03:39:25Z",
            "updatedBy": "dataloader",
            "timestamp": "2020-04-02T03:39:33Z",
            "sourceFile": "proteins_sequence_metadata.csv",
            "fetchInclude": "[this,{sequence:[sequence,id]}]",
            "fetchType": "BiologicalAsset"
        },