Commit ef9e0a80 authored by CI-bot's avatar CI-bot Committed by renku 0.10.4
Browse files

renku rerun data/covidtracking/states-metadata.json data/covidtracking/states-daily.json

parent 1bf60842
Pipeline #87780 passed with stage
in 34 seconds
class: Workflow
cwlVersion: v1.0
hints: []
inputs:
input_1:
default: out_folder
streamable: false
type: string
input_2:
default: data/covidtracking
streamable: false
type: string
input_3:
default:
class: File
path: ../../notebooks/process/download-covidtracking-data.ipynb
streamable: false
type: File
input_4:
default: runs/download-covidtracking-data.runs.ipynb
streamable: false
type: string
input_5:
default: states-daily.json
streamable: false
type: string
input_6:
default: states-metadata.json
streamable: false
type: string
outputs:
output_1:
outputSource: step_1/output_0
streamable: false
type: File
output_2:
outputSource: step_1/output_1
streamable: false
type: Directory
requirements: []
steps:
step_1:
in:
input_1: input_1
input_2: input_2
input_3: input_3
input_4: input_4
out:
- output_0
- output_1
run: a17d560c41a54f5aa307ce5f3c5effe5_papermill.cwl
step_2:
in:
filename: input_5
input_directory: step_1/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-daily.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- &id001
class: InlineJavascriptRequirement
- &id002
class: InitialWorkDirRequirement
listing: $(inputs.input_directory.listing)
successCodes: []
temporaryFailCodes: []
step_3:
in:
filename: input_6
input_directory: step_1/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-metadata.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- *id001
- *id002
successCodes: []
temporaryFailCodes: []
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
%% Cell type:code id: tags:
``` python
import requests
import os
import pandas as pd
```
%% Cell type:code id: tags:parameters
``` python
out_folder = "../data/covidtracking/"
PAPERMILL_OUTPUT_PATH = None
```
%% Cell type:code id: tags:injected-parameters
``` python
# Parameters
PAPERMILL_INPUT_PATH = "/tmp/wkiajyw0/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_INPUT_PATH = "/tmp/msijfsbb/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_OUTPUT_PATH = "runs/download-covidtracking-data.runs.ipynb"
out_folder = "data/covidtracking"
```
%% Cell type:markdown id: tags:
# Download state metadata
Download a dataset of URLs for data for each US state and several territories. See [Google Doc](https://docs.google.com/spreadsheets/d/18oVRrHj3c183mHmq3m89_163yuYltLNlOmPerQ18E8w/htmlview?sle=true).
%% Cell type:code id: tags:
``` python
url = 'http://covidtracking.com/api/states/info'
r = requests.get(url, allow_redirects=True)
states_metadata_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-metadata.json')
with open(out_path, 'wb') as f:
f.write(states_metadata_json)
```
%% Cell type:code id: tags:
``` python
metadata_df = pd.read_json(states_metadata_json)
print(len(metadata_df), "states and territories have metadata")
metadata_df.head(2)
```
%%%% Output: stream
56 states and territories have metadata
%%%% Output: execute_result
state notes \
0 AK Alaska combines PCR and antigen tests in the t...
1 AL Alabama combines PCR and antigen tests in the ...
covid19Site \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 https://alpublichealth.maps.arcgis.com/apps/op...
covid19SiteSecondary \
0 https://experience.arcgis.com/experience/ed1c8...
1 https://alpublichealth.maps.arcgis.com/apps/op...
covid19SiteTertiary covid19SiteQuaternary \
0 https://alaska-dhss.maps.arcgis.com/apps/opsda...
1 https://services7.arcgis.com/4RQmZZ0yaZkGR1zy/...
0 https://alaska-dhss.maps.arcgis.com/apps/opsda... None
1 https://services7.arcgis.com/4RQmZZ0yaZkGR1zy/... None
covid19SiteQuinary twitter \
0 @Alaska_DHSS
1 @alpublichealth
0 None @Alaska_DHSS
1 None @alpublichealth
covid19SiteOld \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 http://www.alabamapublichealth.gov/infectiousd...
covidTrackingProjectPreferredTotalTestUnits \
0 Specimens
1 Unclear units
covidTrackingProjectPreferredTotalTestField totalTestResultsField name \
0 totalTestsViral Total Tests (PCR) Alaska
1 totalTestsViral Total Tests (PCR) Alabama
fips pui pum
0 2 False
1 1 False
covidTrackingProjectPreferredTotalTestField totalTestResultsField pui \
0 totalTestsViral Total Tests (PCR) All data
1 totalTestsViral Total Tests (PCR) No data
pum name fips
0 False Alaska 2
1 False Alabama 1
%% Cell type:markdown id: tags:
# Download daily state data
%% Cell type:code id: tags:
``` python
url = 'https://covidtracking.com/api/states/daily'
r = requests.get(url, allow_redirects=True)
states_daily_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-daily.json')
with open(out_path, 'wb') as f:
f.write(states_daily_json)
```
%% Cell type:code id: tags:
``` python
data_df = pd.read_json(states_daily_json)
print(len(data_df), "data points")
data_df.head(2)
```
%%%% Output: stream
12533 data points
12589 data points
%%%% Output: execute_result
date state positive probableCases negative pending \
0 20201013 AK 11039.0 NaN 503750.0 NaN
1 20201013 AL 167193.0 20110.0 1074124.0 NaN
0 20201014 AK 11183.0 NaN 505994.0 NaN
1 20201014 AL 167977.0 20232.0 1078476.0 NaN
totalTestResults hospitalizedCurrently hospitalizedCumulative \
0 514789.0 59.0 NaN
1 1221207.0 823.0 18440.0
0 517177.0 56.0 NaN
1 1226221.0 834.0 18635.0
inIcuCurrently ... posNeg deathIncrease hospitalizedIncrease \
0 NaN ... 514789 0 0
1 NaN ... 1241317 0 261
0 NaN ... 517177 4 0
1 NaN ... 1246453 41 195
hash commercialScore \
0 881fbdb23caeb53c47e5007c8599ec1624e1c68d 0
1 19ea68f7f6d374d276ce094b7473ceb24e9e0cae 0
0 3e715172385e995a3ce6d9b2672d1b675141de57 0
1 fb08727b5bd834467104578dc753f08ab1551e5a 0
negativeRegularScore negativeScore positiveScore score grade
0 0 0 0 0
1 0 0 0 0
[2 rows x 55 columns]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment