---
title: 'Chapitre 5: Manipulating and analyzing datas'
author: "Armelle de le Court"
date: "10/05/2021"
output: pdf_document
---

## 1: data manipulation using dplyr and tiydr

### Using tidyverse package to read the data and avoid having to se stringAsFactors to False 
```{r,message=FALSE}
library("tidyverse")
```

## 2: What are dplyr and tidyr?

### We're reading our data with read_csv instead of read.csv
```{r,results=FALSE}
surveys <- read_csv("data/portal_data_joined.csv")
str(surveys) # to inspect the data 
```

## some important functions from dplyr:

### select(): subset columns
### filter(): subset rows on conditions
### mutate(): create new columns by using information from other columns
### group_by() and summarize(): create summary statisitcs on grouped data
### arrange(): sort results
### count(): count discrete values

## 3: Selecting columns and filtering rows

### use select() to select colums of a dataframe
```{r,results=FALSE}
select(surveys, plot_id, species_id,weight)
## to select all of the columns except certain ones
select(surveys,-record_id,-species_id)
```
### use **filter** to choose rows based on a specifi criteria 
```{r,results=FALSE}
filter(surveys, year==1995)
```

## 4: Pipes

### To do multiple functions at the same time, we can use pipes, which are %>% (shortcut : Cmd + maj + M)
```{r}
surveys_sml <- surveys %>%
  filter(weight<5) %>%
  select(species_id,sex,weight)
```
### Question : Using pipes, subset the surveys data to include animals collected before 1995 and retain only the columns year, sex, and weight.
```{r,results=FALSE}
surveys %>%
  filter(year>1995) %>%
  select(year,sex,weight)
```

## 5: Mutate 

### To create a new column base on values in existing columns.
```{r}
surveys %>%
  mutate(weight_kg = weight/1000) %>%
  head() # use head() to see only the 1st 6 lines
```
### To remove NAs, we need to use filter too 
```{r}
surveys %>%
  filter(!is.na(weight)) %>%
  mutate(weight_kg = weight/1000) %>%
  head()
```

### Question: Create a new data frame from the surveys data that meets the following criteria: contains only the species_id column and a new column called hindfoot_half containing values that are half the hindfoot_length values. In this hindfoot_half column, there are no NAs and all values are less than 30.
```{r}
surveys_hindfoot_half <- surveys %>%
  filter(!is.na(hindfoot_length)) %>%
  mutate(hindfoot_half = hindfoot_length / 2) %>%
  filter(hindfoot_half < 30) %>%
  select(species_id, hindfoot_half)
surveys_hindfoot_half
```

## 6: Split-apply-combine data analysis

### group_by to split the data into groups, apply some analysis to each group and then combine the results.
```{r}
surveys %>%
  group_by(sex) %>%
  head()
# here it doesnt perform any data processing
```
### Summarize is being used with group_by, which collapses each group into a single-row summary of that group. So to compute the mean weight by sex : 
```{r}
surveys %>%
  group_by(sex) %>%
  summarize(mean_weight=mean(weight,na.rm=TRUE))
# we can also group by multiple cols
surveys %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight= mean(weight, na.rm = TRUE))
```
### When there are NaN, we need to remove them
```{r}
surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight))
```
### We can print with the argument "n" specifying the nmbr of rows to display.
```{r}
surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight)) %>%
  print(n = 5)
```
### we can summarize multiple variables at the same time. + arrange the results, to put the lighter species first
```{r}
surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight),
            min_weight = min(weight)) %>%
  arrange(min_weight)
# in decreasing order we will use arrange(desc(x))
```

### **Counting** 
```{r}
surveys %>%
  count(sex)
# to count the number of rows of data for each sex, with an alphabetical order
surveys %>%
  count(sex,species)
# if we want an descending order of the count
surveys %>%
  count(sex,species) %>%
  arrange(species,desc(n))
```

### Question: 
```{r}
#How many animals were caught in each plot_type surveyed?
surveys %>% 
  count(plot_type)

#Use group_by() and summarize() to find the mean, min, and max hindfoot length for each species (using species_id). Also add the number of observations

surveys %>%
  filter(!is.na(hindfoot_length)) %>% 
  group_by(species) %>%
  summarize(mean_hindfoot = mean(hindfoot_length),
            min_hindfoot = min(hindfoot_length),
            max_hindfoot = max(hindfoot_length),
            n=n())

#What was the heaviest animal measured in each year? Return the columns year, genus, species_id, and weight

surveys %>%
  filter(!is.na(weight)) %>%
  group_by(year) %>%
  filter(weight==max(weight)) %>%
  select(year,genus,species_id,weight) %>%
  arrange(year)
```

## 7: Reshaping data 

### **Pivoting the data into a wider format**

### Pivot_wider takes three main arguments : the data to be transformed, the "names_from" col names whose values will become new col names, the "values_from" col names whose values will fill the new cols.
### We will use that function to transform surveys to find the mean weight of each species in each plot over the entire survey period for example. 
```{r}
surveys_gw <- surveys %>%
  filter(!is.na(weight)) %>%
  group_by(genus, plot_id) %>%
  summarize(mean_weight = mean(weight))
surveys_wide <- surveys_gw %>%
    pivot_wider(names_from = genus,
                values_from = mean_weight)
surveys_wide
```

### **Pivoting data into a longer format**

### pivot_longer() takes four main arguments:the data to be transformed, the new names_to column we wish to create and populate with the current column names, the new values_to column we wish to create and populate with current values, the names of the columns to be used to populate the names_to and values_to variables (or to drop).

### we are going to recreate surveys_gw from surveys_wide, for that we need to drop plot_id.
```{r}
surveys_long <- surveys_wide %>%
    pivot_longer(names_to = "genus",
                 values_to = "mean_weight",
                 -plot_id)

surveys_long
```
### specification for what col to include with the ":" operator. 
```{r}
surveys_wide %>%
    pivot_longer(names_to = "genus",
                 values_to = "mean_weight",
                 Baiomys:Spermophilus)
```
### To re-arraneg the data, we use arrange() function
```{r}
surveys_wide %>%
    pivot_longer(names_to = "genus",
                 values_to = "mean_weight",
                 Baiomys:Spermophilus) %>%
    arrange(genus, plot_id)
```

### Questions
```{r}
# Spread the surveys data frame with year as columns, plot_id as rows, and the number of genera per plot as the values. 
rich_time <- surveys %>%
  group_by(plot_id, year) %>%
  summarize(n_genera = n_distinct(genus)) %>%
  pivot_wider(names_from = "year",
              values_from = "n_genera")
head(rich_time)

#data frame and transform it with pivot_longer() so each row is a unique plot_id by year combination

rich_time %>%
  pivot_longer(names_to = "year",
               values_to = "n_genera",
               -plot_id)
#use pivot_longer() to create a dataset where we have a key column called measurement and a value column that takes on the value of either hindfoot_length or weight
surveys_long <-
  surveys %>%
  pivot_longer(
    names_to = "measurement",
    values_to = "value",
    c(hindfoot_length, weight))
surveys_long

#calculate the average of each measurement in each year for each different plot_type.Then use pivot_wider() to generate a data set with a column for hindfoot_length and weight

surveys_long %>%
  group_by(year, measurement, plot_type) %>%
  summarize(mean_value = mean(value, na.rm=TRUE)) %>%
  pivot_wider(names_from = "measurement",
              values_from = "mean_value")
```

## 8: Exporting data

```{r}
surveys_complete <- surveys %>%
  filter(!is.na(weight),           ## remove missing weight
         !is.na(hindfoot_length),  ## remove missing hindfoot_length
         !is.na(sex))              ## remove missing sex
surveys_complete

## Extract the most common species_id
species_counts <- surveys_complete %>%
  count(species_id) %>%
  filter(n >= 50)
species_counts
## Only keep the most common species
surveys_complete <- surveys_complete %>%
  filter(species_id %in% species_counts$species_id)
surveys_complete

write_csv(surveys_complete, path = "data_output/surveys_complete.csv")
```

## 9: Additional exercises 

### **Question 1**
### We are going to re-analyse beer consumption in 48 individuals using dplyr. The data are available in the rWSBIM1207 package. The data illustrated the fictive beer consumption in litres per year at different age according to gender and employment.

### Directly load the data by typing
```{r,message = FALSE, results=FALSE}
library(rWSBIM1207)
data(beers)
```

### Remove observations with missing values.
```{r}
beers_no_na<- beers %>%
  filter(!is.na(beers))
```

### Using the Year, Month and Year columns, create a new column Date using dplyr::mutate and lubridate::ymd. What is the class of Date ?
```{r,message =FALSE, results=FALSE}
library(lubridate)
library(dplyr)
library(tidyverse)
beers_no_na %>%
  mutate(Date = ymd(paste(Year,Month,Day)))
        
```

### Create a new table, containing observations for women older than 35 years old, employed, and select all columns except Day, Month and Year, and order in descending value of consumption of beers.
```{r}
new_obs_beers<- beers_no_na %>%
  filter(Gender=="Female",Age>35,Work=="Employed") %>%
  select(-Day,-Month,-Year) %>%
  arrange(desc(Consumption))
```

### Export the new table to a csv file.
```{r}
write_csv(new_obs_beers, path = "data_output/new_obs_beers.csv")
```

### Beer consumption analysis:
### Does employment status have an impact on beer consumption?
```{r}
beers_no_na %>%
  group_by(Work) %>%
  summarize(Mean_Consumption= mean(Consumption,na.rm=TRUE))

```

### Do men drink more than women?
```{r}
beers_no_na %>%
  group_by(Gender) %>%
  summarize(mean_cons = mean(Consumption, na.rm = TRUE))
```

### Does employment status have an influence on beer consumption according to gender?
```{r}
beers_no_na %>%
  group_by(Gender,Work) %>%
  summarize(mean_cons = mean(Consumption, na.rm = TRUE))
```

### **Question 2**

### The Cancer Genome Atlas (TCGA) is a large scale effort that has collected high throughput biology data from hundreds of patients samples. In this exercise, we are going to analyse the clinical variables recorded for a subset of the patients.

### Using the clinical1.csv() function from rWSBIM1207, find the path the clinical1.csv file and read it to produce a data.frame named clinical.
```{r,message = FALSE}
library("rWSBIM1207")
clinical1<- read_csv(clinical1.csv())
clinical<- data_frame(clinical1)
class(clinical)
```

### Create a smaller data frame called clinical_mini containing only the columns corresponding to patientID, gender, age_at_diagnosis, smoking_history, number_pack_years_smoked, year_of_tobacco_smoking_onset, and stopped_smoking_year.
```{r}
clinical_mini <- clinical %>%
  select(patientID,gender,age_at_diagnosis,smoking_history,
         number_pack_years_smoked,year_of_tobacco_smoking_onset,stopped_smoking_year)
```

### Calculate the number of males and females in the cohort.
```{r}
clinical_mini %>%
  group_by(gender) %>%
  count(gender)
```

### Create a new variable years_at_diagnosis corresponding to the age at diagnosis converted from days into years.
```{r}
clinical_mini2 <- clinical_mini %>%
  mutate(years_at_diagnosis = age_at_diagnosis/365)
head(clinical_mini2)
```

###Calculate the mean and median age at diagnosis (in years). Pay attention to missing values!
```{r}
clinical_mini2 %>%
  summarize(meanyear = mean(years_at_diagnosis, na.rm=TRUE),
            medianyear = median(years_at_diagnosis,na.rm=TRUE))
```

### Calculate the mean and median age at diagnosis for males and females.
```{r}
clinical_mini2 %>%
  group_by(gender) %>%
  summarize(meanyear = mean(years_at_diagnosis, na.rm=TRUE),
            medianyear = median(years_at_diagnosis,na.rm=TRUE))
```

### How many patient were diagnosed before 50 years?
```{r}
clinical_mini2 %>%
  filter(years_at_diagnosis<50) %>%
  count()
```

### Compare the mean age at diagnosis between current smoker and lifelong non-smoker.
```{r}
clinical_mini2 %>%
  group_by(smoking_history) %>%
  summarize(meanAge = mean(years_at_diagnosis, na.rm = TRUE))
```

### Select patients who stopped smoking more than 15 years ago and calculate the number of smoking years for these cases. Display only cases for which you were able to calculate the data.
```{r}
clinical %>%
  filter(stopped_smoking_year<2006, !is.na(stopped_smoking_year)) %>% 
  mutate(number_of_smoking_years = stopped_smoking_year-year_of_tobacco_smoking_onset) %>% 
  filter(!is.na(number_of_smoking_years))
```

### How many of them smoked less than 5 years?
```{r}
clinical %>%
  filter(stopped_smoking_year<2006, !is.na(stopped_smoking_year)) %>% 
  mutate(number_of_smoking_years = stopped_smoking_year-year_of_tobacco_smoking_onset) %>% 
  filter(!is.na(number_of_smoking_years)) %>%
  filter(number_of_smoking_years<5) %>%
  count()
```

### Try to recreate the following table, reporting the number of smokers and lifelong-non smoker between males and females. Note: the layout can be different.

```{r}
clinictable <- clinical%>%
  select(smoking_history, gender)%>%
  filter(smoking_history %in% c("current smoker", "lifelong non-smoker"))
table(clinictable)
```

### **Question 3**

### Using the interroA.csv() function from the rWSBIM1207 package to get the path to the spreadsheet file, read the data into R using the read_csv function. This data is in the wide format, with the results of each test stored as a separate column.

### Using the appropriate pivot function, convert the data into a long table with a column interro informing which test that line refers to and a column result with the student’s mark.
```{r}
interroA <- read_csv(interroA.csv())
longer <- interroA %>%
  pivot_longer(cols = c(interro1, interro2, interro3, interro4),
               names_to= "interros", values_to = "results")
longer

interro_long<- interroA %>% 
  pivot_longer(names_to = "interro",
               values_to = "res",
               c(interro1, interro2, interro3, interro4))
interro_long
```







