---
title: 'Chapitre 4: Starting with data'
author: "Armelle de le Court"
date: "26/04/2021"
output: pdf_document
---

## 1: Presentation of the survey data

### Dataset stored as a comma separated value (=CSV) file. 
### If we want to download a file :download.file(url="https://ndownloader.figshare.com/files/2292169",destfile = "data/portal_data_joined.csv")
### But I already have it on my data so I just need do to this: 
```{r}
surveys <- read.csv("data/portal_data_joined.csv")
head(surveys) #to see the firsy 6 lines
```

## 2: What are data frames ?

### Its a representation of data in the format of a table where the columns are vectors that all have the same lenght.Columns are vectors, and each column must contain a single type of data. 
### To inspect the structure of a data frame:
```{r}
str(surveys)
```

### 3: Inspectinf data.frame objects

```{r,results = FALSE}
dim(surveys)
nrow(surveys)
ncol(surveys)
head(surveys)
tail(surveys)
names(surveys)
rownames(surveys)
summary(surveys)
```

## Question
```{r}
#class of object surveys?
class(surveys) #its a data.frame 

#How many rows and how many columns are in this object?
ncol(surveys)
nrow(surveys)

#How many species (as defined by the species_id variable)
length(unique(surveys$species_id))
```

## 4: Indexing and subsetting data frames

### If we want to extract some specific data from it, we need to specify the coordinates we want from it. Row number first, and then col numbers. 
```{r}
# first element in the first column of the data frame (as a vector)
surveys[1, 1]
# first element in the 6th column (as a vector)
surveys[1, 6]
```
```{r,results = FALSE}
# first column of the data frame (as a vector)
surveys[, 1]
# first column of the data frame (as a data.frame)
surveys[1]
```
```{r}
# first three elements in the 7th column (as a vector)
surveys[1:3, 7]
# the 3rd row of the data frame (as a data.frame)
surveys[3, ]
# equivalent to head_surveys <- head(surveys)
head_surveys <- surveys[1:6, ]
```

### We can also exclude indices using "-" sign : 
```{r,results=FALSE}
surveys[, -1] ## The whole data frame, except the first column
surveys[-c(7:34786), ] ## Equivalent to head(surveys)
```
### subset by calling indices, but by their names: 
```{r, results = FALSE}
surveys["species_id"] # Result is a data.frame
surveys[, "species_id"] # Result is a vector
surveys[["species_id"]] # Result is a vector
surveys$species_id # Result is a vector
```
### Question
```{r}
#QUESTION1
#data.frame (surveys_200) containing only the data in row 200 of the surveys dataset

surveys_200 <- surveys[200,]
surveys_200

#QUESTION2 Notice how nrow() gave you the number of rows in a data.frame? Use that number to pull out just that last row in the inital surveys data frame

nrow(surveys) #34786
last_row <- surveys[34786,]
last_row

#Compare that with what you see as the last row using tail()
tail(surveys)

#QUESTION3
#Use nrow() to extract the row that is in the middle of the surveys dataframe.Store the content of this row in an object named surveys_middle

n_rows <- nrow(surveys)
surveys_middle <- surveys[n_rows/2,]
surveys_middle

#QUESTION4
#Combine nrow() with the - notation above to reproduce the behavior of head(surveys), keeping just the first through 6th rows of the surveys dataset.

surveys_head <- surveys[-(7:n_rows), ]
surveys_head
```

## 5: Factors

### Stored as integers associated with labels and they can be ordererd or unordered. By default R always sort levels in alphabetical order. 
```{r}
sex <- factor(c("male", "female", "female", "male"))
levels(sex) #level 1 is female bc f comes before m. 
nlevels(sex)
sex
# to reorder: 
sex <- factor(sex, levels= c("male","female"))
sex # it has been reorder
```
### If we need to convert a factor to a character
```{r}
as.character(sex)
```
### If we need do rename factors
```{r}
surv_sex <- factor(surveys$sex)
plot(surv_sex) 
#about 1700 individuals hasnt been recorded for the sex variable. We need to rename that label
head(surv_sex)
levels(surv_sex)
levels(surv_sex)[1] <- "undetermined"
levels(surv_sex) # it now has a name
```
### Question
```{r}
#rename F and M to male and female + barplot 
levels(surv_sex)[2:3] <- c("female", "male")
surv_sex <- factor(surv_sex, levels = c("female", "male", "undetermined"))
plot(surv_sex)
```

### 6: Matrices 

### We want to generate a matrix containing info about all packages installed 
```{r}
ip <- installed.packages()
dim(ip)
head(colnames(ip))
head(rownames(ip))
```
### Question: is lubridate installed? 
```{r}
## 1. get the names of all installed packages
ip <- rownames(installed.packages())
## 2. TRUE if installed, FALSE otherwise
"lubridate" %in% ip
# it is TRUE so it is installed
```
### How to create a matrix
```{r}
m <- matrix(1:9, ncol=3, nrow=3)
```
### Question: construct a matrix of dimension 1000 by 3 of normally distributed data (mean 0, sd 1)
```{r}
set.seed(123)
m <- matrix(rnorm(3000), ncol = 3)
dim(m)
head(m)
```

## 7: Formatting dates

```{r,message=FALSE}
library("lubridate")
```
### ymd() takes a vector representing year,month and day and converts it to a Date vector.
```{r}
my_date <- ymd("2015-01-01")
str(my_date)
# sep indicates the character to use to separate each component
my_date <- ymd(paste("2015", "1", "1", sep = "-"))
str(my_date)
```
### Create a character vector from the year, month and day columns of surveys using paste()
```{r}
head(paste(surveys$year, surveys$month, surveys$day, sep = "-"))
```
### This character vecor can be used as the argument for ymd(): 
```{r}
head(ymd(paste(surveys$year, surveys$month, surveys$day, sep = "-")))
```
### the resulting date vatcor can be added to surveys as a new col called date: 
```{r}
surveys$date <- ymd(paste(surveys$year, surveys$month, surveys$day, sep = "-"))
str(surveys) # we can see there is indeed a new col
# make sure everything worked correctly 
summary(surveys$date)
# some dates have missing values, we need to look where they come from
is_missing_date <- is.na(surveys$date)
date_columns <- c("year", "month", "day")
missing_dates <- surveys[is_missing_date,  date_columns]

head(missing_dates)
```
## 8: Summary of R objects: 

### vector: one dimension (they have a length), single type of data.
###matrix: two dimensions, single type of data.
### data.frame: two dimensions, one type per column.

## 9: Lists

### List is 1D, every item can be different data type. 
### For example, we can create a list with numbers, character, dataframe,...
```{r}
l <- list(1:10, ## numeric
          letters, ## character
          installed.packages(), ## a matrix
          cars, ## a data.frame
          list(1, 2, 3)) ## a list
length(l)
str(l)
```
### List subsetting by using [] for a new sub-list, or [[]] to extract a single element of that list. 
```{r}
l[[1]] ## first element
l[1:2] ## list of length 2
l[1] ## list of length 1
```

## 10: exporting and saving data 

```{r}
# to export the surveys data to the my_surveys.csv file in the data_output directory
write.csv(surveys, file = "data_output/my_surveys.csv")

#to save data as rda
save(surveys, file = "data_output/surveys.rda")
rm(surveys)
load("data_output/surveys.rda")
head(surveys)

#to save as RDS
saveRDS(surveys, file = "data_output/surveys.rds")
rm(surveys)
surveys <- readRDS("data_output/surveys.rds")
head(surveys)
```

## 11: Additional exercises 

## Question 1: You’re doing an colony counting experiment, counting every day how many molds you see in your cell cultures.

### Create a vector named molds containing the results of your counts: 1, 2, 5, 8 and 10. Create a vector days containing the week day, from Monday to Friday. Use these two vector to create a data.frame named molds_study containing two variables, Day and Molds_count.
```{r}
molds <- c(1,2,5,8,10)
days <- c("Monday","Tuesday","Wednesday","Thursday","Friday")
molds_study <- data.frame("Day"= days,"Molds_count"= molds)
molds_study
class(molds_study)
```
### Create a new data.frame that contains the observations where more than 2 colonies were counted. How many observations are there? How many counts are there in total for these observations.
```{r}
m2 <- molds_study$Molds_count > 2 #trouver obs avec >2 colonies
molds2 <- molds_study[m2,] # créer le nouveau data.frame
molds2
nrow(molds2) # compter le nombre d'obs
sum(molds2$Molds_count) # combien de colonies y comptons nous? 
```

### You repeat the molds study experiment the following week and count the following numbers of molds: 1, 6, 6, 5 and 4.
### Add these data as a third column to the molds_study data.frame and rename the variables as Day, Molds_1 and Molds_2.
```{r}
molds_2 <- c(1,6,6,5,4)
molds_study2 <- data.frame("Day"= days,"Molds_1"= molds, "Molds_2"=molds_2)
molds_study2
```
### Calculate for each experiment the total number of molds counted. Check if the first experiment counted more molds than the second one.
```{r}
sum1 <- sum(molds_study2$Molds_1) 
sum2 <- sum(molds_study2$Molds_2)
sum1>sum2 #it returns TRUE so 1st experiment counted more molds 
```
### Save the molds_study variable in a file named molds_study.rda.
```{r}
ave(molds_study, file = "data_output/molds_study.rda")
```

## Question 2: We are going to analyse beer consumption in 48 individuals. The data are available in the rWSBIM1207 package. The data illustrated the fictive beer consumption in liters per year at different age according to gender and employment.

### Using the beers.csv() function from rWSBIM1207, find the path the beers.csv file and read it to produce a data.frame named beers. The spreadsheet uses semi-colons ; to separate cells. Use read.csv2() and read.delim() and set the separator appropriately, and verify that the two variables are identical.
```{r, message = FALSE, results=FALSE}
library(rWSBIM1207)
beers <- read.csv(beers.csv())
beers2 <- read.csv2(beers.csv())
beers2
```
### Check the number of observations and identify the variables that are available. Calculate a summary of each variable using the summary function directly on the data.frame.
```{r}
sum(beers2$Record_ID)
summary(beers2)
```
### Calculate the mean and the median age and consumption.
```{r}
mean(beers2$Age)
median(beers2$Age)
#OU
mean(beers2[,5])
median(beers2[,5])
```
### Do men consume more beer than women on average? To answer this question, calculate the mean consumption for men only, selecting the observations with Gender equal to Male. Then do the same for observations with Gender equal to Female.
```{r}
m <- beers2$Gender == "Male" #extraire lignes correspondant aux hommes
beers2[m, "Consumption"] # extraire consommation des hommes
mean(beers2[m, "Consumption"]) # moyenne 

f <- beers2$Gender == "Female"   #extraire lignes correspondant aux femmes
beers2[f, "Consumption"]       # extraire consommation des femmes
mean(beers2[f, "Consumption"], na.rm=TRUE)     # moyenne 
```
### Calculate a two-way table of gender and employment status.
```{r}
genderXemployment <- table(beers2$Gender,beers2$Work)
genderXemployment
```
### Remove observations with missing values and export the data into a new csv file called beers_no_na.csv.
```{r}
library(dplyr)
beers_no_na <- beers2 %>% 
  na.omit

save(beers_no_na, file = "data_output/beers_no_na.csv")
```

## Question 3: We are going to analyse clinical data from The Cancer Genome Atlas (TCGA). The data are available in the rWSBIM1207 package.

### Obtain the path to th csv file containing the clinical data need for this exercise using the clinical1.csv function and read it into R as a data.frame called clinical.
```{r}
clinical <- read.csv(clinical1.csv())
class(clinical)
```
### Inspect the data using str and View. How many patients are recorded in the table?
```{r, results=FALSE}
# str(clinical)
# View(clinical)
nrow(clinical)
```
### Print the column names using two different functions.
```{r}
#?
```
### Create a smaller data frame called clinical_mini containing only the columns corresponding to the patientID, gender, age_at_diagnosis and smoking_history. Try to do this using column indices and column names.
```{r}
clinical_mini <- data_frame(clinical$patientID,clinical$gender,clinical$age_at_diagnosis,clinical$smoking_history)

library(tidyverse)
names(clinical_mini)[1] <- "patientID"
names(clinical_mini)[2] <- "gender"
names(clinical_mini)[3] <- "age_at_diagnosis"
names(clinical_mini)[4] <- "smoking_history"
```
### Inspect the smoking_history column. How many categories are recorded? How many observations are there for each category?
```{r}
tablesm<- table(clinical_mini$smoking_history)
length(unique(tablesm))
```
### The column age at diagnosis is recorded in days. Create a new column years_at_diagnosis corresponding to the age at diagnosis converted in years.
```{r}
new_clinical_mini <- clinical_mini %>% 
  mutate(years_at_diagnosis = age_at_diagnosis/365)
```
### Calculate the mean and median age at diagnosis. Hint: pay attention to missing values!
```{r}
mean(new_clinical_mini$years_at_diagnosis, na.rm=TRUE)
median(new_clinical_mini$years_at_diagnosis, na.rm=TRUE)
```
### Is there a difference between the years_at_diagnosis for male and female patients?
```{r,message=FALSE}
library(tidyverse)
```
```{r}
male<- new_clinical_mini %>% 
  group_by(gender="male", na.rm=TRUE) 
mean(male$years_at_diagnosis, na.rm=TRUE)

female<- new_clinical_mini %>% 
  group_by(gender="female", na.rm=TRUE) 
mean(female$years_at_diagnosis, na.rm=TRUE)
```
### Use the quantile function to calculate the first and last quartile of age at diagnosis. Use the help function (?quantile) to see how to use the quantile function.
```{r}
?quantile
quantile(new_clinical_mini$years_at_diagnosis, na.rm=TRUE)
```
### Use the summary function to confirm your previous results.
```{r}
summary(new_clinical_mini$years_at_diagnosis)
```











