Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
Christoph Räss
HSLU deep Learning
Commits
5fbdc4c1
Commit
5fbdc4c1
authored
Nov 18, 2020
by
Simon van Hemert
Committed by
renku 0.12.0
Nov 18, 2020
Browse files
renku init --force
parent
82b91540
Changes
16
Hide whitespace changes
Inline
Side-by-side
.gitlab-ci.yml
View file @
5fbdc4c1
# renku: v1.dev20180723
variables
:
GIT_STRATEGY
:
fetch
GIT_SSL_NO_VERIFY
:
"
true"
...
...
@@ -11,7 +13,9 @@ image_build:
image
:
docker:stable
before_script
:
-
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN http://$CI_REGISTRY
script
:
|
CI_COMMIT_SHA_7=$(echo $CI_COMMIT_SHA | cut -c1-7)
docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7 .
docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7
\ No newline at end of file
script
:
-
CI_COMMIT_SHA_7=$(echo $CI_COMMIT_SHA | cut -c1-7)
-
docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7 .
-
docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7
tags
:
-
image-build
\ No newline at end of file
.renku/metadata.yml
View file @
5fbdc4c1
...
...
@@ -19,12 +19,10 @@ http://schema.org/schemaVersion: '8'
https://swissdatasciencecenter.github.io/renku-ontology#automatedTemplateUpdate:
false
https://swissdatasciencecenter.github.io/renku-ontology#immutableTemplateFiles: []
https://swissdatasciencecenter.github.io/renku-ontology#templateId: python-minimal
https://swissdatasciencecenter.github.io/renku-ontology#templateMetadata: '{"__template_source__"
:
"
https://github.com/SwissDataScienceCenter/renku-project-template"
,
"
__template_ref__"
:
"
0.1.13"
,
"
__template_id__"
:
"
python-minimal"
,
"
__namespace__"
:
"
simon.vanhemert"
,
"
__automated_update__"
:
true
, "__repository__"
:
"
https://renkulab.io/gitlab"
,
"
__sanitized_project_name__"
:
"
hslu-deep-learning"
,
"
__project_slug__"
:
"
simon.vanhemert/hslu-deep-learning"
,
"
description"
:
"
"
,
"
name"
:
"
HSLU
deep
Learning"
}
'
https://swissdatasciencecenter.github.io/renku-ontology#templateReference:
0.1.13
https://swissdatasciencecenter.github.io/renku-ontology#templateSource:
https://github.com/SwissDataScienceCenter/renku-project-template
https://swissdatasciencecenter.github.io/renku-ontology#templateVersion:
null
https://swissdatasciencecenter.github.io/renku-ontology#templateMetadata: '{"description"
:
"
Deep
learning
project
at
Renkulab"
,
"
__template_source__"
:
"
renku"
,
"
__template_ref__"
:
"
master"
,
"
__template_id__"
:
"
python-minimal"
,
"
__namespace__"
:
"
"
,
"
__sanitized_project_name__"
:
"
"
,
"
__repository__"
:
"
"
,
"
__project_slug__"
:
"
"
,
"
name"
:
"
hslu-deep-learning"
}
'
https://swissdatasciencecenter.github.io/renku-ontology#templateReference:
master
https://swissdatasciencecenter.github.io/renku-ontology#templateSource:
renku
https://swissdatasciencecenter.github.io/renku-ontology#templateVersion:
0.12.0
.renku/renku.ini
View file @
5fbdc4c1
[renku "interactive"]
default_url
=
/lab
[renku]
lfs_threshold
=
3mb
default_url
=
/lab
\ No newline at end of file
.renku/template_checksums.json
View file @
5fbdc4c1
{
".renkulfsignore"
:
"25702085882005ad576e6e51259eba461306b6b4be069361d54f3926d941ecfb"
,
".gitignore"
:
"d890ab1ae56fa2761ba26961529e8e010fbf0e607a0fbe5364d34eeba38beab0"
,
"requirements.txt"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
,
".dockerignore"
:
"46625726a43957a07c38550b006dd352ef85af18f7b984a1a87284001d03713a"
,
"README.md"
:
"da03529d4172b0054eaa7065129f3b97574ffa8f2085973f5d634ba5300e2247"
,
"Dockerfile"
:
"24550d7598c2cb8038c294c7ef2b2cae827457abf92d58121d8d168a03a4e0a7"
,
"environment.yml"
:
"f91b4d74abed097afebcfa125fd3de20614384a6555d864e4ac304fee375b27e"
,
".gitlab-ci.yml"
:
"309bfbc0e2851600908f0fc430e8a41345146dca2b98eb0908f21cc1796db8ce"
,
".renku/renku.ini"
:
"187a51794112821730b021723717f6680a277cc25fa07838b2e2c541a619b0a1"
,
"data/.gitkeep"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
,
"notebooks/.gitkeep"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
\ No newline at end of file
{
"README.md"
:
"4db2493809b0c67da4d85e00765901a5d99c8663c7dcee44e6794b700993fddb"
,
".gitlab-ci.yml"
:
"7dc37e92ee3349316d423e293da71538bae50bb386379515aba59cae84955f06"
,
".gitignore"
:
"d890ab1ae56fa2761ba26961529e8e010fbf0e607a0fbe5364d34eeba38beab0"
,
".renkulfsignore"
:
"25702085882005ad576e6e51259eba461306b6b4be069361d54f3926d941ecfb"
,
".dockerignore"
:
"46625726a43957a07c38550b006dd352ef85af18f7b984a1a87284001d03713a"
,
"requirements.txt"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
,
"environment.yml"
:
"f91b4d74abed097afebcfa125fd3de20614384a6555d864e4ac304fee375b27e"
,
"Dockerfile"
:
"093414748a7a7c3bdd2b7277220df677ccf6deeb212de61f9f1c61c6a2b893d4"
,
".renku/renku.ini"
:
"187a51794112821730b021723717f6680a277cc25fa07838b2e2c541a619b0a1"
,
".renku/metadata.yml"
:
"380aa633a150dc4df38537167986979a79e74c19ff3e074c950e1f4d8efca4b0"
,
"data/.gitkeep"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
,
"notebooks/.gitkeep"
:
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
\ No newline at end of file
Dockerfile
View file @
5fbdc4c1
# For finding latest versions of the base image see
# https://github.com/SwissDataScienceCenter/renkulab-docker
ARG
RENKU_BASE_IMAGE=renku/renkulab-py:3.7-0.7.2
FROM
${RENKU_BASE_IMAGE}
FROM
renku/renkulab-py:3.7-renku0.10.4-0.6.3
# see https://github.com/SwissDataScienceCenter/renkulab-docker
# to swap this image for the latest version available
# Uncomment and adapt if code is to be included in the image
# COPY src /code/src
...
...
@@ -22,19 +21,4 @@ COPY requirements.txt environment.yml /tmp/
RUN
conda
env
update
-q
-f
/tmp/environment.yml
&&
\
/opt/conda/bin/pip
install
-r
/tmp/requirements.txt
&&
\
conda clean
-y
--all
&&
\
conda
env export
-n
"root"
# RENKU_VERSION determines the version of the renku CLI
# that will be used in this image. To find the latest version,
# visit https://pypi.org/project/renku/#history.
ARG
RENKU_VERSION=0.12.0
########################################################
# Do not edit this section and do not add anything below
RUN if
[
-n
"
$RENKU_VERSION
"
]
;
then
\
pipx uninstall renku
&&
\
pipx
install
--force
renku
==
${
RENKU_VERSION
}
\
;
fi
########################################################
\ No newline at end of file
conda
env export
-n
"root"
\ No newline at end of file
README.md
View file @
5fbdc4c1
# HSLU deep Learning
# hslu-deep-learning
Deep learning project at Renkulab
## Introduction
...
...
notebooks/Block 1/Jupyter Notebook Block 1 - Introduction to Image Classification.ipynb
View file @
5fbdc4c1
%% Cell type:markdown id: tags:
# 1. Importing and Visualization of CIFAR-10 Dataset
%% Cell type:code id: tags:
```
python
import
numpy
as
np
from
tensorflow.keras.datasets
import
cifar10
(
X_train
,
y_train
),
(
X_test
,
y_test
)
=
cifar10
.
load_data
()
```
%% Cell type:markdown id: tags:
### REMARK:
If you have the data set downloaded on your own drive, you could unpack and inspect using the following cells.
If you are running on Renkulab, you can skip to the next header
%% Cell type:code id: tags:
```
python
# function to import CIFAR-10 data set
def
unpickle
(
file
):
import
pickle
with
open
(
file
,
'rb'
)
as
fo
:
dict
=
pickle
.
load
(
fo
,
encoding
=
'bytes'
)
return
dict
data_batch_1
=
unpickle
(
"./Daten/data_batch_1"
)
data_batch_2
=
unpickle
(
"./Daten/data_batch_2"
)
data_batch_3
=
unpickle
(
"./Daten/data_batch_3"
)
data_batch_4
=
unpickle
(
"./Daten/data_batch_4"
)
data_batch_5
=
unpickle
(
"./Daten/data_batch_5"
)
test_batch
=
unpickle
(
"./Daten/test_batch"
)
```
%%%% Output: error
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-
1-d20dcc958bb0
> in <module>
7
dict = pickle.load(fo, encoding='bytes')
8
return dict
---->
9
data_batch_1 = unpickle("./Daten/data_batch_1")
10
data_batch_2 = unpickle("./Daten/data_batch_2")
11
data_batch_3 = unpickle("./Daten/data_batch_3")
<ipython-input-
1-d20dcc958bb0
> in unpickle(file)
4
def unpickle(file):
5
import pickle
---->
6
with open(file, 'rb') as fo:
7
dict = pickle.load(fo, encoding='bytes')
8
return dict
<ipython-input-
2-e287de61b2d8
> in <module>
5
dict = pickle.load(fo, encoding='bytes')
6
return dict
---->
7
data_batch_1 = unpickle("./Daten/data_batch_1")
8
data_batch_2 = unpickle("./Daten/data_batch_2")
9
data_batch_3 = unpickle("./Daten/data_batch_3")
<ipython-input-
2-e287de61b2d8
> in unpickle(file)
2
def unpickle(file):
3
import pickle
---->
4
with open(file, 'rb') as fo:
5
dict = pickle.load(fo, encoding='bytes')
6
return dict
FileNotFoundError: [Errno 2] No such file or directory: './Daten/data_batch_1'
%% Cell type:markdown id: tags:
What is the data structure of e.g. data_batch_1 ?
%% Cell type:code id: tags:
```
python
type
(
data_batch_1
)
```
%%%% Output: execute_result
dict
%% Cell type:markdown id: tags:
What are the keys of e.g. data_batch_1 ?
%% Cell type:code id: tags:
```
python
data_batch_1
.
keys
()
```
%%%% Output: execute_result
dict_keys([b'batch_label', b'labels', b'data', b'filenames'])
%% Cell type:markdown id: tags:
What is the data structure of data_batch_1[b'data'] ?
%% Cell type:code id: tags:
```
python
type
(
data_batch_1
[
b
'data'
])
```
%%%% Output: execute_result
numpy.ndarray
%% Cell type:markdown id: tags:
What is the data structure of data_batch_1[b'labels'] ?
%% Cell type:code id: tags:
```
python
type
(
data_batch_1
[
b
'labels'
])
```
%%%% Output: execute_result
list
%% Cell type:markdown id: tags:
What is the shape of data_batch_1[b'data'] ?
%% Cell type:code id: tags:
```
python
data_batch_1
[
b
'data'
].
shape
```
%%%% Output: execute_result
(10000, 3072)
%% Cell type:markdown id: tags:
What is the size of data_batch_1[b'labels'] ?
%% Cell type:code id: tags:
```
python
len
(
data_batch_1
[
b
'labels'
])
```
%%%% Output: execute_result
10000
%% Cell type:markdown id: tags:
What are the first 10 elements of data_batch_1[b'labels'] ?
%% Cell type:code id: tags:
```
python
data_batch_1
[
b
'labels'
][:
10
]
```
%%%% Output: execute_result
[6, 9, 9, 4, 1, 1, 2, 7, 8, 3]
%% Cell type:markdown id: tags:
What is the data type of data_batch_1[b'data'] ?
%% Cell type:code id: tags:
```
python
data_batch_1
[
b
'data'
].
dtype
```
%%%% Output: execute_result
dtype('uint8')
%% Cell type:markdown id: tags:
Let us concatenate the batch training data
%% Cell type:code id: tags:
```
python
X_train
=
np
.
concatenate
([
data_batch_1
[
b
'data'
],
data_batch_2
[
b
'data'
],
data_batch_3
[
b
'data'
],
data_batch_4
[
b
'data'
],
data_batch_5
[
b
'data'
]],
axis
=
0
)
```
%% Cell type:markdown id: tags:
Let us concatenate the training labels
%% Cell type:code id: tags:
```
python
y_train
=
np
.
concatenate
([
data_batch_1
[
b
'labels'
]
,
data_batch_2
[
b
'labels'
],
data_batch_3
[
b
'labels'
],
data_batch_4
[
b
'labels'
],
data_batch_5
[
b
'labels'
]],
axis
=
0
)
```
%%%% Output: error
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-5-0db84ead4b15> in <module>
----> 1 y_train=np.concatenate([data_batch_1[b'labels'] ,
2 data_batch_2[b'labels'],
3 data_batch_3[b'labels'],
4 data_batch_4[b'labels'],
5 data_batch_5[b'labels']],
NameError: name 'data_batch_1' is not defined
%% Cell type:markdown id: tags:
Let us define the test data as X_test
%% Cell type:code id: tags:
```
python
X_test
=
test_batch
[
b
'data'
]
X_test
.
shape
```
%%%% Output: error
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-8-95e1076f079d> in <module>
----> 1 X_test=test_batch[b'data']
2 X_test.shape
NameError: name 'test_batch' is not defined
%% Cell type:markdown id: tags:
Let us cast the test labels as ndarray
%% Cell type:code id: tags:
```
python
y_test
=
np
.
array
(
test_batch
[
b
'labels'
])
y_test
.
shape
```
%%%% Output: execute_result
(10000,)
%% Cell type:markdown id: tags:
# Continue from here in Renkulab
%% Cell type:markdown id: tags:
What is the shape of X_train ?
%% Cell type:code id: tags:
```
python
X_train
.
shape
```
%%%% Output: execute_result
(50000, 32, 32, 3)
%% Cell type:markdown id: tags:
What is the shape of Y_train ?
%% Cell type:code id: tags:
```
python
y_train
.
shape
```
%%%% Output: execute_result
(50000, 1)
%% Cell type:markdown id: tags:
Let us visualize an image.
By default, NumPy arrays are created in row major order.
Spatially this means, that if we have a two-dimensional
array of data, the items in each row of the array are stored
in adjacent memory locations. In the case of a three-dimensional
array of data, the items along
`axis=2`
are stored in adjacent order.
Since the first $32$ entries of the array $X
\_
train[0]$ are the
red channel values of the first row of the image, etc., we need to
pass the tuple $(3,32,32)$ to reshape.
By default, NumPy arrays are created in row major order, that is, when
reshaping the array, higher order dimensions are traversed first
(e.g.
`axis=2`
before advancing on
`axis=1`
.)
`plt.imshow`
needs for each inner list the values representing
a pixel. Here, with an RGB image, there are 3 values. We thus
need to transpose the array : the RGB values need to be located
along
`axis=2`
.
%% Cell type:code id: tags:
```
python
%
matplotlib
inline
import
matplotlib.pyplot
as
plt
# plt.imshow(X_train[20].reshape((3,32,32)).transpose((1,2,0)).astype('uint8'))
plt
.
imshow
(
X_train
[
20
])
plt
.
show
()
```
%%%% Output: display_data
%% Cell type:markdown id: tags:
We visualize some examples from the dataset.
We show a few examples of training images from each class.
%% Cell type:code id: tags:
```
python
%
matplotlib
inline
classes
=
[
'plane'
,
'car'
,
'bird'
,
'cat'
,
'deer'
,
'dog'
,
'frog'
,
'horse'
,
'ship'
,
'truck'
]
num_classes
=
len
(
classes
)
samples_per_class
=
7
for
y
,
cls
in
enumerate
(
classes
):
idxs
=
np
.
flatnonzero
(
y_train
==
y
)
idxs
=
np
.
random
.
choice
(
idxs
,
samples_per_class
,
replace
=
False
)
for
i
,
idx
in
enumerate
(
idxs
):
plt_idx
=
i
*
num_classes
+
y
+
1
plt
.
subplot
(
samples_per_class
,
num_classes
,
plt_idx
)
plt
.
imshow
(
X_train
[
idx
])
plt
.
axis
(
'off'
)
if
i
==
0
:
plt
.
title
(
cls
)
plt
.
show
()
```
%%%% Output: display_data
%% Cell type:markdown id: tags:
# 2. K-Nearest-Neighbour Classifier
%% Cell type:markdown id: tags:
We subsample the data for more efficient code execution in this exercise.
%% Cell type:code id: tags:
```
python
num_training
=
5000
mask
=
range
(
num_training
)
X_train
=
X_train
[
mask
]
y_train
=
y_train
[
mask
]
num_test
=
500
mask
=
range
(
num_test
)
X_test
=
X_test
[
mask
]
y_test
=
y_test
[
mask
]
```
%% Cell type:markdown id: tags:
We define Class KNearestNeighbor.
%% Cell type:code id: tags:
