Newer
Older
{
"cells": [
{
"cell_type": "markdown",
"id": "cac88086-1bc3-4916-809f-85e7e94f8fe0",
"metadata": {},
"source": [
"## Renku MLS Plug-in demo"
]
},
{
"cell_type": "markdown",
"id": "2b928da5-636d-463c-a081-98586cf8c468",
"metadata": {},
"source": [
"This plug-in allows you to compare across different `renku run` iterations in terms of pre-defined metrics."
]
},
{
"cell_type": "markdown",
"id": "a13b4e8a-660f-402a-8e53-72b68b198985",
"metadata": {},
"source": [
"See `src/train.py` for the demonstration training file."
]
},
{
"cell_type": "markdown",
"id": "160b5662-463a-4db4-84c3-1cbf1ffe472e",
"metadata": {},
"source": [
"### Note: The following `renku run` commands have already been run and are shown for demonstration purposes. For cells to run, skip below to "
]
},
"id": "4b06791b-a3c2-45db-a788-69c8c39d0cf0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"You chose the RandomForestClassifier model.\n",
"Accuracy: 1.0\n",
"Info: Adding these files to Git LFS:\n",
"\tmodels/RFC\n",
"To disable this message in the future, run:\n",
"\trenku config set show_lfs_message False\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/conda/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
" from pandas import MultiIndex, Int64Index\n"
]
}
],
"source": [
"%%bash\n",
"cd ../ # Return to the main repository\n",
"renku run -- python src/train.py data/wine/wine.data label RandomForestClassifier models/RFC"
"id": "c6544915-fda6-4728-92a4-faa27acb2be0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"You chose the LinearSVC model.\n",
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/conda/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
" from pandas import MultiIndex, Int64Index\n",
"/opt/conda/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
" warnings.warn(\n"
]
}
],
"source": [
"%%bash\n",
"cd ../ # Return to the main repository\n",
"renku run -- python src/train.py data/wine/wine.data label LinearSVC models/SVC"
"id": "7d2b84bb-0c7d-4095-8af5-83ff05e50c4f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"You chose the XGBClassifier model.\n",
"[02:52:22] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"Accuracy: 0.9830508474576272\n",
"Info: Adding these files to Git LFS:\n",
"\tmodels/XGB\n",
"To disable this message in the future, run:\n",
"\trenku config set show_lfs_message False\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/conda/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
" from pandas import MultiIndex, Int64Index\n",
"/opt/conda/lib/python3.9/site-packages/xgboost/sklearn.py:1224: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].\n",
" warnings.warn(label_encoder_deprecation_msg, UserWarning)\n"
]
}
],
"source": [
"%%bash\n",
"cd ../ # Return to the main repository\n",
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
"renku run -- python src/train.py data/wine/wine.data label XGBClassifier models/XGB"
]
},
{
"cell_type": "markdown",
"id": "9dab8308-6b91-4338-ab33-d44cb6e15459",
"metadata": {},
"source": [
"## View the leaderboard"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "42914a63-4354-461f-8f1d-54a5c647d41d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+----------------------------------+-------------------------------------------------+-----------------------------------------+--------------------+\n",
"| Run ID | Model | Inputs | accuracy |\n",
"+----------------------------------+-------------------------------------------------+-----------------------------------------+--------------------+\n",
"| 83dbed2912bd440e97681720ac2b588e | sklearn.ensemble._forest.RandomForestClassifier | ['data/wine/wine.data', 'src/train.py'] | 1.0 |\n",
"| 150a59441ca54dfcba3365db804fab99 | xgboost.sklearn.XGBClassifier | ['data/wine/wine.data', 'src/train.py'] | 0.9830508474576272 |\n",
"| 148fc15412a7430e9268849c8bb1df84 | sklearn.svm._classes.LinearSVC | ['data/wine/wine.data', 'src/train.py'] | 0.8305084745762712 |\n",
"+----------------------------------+-------------------------------------------------+-----------------------------------------+--------------------+\n"
]
}
],
"source": [
"! cd ../; renku mls leaderboard"
]
},
{
"cell_type": "markdown",
"id": "5e6e8e44-ce99-40fd-a11e-3225f28bb7ff",
"metadata": {},
"source": [
"## View the hyper-parameters in each of the models"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "879cf6b8-890a-43d3-8034-5ed3dfa2b71e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+----------------------------------+-------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Run ID | Model | Hyper-Parameters |\n",
"+----------------------------------+-------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| 148fc15412a7430e9268849c8bb1df84 | sklearn.svm._classes.LinearSVC | {\"C\": \"1.0\", \"dual\": \"true\", \"fit_intercept\": \"true\", \"intercept_scaling\": \"1\", \"loss\": \"squared_hinge\", \"max_iter\": \"1000\", \"multi_class\": \"ovr\", \"penalty\": \"l2\", \"tol\": \"0.0001\", \"verbose\": \"0\"} |\n",
"| 150a59441ca54dfcba3365db804fab99 | xgboost.sklearn.XGBClassifier | {\"base_score\": \"0.5\", \"booster\": \"gbtree\", \"colsample_bylevel\": \"1\", \"colsample_bynode\": \"1\", \"colsample_bytree\": \"1\", \"enable_categorical\": \"false\", \"gamma\": \"0\", \"gpu_id\": \"-1\", \"interaction_constraints\": \"\", \"learning_rate\": \"0.300000012\", \"max_delta_step\": \"0\", \"max_depth\": \"6\", \"min_child_weight\": \"1\", \"missing\": \"nan\", \"monotone_constraints\": \"()\", \"n_estimators\": \"100\", \"n_jobs\": \"8\", \"num_parallel_tree\": \"1\", \"objective\": \"multi:softprob\", \"predictor\": \"auto\", \"random_state\": \"0\", \"reg_alpha\": \"0\", \"reg_lambda\": \"1\", \"subsample\": \"1\", \"tree_method\": \"exact\", \"use_label_encoder\": \"true\", \"validate_parameters\": \"1\"} |\n",
"| 83dbed2912bd440e97681720ac2b588e | sklearn.ensemble._forest.RandomForestClassifier | {\"bootstrap\": \"true\", \"ccp_alpha\": \"0.0\", \"criterion\": \"gini\", \"max_features\": \"auto\", \"min_impurity_decrease\": \"0.0\", \"min_samples_leaf\": \"1\", \"min_samples_split\": \"2\", \"min_weight_fraction_leaf\": \"0.0\", \"n_estimators\": \"100\", \"oob_score\": \"false\", \"verbose\": \"0\", \"warm_start\": \"false\"} |\n",
"+----------------------------------+-------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
],
"source": [
"! cd ../; renku mls params"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}