When I studied in mathematical lyceum, my teacher taught me that the best way to understand something is to visualize it. For example, we had a wooden board, plasticine, and metal wire to be able to visualize stereometry problems. It helped a lot to develop visual thinking and skills in solving challenging tasks.
I truly believe that real data scientists should understand algorithms and have a feeling on how to improve it if something works not fine. Especially in the area of deep learning. In my mind, the best way to develop these skills is to see how the model is trained, what happens when you change hyperparameters. This is the reason why I want to share how to build a simple dashboard for CNN live training with the opportunity to tune a few hyperparameters online.
There is common knowledge that if we choose too big learning rate, we will see how our loss function explodes (our model will not converge); if we choose too small learning rate, the training process can last too long. What about dropout? There is an opinion that dropout reduces overfitting. I get to check everything myself even if I believe, because to know and to believe are different things.
Below is the short demo of my dashboard. Red dots on loss function & accuracy plots represent the training dataset, blue dots represent the test dataset.
Dashboard displays the following statistics:
For this task, I am using AlexNet architecture to classify images on 10 classes: Alaskan malamute, baboon, echidna, giant panda, hippo, king penguin, llama, otter, red panda, and wombat. Images are downloaded from the ImageNet. I will not go into details in this post, but you can explore file get_dataset.py. During training, the following parameters can be tweaked:
Script can be easily changed to add additional functionality.
I assume that you understand what is CNN and have basic knowledge of the following:
There are four main parts of the system: dataset, model, database, and dashboard/UI. These parts interact with each other to successfully run the system. Firstly I will describe each of these parts and after that, I will give a short description of how they interact with each other.
For this exercise, I use a dataset from the ImageNet that contains the following ten classes: Alaskan malamute, baboon, echidna, giant panda, hippo, king penguin, llama, otter, red panda, and wombat. To download all images from ImageNet, I can run python board.py from the following location: ../cnn_live_training.
Firstly, I have to find classes ids and save them to some variable:
The ImageNet stores URLs to images. Some URLs/images might not exist anymore. To get these URLs based on class id, I use the following function:
To download all images I use a loop where I download image by image. Below is the function to download image by URL:
The full version of the code can be seen in the file get_dataset.py. You can easily change these classes to other classes or you can even change the ImageNet to your custom dataset.
For the training, I am using by default the AlexNet architecture with Adam or SGD with Nesterov momentum optimizer. Optionally, the VGG16 can be chosen. Models can be imported either from the file models.py or from torchvision.models. The second option has the opportunity to use pre-trained weights. Dataset preparation happens in the file data_preparation.py. The training process happens in the file train.py.
I don’t have the goal to explain in this article how to build a pipeline for training CNN that is why I am not going into detail in this part. But I am happy to recommend the amazing course CS231n from Stanford and particularly HW2(Q4), where you can learn step by step how to build this pipeline. This homework can be found here.
Before running the system, we have to create dl_playground DB in PostgreSQL with the schema cnn_live_training that contains three following tables: parameters, statistics, activations.
parameters
This table contains only one row with current parameters for the training CNN model. When we change any parameters in our dashboard (file board.py), this data will be updated in the parameters SQL table. The table contains the following columns:
statistics
This table contains statistics of the training process. Data is updated every --n-print step. The table contains the following columns:
activations
This table contains the current distribution of weights in activation maps for all convolutional and fully connected layers. The table contains the following columns:
The dashboard consists of three main blocks: control panel, loss function & accuracy, and activation maps (distribution). These blocks are built using dash containers.
Control panel contains filters of parameters and “submit parameters” button that can be used to send chosen parameters to described above table “parameters”.
There are four filters: optimizer, learning rate, weight decay, and dropout.
Below is the script, how to create an optimizer filter (other filters are similar):
After that I create a container that contains all four filters:
How to create other parts of the control panel can be found in the file board.py.
Loss function & Accuracy contains a table with the history of used parameters and two plots with train/test loss function and accuracy values in time. Data is updated every one second (time interval can be changed) automatically.
Below is the script on how to create a table and button to stop training in the dashboard (I replaced real styles with short names for reading convenience):
Script to create plot template can be seen below:
Values are uploaded dynamically from PostgreSQL using callbacks (I provide only template for reading convenience):
I need to use a callback here because I want to update the plot and the table every 1 second. So, I have to use this variable as an input.
Activation maps (distribution) contains plots with distribution of activation map for each layer for the last step. Data is updated every one second (time interval can be changed) automatically.
The activations of the first two layers look similar to a normal distribution with the mean value in 0. The reason for this is for the first two layers we apply normalization. To understand more, I encourage you to watch a lecture from Stanford here.
Below is the script to create a container with the plots. It is similar to the previous container with loss function and accuracy plots:
The callback for the activation maps is similar to the “loss function & accuracy”:
It’s time to wrap everything up. To recall back, my goal is to train CNN live and being able to control this process by changing hyperparameters. So how does it happen? I have a dashboard where we can see the progress of the CNN training and where we have some filters that we can choose and activate by pushing the button “Submit parameters”.
What happens after that? All these parameters are sent to the table parameters in my database in PostgreSQL, using callback in the file board.py and function update_params:
At the same time, the script train.py connects to a database at the end of each training step, seeking to update the optimizer if parameters get updated:
Every n_step step data from training is saved to statistics and activations tables in database in PostgreSQL:
And this data simultaneously displayed in the dashboard because the script board.py every 1 sec. connects to the same tables:
All parameters are displayed in the table by extracting this information from the table :
If we want to stop training beforehand, we can push the button “Stop Training” below the table. After pushing the button, the callback will change the variable stop_train from False to True in the parameters table in my database:
At the same time, the script train.py check this parameter every training step and if it is True, training will be interrupted.
Without practical recommendations on what parameters to use to start training, this post will not be complete. If you want to see that everything works, but don’t have time for experiments, you can start from the following parameters:
If you want to see how the model explodes, just increase the learning rate to 0.01. Good luck with your experiments.
I will give a short description for Ubuntu, using a virtual environment (venv).
Run from the ../cnn_live_training command python get_dataset.py
Run from the ../cnn_live_training folder two following commands
python board.py
python train.py
In this story, I wanted to share my idea on how to nurture the feeling of training CNN. From one side, the idea is simple: build a training pipeline, create a dashboard and connect them using a database. But there are many annoying details that not possible to put in one small story. All script and additional details can be found in my git repository.
If this post makes someone interested and give additional knowledge, I will become slightly happier because it means that I reached my goal. I will appreciate any comments, constructive criticism, or questions, feel free to leave your feedback below or you can reach me via LinkedIn.
[1] L. Fei-Fei, R. Krishna and D. Xu, CS231n: Convolutional Neural Networks for Visual Recognition (2020), Stanford University
[2] A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks (2012), NeurIPS 2012
[3] A. Nagpal, L1 and L2 Regularization Methods (2017), Towards Data Science
<hr><p>Live CNN Training Dashboard: Hyperparameters Tuning was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>