Skip to content

D
dlib
Commit c20c11af authored by Davis King's avatar Davis King
Browse files
Options

Added test_one_step() to the dnn_trainer. This allows you to do automatic 

early stopping based on observing the loss on held out data.
parent 5f5684a8
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 165 additions and 0 deletions
dlib/dnn/trainer.h
View file @ c20c11af
This diff is collapsed.
dlib/dnn/trainer_abstract.h
View file @ c20c11af
......@@ -76,6 +76,7 @@ namespace dlib
- #get_learning_rate() == 1e-2
- #get_min_learning_rate() == 1e-5
- #get_iterations_without_progress_threshold() == 2000
- #get_test_iterations_without_progress_threshold() == 200
- #get_learning_rate_shrink_factor() == 0.1
- #get_learning_rate_schedule().size() == 0
- #get_train_one_step_calls() == 0
......@@ -536,6 +537,170 @@ namespace dlib
stopped touching the net.
!*/
// ----------------------
double get_average_test_loss (
) const;
/*!
ensures
- returns the average loss value observed during previous calls to
test_one_step().
- This function blocks until all threads inside the dnn_trainer have
stopped touching the net.
!*/
void test_one_step (
const std::vector<input_type>& data,
const std::vector<training_label_type>& labels
);
/*!
requires
- data.size() == labels.size()
- data.size() > 0
- net_type uses a supervised loss.
i.e. net_type::training_label_type != no_label_type.
ensures
- Runs the given data through the network and computes and records the loss.
- This call does not modify network parameters. The point of
test_one_step() is two fold, to allow you to observe the accuracy of the
network on hold out data during training, and to allow the trainer to
automatically adjust the learning rate when the test loss stops
improving. It should be noted that you are not required to use
test_one_step() at all, but if you want to do this kind of thing it is
available.
- You can observe the current average loss value by calling get_average_test_loss().
- The computation will happen in another thread. Therefore, after calling
this function you should call get_net() before you touch the net object
from the calling thread to ensure no other threads are still accessing
the network.
!*/
template <
typename data_iterator,
typename label_iterator
>
void test_one_step (
data_iterator dbegin,
data_iterator dend,
label_iterator lbegin
);
/*!
requires
- std::advance(lbegin, std::distance(dbegin, dend) - 1) is dereferencable
- std::distance(dbegin, dend) > 0
- net_type uses a supervised loss.
i.e. net_type::training_label_type != no_label_type.
ensures
- Runs the given data through the network and computes and records the loss.
- This call does not modify network parameters. The point of
test_one_step() is two fold, to allow you to observe the accuracy of the
network on hold out data during training, and to allow the trainer to
automatically adjust the learning rate when the test loss stops
improving. It should be noted that you are not required to use
test_one_step() at all, but if you want to do this kind of thing it is
available.
- You can observe the current average loss value by calling get_average_test_loss().
- The computation will happen in another thread. Therefore, after calling
this function you should call get_net() before you touch the net object
from the calling thread to ensure no other threads are still accessing
the network.
!*/
void test_one_step (
const std::vector<input_type>& data
);
/*!
requires
- data.size() > 0
- net_type uses an unsupervised loss.
i.e. net_type::training_label_type == no_label_type.
ensures
- Runs the given data through the network and computes and records the loss.
- This call does not modify network parameters. The point of
test_one_step() is two fold, to allow you to observe the accuracy of the
network on hold out data during training, and to allow the trainer to
automatically adjust the learning rate when the test loss stops
improving. It should be noted that you are not required to use
test_one_step() at all, but if you want to do this kind of thing it is
available.
- You can observe the current average loss value by calling get_average_test_loss().
- The computation will happen in another thread. Therefore, after calling
this function you should call get_net() before you touch the net object
from the calling thread to ensure no other threads are still accessing
the network.
!*/
template <
typename data_iterator
>
void test_one_step (
data_iterator dbegin,
data_iterator dend
);
/*!
requires
- std::distance(dbegin, dend) > 0
- net_type uses an unsupervised loss.
i.e. net_type::training_label_type == no_label_type.
ensures
- Runs the given data through the network and computes and records the loss.
- This call does not modify network parameters. The point of
test_one_step() is two fold, to allow you to observe the accuracy of the
network on hold out data during training, and to allow the trainer to
automatically adjust the learning rate when the test loss stops
improving. It should be noted that you are not required to use
test_one_step() at all, but if you want to do this kind of thing it is
available.
- You can observe the current average loss value by calling get_average_test_loss().
- The computation will happen in another thread. Therefore, after calling
this function you should call get_net() before you touch the net object
from the calling thread to ensure no other threads are still accessing
the network.
!*/
void set_test_iterations_without_progress_threshold (
unsigned long thresh
);
/*!
ensures
- #get_test_iterations_without_progress_threshold() == thresh
- #get_learning_rate_schedule().size() == 0
- This function blocks until all threads inside the dnn_trainer have
stopped touching the net.
!*/
unsigned long get_test_iterations_without_progress_threshold (
) const;
/*!
ensures
- This object monitors the progress of training and estimates if the
testing error is being reduced. It does this by looking at the previous
get_test_iterations_without_progress_threshold() mini-batch results from
test_one_step() and applying the statistical test defined by the
running_gradient object to see if the testing error is getting smaller.
If it isn't being reduced then get_learning_rate() is made smaller by a
factor of get_learning_rate_shrink_factor().
Therefore, get_test_iterations_without_progress_threshold() should always be
set to something sensibly large so that this test can be done with
reasonably high confidence. Think of this test as saying "if the testing loss
hasn't decreased for the previous get_test_iterations_without_progress_threshold()
calls to test_one_step() then shrink the learning rate".
!*/
unsigned long get_test_steps_without_progress (
) const;
/*!
ensures
- if (get_learning_rate_shrink_factor() != 1) then
- returns an estimate of how many mini-batches have executed without us
observing a statistically significant decrease in the testing error
(i.e. the error on the data given to the trainer via test_one_step()
calls).
- else
- returns 0
!*/
};
// ----------------------------------------------------------------------------------------
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment