state to disk. If it detects that the loss has been going up then instead of
saving to disk it recalls the previously good state.  This way, if we hit a
bad mini-batch during training which negatively effects the model in a
significant way, the dnn_trainer will automatically revert back to an earlier
good state.