View source for Dropout: A Simple Way to Prevent Neural Networks from Overfitting/paper

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = Machine Learning
 | difficulty  = Research
 | authors     = Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R.
 | year        = 2014
 | venue       = Journal of Machine Learning Research
 | source_url  = https://jmlr.org/papers/v15/srivastava14a.html
 | pdf_url     = http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf
}}
{{ContentMeta
 | generated_by   = claude-code-direct
 | model_used     = claude-opus-4-7
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
'''Dropout: A Simple Way to Prevent Neural Networks from Overfitting'''

<!--T:2-->
''Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov''

<!--T:3-->
Department of Computer Science, University of Toronto. Editor: Yoshua Bengio.

<!--T:4-->
''Journal of Machine Learning Research'' '''15''' (2014) 1929–1958. Submitted 11/13; Published 6/14.

== Abstract == <!--T:5-->

<!--T:6-->
Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.

<!--T:7-->
'''Keywords:''' neural networks, regularization, model combination, deep learning

== Introduction == <!--T:8-->

<!--T:9-->
Deep neural networks contain multiple non-linear hidden layers and this makes them very expressive models that can learn very complicated relationships between their inputs and outputs. With limited training data, however, many of these complicated relationships will be the result of sampling noise, so they will exist in the training set but not in real test data even if it is drawn from the same distribution. This leads to overfitting and many methods have been developed for reducing it. These include stopping the training as soon as performance on a validation set starts to get worse, introducing weight penalties of various kinds such as L1 and L2 regularization and soft weight sharing (Nowlan and Hinton, 1992).

<!--T:10-->
With unlimited computation, the best way to "regularize" a fixed-sized model is to average the predictions of all possible settings of the parameters, weighting each setting by its posterior probability given the training data. This can sometimes be approximated quite well for simple or small models (Xiong et al., 2011; Salakhutdinov and Mnih, 2008), but we would like to approach the performance of the Bayesian gold standard using considerably less computation. We propose to do this by approximating an equally weighted geometric mean of the predictions of an exponential number of learned models that share parameters.

<!--T:11-->
Model combination nearly always improves the performance of machine learning methods. With large neural networks, however, the obvious idea of averaging the outputs of many separately trained nets is prohibitively expensive. Combining several models is most helpful when the individual models are different from each other and in order to make neural net models different, they should either have different architectures or be trained on different data. Training many different architectures is hard because finding optimal hyperparameters for each architecture is a daunting task and training each large network requires a lot of computation. Moreover, large networks normally require large amounts of training data and there may not be enough data available to train different networks on different subsets of the data. Even if one was able to train many different large networks, using them all at test time is infeasible in applications where it is important to respond quickly.

<!--T:12-->
Dropout is a technique that addresses both these issues. It prevents overfitting and provides a way of approximately combining exponentially many different neural network architectures efficiently. The term "dropout" refers to dropping out units (hidden and visible) in a neural network. By dropping a unit out, we mean temporarily removing it from the network, along with all its incoming and outgoing connections. The choice of which units to drop is random. In the simplest case, each unit is retained with a fixed probability <math>p</math> independent of other units, where <math>p</math> can be chosen using a validation set or can simply be set at 0.5, which seems to be close to optimal for a wide range of networks and tasks. For the input units, however, the optimal probability of retention is usually closer to 1 than to 0.5.

<!--T:13-->
Applying dropout to a neural network amounts to sampling a "thinned" network from it. The thinned network consists of all the units that survived dropout. A neural net with <math>n</math> units can be seen as a collection of <math>2^n</math> possible thinned neural networks. These networks all share weights so that the total number of parameters is still <math>O(n^2)</math>, or less. For each presentation of each training case, a new thinned network is sampled and trained. So training a neural network with dropout can be seen as training a collection of <math>2^n</math> thinned networks with extensive weight sharing, where each thinned network gets trained very rarely, if at all.

<!--T:14-->
At test time, it is not feasible to explicitly average the predictions from exponentially many thinned models. However, a very simple approximate averaging method works well in practice. The idea is to use a single neural net at test time without dropout. The weights of this network are scaled-down versions of the trained weights. If a unit is retained with probability <math>p</math> during training, the outgoing weights of that unit are multiplied by <math>p</math> at test time. This ensures that for any hidden unit the expected output (under the distribution used to drop units at training time) is the same as the actual output at test time. By doing this scaling, <math>2^n</math> networks with shared weights can be combined into a single neural network to be used at test time. We found that training a network with dropout and using this approximate averaging method at test time leads to significantly lower generalization error on a wide variety of classification problems compared to training with other regularization methods.

<!--T:15-->
The idea of dropout is not limited to feed-forward neural nets. It can be more generally applied to graphical models such as Boltzmann Machines. In this paper, we introduce the dropout Restricted Boltzmann Machine model and compare it to standard Restricted Boltzmann Machines (RBM). Our experiments show that dropout RBMs are better than standard RBMs in certain respects.

== Motivation == <!--T:16-->

<!--T:17-->
A motivation for dropout comes from a theory of the role of sex in evolution (Livnat et al., 2010). Sexual reproduction involves taking half the genes of one parent and half of the other, adding a very small amount of random mutation, and combining them to produce an offspring. The asexual alternative is to create an offspring with a slightly mutated copy of the parent's genes. It seems plausible that asexual reproduction should be a better way to optimize individual fitness because a good set of genes that have come to work well together can be passed on directly to the offspring. On the other hand, sexual reproduction is likely to break up these co-adapted sets of genes, especially if these sets are large and, intuitively, this should decrease the fitness of organisms that have already evolved complicated co-adaptations. However, sexual reproduction is the way most advanced organisms evolved.

<!--T:18-->
One possible explanation for the superiority of sexual reproduction is that, over the long term, the criterion for natural selection may not be individual fitness but rather mix-ability of genes. The ability of a set of genes to be able to work well with another random set of genes makes them more robust. Since a gene cannot rely on a large set of partners to be present at all times, it must learn to do something useful on its own or in collaboration with a small number of other genes. According to this theory, the role of sexual reproduction is not just to allow useful new genes to spread throughout the population, but also to facilitate this process by reducing complex co-adaptations that would reduce the chance of a new gene improving the fitness of an individual. Similarly, each hidden unit in a neural network trained with dropout must learn to work with a randomly chosen sample of other units. This should make each hidden unit more robust and drive it towards creating useful features on its own without relying on other hidden units to correct its mistakes. However, the hidden units within a layer will still learn to do different things from each other.

<!--T:19-->
A closely related, but slightly different motivation for dropout comes from thinking about successful conspiracies. Ten conspiracies each involving five people is probably a better way to create havoc than one big conspiracy that requires fifty people to all play their parts correctly. If conditions do not change and there is plenty of time for rehearsal, a big conspiracy can work well, but with non-stationary conditions, the smaller the conspiracy the greater its chance of still working. Complex co-adaptations can be trained to work well on a training set, but on novel test data they are far more likely to fail than multiple simpler co-adaptations that achieve the same thing.

== Related Work == <!--T:20-->

<!--T:21-->
Dropout can be interpreted as a way of regularizing a neural network by adding noise to its hidden units. The idea of adding noise to the states of units has previously been used in the context of Denoising Autoencoders (DAEs) by Vincent et al. (2008, 2010) where noise is added to the input units of an autoencoder and the network is trained to reconstruct the noise-free input. Our work extends this idea by showing that dropout can be effectively applied in the hidden layers as well and that it can be interpreted as a form of model averaging. We also show that adding noise is not only useful for unsupervised feature learning but can also be extended to supervised learning problems. In fact, our method can be applied to other neuron-based architectures, for example, Boltzmann Machines. While 5% noise typically works best for DAEs, we found that our weight scaling procedure applied at test time enables us to use much higher noise levels. Dropping out 20% of the input units and 50% of the hidden units was often found to be optimal.

<!--T:22-->
Since dropout can be seen as a stochastic regularization technique, it is natural to consider its deterministic counterpart which is obtained by marginalizing out the noise. In this paper, we show that, in simple cases, dropout can be analytically marginalized out to obtain deterministic regularization methods. Recently, van der Maaten et al. (2013) also explored deterministic regularizers corresponding to different exponential-family noise distributions, including dropout (which they refer to as "blankout noise"). However, they apply noise to the inputs and only explore models with no hidden layers. Wang and Manning (2013) proposed a method for speeding up dropout by marginalizing dropout noise. Chen et al. (2012) explored marginalization in the context of denoising autoencoders.

<!--T:23-->
In dropout, we minimize the loss function stochastically under a noise distribution. This can be seen as minimizing an expected loss function. Previous work of Globerson and Roweis (2006); Dekel et al. (2010) explored an alternate setting where the loss is minimized when an adversary gets to pick which units to drop. Here, instead of a noise distribution, the maximum number of units that can be dropped is fixed. However, this work also does not explore models with hidden units.

== Model Description == <!--T:24-->

<!--T:25-->
This section describes the dropout neural network model. Consider a neural network with <math>L</math> hidden layers. Let <math>l \in \{1, \dots, L\}</math> index the hidden layers of the network. Let <math>z^{(l)}</math> denote the vector of inputs into layer <math>l</math>, <math>y^{(l)}</math> denote the vector of outputs from layer <math>l</math> (<math>y^{(0)} = x</math> is the input). <math>W^{(l)}</math> and <math>b^{(l)}</math> are the weights and biases at layer <math>l</math>. The feed-forward operation of a standard neural network can be described as (for <math>l \in \{0, \dots, L-1\}</math> and any hidden unit <math>i</math>):

<!--T:26-->
:<math>z^{(l+1)}_i = w^{(l+1)}_i\, y^l + b^{(l+1)}_i, \qquad y^{(l+1)}_i = f(z^{(l+1)}_i),</math>

<!--T:27-->
where <math>f</math> is any activation function, for example, <math>f(x) = 1/(1 + \exp(-x))</math>.

<!--T:28-->
With dropout, the feed-forward operation becomes:

<!--T:29-->
:<math>r^{(l)}_j \sim \mathrm{Bernoulli}(p), \qquad \tilde{y}^{(l)} = r^{(l)} \ast y^{(l)},</math>

<!--T:30-->
:<math>z^{(l+1)}_i = w^{(l+1)}_i\, \tilde{y}^l + b^{(l+1)}_i, \qquad y^{(l+1)}_i = f(z^{(l+1)}_i).</math>

<!--T:31-->
Here <math>\ast</math> denotes an element-wise product. For any layer <math>l</math>, <math>r^{(l)}</math> is a vector of independent Bernoulli random variables each of which has probability <math>p</math> of being 1. This vector is sampled and multiplied element-wise with the outputs of that layer, <math>y^{(l)}</math>, to create the thinned outputs <math>\tilde{y}^{(l)}</math>. The thinned outputs are then used as input to the next layer. This process is applied at each layer. This amounts to sampling a sub-network from a larger network. For learning, the derivatives of the loss function are backpropagated through the sub-network. At test time, the weights are scaled as <math>W^{(l)}_{\text{test}} = p\, W^{(l)}</math>. The resulting neural network is used without dropout.

== Learning Dropout Nets == <!--T:32-->

<!--T:33-->
This section describes a procedure for training dropout neural nets.

=== Backpropagation === <!--T:34-->

<!--T:35-->
Dropout neural networks can be trained using stochastic gradient descent in a manner similar to standard neural nets. The only difference is that for each training case in a mini-batch, we sample a thinned network by dropping out units. Forward and backpropagation for that training case are done only on this thinned network. The gradients for each parameter are averaged over the training cases in each mini-batch. Any training case which does not use a parameter contributes a gradient of zero for that parameter. Many methods have been used to improve stochastic gradient descent such as momentum, annealed learning rates and L2 weight decay. Those were found to be useful for dropout neural networks as well.

<!--T:36-->
One particular form of regularization was found to be especially useful for dropout—constraining the norm of the incoming weight vector at each hidden unit to be upper bounded by a fixed constant <math>c</math>. In other words, if <math>w</math> represents the vector of weights incident on any hidden unit, the neural network was optimized under the constraint <math>\|w\|_2 \leq c</math>. This constraint was imposed during optimization by projecting <math>w</math> onto the surface of a ball of radius <math>c</math>, whenever <math>w</math> went out of it. This is also called max-norm regularization since it implies that the maximum value that the norm of any weight can take is <math>c</math>. The constant <math>c</math> is a tunable hyperparameter, which is determined using a validation set. Max-norm regularization has been previously used in the context of collaborative filtering (Srebro and Shraibman, 2005). It typically improves the performance of stochastic gradient descent training of deep neural nets, even when no dropout is used.

<!--T:37-->
Although dropout alone gives significant improvements, using dropout along with max-norm regularization, large decaying learning rates and high momentum provides a significant boost over just using dropout. A possible justification is that constraining weight vectors to lie inside a ball of fixed radius makes it possible to use a huge learning rate without the possibility of weights blowing up. The noise provided by dropout then allows the optimization process to explore different regions of the weight space that would have otherwise been difficult to reach. As the learning rate decays, the optimization takes shorter steps, thereby doing less exploration and eventually settles into a minimum.

=== Unsupervised Pretraining === <!--T:38-->

<!--T:39-->
Neural networks can be pretrained using stacks of RBMs (Hinton and Salakhutdinov, 2006), autoencoders (Vincent et al., 2010) or Deep Boltzmann Machines (Salakhutdinov and Hinton, 2009). Pretraining is an effective way of making use of unlabeled data. Pretraining followed by finetuning with backpropagation has been shown to give significant performance boosts over finetuning from random initializations in certain cases.

<!--T:40-->
Dropout can be applied to finetune nets that have been pretrained using these techniques. The pretraining procedure stays the same. The weights obtained from pretraining should be scaled up by a factor of <math>1/p</math>. This makes sure that for each unit, the expected output from it under random dropout will be the same as the output during pretraining. We were initially concerned that the stochastic nature of dropout might wipe out the information in the pretrained weights. This did happen when the learning rates used during finetuning were comparable to the best learning rates for randomly initialized nets. However, when the learning rates were chosen to be smaller, the information in the pretrained weights seemed to be retained and we were able to get improvements in terms of the final generalization error compared to not using dropout when finetuning.

== Experimental Results == <!--T:41-->

<!--T:42-->
We trained dropout neural networks for classification problems on data sets in different domains. We found that dropout improved generalization performance on all data sets compared to neural networks that did not use dropout. The data sets are:

<!--T:43-->
* '''MNIST''': A standard toy data set of handwritten digits.
* '''TIMIT''': A standard speech benchmark for clean speech recognition.
* '''CIFAR-10 and CIFAR-100''': Tiny natural images (Krizhevsky, 2009).
* '''Street View House Numbers data set (SVHN)''': Images of house numbers collected by Google Street View (Netzer et al., 2011).
* '''ImageNet''': A large collection of natural images.
* '''Reuters-RCV1''': A collection of Reuters newswire articles.
* '''Alternative Splicing data set''': RNA features for predicting alternative gene splicing (Xiong et al., 2011).

<!--T:44-->
We chose a diverse set of data sets to demonstrate that dropout is a general technique for improving neural nets and is not specific to any particular application domain.

=== Results on Image Data Sets === <!--T:45-->

<!--T:46-->
We used five image data sets to evaluate dropout—MNIST, SVHN, CIFAR-10, CIFAR-100 and ImageNet. These data sets include different image types and training set sizes. Models which achieve state-of-the-art results on all of these data sets use dropout.

==== MNIST ==== <!--T:47-->

<!--T:48-->
The MNIST data set consists of 28 × 28 pixel handwritten digit images. The task is to classify the images into 10 digit classes. The best performing neural networks for the permutation invariant setting that do not use dropout or unsupervised pretraining achieve an error of about 1.60% (Simard et al., 2003). With dropout the error reduces to 1.35%. Replacing logistic units with rectified linear units (ReLUs) (Jarrett et al., 2009) further reduces the error to 1.25%. Adding max-norm regularization again reduces it to 1.06%. Increasing the size of the network leads to better results. A neural net with 2 layers and 8192 units per layer gets down to 0.95% error. Note that this network has more than 65 million parameters and is being trained on a data set of size 60,000. Training a network of this size to give good generalization error is very hard with standard regularization methods and early stopping. Dropout, on the other hand, prevents overfitting, even in this case. It does not even need early stopping. Goodfellow et al. (2013) showed that results can be further improved to 0.94% by replacing ReLU units with maxout units. All dropout nets use <math>p = 0.5</math> for hidden units and <math>p = 0.8</math> for input units.

<!--T:49-->
Dropout nets pretrained with stacks of RBMs and Deep Boltzmann Machines also give improvements. DBM-pretrained dropout nets achieve a test error of 0.79% which is the best performance ever reported for the permutation invariant setting.

<!--T:50-->
In order to test the robustness of dropout, classification experiments were done with networks of many different architectures keeping all hyperparameters, including <math>p</math>, fixed. The same architectures trained with and without dropout have drastically different test errors as seen as by the two separate clusters of trajectories. Dropout gives a huge improvement across all architectures, without using hyperparameters that were tuned specifically for each architecture.

==== Street View House Numbers ==== <!--T:51-->

<!--T:52-->
The Street View House Numbers (SVHN) Data Set (Netzer et al., 2011) consists of color images of house numbers collected by Google Street View. The part of the data set that we use in our experiments consists of 32 × 32 color images roughly centered on a digit in a house number. The task is to identify that digit.

<!--T:53-->
For this data set, we applied dropout to convolutional neural networks (LeCun et al., 1989). The best architecture that we found has three convolutional layers followed by 2 fully connected hidden layers. All hidden units were ReLUs. Each convolutional layer was followed by a max-pooling layer. Dropout was applied to all the layers of the network with the probability of retaining a hidden unit being <math>p = (0.9, 0.75, 0.75, 0.5, 0.5, 0.5)</math> for the different layers of the network (going from input to convolutional layers to fully connected layers). Max-norm regularization was used for weights in both convolutional and fully connected layers. We find that convolutional nets outperform other methods. The best performing convolutional nets that do not use dropout achieve an error rate of 3.95%. Adding dropout only to the fully connected layers reduces the error to 3.02%. Adding dropout to the convolutional layers as well further reduces the error to 2.55%. Even more gains can be obtained by using maxout units.

<!--T:54-->
The additional gain in performance obtained by adding dropout in the convolutional layers (3.02% to 2.55%) is worth noting. One may have presumed that since the convolutional layers don't have a lot of parameters, overfitting is not a problem and therefore dropout would not have much effect. However, dropout in the lower layers still helps because it provides noisy inputs for the higher fully connected layers which prevents them from overfitting.

==== CIFAR-10 and CIFAR-100 ==== <!--T:55-->

<!--T:56-->
The CIFAR-10 and CIFAR-100 data sets consist of 32 × 32 color images drawn from 10 and 100 categories respectively. Without any data augmentation, Snoek et al. (2012) used Bayesian hyperparameter optimization to obtained an error rate of 14.98% on CIFAR-10. Using dropout in the fully connected layers reduces that to 14.32% and adding dropout in every layer further reduces the error to 12.61%. Goodfellow et al. (2013) showed that the error is further reduced to 11.68% by replacing ReLU units with maxout units. On CIFAR-100, dropout reduces the error from 43.48% to 37.20% which is a huge improvement. No data augmentation was used for either data set (apart from the input dropout).

==== ImageNet ==== <!--T:57-->

<!--T:58-->
ImageNet is a data set of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. A subset of ImageNet with roughly 1000 images in each of 1000 categories is used in this challenge. Since the number of categories is rather large, it is conventional to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.

<!--T:59-->
ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so most of our experiments were performed on this data set. Convolutional nets with dropout outperform other methods by a large margin. The architecture and implementation details are described in detail in Krizhevsky et al. (2012). Our model based on convolutional nets and dropout won the ILSVRC-2012 competition. While the best methods based on standard vision features achieve a top-5 error rate of about 26%, convolutional nets with dropout achieve a test error of about 16% which is a staggering difference.

=== Results on TIMIT === <!--T:60-->

<!--T:61-->
Next, we applied dropout to a speech recognition task. We use the TIMIT data set which consists of recordings from 680 speakers covering 8 major dialects of American English reading ten phonetically-rich sentences in a controlled noise-free environment. Dropout neural networks were trained on windows of 21 log-filter bank frames to predict the label of the central frame. No speaker dependent operations were performed. A 6-layer net gives a phone error rate of 23.4%. Dropout further improves it to 21.8%. We also trained dropout nets starting from pretrained weights. A 4-layer net pretrained with a stack of RBMs gets a phone error rate of 22.7%. With dropout, this reduces to 19.7%. Similarly, for an 8-layer net the error reduces from 20.5% to 19.7%.

=== Results on a Text Data Set === <!--T:62-->

<!--T:63-->
To test the usefulness of dropout in the text domain, we used dropout networks to train a document classifier. We used a subset of the Reuters-RCV1 data set which is a collection of over 800,000 newswire articles from Reuters. These articles cover a variety of topics. The task is to take a bag of words representation of a document and classify it into 50 disjoint topics. Our best neural net which did not use dropout obtained an error rate of 31.05%. Adding dropout reduced the error to 29.62%. We found that the improvement was much smaller compared to that for the vision and speech data sets.

=== Comparison with Bayesian Neural Networks === <!--T:64-->

<!--T:65-->
Dropout can be seen as a way of doing an equally-weighted averaging of exponentially many models with shared weights. On the other hand, Bayesian neural networks (Neal, 1996) are the proper way of doing model averaging over the space of neural network structures and parameters. In dropout, each model is weighted equally, whereas in a Bayesian neural network each model is weighted taking into account the prior and how well the model fits the data, which is the more correct approach. Bayesian neural nets are extremely useful for solving problems in domains where data is scarce such as medical diagnosis, genetics, drug discovery and other computational biology applications. However, Bayesian neural nets are slow to train and difficult to scale to very large network sizes. Besides, it is expensive to get predictions from many large nets at test time. On the other hand, dropout neural nets are much faster to train and use at test time.

<!--T:66-->
The data set that we use (Xiong et al., 2011) comes from the domain of genetics. The task is to predict the occurrence of alternative splicing based on RNA features. Alternative splicing is a significant cause of cellular diversity in mammalian tissues. Predicting the occurrence of alternate splicing in certain tissues under different conditions is important for understanding many human diseases. Given the RNA features, the task is to predict the probability of three splicing related events that biologists care about. The evaluation metric is Code Quality which is a measure of the negative KL divergence between the target and the predicted probability distributions (higher is better). As expected, we found that Bayesian neural nets perform better than dropout. However, we see that dropout improves significantly upon the performance of standard neural nets and outperforms all other methods. The challenge in this data set is to prevent overfitting since the size of the training set is small. With dropout we were able to prevent overfitting without the need to do dimensionality reduction. The dropout nets are very large (1000s of hidden units) compared to a few tens of units in the Bayesian network. This shows that dropout has a strong regularizing effect.

=== Comparison with Standard Regularizers === <!--T:67-->

<!--T:68-->
Several regularization methods have been proposed for preventing overfitting in neural networks. These include L2 weight decay (more generally Tikhonov regularization (Tikhonov, 1943)), lasso (Tibshirani, 1996), KL-sparsity and max-norm regularization. Dropout can be seen as another way of regularizing neural networks. We compare dropout with some of these regularization methods using the MNIST data set. The same network architecture (784-1024-1024-2048-10) with ReLUs was trained using stochastic gradient descent with different regularizations. The values of different hyperparameters associated with each kind of regularization (decay constants, target sparsity, dropout rate, max-norm upper bound) were obtained using a validation set. We found that dropout combined with max-norm regularization gives the lowest generalization error.

== Salient Features == <!--T:69-->

<!--T:70-->
The experiments described in the previous section provide strong evidence that dropout is a useful technique for improving neural networks. In this section, we closely examine how dropout affects a neural network. We analyze the effect of dropout on the quality of features produced. We see how dropout affects the sparsity of hidden unit activations. We also see how the advantages obtained from dropout vary with the probability of retaining units, size of the network and the size of the training set. These observations give some insight into why dropout works so well.

=== Effect on Features === <!--T:71-->

<!--T:72-->
In a standard neural network, the derivative received by each parameter tells it how it should change so the final loss function is reduced, given what all other units are doing. Therefore, units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This in turn leads to overfitting because these co-adaptations do not generalize to unseen data. We hypothesize that for each hidden unit, dropout prevents co-adaptation by making the presence of other hidden units unreliable. Therefore, a hidden unit cannot rely on other specific units to correct its mistakes. It must perform well in a wide variety of different contexts provided by the other hidden units. To observe this effect directly, we look at the first level features learned by neural networks trained on visual tasks with and without dropout.

<!--T:73-->
Features learned by an autoencoder on MNIST with a single hidden layer of 256 rectified linear units without dropout exhibit co-adaptation: each hidden unit on its own does not seem to be detecting a meaningful feature. With dropout in the hidden layer with <math>p = 0.5</math>, the hidden units seem to detect edges, strokes and spots in different parts of the image. This shows that dropout does break up co-adaptations, which is probably the main reason why it leads to lower generalization errors.

=== Effect on Sparsity === <!--T:74-->

<!--T:75-->
We found that as a side-effect of doing dropout, the activations of the hidden units become sparse, even when no sparsity inducing regularizers are present. Thus, dropout automatically leads to sparse representations. To observe this effect, we take the autoencoders trained in the previous section and look at the sparsity of hidden unit activations on a random mini-batch taken from the test set. In a good sparse model, there should only be a few highly activated units for any data case. Moreover, the average activation of any unit across data cases should be low. Comparing the histograms of activations we can see that fewer hidden units have high activations in the dropout net compared to the non-dropout net. The mean activations are also smaller for the dropout net. The overall mean activation of hidden units is close to 2.0 for the autoencoder without dropout but drops to around 0.7 when dropout is used.

=== Effect of Dropout Rate === <!--T:76-->

<!--T:77-->
Dropout has a tunable hyperparameter <math>p</math> (the probability of retaining a unit in the network). We explore the effect of varying this hyperparameter. The comparison is done in two situations:

<!--T:78-->
# The number of hidden units is held constant.
# The number of hidden units is changed so that the expected number of hidden units that will be retained after dropout is held constant.

<!--T:79-->
In the first case, we train the same network architecture with different amounts of dropout. We use a 784-2048-2048-2048-10 architecture. No input dropout was used. If the architecture is held constant, having a small <math>p</math> means very few units will turn on during training. It can be seen that this has led to underfitting since the training error is also high. As <math>p</math> increases, the error goes down. It becomes flat when <math>0.4 \leq p \leq 0.8</math> and then increases as <math>p</math> becomes close to 1.

<!--T:80-->
Another interesting setting is the second case in which the quantity <math>pn</math> is held constant where <math>n</math> is the number of hidden units in any particular layer. This means that networks that have small <math>p</math> will have a large number of hidden units. Therefore, after applying dropout, the expected number of units that are present will be the same across different architectures. However, the test networks will be of different sizes. In our experiments, we set <math>pn = 256</math> for the first two hidden layers and <math>pn = 512</math> for the last hidden layer. The magnitude of errors for small values of <math>p</math> has reduced by a lot compared to the previous setting (for <math>p = 0.1</math> it fell from 2.7% to 1.7%). Values of <math>p</math> that are close to 0.6 seem to perform best for this choice of <math>pn</math> but our usual default value of 0.5 is close to optimal.

=== Effect of Data Set Size === <!--T:81-->

<!--T:82-->
One test of a good regularizer is that it should make it possible to get good generalization error from models with a large number of parameters trained on small data sets. This section explores the effect of changing the data set size when dropout is used with feed-forward networks. Huge neural networks trained in the standard way overfit massively on small data sets. To see if dropout can help, we run classification experiments on MNIST and vary the amount of data given to the network. The network was given data sets of size 100, 500, 1K, 5K, 10K and 50K chosen randomly from the MNIST training set. The same network architecture (784-1024-1024-2048-10) was used for all data sets. Dropout with <math>p = 0.5</math> was performed at all the hidden layers and <math>p = 0.8</math> at the input layer. It can be observed that for extremely small data sets (100, 500) dropout does not give any improvements. The model has enough parameters that it can overfit on the training data, even with all the noise coming from dropout. As the size of the data set is increased, the gain from doing dropout increases up to a point and then declines. This suggests that for any given architecture and dropout rate, there is a "sweet spot" corresponding to some amount of data that is large enough to not be memorized in spite of the noise but not so large that overfitting is not a problem anyways.

=== Monte-Carlo Model Averaging vs. Weight Scaling === <!--T:83-->

<!--T:84-->
The efficient test time procedure that we propose is to do an approximate model combination by scaling down the weights of the trained neural network. An expensive but more correct way of averaging the models is to sample <math>k</math> neural nets using dropout for each test case and average their predictions. As <math>k \to \infty</math>, this Monte-Carlo model average gets close to the true model average. It is interesting to see empirically how many samples <math>k</math> are needed to match the performance of the approximate averaging method.

<!--T:85-->
We again use the MNIST data set and do classification by averaging the predictions of <math>k</math> randomly sampled neural networks. It can be seen that around <math>k = 50</math>, the Monte-Carlo method becomes as good as the approximate method. Thereafter, the Monte-Carlo method is slightly better than the approximate method but well within one standard deviation of it. This suggests that the weight scaling method is a fairly good approximation of the true model average.

== Dropout Restricted Boltzmann Machines == <!--T:86-->

<!--T:87-->
Besides feed-forward neural networks, dropout can also be applied to Restricted Boltzmann Machines (RBM). In this section, we formally describe this model and show some results to illustrate its key properties.

=== Model Description === <!--T:88-->

<!--T:89-->
Consider an RBM with visible units <math>v \in \{0, 1\}^D</math> and hidden units <math>h \in \{0, 1\}^F</math>. It defines the following probability distribution:

<!--T:90-->
:<math>P(h, v;\, \theta) = \frac{1}{Z(\theta)} \exp(v^\top W h + a^\top h + b^\top v).</math>

<!--T:91-->
Where <math>\theta = \{W, a, b\}</math> represents the model parameters and <math>Z</math> is the partition function. Dropout RBMs are RBMs augmented with a vector of binary random variables <math>r \in \{0, 1\}^F</math>. Each random variable <math>r_j</math> takes the value 1 with probability <math>p</math>, independent of others. If <math>r_j</math> takes the value 1, the hidden unit <math>h_j</math> is retained, otherwise it is dropped from the model. The joint distribution defined by a Dropout RBM can be expressed as:

<!--T:92-->
:<math>P(r, h, v;\, p, \theta) = P(r;\, p)\, P(h, v\mid r;\, \theta),</math>

<!--T:93-->
:<math>P(r;\, p) = \prod_{j=1}^{F} p^{r_j} (1 - p)^{1 - r_j},</math>

<!--T:94-->
:<math>P(h, v\mid r;\, \theta) = \frac{1}{Z'(\theta, r)} \exp(v^\top W h + a^\top h + b^\top v) \prod_{j=1}^{F} g(h_j, r_j),</math>

<!--T:95-->
:<math>g(h_j, r_j) = \mathbf{1}(r_j = 1) + \mathbf{1}(r_j = 0)\, \mathbf{1}(h_j = 0).</math>

<!--T:96-->
<math>Z'(\theta, r)</math> is the normalization constant. <math>g(h_j, r_j)</math> imposes the constraint that if <math>r_j = 0</math>, <math>h_j</math> must be 0. The distribution over <math>h</math>, conditioned on <math>v</math> and <math>r</math> is factorial:

<!--T:97-->
:<math>P(h\mid r, v) = \prod_{j=1}^{F} P(h_j\mid r_j, v),</math>

<!--T:98-->
:<math>P(h_j = 1\mid r_j, v) = \mathbf{1}(r_j = 1)\, \sigma\!\left(b_j + \sum_i W_{ij} v_i\right).</math>

<!--T:99-->
The distribution over <math>v</math> conditioned on <math>h</math> is same as that of an RBM:

<!--T:100-->
:<math>P(v\mid h) = \prod_{i=1}^{D} P(v_i\mid h), \qquad P(v_i = 1\mid h) = \sigma\!\left(a_i + \sum_j W_{ij} h_j\right).</math>

<!--T:101-->
Conditioned on <math>r</math>, the distribution over <math>\{v, h\}</math> is same as the distribution that an RBM would impose, except that the units for which <math>r_j = 0</math> are dropped from <math>h</math>. Therefore, the Dropout RBM model can be seen as a mixture of exponentially many RBMs with shared weights each using a different subset of <math>h</math>.

=== Learning Dropout RBMs === <!--T:102-->

<!--T:103-->
Learning algorithms developed for RBMs such as Contrastive Divergence (Hinton et al., 2006) can be directly applied for learning Dropout RBMs. The only difference is that <math>r</math> is first sampled and only the hidden units that are retained are used for training. Similar to dropout neural networks, a different <math>r</math> is sampled for each training case in every minibatch. In our experiments, we use CD-1 for training dropout RBMs.

=== Effect on Features === <!--T:104-->

<!--T:105-->
Dropout in feed-forward networks improved the quality of features by reducing co-adaptations. This section explores whether this effect transfers to Dropout RBMs as well. Features learned by a binary RBM with 256 hidden units differ qualitatively from those learned by a dropout RBM with the same number of hidden units. Features learned by the dropout RBM appear qualitatively different in the sense that they seem to capture features that are coarser compared to the sharply defined stroke-like features in the standard RBM. There seem to be very few dead units in the dropout RBM relative to the standard RBM.

=== Effect on Sparsity === <!--T:106-->

<!--T:107-->
Next, we investigate the effect of dropout RBM training on sparsity of the hidden unit activations. The histograms of hidden unit activations and their means on a test mini-batch after training an RBM versus a dropout RBM clearly indicate that the dropout RBMs learn much sparser representations than standard RBMs even when no additional sparsity inducing regularizer is present.

== Marginalizing Dropout == <!--T:108-->

<!--T:109-->
Dropout can be seen as a way of adding noise to the states of hidden units in a neural network. In this section, we explore the class of models that arise as a result of marginalizing this noise. These models can be seen as deterministic versions of dropout. In contrast to standard ("Monte-Carlo") dropout, these models do not need random bits and it is possible to get gradients for the marginalized loss functions. Deterministic algorithms have been proposed that try to learn models that are robust to feature deletion at test time (Globerson and Roweis, 2006). Marginalization in the context of denoising autoencoders has been explored previously (Chen et al., 2012). The marginalization of dropout noise in the context of linear regression was discussed in Srivastava (2013). Wang and Manning (2013) further explored the idea of marginalizing dropout to speed-up training. van der Maaten et al. (2013) investigated different input noise distributions and the regularizers obtained by marginalizing this noise. Wager et al. (2013) describes how dropout can be seen as an adaptive regularizer.

=== Linear Regression === <!--T:110-->

<!--T:111-->
First we explore a very simple case of applying dropout to the classical problem of linear regression. Let <math>X \in \mathbb{R}^{N \times D}</math> be a data matrix of <math>N</math> data points. <math>y \in \mathbb{R}^N</math> be a vector of targets. Linear regression tries to find a <math>w \in \mathbb{R}^D</math> that minimizes <math>\|y - Xw\|^2</math>. When the input <math>X</math> is dropped out such that any input dimension is retained with probability <math>p</math>, the input can be expressed as <math>R \ast X</math> where <math>R \in \{0, 1\}^{N \times D}</math> is a random matrix with <math>R_{ij} \sim \mathrm{Bernoulli}(p)</math> and <math>\ast</math> denotes an element-wise product. Marginalizing the noise, the objective function becomes:

<!--T:112-->
:<math>\underset{w}{\mathrm{minimize}}\; \mathbb{E}_{R \sim \mathrm{Bernoulli}(p)}\!\left[\, \|y - (R \ast X) w\|^2 \,\right].</math>

<!--T:113-->
This reduces to:

<!--T:114-->
:<math>\underset{w}{\mathrm{minimize}}\; \|y - p X w\|^2 + p(1 - p)\, \|\Gamma w\|^2,</math>

<!--T:115-->
where <math>\Gamma = (\mathrm{diag}(X^\top X))^{1/2}</math>. Therefore, dropout with linear regression is equivalent, in expectation, to ridge regression with a particular form for <math>\Gamma</math>. This form of <math>\Gamma</math> essentially scales the weight cost for weight <math>w_i</math> by the standard deviation of the <math>i</math>th dimension of the data. If a particular data dimension varies a lot, the regularizer tries to squeeze its weight more.

<!--T:116-->
Another interesting way to look at this objective is to absorb the factor of <math>p</math> into <math>w</math>. This leads to the form <math>\|y - X \tilde{w}\|^2 + \frac{1 - p}{p}\, \|\Gamma \tilde{w}\|^2</math>, where <math>\tilde{w} = pw</math>. This makes the dependence of the regularization constant on <math>p</math> explicit. For <math>p</math> close to 1, all the inputs are retained and the regularization constant is small. As more dropout is done (by decreasing <math>p</math>), the regularization constant grows larger.

=== Logistic Regression and Deep Networks === <!--T:117-->

<!--T:118-->
For logistic regression and deep neural nets, it is hard to obtain a closed form marginalized model. However, Wang and Manning (2013) showed that in the context of dropout applied to logistic regression, the corresponding marginalized model can be trained approximately. Under reasonable assumptions, the distributions over the inputs to the logistic unit and over the gradients of the marginalized model are Gaussian. Their means and variances can be computed efficiently. This approximate marginalization outperforms Monte-Carlo dropout in terms of training time and generalization performance. However, the assumptions involved in this technique become successively weaker as more layers are added. Therefore, the results are not directly applicable to deep networks.

== Multiplicative Gaussian Noise == <!--T:119-->

<!--T:120-->
Dropout involves multiplying hidden activations by Bernoulli distributed random variables which take the value 1 with probability <math>p</math> and 0 otherwise. This idea can be generalized by multiplying the activations with random variables drawn from other distributions. We recently discovered that multiplying by a random variable drawn from <math>\mathcal{N}(1, 1)</math> works just as well, or perhaps better than using Bernoulli noise. This new form of dropout amounts to adding a Gaussian distributed random variable with zero mean and standard deviation equal to the activation of the unit. That is, each hidden activation <math>h_i</math> is perturbed to <math>h_i + h_i r</math> where <math>r \sim \mathcal{N}(0, 1)</math>, or equivalently <math>h_i r'</math> where <math>r' \sim \mathcal{N}(1, 1)</math>. We can generalize this to <math>r' \sim \mathcal{N}(1, \sigma^2)</math> where <math>\sigma</math> becomes an additional hyperparameter to tune, just like <math>p</math> was in the standard (Bernoulli) dropout. The expected value of the activations remains unchanged, therefore no weight scaling is required at test time.

<!--T:121-->
In this paper, we described dropout as a method where we retain units with probability <math>p</math> at training time and scale down the weights by multiplying them by a factor of <math>p</math> at test time. Another way to achieve the same effect is to scale up the retained activations by multiplying by <math>1/p</math> at training time and not modifying the weights at test time. These methods are equivalent with appropriate scaling of the learning rate and weight initializations at each layer.

<!--T:122-->
Therefore, dropout can be seen as multiplying <math>h_i</math> by a Bernoulli random variable <math>r_b</math> that takes the value <math>1/p</math> with probability <math>p</math> and 0 otherwise. <math>\mathbb{E}[r_b] = 1</math> and <math>\mathrm{Var}[r_b] = (1 - p)/p</math>. For the Gaussian multiplicative noise, if we set <math>\sigma^2 = (1 - p)/p</math>, we end up multiplying <math>h_i</math> by a random variable <math>r_g</math>, where <math>\mathbb{E}[r_g] = 1</math> and <math>\mathrm{Var}[r_g] = (1 - p)/p</math>. Therefore, both forms of dropout can be set up so that the random variable being multiplied by has the same mean and variance. However, given these first and second order moments, <math>r_g</math> has the highest entropy and <math>r_b</math> has the lowest. Both these extremes work well, although preliminary experimental results suggest that the high entropy case might work slightly better. For each layer, the value of <math>\sigma</math> in the Gaussian model was set to be <math>\sqrt{(1 - p)/p}</math> using the <math>p</math> from the corresponding layer in the Bernoulli model.

== Conclusion == <!--T:123-->

<!--T:124-->
Dropout is a technique for improving neural networks by reducing overfitting. Standard backpropagation learning builds up brittle co-adaptations that work for the training data but do not generalize to unseen data. Random dropout breaks up these co-adaptations by making the presence of any particular hidden unit unreliable. This technique was found to improve the performance of neural nets in a wide variety of application domains including object classification, digit recognition, speech recognition, document classification and analysis of computational biology data. This suggests that dropout is a general technique and is not specific to any domain. Methods that use dropout achieve state-of-the-art results on SVHN, ImageNet, CIFAR-100 and MNIST. Dropout considerably improved the performance of standard neural nets on other data sets as well.

<!--T:125-->
This idea can be extended to Restricted Boltzmann Machines and other graphical models. The central idea of dropout is to take a large model that overfits easily and repeatedly sample and train smaller sub-models from it. RBMs easily fit into this framework. We developed Dropout RBMs and empirically showed that they have certain desirable properties.

<!--T:126-->
One of the drawbacks of dropout is that it increases training time. A dropout network typically takes 2–3 times longer to train than a standard neural network of the same architecture. A major cause of this increase is that the parameter updates are very noisy. Each training case effectively tries to train a different random architecture. Therefore, the gradients that are being computed are not gradients of the final architecture that will be used at test time. Therefore, it is not surprising that training takes a long time. However, it is likely that this stochasticity prevents overfitting. This creates a trade-off between overfitting and training time. With more training time, one can use high dropout and suffer less overfitting. However, one way to obtain some of the benefits of dropout without stochasticity is to marginalize the noise to obtain a regularizer that does the same thing as the dropout procedure, in expectation. We showed that for linear regression this regularizer is a modified form of L2 regularization. For more complicated models, it is not obvious how to obtain an equivalent regularizer. Speeding up dropout is an interesting direction for future work.

== Acknowledgments == <!--T:127-->

<!--T:128-->
This research was supported by OGS, NSERC and an Early Researcher Award.

== Appendix A. A Practical Guide for Training Dropout Networks == <!--T:129-->

<!--T:130-->
Neural networks are infamous for requiring extensive hyperparameter tuning. Dropout networks are no exception. In this section, we describe heuristics that might be useful for applying dropout.

=== Network Size === <!--T:131-->

<!--T:132-->
It is to be expected that dropping units will reduce the capacity of a neural network. If <math>n</math> is the number of hidden units in any layer and <math>p</math> is the probability of retaining a unit, then instead of <math>n</math> hidden units, only <math>pn</math> units will be present after dropout, in expectation. Moreover, this set of <math>pn</math> units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an <math>n</math>-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least <math>n/p</math> units. We found this to be a useful heuristic for setting the number of hidden units in both convolutional and fully connected networks.

=== Learning Rate and Momentum === <!--T:133-->

<!--T:134-->
Dropout introduces a significant amount of noise in the gradients compared to standard stochastic gradient descent. Therefore, a lot of gradients tend to cancel each other. In order to make up for this, a dropout net should typically use 10–100 times the learning rate that was optimal for a standard neural net. Another way to reduce the effect the noise is to use a high momentum. While momentum values of 0.9 are common for standard nets, with dropout we found that values around 0.95 to 0.99 work quite a lot better. Using high learning rate and/or momentum significantly speed up learning.

=== Max-norm Regularization === <!--T:135-->

<!--T:136-->
Though large momentum and learning rate speed up learning, they sometimes cause the network weights to grow very large. To prevent this, we can use max-norm regularization. This constrains the norm of the vector of incoming weights at each hidden unit to be bound by a constant <math>c</math>. Typical values of <math>c</math> range from 3 to 4.

=== Dropout Rate === <!--T:137-->

<!--T:138-->
Dropout introduces an extra hyperparameter—the probability of retaining a unit <math>p</math>. This hyperparameter controls the intensity of dropout. <math>p = 1</math> implies no dropout and low values of <math>p</math> mean more dropout. Typical values of <math>p</math> for hidden units are in the range 0.5 to 0.8. For input layers, the choice depends on the kind of input. For real-valued inputs (image patches or speech frames), a typical value is 0.8. For hidden layers, the choice of <math>p</math> is coupled with the choice of number of hidden units <math>n</math>. Smaller <math>p</math> requires big <math>n</math> which slows down the training and leads to underfitting. Large <math>p</math> may not produce enough dropout to prevent overfitting.

== Appendix B. Detailed Description of Experiments and Data Sets == <!--T:139-->

<!--T:140-->
This section describes the network architectures and training details for the experimental results reported in this paper. The code for reproducing these results can be obtained from <code>http://www.cs.toronto.edu/~nitish/dropout</code>. The implementation is GPU-based. We used the excellent CUDA libraries—cudamat (Mnih, 2009) and cuda-convnet (Krizhevsky et al., 2012) to implement our networks.

=== MNIST === <!--T:141-->

<!--T:142-->
The MNIST data set consists of 60,000 training and 10,000 test examples each representing a 28×28 digit image. We held out 10,000 random training images for validation. Hyperparameters were tuned on the validation set such that the best validation error was produced after 1 million weight updates. The validation set was then combined with the training set and training was done for 1 million weight updates. This net was used to evaluate the performance on the test set. This way of using the validation set was chosen because we found that it was easy to set up hyperparameters so that early stopping was not required at all. Therefore, once the hyperparameters were fixed, it made sense to combine the validation and training sets and train for a very long time.

<!--T:143-->
The architectures include all combinations of 2, 3, and 4 layer networks with 1024 and 2048 units in each layer. Thus, there are six architectures in all. For all the architectures, we used <math>p = 0.5</math> in all hidden layers and <math>p = 0.8</math> in the input layer. A final momentum of 0.95 and weight constraints with <math>c = 2</math> was used in all the layers. To test the limits of dropout's regularization power, we also experimented with 2 and 3 layer nets having 4096 and 8192 units. 2 layer nets gave improvements. However, the three layer nets performed slightly worse than 2 layer ones with the same level of dropout. When we increased dropout, performance improved but not enough to outperform the 2 layer nets.

=== SVHN === <!--T:144-->

<!--T:145-->
The SVHN data set consists of approximately 600,000 training images and 26,000 test images. The training set consists of two parts—a standard labeled training set and another set of labeled examples that are easy. A validation set was constructed by taking examples from both the parts. Two-thirds of it were taken from the standard set (400 per class) and one-third from the extra set (200 per class), a total of 6000 samples. This same process is used by Sermanet et al. (2012). The inputs were RGB pixels normalized to have zero mean and unit variance. Other preprocessing techniques such as global or local contrast normalization or ZCA whitening did not give any noticeable improvements.

<!--T:146-->
The best architecture that we found uses three convolutional layers each followed by a max-pooling layer. The convolutional layers have 96, 128 and 256 filters respectively. Each convolutional layer has a 5 × 5 receptive field applied with a stride of 1 pixel. Each max pooling layer pools 3 × 3 regions at strides of 2 pixels. The convolutional layers are followed by two fully connected hidden layers having 2048 units each. All units use the rectified linear activation function. Dropout was applied to all the layers of the network with the probability of retaining the unit being <math>p = (0.9, 0.75, 0.75, 0.5, 0.5, 0.5)</math> for the different layers of the network (going from input to convolutional layers to fully connected layers). In addition, the max-norm constraint with <math>c = 4</math> was used for all the weights. A momentum of 0.95 was used in all the layers. These hyperparameters were tuned using a validation set. Since the training set was quite large, we did not combine the validation set with the training set for final training. We reported test error of the model that had smallest validation error.

=== CIFAR-10 and CIFAR-100 === <!--T:147-->

<!--T:148-->
The CIFAR-10 and CIFAR-100 data sets consist of 50,000 training and 10,000 test images each. They have 10 and 100 image categories respectively. These are 32 × 32 color images. We used 5,000 of the training images for validation. We followed the procedure similar to MNIST, where we found the best hyperparameters using the validation set and then combined it with the training set. The images were preprocessed by doing global contrast normalization in each color channel followed by ZCA whitening. Global contrast normalization means that for image and each color channel in that image, we compute the mean of the pixel intensities and subtract it from the channel. ZCA whitening means that we mean center the data, rotate it onto its principle components, normalize each component and then rotate it back. The network architecture and dropout rates are same as that for SVHN, except the learning rates for the input layer which had to be set to smaller values.

=== TIMIT === <!--T:149-->

<!--T:150-->
The open source Kaldi toolkit (Povey et al., 2011) was used to preprocess the data into log-filter banks. A monophone system was trained to do a forced alignment and to get labels for speech frames. Dropout neural networks were trained on windows of 21 consecutive frames to predict the label of the central frame. No speaker dependent operations were performed. The inputs were mean centered and normalized to have unit variance. We used probability of retention <math>p = 0.8</math> in the input layers and 0.5 in the hidden layers. Max-norm constraint with <math>c = 4</math> was used in all the layers. A momentum of 0.95 with a high learning rate of 0.1 was used. The learning rate was decayed as <math>\epsilon_0 (1 + t/T)^{-1}</math>. For DBN pretraining, we trained RBMs using CD-1. The variance of each input unit for the Gaussian RBM was fixed to 1. For finetuning the DBN with dropout, we found that in order to get the best results it was important to use a smaller learning rate (about 0.01). Adding max-norm constraints did not give any improvements.

=== Reuters === <!--T:151-->

<!--T:152-->
The Reuters RCV1 corpus contains more than 800,000 documents categorized into 103 classes. These classes are arranged in a tree hierarchy. We created a subset of this data set consisting of 402,738 articles and a vocabulary of 2000 words comprising of 50 categories in which each document belongs to exactly one class. The data was split into equal sized training and test sets. We tried many network architectures and found that dropout gave improvements in classification accuracy over all of them. However, the improvement was not as significant as that for the image and speech data sets. This might be explained by the fact that this data set is quite big (more than 200,000 training examples) and overfitting is not a very serious problem.

=== Alternative Splicing === <!--T:153-->

<!--T:154-->
The alternative splicing data set consists of data for 3665 cassette exons, 1014 RNA features and 4 tissue types derived from 27 mouse tissues. For each input, the target consists of 4 softmax units (one for tissue type). Each softmax unit has 3 states (inc, exc, nc) which are of the biological importance. For each softmax unit, the aim is to predict a distribution over these 3 states that matches the observed distribution from wet lab experiments as closely as possible. The evaluation metric is Code Quality which is defined as:

<!--T:155-->
:<math>\sum_{i=1}^{|\text{data points}|}\; \sum_{t \in \text{tissue types}}\; \sum_{s \in \{\text{inc, exc, nc}\}} p^{s}_{i,t}\, \log\!\left(\frac{q^{s}_{t}(r_i)}{\bar{p}^{s}}\right),</math>

<!--T:156-->
where <math>p^{s}_{i,t}</math> is the target probability for state <math>s</math> and tissue type <math>t</math> in input <math>i</math>; <math>q^{s}_{t}(r_i)</math> is the predicted probability for state <math>s</math> in tissue type <math>t</math> for input <math>r_i</math> and <math>\bar{p}^{s}</math> is the average of <math>p^{s}_{i,t}</math> over <math>i</math> and <math>t</math>. A two layer dropout network with 1024 units in each layer was trained on this data set. A value of <math>p = 0.5</math> was used for the hidden layer and <math>p = 0.7</math> for the input layer. Max-norm regularization with high decaying learning rates was used. Results were averaged across the same 5 folds used by Xiong et al. (2011).

== References == <!--T:157-->

<!--T:158-->
* M. Chen, Z. Xu, K. Weinberger, and F. Sha. Marginalized denoising autoencoders for domain adaptation. In ''Proceedings of the 29th International Conference on Machine Learning'', pages 767–774. ACM, 2012.
* G. E. Dahl, M. Ranzato, A. Mohamed, and G. E. Hinton. Phone recognition with the mean-covariance restricted Boltzmann machine. In ''Advances in Neural Information Processing Systems 23'', pages 469–477, 2010.
* O. Dekel, O. Shamir, and L. Xiao. Learning to classify with missing and corrupted features. ''Machine Learning'', 81(2):149–178, 2010.
* A. Globerson and S. Roweis. Nightmare at test time: robust learning by feature deletion. In ''Proceedings of the 23rd International Conference on Machine Learning'', pages 353–360. ACM, 2006.
* I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ''Proceedings of the 30th International Conference on Machine Learning'', pages 1319–1327. ACM, 2013.
* G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. ''Science'', 313(5786):504–507, 2006.
* G. E. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for deep belief nets. ''Neural Computation'', 18:1527–1554, 2006.
* K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In ''Proceedings of the International Conference on Computer Vision (ICCV'09)''. IEEE, 2009.
* A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
* A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In ''Advances in Neural Information Processing Systems 25'', pages 1106–1114, 2012.
* Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. ''Neural Computation'', 1(4):541–551, 1989.
* Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, Z. Li, M.-H. Tsai, X. Zhou, T. Huang, and T. Zhang. Imagenet classification: fast descriptor coding and large-scale SVM training. ''Large scale visual recognition challenge'', 2010.
* A. Livnat, C. Papadimitriou, N. Pippenger, and M. W. Feldman. Sex, mixability, and modularity. ''Proceedings of the National Academy of Sciences'', 107(4):1452–1457, 2010.
* V. Mnih. CUDAMat: a CUDA-based matrix class for Python. Technical Report UTML TR 2009-004, Department of Computer Science, University of Toronto, November 2009.
* A. Mohamed, G. E. Dahl, and G. E. Hinton. Acoustic modeling using deep belief networks. ''IEEE Transactions on Audio, Speech, and Language Processing'', 2010.
* R. M. Neal. ''Bayesian Learning for Neural Networks''. Springer-Verlag New York, Inc., 1996.
* Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In ''NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011'', 2011.
* S. J. Nowlan and G. E. Hinton. Simplifying neural networks by soft weight-sharing. ''Neural Computation'', 4(4), 1992.
* D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely. The Kaldi Speech Recognition Toolkit. In ''IEEE 2011 Workshop on Automatic Speech Recognition and Understanding''. IEEE Signal Processing Society, 2011.
* R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. In ''Proceedings of the International Conference on Artificial Intelligence and Statistics'', volume 5, pages 448–455, 2009.
* R. Salakhutdinov and A. Mnih. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In ''Proceedings of the 25th International Conference on Machine Learning''. ACM, 2008.
* J. Sanchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In ''Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition'', pages 1665–1672, 2011.
* P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house numbers digit classification. In ''International Conference on Pattern Recognition (ICPR 2012)'', 2012.
* P. Simard, D. Steinkraus, and J. Platt. Best practices for convolutional neural networks applied to visual document analysis. In ''Proceedings of the Seventh International Conference on Document Analysis and Recognition'', volume 2, pages 958–962, 2003.
* J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In ''Advances in Neural Information Processing Systems 25'', pages 2960–2968, 2012.
* N. Srebro and A. Shraibman. Rank, trace-norm and max-norm. In ''Proceedings of the 18th annual conference on Learning Theory, COLT'05'', pages 545–560. Springer-Verlag, 2005.
* N. Srivastava. Improving Neural Networks with Dropout. Master's thesis, University of Toronto, January 2013.
* R. Tibshirani. Regression shrinkage and selection via the lasso. ''Journal of the Royal Statistical Society. Series B. Methodological'', 58(1):267–288, 1996.
* A. N. Tikhonov. On the stability of inverse problems. ''Doklady Akademii Nauk SSSR'', 39(5):195–198, 1943.
* L. van der Maaten, M. Chen, S. Tyree, and K. Q. Weinberger. Learning with marginalized corrupted features. In ''Proceedings of the 30th International Conference on Machine Learning'', pages 410–418. ACM, 2013.
* P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In ''Proceedings of the 25th International Conference on Machine Learning'', pages 1096–1103. ACM, 2008.
* P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. In ''Proceedings of the 27th International Conference on Machine Learning'', pages 3371–3408. ACM, 2010.
* S. Wager, S. Wang, and P. Liang. Dropout training as adaptive regularization. In ''Advances in Neural Information Processing Systems 26'', pages 351–359, 2013.
* S. Wang and C. D. Manning. Fast dropout training. In ''Proceedings of the 30th International Conference on Machine Learning'', pages 118–126. ACM, 2013.
* H. Y. Xiong, Y. Barash, and B. J. Frey. Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context. ''Bioinformatics'', 27(18):2554–2562, 2011.
* M. D. Zeiler and R. Fergus. Stochastic pooling for regularization of deep convolutional neural networks. ''CoRR'', abs/1301.3557, 2013.
</translate>

[[Category:Machine Learning]]
[[Category:Research]]
[[Category:Research Papers]]