View source for A Theoretically Grounded Application of Dropout in Recurrent Neural Networks/paper

<languages />
{{PaperTabs}}
{{PaperInfobox
 | topic_area  = Machine Learning
 | difficulty  = Research
 | authors     = Yarin Gal; Zoubin Ghahramani
 | year        = 2015
 | arxiv_id    = 1512.05287
 | source_url  = https://arxiv.org/abs/1512.05287
 | pdf_url     = https://arxiv.org/pdf/1512.05287.pdf
}}
{{ContentMeta
 | generated_by   = arxiv-importer-raw
 | model_used     = strategy:html5
 | generated_date = 2026-04-27
}}

<translate>
<!--T:1-->
<span id="a-theoretically-grounded-application-of-dropout-in-recurrent-neural-networks"></span>

= A Theoretically Grounded Application of Dropout in Recurrent Neural Networks = <!--T:2-->

<!--T:3-->
&amp;Yarin Gal &amp;<br />
University of Cambridge<br />
<code>{yg279,zg201}@cam.ac.uk</code> &amp;Zoubin Ghahramani

<!--T:4-->
<span id="abstract"></span>

== Abstract == <!--T:5-->

<!--T:6-->
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

<!--T:7-->
<div id="S1" class="section">

<!--T:8-->
<span id="introduction"></span>

== 1 Introduction == <!--T:9-->

<!--T:10-->
Recurrent neural networks (RNNs) are sequence-based models of key importance for natural language understanding, language generation, video processing, and many other tasks [<nowiki/>[[#bib.bib1|1]], [[#bib.bib2|2]], [[#bib.bib3|3]]]. The model’s input is a sequence of symbols, where at each time step a simple neural network (RNN unit) is applied to a single symbol, as well as to the network’s output from the previous time step. RNNs are powerful models, showing superb performance on many tasks, but overfit quickly. Lack of regularisation in RNN models makes it difficult to handle small data, and to avoid overfitting researchers often use early stopping, or small and under-specified models [<nowiki/>[[#bib.bib4|4]]].

<!--T:11-->
Dropout is a popular regularisation technique with deep networks [<nowiki/>[[#bib.bib5|5]], [[#bib.bib6|6]]] where network units are randomly masked during training (dropped). But the technique has never been applied successfully to RNNs. Empirical results have led many to believe that noise added to recurrent layers (connections between RNN units) will be amplified for long sequences, and drown the signal [<nowiki/>[[#bib.bib4|4]]]. Consequently, existing research has concluded that the technique should be used with the inputs and outputs of the RNN alone [<nowiki/>[[#bib.bib4|4]], [[#bib.bib7|7]], [[#bib.bib8|8]], [[#bib.bib9|9]], [[#bib.bib10|10]]]. But this approach still leads to overfitting, as is shown in our experiments.

<!--T:12-->
Recent results at the intersection of Bayesian research and deep learning offer interpretation of common deep learning techniques through Bayesian eyes [<nowiki/>[[#bib.bib11|11]], [[#bib.bib12|12]], [[#bib.bib13|13]], [[#bib.bib14|14]], [[#bib.bib15|15]], [[#bib.bib16|16]]]. This Bayesian view of deep learning allowed the introduction of new techniques into the field, such as methods to obtain principled uncertainty estimates from deep learning networks [<nowiki/>[[#bib.bib14|14]], [[#bib.bib17|17]]]. Gal and Ghahramani [<nowiki/>[[#bib.bib14|14]]] for example showed that dropout can be interpreted as a variational approximation to the posterior of a Bayesian neural network (NN). Their variational approximating distribution is a mixture of two Gaussians with small variances, with the mean of one Gaussian fixed at zero. This grounding of dropout in approximate Bayesian inference suggests that an extension of the theoretical results might offer insights into the use of the technique with RNN models.

<!--T:13-->
Here we focus on common RNN models in the field (LSTM [<nowiki/>[[#bib.bib18|18]]], GRU [<nowiki/>[[#bib.bib19|19]]]) and interpret these as probabilistic models, i.e. as RNNs with network weights treated as random variables, and with suitably defined likelihood functions. We then perform approximate variational inference in these probabilistic Bayesian models (which we will refer to as Variational RNNs). Approximating the posterior distribution over the weights with a mixture of Gaussians (with one component fixed at zero and small variances) will lead to a tractable optimisation objective. Optimising this objective is identical to performing a new variant of dropout in the respective RNNs.

<!--T:14-->
<div id="S1.F1" class="figure">

<!--T:15-->
<div id="S1.F1.sf1" class="figure">

<!--T:16-->
[[File:data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjM0Ny4yOSIgaWQ9IlMxLkYxLnNmMS5waWMxIiBvdmVyZmxvdz0idmlzaWJsZSIgdmVyc2lvbj0iMS4xIiB3aWR0aD0iMjYzLjA5Ij48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsMzQ3LjI5KSBtYXRyaXgoMSAwIDAgLTEgMCAwKSB0cmFuc2xhdGUoMTMwLjY4LDApIHRyYW5zbGF0ZSgwLDI4Ljg1KSI+PGcgc3Ryb2tlLXdpZHRoPSIwLjRwdCI+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTExLjcyIC0xMC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuMzIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgOS40NikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTIuMTUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjMuNDUiPlxwZ2ZtYXRocmVzdWx0cHQ8L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgMTguMTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSA2LjM3IDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguMzQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTAuNzEiPjxtYXRoIGFsdHRleHQ9Inhfe3R9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPng8L21pPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+dDwvbWk+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxjaSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+8J2RoTwvY2k+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dH08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNC42MSA1OS43NiBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAtNC42MSAxNTMuMjYgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC01LjA0IDI1MS4zNykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuNjk1KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDQ1LjMzKSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAwKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjY1IiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEwLjA4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0fSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj55PC9taT48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPnQ8L21pPjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPvCdkaE8L2NpPjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnlfe3R9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04My4wMiAtMjMuOTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDQzLjU5NSkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC40NSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxNy4xNiI+PG1hdGggYWx0dGV4dD0ieF97dC0xfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+4oiSPC9tbz48bW4gaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PG1pbnVzIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9taW51cz48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIj7wnZGhPC9jaT48Y24gaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIj4xPC9jbj48L2FwcGx5PjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnhfe3QtMX08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNzkuMDUgNzMuNDMgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gLTc5LjA1IDE2Ni45MyBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjcgMjY1LjA0KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My42OTUpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguNjUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTYuNTQiPjxtYXRoIGFsdHRleHQ9Inlfe3QtMX0iIGRpc3BsYXk9ImlubGluZSIgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtc3ViIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCI+eTwvbWk+PG1yb3cgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiPnQ8L21pPjxtbyBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiPuKIkjwvbW8+PG1uIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxtaW51cyBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiPjwvbWludXM+PGNpIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0LTF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDY1Ljg1IC0yNC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuOTIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjkuMSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIyMC42MSI+PG1hdGggYWx0dGV4dD0ieF97dCsxfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+KzwvbW8+PG1uIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGlPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxwbHVzIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9wbHVzPjxjaSBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiPvCdkaE8L2NpPjxjbiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMiPjE8L2NuPjwvYXBwbHk+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dCsxfTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PC9nPjwvZz48cGF0aCBkPSJNIDcxLjU0IDczLjc1IGggOS4yMiB2IDQ4LjU5IGggLTkuMjIgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDcxLjU0IDE2Ny4yNSBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgNjYuMTcgMjY1LjM2KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My45MikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOS4xIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjE5Ljk4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0KzF9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPnk8L21pPjxtcm93IGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIj50PC9taT48bW8gaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIj4rPC9tbz48bW4gaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaY8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PHBsdXMgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIj48L3BsdXM+PGNpIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0KzF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwMDBGRiI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMzEuMjkgTCAtNzQuNDMgNjguOTkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAtNzQuNDMgNjguOTkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTcuNjIgTCAwIDU1LjMyIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgMCA1NS4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiM4MDgwODAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMzEuNjEgTCA3Ni4xNiA2OS4zMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIDc2LjE2IDY5LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwRkYwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMTI0Ljc5IEwgLTc0LjQzIDE2Mi40OSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAxNjIuNDkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTExLjEyIEwgMCAxNDguODIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDE0OC44MikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRkZGMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMTI1LjExIEwgNzYuMTYgMTYyLjgxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMTYyLjgxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGMDAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMjE4LjI5IEwgLTc0LjQzIDI1NS45OCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAyNTUuOTgpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDBGRkZGIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMjA0LjYyIEwgMCAyNDIuMzIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDI0Mi4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRjAwRkYiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMjE4LjYxIEwgNzYuMTYgMjU2LjMxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMjU2LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9IjMuMHB0LDMuMHB0IiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiPjxwYXRoIGQ9Ik0gLTEyMS4xOCA5Ny43MyBMIC04Mi40NSA5Ny43MyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjQ1IDk3LjczKSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSAtNjcuMDUgOTYuMzcgTCAtOC4wMSA4NS41MyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU1IC0wLjE4MDYzIDAuMTgwNjMgMC45ODM1NSAtOC4wMSA4NS41MykiPjxwYXRoIGQ9Ik0gLTEuNjYgMi4yMSBDIC0xLjUyIDEuMzggMCAwLjE0IDAuNDIgMCBDIDAgLTAuMTQgLTEuNTIgLTEuMzggLTEuNjYgLTIuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9IjMuMHB0LDMuMHB0IiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiPjxwYXRoIGQ9Ik0gNy4zOCA4NS40MSBMIDY4LjE1IDk2LjU4IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIwLjMycHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTQgMC4xODA3MyAtMC4xODA3MyAwLjk4MzU0IDY4LjE1IDk2LjU4KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSA4My41MyA5OC4wNSBMIDEyMi4yNyA5OC4wNSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTIyLjI3IDk4LjA1KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSAtMTIxLjE4IDE5MS4yMiBMIC04Mi40NSAxOTEuMjIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjAuMzJwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04Mi40NSAxOTEuMjIpIj48cGF0aCBkPSJNIC0xLjY2IDIuMjEgQyAtMS41MiAxLjM4IDAgMC4xNCAwLjQyIDAgQyAwIC0wLjE0IC0xLjUyIC0xLjM4IC0xLjY2IC0yLjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSIzLjBwdCwzLjBwdCIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0Ij48cGF0aCBkPSJNIC02Ny4wNSAxODkuODcgTCAtOC4wMSAxNzkuMDMiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjAuMzJwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC45ODM1NSAtMC4xODA2MyAwLjE4MDYzIDAuOTgzNTUgLTguMDEgMTc5LjAzKSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSA3LjM4IDE3OC45MSBMIDY4LjE1IDE5MC4wOCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU0IDAuMTgwNzMgLTAuMTgwNzMgMC45ODM1NCA2OC4xNSAxOTAuMDgpIj48cGF0aCBkPSJNIC0xLjY2IDIuMjEgQyAtMS41MiAxLjM4IDAgMC4xNCAwLjQyIDAgQyAwIC0wLjE0IC0xLjUyIC0xLjM4IC0xLjY2IC0yLjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSIzLjBwdCwzLjBwdCIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0Ij48cGF0aCBkPSJNIDgzLjUzIDE5MS41NSBMIDEyMi4yNyAxOTEuNTUiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIwLjMycHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAxMjIuMjcgMTkxLjU1KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjwvc3ZnPg==]]

<!--T:17-->
<div id="S1.F1.sf2" class="figure">

<!--T:18-->
[[File:data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjM0Ny4yOSIgaWQ9IlMxLkYxLnNmMi5waWMxIiBvdmVyZmxvdz0idmlzaWJsZSIgdmVyc2lvbj0iMS4xIiB3aWR0aD0iMjYzLjA5Ij48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMC40cHQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsMzQ3LjI5KSBtYXRyaXgoMSAwIDAgLTEgMCAwKSB0cmFuc2xhdGUoMTMwLjY4LDApIHRyYW5zbGF0ZSgwLDI4Ljg1KSI+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTExLjcyIC0xMC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuMzIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgOS40NikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTIuMTUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjMuNDUiPlxwZ2ZtYXRocmVzdWx0cHQ8L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgMTguMTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSA2LjM3IDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguMzQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTAuNzEiPjxtYXRoIGFsdHRleHQ9Inhfe3R9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPng8L21pPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+dDwvbWk+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxjaSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+8J2RoTwvY2k+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dH08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNC42MSA1OS43NiBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAtNC42MSAxNTMuMjYgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC01LjA0IDI1MS4zNykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuNjk1KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDQ1LjMzKSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAwKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjY1IiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEwLjA4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0fSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj55PC9taT48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPnQ8L21pPjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPvCdkaE8L2NpPjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnlfe3R9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04My4wMiAtMjMuOTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDQzLjU5NSkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC40NSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxNy4xNiI+PG1hdGggYWx0dGV4dD0ieF97dC0xfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+4oiSPC9tbz48bW4gaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PG1pbnVzIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9taW51cz48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIj7wnZGhPC9jaT48Y24gaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIj4xPC9jbj48L2FwcGx5PjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnhfe3QtMX08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNzkuMDUgNzMuNDMgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gLTc5LjA1IDE2Ni45MyBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjcgMjY1LjA0KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My42OTUpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguNjUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTYuNTQiPjxtYXRoIGFsdHRleHQ9Inlfe3QtMX0iIGRpc3BsYXk9ImlubGluZSIgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtc3ViIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCI+eTwvbWk+PG1yb3cgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiPnQ8L21pPjxtbyBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiPuKIkjwvbW8+PG1uIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxtaW51cyBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiPjwvbWludXM+PGNpIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0LTF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDY1Ljg1IC0yNC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuOTIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjkuMSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIyMC42MSI+PG1hdGggYWx0dGV4dD0ieF97dCsxfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+KzwvbW8+PG1uIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGlPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxwbHVzIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9wbHVzPjxjaSBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiPvCdkaE8L2NpPjxjbiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMiPjE8L2NuPjwvYXBwbHk+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dCsxfTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PC9nPjwvZz48cGF0aCBkPSJNIDcxLjU0IDczLjc1IGggOS4yMiB2IDQ4LjU5IGggLTkuMjIgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDcxLjU0IDE2Ny4yNSBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgNjYuMTcgMjY1LjM2KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My45MikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOS4xIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjE5Ljk4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0KzF9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPnk8L21pPjxtcm93IGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIj50PC9taT48bW8gaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIj4rPC9tbz48bW4gaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaY8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PHBsdXMgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIj48L3BsdXM+PGNpIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0KzF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwMDBGRiI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMzEuMjkgTCAtNzQuNDMgNjguOTkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAtNzQuNDMgNjguOTkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDAwMEZGIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTcuNjIgTCAwIDU1LjMyIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgMCA1NS4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiMwMDAwRkYiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMzEuNjEgTCA3Ni4xNiA2OS4zMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIDc2LjE2IDY5LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwRkYwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMTI0Ljc5IEwgLTc0LjQzIDE2Mi40OSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAxNjIuNDkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDBGRjAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTExLjEyIEwgMCAxNDguODIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDE0OC44MikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiMwMEZGMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMTI1LjExIEwgNzYuMTYgMTYyLjgxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMTYyLjgxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGMDAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMjE4LjI5IEwgLTc0LjQzIDI1NS45OCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAyNTUuOTgpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkYwMDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMjA0LjYyIEwgMCAyNDIuMzIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDI0Mi4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRjAwMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMjE4LjYxIEwgNzYuMTYgMjU2LjMxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMjU2LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGODAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtMTIxLjE4IDk3LjczIEwgLTgzLjQ5IDk3LjczIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtODMuNDkgOTcuNzMpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIC02Ny4wNSA5Ni4zNyBMIC05LjAzIDg1LjcxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTUgLTAuMTgwNjMgMC4xODA2MyAwLjk4MzU1IC05LjAzIDg1LjcxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGODAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSA3LjM4IDg1LjQxIEwgNjcuMTMgOTYuMzkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC45ODM1NCAwLjE4MDczIC0wLjE4MDczIDAuOTgzNTQgNjcuMTMgOTYuMzkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDgzLjUzIDk4LjA1IEwgMTIxLjIzIDk4LjA1IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAxMjEuMjMgOTguMDUpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIC0xMjEuMTggMTkxLjIyIEwgLTgzLjQ5IDE5MS4yMiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgzLjQ5IDE5MS4yMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNCRjAwNDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gLTY3LjA1IDE4OS44NyBMIC05LjAzIDE3OS4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU1IC0wLjE4MDYzIDAuMTgwNjMgMC45ODM1NSAtOS4wMyAxNzkuMjEpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDcuMzggMTc4LjkxIEwgNjcuMTMgMTg5Ljg5IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTQgMC4xODA3MyAtMC4xODA3MyAwLjk4MzU0IDY3LjEzIDE4OS44OSkiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNCRjAwNDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gODMuNTMgMTkxLjU1IEwgMTIxLjIzIDE5MS41NSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTIxLjIzIDE5MS41NSkiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48L2c+PC9zdmc+]]

<!--T:19-->
In the new dropout variant, we repeat the same dropout mask at each time step for both inputs, outputs, and recurrent layers (drop the same network units at each time step). This is in contrast to the existing ad hoc techniques where different dropout masks are sampled at each time step for the inputs and outputs alone (no dropout is used with the recurrent connections since the use of different masks with these connections leads to deteriorated performance). Our method and its relation to existing techniques is depicted in figure [[#S1.F1|1]]. When used with discrete inputs (i.e. words) we place a distribution over the word embeddings as well. Dropout in the word-based model corresponds then to randomly dropping word types in the sentence, and might be interpreted as forcing the model not to rely on single words for its task.

<!--T:20-->
We next survey related literature and background material, and then formalise our approximate inference for the Variational RNN, resulting in the dropout variant proposed above. Experimental results are presented thereafter.

<!--T:21-->
<div id="S2" class="section">

<!--T:22-->
<span id="related-research"></span>

== 2 Related Research == <!--T:23-->

<!--T:24-->
In the past few years a considerable body of work has been collected demonstrating the negative effects of a naive application of dropout in RNNs’ recurrent connections. Pachitariu and Sahani [<nowiki/>[[#bib.bib7|7]]], working with language models, reason that noise added in the recurrent connections of an RNN leads to model instabilities. Instead, they add noise to the decoding part of the model alone. Bayer et al. [<nowiki/>[[#bib.bib8|8]]] apply a deterministic approximation of dropout (fast dropout) in RNNs. They reason that with dropout, the RNN’s dynamics change dramatically, and that dropout should be applied to the “non-dynamic” parts of the model – connections feeding from the hidden layer to the output layer. Pham et al. [<nowiki/>[[#bib.bib9|9]]] assess dropout with handwriting recognition tasks. They conclude that dropout in recurrent layers disrupts the RNN’s ability to model sequences, and that dropout should be applied to feed-forward connections and not to recurrent connections. The work by Zaremba, Sutskever, and Vinyals [<nowiki/>[[#bib.bib4|4]]] was developed in parallel to Pham et al. [<nowiki/>[[#bib.bib9|9]]]. Zaremba et al. [<nowiki/>[[#bib.bib4|4]]] assess the performance of dropout in RNNs on a wide series of tasks. They show that applying dropout to the non-recurrent connections alone results in improved performance, and provide (as yet unbeaten) state-of-the-art results in language modelling on the Penn Treebank. They reason that without dropout only small models were used in the past in order to avoid overfitting, whereas with the application of dropout larger models can be used, leading to improved results. This work is considered a reference implementation by many (and we compare to this as a baseline below). Bluche et al. [<nowiki/>[[#bib.bib10|10]]] extend on the previous body of work and perform exploratory analysis of the performance of dropout before, inside, and after the RNN’s unit. They provide mixed results, not showing significant improvement on existing techniques. More recently, and done in parallel to this work, Moon et al. [<nowiki/>[[#bib.bib20|20]]] suggested a new variant of dropout in RNNs in the speech recognition community. They randomly drop elements in the LSTM’s internal cell <math display="inline">\mathbf{c}_{t}</math> and use the same mask at every time step. This is the closest to our proposed approach (although fundamentally different to the approach we suggest, explained in §[[#S4.SS1|4.1]]), and we compare to this variant below as well.

<!--T:25-->
Existing approaches are based on an empirical experimentation with different flavours of dropout, following a process of trial-and-error. These approaches have led many to believe that dropout cannot be extended to a large number of parameters within the recurrent layers, leaving them with no regularisation. In contrast to these conclusions, we show that it is possible to derive a variational inference based variant of dropout which successfully regularises such parameters, by grounding our approach in recent theoretical research.

<!--T:26-->
<div id="S3" class="section">

<!--T:27-->
<span id="background"></span>

== 3 Background == <!--T:28-->

<!--T:29-->
We review necessary background in Bayesian neural networks and approximate variational inference. Building on these ideas, in the next section we propose approximate inference in the probabilistic RNN which will lead to a new variant of dropout.

<!--T:30-->
<div id="S3.SS1" class="section">

<!--T:31-->
<span id="bayesian-neural-networks"></span>

=== 3.1 Bayesian Neural Networks === <!--T:32-->

<!--T:33-->
Given training inputs <math display="inline">\mathbf{X} = {\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}}</math> and their corresponding outputs <math display="inline">\mathbf{Y} = {\{\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\}}</math>, in Bayesian (parametric) regression we would like to infer parameters <math display="inline">\mathbf{ω}</math> of a function <math display="inline">\mathbf{y} = {\mathbf{f}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}}</math> that are likely to have generated our outputs. What parameters are likely to have generated our data? Following the Bayesian approach we would put some prior distribution over the space of parameters, <math display="inline">p\hspace{0pt}{({\mathbf{ω}})}</math>. This distribution represents our prior belief as to which parameters are likely to have generated our data. We further need to define a likelihood distribution <math display="inline">p\hspace{0pt}{(\left. \mathbf{y} \middle| {\mathbf{x},{\mathbf{ω}}} \right.)}</math>. For classification tasks we may assume a softmax likelihood,

<!--T:34-->
{| id="A3.EGx1" class="wikitable"
|-
|
| <math display="inline">{p\hspace{0pt}\left( {y = \left. d \middle| {\mathbf{x},{\mathbf{ω}}} \right.} \right)} = {\text{Categorical}\hspace{0pt}\left( {{\exp{({f_{d}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}})}}/{\sum\limits_{d^{\prime}}{\exp{({f_{d^{\prime}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}})}}}} \right)}</math>
|
|}

<!--T:35-->
or a Gaussian likelihood for regression. Given a dataset <math display="inline">\mathbf{X},\mathbf{Y}</math>, we then look for the posterior distribution over the space of parameters: <math display="inline">p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}</math>. This distribution captures how likely various function parameters are given our observed data. With it we can predict an output for a new input point <math display="inline">\mathbf{x}^{\ast}</math> by integrating

<!--T:36-->
{| id="A3.EGx2" class="wikitable"
|-
|
| <math display="inline">{{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},\mathbf{X},\mathbf{Y}} \right.)}} = {\int{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\mathbf{ω}}} \right.)}\hspace{0pt}p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}\hspace{0pt}{\text{d}{\mathbf{ω}}}}}}.</math>
|
| (1)
|}

<!--T:37-->
One way to define a distribution over a parametric set of functions is to place a prior distribution over a neural network’s weights, resulting in a Bayesian NN [<nowiki/>[[#bib.bib21|21]], [[#bib.bib22|22]]]. Given weight matrices <math display="inline">\mathbf{W}_{i}</math> and bias vectors <math display="inline">\mathbf{b}_{i}</math> for layer <math display="inline">i</math>, we often place standard matrix Gaussian prior distributions over the weight matrices, <math display="inline">{p\hspace{0pt}{(\mathbf{W}_{i})}} = {\mathcal{N}\hspace{0pt}{(\mathbf{0},\mathbf{I})}}</math> and often assume a point estimate for the bias vectors for simplicity.

<!--T:38-->
<div id="S3.SS2" class="section">

<!--T:39-->
<span id="approximate-variational-inference-in-bayesian-neural-networks"></span>

=== 3.2 Approximate Variational Inference in Bayesian Neural Networks === <!--T:40-->

<!--T:41-->
We are interested in finding the distribution of weight matrices (parametrising our functions) that have generated our data. This is the posterior over the weights given our observables <math display="inline">\mathbf{X},\mathbf{Y}</math>: <math display="inline">p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}</math>. This posterior is not tractable in general, and we may use variational inference to approximate it (as was done in [<nowiki/>[[#bib.bib23|23]], [[#bib.bib24|24]], [[#bib.bib25|25]], [[#bib.bib12|12]]]). We need to define an approximating variational distribution <math display="inline">q\hspace{0pt}{({\mathbf{ω}})}</math>, and then minimise the KL divergence between the approximating distribution and the full posterior:

<!--T:42-->
{| id="A3.EGx3" class="wikitable"
|-
|
| <math display="inline">\text{KL}\left( q{({\mathbf{ω}})}||p{({\mathbf{ω}}|\mathbf{X},\mathbf{Y})} \right)</math>
| <math display="inline">\propto - \int q{({\mathbf{ω}})}\log p{(\mathbf{Y}|\mathbf{X},{\mathbf{ω}})}{\text{d}{\mathbf{ω}}} + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}</math>
|
|
|-
|
|
| <math display="inline">= - \sum\limits_{i = 1}^{N}\int q{({\mathbf{ω}})}\log p{(\mathbf{y}_{i}|\mathbf{f}^{\mathbf{ω}}{(\mathbf{x}_{i})})}{\text{d}{\mathbf{ω}}} + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}.</math>
|
| (2)
|}

<!--T:43-->
We next extend this approximate variational inference to probabilistic RNNs, and use a <math display="inline">q\hspace{0pt}{({\mathbf{ω}})}</math> distribution that will give rise to a new variant of dropout in RNNs.

<!--T:44-->
<div id="S4" class="section">

<!--T:45-->
<span id="variational-inference-in-recurrent-neural-networks"></span>

== 4 Variational Inference in Recurrent Neural Networks == <!--T:46-->

<!--T:47-->
In this section we will concentrate on simple RNN models for brevity of notation. Derivations for LSTM and GRU follow similarly. Given input sequence <math display="inline">\mathbf{x} = {\lbrack\mathbf{x}_{1},\ldots,\mathbf{x}_{T}\rbrack}</math> of length <math display="inline">T</math>, a simple RNN is formed by a repeated application of a function <math display="inline">\mathbf{f}_{\mathbf{h}}</math>. This generates a hidden state <math display="inline">\mathbf{h}_{t}</math> for time step <math display="inline">t</math>:

<!--T:48-->
{| id="A3.EGx4" class="wikitable"
|-
|
| <math display="inline">\mathbf{h}_{t}</math>
| <math display="inline">= {\mathbf{f}_{\mathbf{h}}\hspace{0pt}{(\mathbf{x}_{t},\mathbf{h}_{t - 1})}} = {\sigma\hspace{0pt}{({{\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{\mathbf{h}}} + {\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{\mathbf{h}}} + \mathbf{b}_{\mathbf{h}}})}}</math>
|
|}

<!--T:49-->
for some non-linearity <math display="inline">\sigma</math>. The model output can be defined, for example, as <math display="inline">{\mathbf{f}_{\mathbf{y}}\hspace{0pt}{(\mathbf{h}_{T})}} = {{\mathbf{h}_{T}\hspace{0pt}\mathbf{W}_{\mathbf{y}}} + \mathbf{b}_{\mathbf{y}}}</math>. We view this RNN as a probabilistic model by regarding <math display="inline">{\mathbf{ω}} = {\{\mathbf{W}_{\mathbf{h}},\mathbf{U}_{\mathbf{h}},\mathbf{b}_{\mathbf{h}},\mathbf{W}_{\mathbf{y}},\mathbf{b}_{\mathbf{y}}\}}</math> as random variables (following normal prior distributions). To make the dependence on <math display="inline">\mathbf{ω}</math> clear, we write <math display="inline">\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}</math> for <math display="inline">\mathbf{f}_{\mathbf{y}}</math> and similarly for <math display="inline">\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}</math>. We define our probabilistic model’s likelihood as above (section [[#S3.SS1|3.1]]). The posterior over random variables <math display="inline">\mathbf{ω}</math> is rather complex, and we use variational inference with approximating distribution <math display="inline">q\hspace{0pt}{({\mathbf{ω}})}</math> to approximate it.

<!--T:50-->
Evaluating each sum term in eq. ([[#S3.Ex2|3.2]]) above with our RNN model we get

<!--T:51-->
{| id="A3.EGx5" class="wikitable"
|-
|
| <math display="inline">\int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}{(\left. \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{h}_{T})}} \right.)}\hspace{0pt}{\text{d}{\mathbf{ω}}}}</math>
| <math display="inline">= {\int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{T},\mathbf{h}_{T - 1})}} \right)} \right)\hspace{0pt}{\text{d}{\mathbf{ω}}}}}</math>
|
|-
|
|
| <math display="inline">= {\int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{T},{\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{({\ldots\hspace{0pt}\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{1},\mathbf{h}_{0})}\hspace{0pt}\ldots})}})}} \right)} \right)\hspace{0pt}{\text{d}{\mathbf{ω}}}}}</math>
|
|}

<!--T:52-->
with <math display="inline">\mathbf{h}_{0} = \text{0}</math>. We approximate this with Monte Carlo (MC) integration with a single sample:

<!--T:53-->
{| id="A3.EGx6" class="wikitable"
|-
|
| <math display="inline">{\approx {{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\hat{\mathbf{ω}}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{(\mathbf{x}_{T},{\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{({\ldots\hspace{0pt}\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{(\mathbf{x}_{1},\mathbf{h}_{0})}\hspace{0pt}\ldots})}})}} \right)} \right)}},</math>
| <math display="inline">\hat{\mathbf{ω}} \sim {q\hspace{0pt}{({\mathbf{ω}})}}</math>
|
|}

<!--T:54-->
resulting in an unbiased estimator to each sum term.

<!--T:55-->
This estimator is plugged into equation ([[#S3.Ex2|3.2]]) to obtain our minimisation objective

<!--T:56-->
{| id="A3.EGx7" class="wikitable"
|-
|
| <math display="inline">\mathcal{L}</math>
| <math display="inline">\approx - \sum\limits_{i = 1}^{N}\log p\left( \mathbf{y}_{i} \middle| \mathbf{f}_{\mathbf{y}}^{{\hat{\mathbf{ω}}}_{i}}\left( \mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\mathbf{x}_{i,T},\mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\ldots\mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\mathbf{x}_{i,1},\mathbf{h}_{0})}\ldots)})} \right) \right) + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}.</math>
|
| (3)
|}

<!--T:57-->
Note that for each sequence <math display="inline">\mathbf{x}_{i}</math> we sample a new realisation <math display="inline">{\hat{\mathbf{ω}}}_{i} = {\{{\hat{\mathbf{W}}}_{\mathbf{h}}^{i},{\hat{\mathbf{U}}}_{\mathbf{h}}^{i},{\hat{\mathbf{b}}}_{\mathbf{h}}^{i},{\hat{\mathbf{W}}}_{\mathbf{y}}^{i},{\hat{\mathbf{b}}}_{\mathbf{y}}^{i}\}}</math>, and that each symbol in the sequence <math display="inline">\mathbf{x}_{i} = {\lbrack\mathbf{x}_{i,1},\ldots,\mathbf{x}_{i,T}\rbrack}</math> is passed through the function <math display="inline">\mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}</math> with the same weight realisations <math display="inline">{\hat{\mathbf{W}}}_{\mathbf{h}}^{i},{\hat{\mathbf{U}}}_{\mathbf{h}}^{i},{\hat{\mathbf{b}}}_{\mathbf{h}}^{i}</math> used at every time step <math display="inline">t \leq T</math>.

<!--T:58-->
Following [<nowiki/>[[#bib.bib17|17]]] we define our approximating distribution to factorise over the weight matrices and their rows in <math display="inline">\mathbf{ω}</math>. For every weight matrix row <math display="inline">\mathbf{w}_{k}</math> the approximating distribution is:

<!--T:59-->
{| id="A3.EGx8" class="wikitable"
|-
|
| <math display="inline">q\hspace{0pt}{(\mathbf{w}_{k})}</math>
| <math display="inline">= {{p\hspace{0pt}\mathcal{N}\hspace{0pt}{(\mathbf{w}_{k};\text{0},{\sigma^{2}\hspace{0pt}I})}} + {{({1 - p})}\hspace{0pt}\mathcal{N}\hspace{0pt}{(\mathbf{w}_{k};\mathbf{m}_{k},{\sigma^{2}\hspace{0pt}I})}}}</math>
|
|}

<!--T:60-->
with <math display="inline">\mathbf{m}_{k}</math> variational parameter (row vector), <math display="inline">p</math> given in advance (the dropout probability), and small <math display="inline">\sigma^{2}</math>. We optimise over <math display="inline">\mathbf{m}_{k}</math> the variational parameters of the random weight matrices; these correspond to the RNN’s weight matrices in the standard view<sup>11</sup>1Graves et al. [<nowiki/>[[#bib.bib26|26]]] further factorise the approximating distribution over the elements of each row, and use a Gaussian approximating distribution with each element (rather than a mixture); the approximating distribution above seems to give better performance, and has a close relation with dropout [<nowiki/>[[#bib.bib17|17]]].. The KL in eq. ([[#S4.E3|3]]) can be approximated as <math display="inline">L_{2}</math> regularisation over the variational parameters <math display="inline">\mathbf{m}_{k}</math> [<nowiki/>[[#bib.bib17|17]]].

<!--T:61-->
Evaluating the model output <math display="inline">\mathbf{f}_{\mathbf{y}}^{\hat{\mathbf{ω}}}\hspace{0pt}{( \cdot )}</math> with sample <math display="inline">\hat{\mathbf{ω}} \sim {q\hspace{0pt}{({\mathbf{ω}})}}</math> corresponds to randomly zeroing (masking) rows in each weight matrix <math display="inline">\mathbf{W}</math> during the forward pass – i.e. performing dropout. Our objective <math display="inline">\mathcal{L}</math> is identical to that of the standard RNN. In our RNN setting with a sequence input, each weight matrix row is randomly masked once, and importantly the same mask is used through all time steps.<sup>22</sup>2In appendix [[#A1|A]] we discuss the relation of our dropout interpretation to the ensembling one.

<!--T:62-->
Predictions can be approximated by either propagating the mean of each layer to the next (referred to as the standard dropout approximation), or by approximating the posterior in eq. ([[#S3.E1|1]]) with <math display="inline">q\hspace{0pt}{({\mathbf{ω}})}</math>,

<!--T:63-->
{| id="A3.EGx9" class="wikitable"
|-
|
| <math display="inline">p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},\mathbf{X},\mathbf{Y}} \right.)}</math>
| <math display="inline">\approx {\int{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\mathbf{ω}}} \right.)}\hspace{0pt}q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\text{d}{\mathbf{ω}}}}} \approx {\frac{1}{K}\hspace{0pt}{\sum\limits_{k = 1}^{K}{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\hat{\mathbf{ω}}}_{k}} \right.)}}}}</math>
|
| (4)
|}

<!--T:64-->
with <math display="inline">{\hat{\mathbf{ω}}}_{k} \sim {q\hspace{0pt}{({\mathbf{ω}})}}</math>, i.e. by performing dropout at test time and averaging results (MC dropout).

<!--T:65-->
<div id="S4.SS1" class="section">

<!--T:66-->
<span id="implementation-and-relation-to-dropout-in-rnns"></span>

=== 4.1 Implementation and Relation to Dropout in RNNs === <!--T:67-->

<!--T:68-->
Implementing our approximate inference is identical to implementing dropout in RNNs with the same network units dropped at each time step, randomly dropping inputs, outputs, and recurrent connections. This is in contrast to existing techniques, where different network units would be dropped at different time steps, and no dropout would be applied to the recurrent connections (fig. [[#S1.F1|1]]).

<!--T:69-->
Certain RNN models such as LSTMs and GRUs use different gates within the RNN units. For example, an LSTM is defined using four gates: “input”, “forget”, “output”, and “input modulation”,

<!--T:70-->
{| id="A3.EGx10" class="wikitable"
|-
|
| <math display="inline">\underset{¯}{\mathbf{i}}</math>
| <math display="inline">= {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{i}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{i}}} \right)}</math>
| <math display="inline">\underset{¯}{\mathbf{f}} = {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{f}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{f}}} \right)}</math>
|
|
|-
|
| <math display="inline">\underset{¯}{\mathbf{o}}</math>
| <math display="inline">= {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{o}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{o}}} \right)}</math>
| <math display="inline">\underset{¯}{\mathbf{g}} = {\text{tanh}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{g}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{g}}} \right)}</math>
|
|
|-
|
| <math display="inline">\mathbf{c}_{t}</math>
| <math display="inline">= {{\underset{¯}{\mathbf{f}} \circ \mathbf{c}_{t - 1}} + {\underset{¯}{\mathbf{i}} \circ \underset{¯}{\mathbf{g}}}}</math>
| <math display="inline">\mathbf{h}_{t} = {{\underset{¯}{\mathbf{o}} \circ \text{tanh}}\hspace{0pt}{(\mathbf{c}_{t})}}</math>
|
| (5)
|}

<!--T:71-->
with <math display="inline">{\mathbf{ω}} = {\{\mathbf{W}_{i},\mathbf{U}_{i},\mathbf{W}_{f},\mathbf{U}_{f},\mathbf{W}_{o},\mathbf{U}_{o},\mathbf{W}_{g},\mathbf{U}_{g}\}}</math> weight matrices and <math display="inline">\circ</math> the element-wise product. Here an internal state <math display="inline">\mathbf{c}_{t}</math> (also referred to as cell) is updated additively.

<!--T:72-->
Alternatively, the model could be re-parametrised as in [<nowiki/>[[#bib.bib26|26]]]:

<!--T:73-->
{| id="A3.EGx11" class="wikitable"
|-
|
| <math display="inline">\begin{pmatrix}
\underset{¯}{\mathbf{i}} \\
\underset{¯}{\mathbf{f}} \\
\underset{¯}{\mathbf{o}} \\
\underset{¯}{\mathbf{g}}
\end{pmatrix} = {\begin{pmatrix}
\text{sigm} \\
\text{sigm} \\
\text{sigm} \\
\text{tanh}
\end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix}
\mathbf{x}_{t} \\
\mathbf{h}_{t - 1}
\end{pmatrix} \cdot \mathbf{W}} \right)}</math>
|
| (6)
|}

<!--T:74-->
with <math display="inline">{\mathbf{ω}} = {\{\mathbf{W}\}}</math>, <math display="inline">\mathbf{W}</math> a matrix of dimensions <math display="inline">2\hspace{0pt}K</math> by <math display="inline">4\hspace{0pt}K</math> (<math display="inline">K</math> being the dimensionality of <math display="inline">\mathbf{x}_{t}</math>). We name this parametrisation a tied-weights LSTM (compared to the untied-weights LSTM in eq. ([[#S4.Ex8|4.1]])).

<!--T:75-->
Even though these two parametrisations result in the same deterministic model, they lead to different approximating distributions <math display="inline">q\hspace{0pt}{({\mathbf{ω}})}</math>. With the first parametrisation one could use different dropout masks for different gates (even when the same input <math display="inline">\mathbf{x}_{t}</math> is used). This is because the approximating distribution is placed over the matrices rather than the inputs: we might drop certain rows in one weight matrix <math display="inline">\mathbf{W}</math> applied to <math display="inline">\mathbf{x}_{t}</math> and different rows in another matrix <math display="inline">\mathbf{W}^{\prime}</math> applied to <math display="inline">\mathbf{x}_{t}</math>. With the second parametrisations we would place a distribution over the single matrix <math display="inline">\mathbf{W}</math>. This leads to a faster forward-pass, but with slightly diminished results as we will see in the experiments section.

<!--T:76-->
In more concrete terms, we may write our dropout variant with the second parametrisation (eq. ([[#S4.E6|6]])) as

<!--T:77-->
{| id="A3.EGx12" class="wikitable"
|-
|
| <math display="inline">\begin{pmatrix}
\underset{¯}{\mathbf{i}} \\
\underset{¯}{\mathbf{f}} \\
\underset{¯}{\mathbf{o}} \\
\underset{¯}{\mathbf{g}}
\end{pmatrix} = {\begin{pmatrix}
\text{sigm} \\
\text{sigm} \\
\text{sigm} \\
\text{tanh}
\end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix}
{\mathbf{x}_{t} \circ \mathtt{z}_{\mathbf{x}}} \\
{\mathbf{h}_{t - 1} \circ \mathtt{z}_{\mathbf{h}}}
\end{pmatrix} \cdot \mathbf{W}} \right)}</math>
|
| (7)
|}

<!--T:78-->
with <math display="inline">\mathtt{z}_{\mathbf{x}},\mathtt{z}_{\mathbf{h}}</math> random masks repeated at all time steps (and similarly for the parametrisation in eq. ([[#S4.Ex8|4.1]])).

<!--T:79-->
In comparison, Zaremba et al. [<nowiki/>[[#bib.bib4|4]]]’s dropout variant replaces <math display="inline">\mathtt{z}_{\mathbf{x}}</math> in eq. ([[#S4.E7|7]]) with the time-dependent <math display="inline">\mathtt{z}_{\mathbf{x}}^{t}</math> which is sampled anew every time step (whereas <math display="inline">\mathtt{z}_{\mathbf{h}}</math> is removed and the recurrent connection <math display="inline">\mathbf{h}_{t - 1}</math> is not dropped):

<!--T:80-->
{| id="A3.EGx13" class="wikitable"
|-
|
| <math display="inline">{\begin{pmatrix}
\underset{¯}{\mathbf{i}} \\
\underset{¯}{\mathbf{f}} \\
\underset{¯}{\mathbf{o}} \\
\underset{¯}{\mathbf{g}}
\end{pmatrix} = {\begin{pmatrix}
\text{sigm} \\
\text{sigm} \\
\text{sigm} \\
\text{tanh}
\end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix}
{\mathbf{x}_{t} \circ \mathtt{z}_{\mathbf{x}}^{t}} \\
\mathbf{h}_{t - 1}
\end{pmatrix} \cdot \mathbf{W}} \right)}}.</math>
|
| (8)
|}

<!--T:81-->
On the other hand, Moon et al. [<nowiki/>[[#bib.bib20|20]]]’s dropout variant changes eq. ([[#S4.Ex8|4.1]]) by adapting the internal cell

<!--T:82-->
{| id="A3.EGx14" class="wikitable"
|-
|
| <math display="inline">\mathbf{c}_{t} = {\mathbf{c}_{t} \circ \mathtt{z}_{\mathbf{c}}}</math>
|
| (9)
|}

<!--T:83-->
with the same mask <math display="inline">\mathtt{z}_{\mathbf{c}}</math> used at all time steps. Note that unlike [<nowiki/>[[#bib.bib20|20]]], by viewing dropout as an operation over the weights our technique trivially extends to RNNs and GRUs.

<!--T:84-->
<div id="S4.SS2" class="section">

<!--T:85-->
<span id="word-embeddings-dropout"></span>

=== 4.2 Word Embeddings Dropout === <!--T:86-->

<!--T:87-->
In datasets with continuous inputs we often apply dropout to the input layer – i.e. to the input vector itself. This is equivalent to placing a distribution over the weight matrix which follows the input and approximately integrating over it (the matrix is optimised, therefore prone to overfitting otherwise).

<!--T:88-->
But for models with discrete inputs such as words (where every word is mapped to a continuous vector – a word embedding) this is seldom done. With word embeddings the input can be seen as either the word embedding itself, or, more conveniently, as a “one-hot” encoding (a vector of zeros with <math display="inline">1</math> at a single position). The product of the one-hot encoded vector with an embedding matrix <math display="inline">\mathbf{W}_{E} \in {\mathbb{R}}^{V \times D}</math> (where <math display="inline">D</math> is the embedding dimensionality and <math display="inline">V</math> is the number of words in the vocabulary) then gives a word embedding. Curiously, this parameter layer is the largest layer in most language applications, yet it is often not regularised. Since the embedding matrix is optimised it can lead to overfitting, and it is therefore desirable to apply dropout to the one-hot encoded vectors. This in effect is identical to dropping words at random throughout the input sentence, and can also be interpreted as encouraging the model to not “depend” on single words for its output.

<!--T:89-->
Note that as before, we randomly set rows of the matrix <math display="inline">\mathbf{W}_{E} \in {\mathbb{R}}^{V \times D}</math> to zero. Since we repeat the same mask at each time step, we drop the same words throughout the sequence – i.e. we drop word types at random rather than word tokens (as an example, the sentence “the dog and the cat” might become “— dog and — cat” or “the — and the cat”, but never “— dog and the cat”). A possible inefficiency implementing this is the requirement to sample <math display="inline">V</math> Bernoulli random variables, where <math display="inline">V</math> might be large. This can be solved by the observation that for sequences of length <math display="inline">T</math>, at most <math display="inline">T</math> embeddings could be dropped (other dropped embeddings have no effect on the model output). For <math display="inline">T \ll V</math> it is therefore more efficient to first map the words to the word embeddings, and only then to zero-out word embeddings based on their word type.

<!--T:90-->
<div id="S5" class="section">

<!--T:91-->
<span id="experimental-evaluation"></span>

== 5 Experimental Evaluation == <!--T:92-->

<!--T:93-->
We start by implementing our proposed dropout variant into the Torch implementation of Zaremba et al. [<nowiki/>[[#bib.bib4|4]]], that has become a reference implementation for many in the field. Zaremba et al. [<nowiki/>[[#bib.bib4|4]]] have set a benchmark on the Penn Treebank that to the best of our knowledge hasn’t been beaten for the past 2 years. We improve on [<nowiki/>[[#bib.bib4|4]]]’s results, and show that our dropout variant improves model performance compared to early-stopping and compared to using under-specified models. We continue to evaluate our proposed dropout variant with both LSTM and GRU models on a sentiment analysis task where labelled data is scarce. We finish by giving an in-depth analysis of the properties of the proposed method, with code and many experiments deferred to the appendix due to space constraints.

<!--T:94-->
<div id="S5.SS1" class="section">

<!--T:95-->
<span id="language-modelling"></span>

=== 5.1 Language Modelling === <!--T:96-->

<!--T:97-->
We replicate the language modelling experiment of Zaremba, Sutskever, and Vinyals [<nowiki/>[[#bib.bib4|4]]]. The experiment uses the Penn Treebank, a standard benchmark in the field. This dataset is considered a small one in the language processing community, with <math display="inline">887,521</math> tokens (words) in total, making overfitting a considerable concern. Throughout the experiments we refer to LSTMs with the dropout technique proposed following our Bayesian interpretation as Variational LSTMs, and refer to existing dropout techniques as naive dropout LSTMs (eq. ([[#S4.E8|8]]), different masks at different steps, applied to the input and output of the LSTM alone). We refer to LSTMs with no dropout as standard LSTMs.

<!--T:98-->
We implemented a Variational LSTM for both the medium model of [<nowiki/>[[#bib.bib4|4]]] (2 layers with 650 units in each layer) as well as their large model (2 layers with 1500 units in each layer). The only changes we’ve made to [<nowiki/>[[#bib.bib4|4]]]’s setting are 1) using our proposed dropout variant instead of naive dropout, and 2) tuning weight decay (which was chosen to be zero in [<nowiki/>[[#bib.bib4|4]]]). All other hyper-parameters are kept identical to [<nowiki/>[[#bib.bib4|4]]]: learning rate decay was not tuned for our setting and is used following [<nowiki/>[[#bib.bib4|4]]]. Dropout parameters were optimised with grid search (tying the dropout probability over the embeddings together with the one over the recurrent layers, and tying the dropout probability for the inputs and outputs together as well). These are chosen to minimise validation perplexity<sup>33</sup>3Optimal probabilities are 0.3 and 0.5 respectively for the large model, compared [<nowiki/>[[#bib.bib4|4]]]’s 0.6 dropout probability, and 0.2 and 0.35 respectively for the medium model, compared [<nowiki/>[[#bib.bib4|4]]]’s 0.5 dropout probability.. We further compared to Moon et al. [<nowiki/>[[#bib.bib20|20]]] who only drop elements in the LSTM internal state using the same mask at all time steps (in addition to performing dropout on the inputs and outputs, eq. ([[#S4.E9|9]])). We implemented their dropout variant with each model size, and repeated the procedure above to find optimal dropout probabilities (0.3 with the medium model, and 0.5 with the large model). We had to use early stopping for the large model with [<nowiki/>[[#bib.bib20|20]]]’s variant as the model starts overfitting after 16 epochs. Moon et al. [<nowiki/>[[#bib.bib20|20]]] proposed their dropout variant within the speech recognition community, where they did not have to consider embeddings overfitting (which, as we will see below, affect the recurrent layers considerably). We therefore performed an additional experiment using [<nowiki/>[[#bib.bib20|20]]]’s variant together with our embedding dropout (referred to as Moon et al. [<nowiki/>[[#bib.bib20|20]]]+emb dropout).

<!--T:99-->
Our results are given in table [[#S5.T1|1]]. For the variational LSTM we give results using both the tied weights model (eq. ([[#S4.E6|6]])–([[#S4.E7|7]]), Variational (tied weights)), and without weight tying (eq. ([[#S4.Ex8|4.1]]), Variational (untied weights)). For each model we report performance using both the standard dropout approximation (averaging the weights at test time – propagating the mean of each approximating distribution as input to the next layer), and using MC dropout (obtained by performing dropout at test time 1000 times, and averaging the model outputs following eq. ([[#S4.E4|4]]), denoted MC). For each model we report average perplexity and standard deviation (each experiment was repeated 3 times with different random seeds and the results were averaged). Model training time is given in words per second (WPS).

<!--T:100-->
It is interesting that using the dropout approximation, weight tying results in lower validation error and test error than the untied weights model. But with MC dropout the untied weights model performs much better. Validation perplexity for the large model is improved from [<nowiki/>[[#bib.bib4|4]]]’s <math display="inline">82.2</math> down to <math display="inline">77.3</math> (with weight tying), or <math display="inline">77.9</math> without weight tying. Test perplexity is reduced from <math display="inline">78.4</math> down to <math display="inline">73.4</math> (with MC dropout and untied weights). To the best of our knowledge, these are currently the best single model perplexities on the Penn Treebank.

<!--T:101-->
It seems that Moon et al. [<nowiki/>[[#bib.bib20|20]]] underperform even compared to [<nowiki/>[[#bib.bib4|4]]]. With no embedding dropout the large model overfits and early stopping is required (with no early stopping the model’s validation perplexity goes up to 131 within 30 epochs). Adding our embedding dropout, the model performs much better, but still underperforms compared to applying dropout on the inputs and outputs alone.

<!--T:102-->
Comparing our results to the non-regularised LSTM (evaluated with early stopping, giving similar performance as the early stopping experiment in [<nowiki/>[[#bib.bib4|4]]]) we see that for either model size an improvement can be obtained by using our dropout variant. Comparing the medium sized Variational model to the large one we see that a significant reduction in perplexity can be achieved by using a larger model. This cannot be done with the non-regularised LSTM, where a larger model leads to worse results. This shows that reducing the complexity of the model, a possible approach to avoid overfitting, actually leads to a worse fit when using dropout.

<!--T:103-->
<div id="S5.T1" class="figure">

<!--T:104-->
{| id="S5.T1.48" class="wikitable"
|- id="S5.T1.48.49.1"
! id="S5.T1.48.49.1.1" style="padding: 1pt 4.0pt"|
! id="S5.T1.48.49.1.2" colspan="3" style="padding: 1pt 4.0pt"| Medium LSTM
! id="S5.T1.48.49.1.3" colspan="3" style="padding: 1pt 4.0pt"| Large LSTM
|- id="S5.T1.48.50.2"
! id="S5.T1.48.50.2.1" style="padding: 1pt 4.0pt"|
! id="S5.T1.48.50.2.2" style="padding: 1pt 4.0pt"| Validation
! id="S5.T1.48.50.2.3" style="padding: 1pt 4.0pt"| Test
! id="S5.T1.48.50.2.4" style="padding: 1pt 4.0pt"| WPS
! id="S5.T1.48.50.2.5" style="padding: 1pt 4.0pt"| Validation
! id="S5.T1.48.50.2.6" style="padding: 1pt 4.0pt"| Test
! id="S5.T1.48.50.2.7" style="padding: 1pt 4.0pt"| WPS
|- id="S5.T1.6.6"
! id="S5.T1.6.6.7" style="padding: 1pt 4.0pt"| Non-regularized (early stopping)
| id="S5.T1.1.1.1" style="padding: 1pt 4.0pt"| <math display="inline">121.1</math>
| id="S5.T1.2.2.2" style="padding: 1pt 4.0pt"| <math display="inline">121.7</math>
| id="S5.T1.3.3.3" style="padding: 1pt 4.0pt"| <math display="inline">5.5</math>K
| id="S5.T1.4.4.4" style="padding: 1pt 4.0pt"| <math display="inline">128.3</math>
| id="S5.T1.5.5.5" style="padding: 1pt 4.0pt"| <math display="inline">127.4</math>
| id="S5.T1.6.6.6" style="padding: 1pt 4.0pt"| <math display="inline">2.5</math>K
|- id="S5.T1.12.12"
! id="S5.T1.12.12.7" style="padding: 1pt 4.0pt"| Moon et al. [<nowiki/>[[#bib.bib20|20]]]
| id="S5.T1.7.7.1" style="padding: 1pt 4.0pt"| <math display="inline">100.7</math>
| id="S5.T1.8.8.2" style="padding: 1pt 4.0pt"| <math display="inline">97.0</math>
| id="S5.T1.9.9.3" style="padding: 1pt 4.0pt"| <math display="inline">4.8</math>K
| id="S5.T1.10.10.4" style="padding: 1pt 4.0pt"| <math display="inline">122.9</math>
| id="S5.T1.11.11.5" style="padding: 1pt 4.0pt"| <math display="inline">118.7</math>
| id="S5.T1.12.12.6" style="padding: 1pt 4.0pt"| <math display="inline">3</math>K
|- id="S5.T1.18.18"
! id="S5.T1.18.18.7" style="padding: 1pt 4.0pt"| Moon et al. [<nowiki/>[[#bib.bib20|20]]] +emb dropout
| id="S5.T1.13.13.1" style="padding: 1pt 4.0pt"| <math display="inline">88.9</math>
| id="S5.T1.14.14.2" style="padding: 1pt 4.0pt"| <math display="inline">86.5</math>
| id="S5.T1.15.15.3" style="padding: 1pt 4.0pt"| <math display="inline">4.8</math>K
| id="S5.T1.16.16.4" style="padding: 1pt 4.0pt"| <math display="inline">88.8</math>
| id="S5.T1.17.17.5" style="padding: 1pt 4.0pt"| <math display="inline">86.0</math>
| id="S5.T1.18.18.6" style="padding: 1pt 4.0pt"| <math display="inline">3</math>K
|- id="S5.T1.24.24"
! id="S5.T1.24.24.7" style="padding: 1pt 4.0pt"| Zaremba et al. [<nowiki/>[[#bib.bib4|4]]]
| id="S5.T1.19.19.1" style="padding: 1pt 4.0pt"| <math display="inline">86.2</math>
| id="S5.T1.20.20.2" style="padding: 1pt 4.0pt"| <math display="inline">82.7</math>
| id="S5.T1.21.21.3" style="padding: 1pt 4.0pt"| <math display="inline">5.5</math>K
| id="S5.T1.22.22.4" style="padding: 1pt 4.0pt"| <math display="inline">82.2</math>
| id="S5.T1.23.23.5" style="padding: 1pt 4.0pt"| <math display="inline">78.4</math>
| id="S5.T1.24.24.6" style="padding: 1pt 4.0pt"| <math display="inline">2.5</math>K
|- id="S5.T1.30.30"
! id="S5.T1.30.30.7" style="padding: 1pt 4.0pt"| Variational (tied weights)
| id="S5.T1.25.25.1" style="padding: 1pt 4.0pt"| <math display="inline">81.8 \pm 0.2</math>
| id="S5.T1.26.26.2" style="padding: 1pt 4.0pt"| <math display="inline">79.7 \pm 0.1</math>
| id="S5.T1.27.27.3" style="padding: 1pt 4.0pt"| <math display="inline">4.7</math>K
| id="S5.T1.28.28.4" style="padding: 1pt 4.0pt"| <math display="inline">77.3 \pm 0.2</math>
| id="S5.T1.29.29.5" style="padding: 1pt 4.0pt"| <math display="inline">75.0 \pm 0.1</math>
| id="S5.T1.30.30.6" style="padding: 1pt 4.0pt"| <math display="inline">2.4</math>K
|- id="S5.T1.36.36"
! id="S5.T1.36.36.7" style="padding: 1pt 4.0pt"| Variational (tied weights, MC)
| id="S5.T1.31.31.1" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.32.32.2" style="padding: 1pt 4.0pt"| <math display="inline">79.0 \pm 0.1</math>
| id="S5.T1.33.33.3" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.34.34.4" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.35.35.5" style="padding: 1pt 4.0pt"| <math display="inline">74.1 \pm 0.0</math>
| id="S5.T1.36.36.6" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
|- id="S5.T1.42.42"
! id="S5.T1.42.42.7" style="padding: 1pt 4.0pt"| Variational (untied weights)
| id="S5.T1.37.37.1" style="padding: 1pt 4.0pt"| <math display="inline">81.9 \pm 0.2</math>
| id="S5.T1.38.38.2" style="padding: 1pt 4.0pt"| <math display="inline">79.7 \pm 0.1</math>
| id="S5.T1.39.39.3" style="padding: 1pt 4.0pt"| <math display="inline">2.7</math>K
| id="S5.T1.40.40.4" style="padding: 1pt 4.0pt"| <math display="inline">77.9 \pm 0.3</math>
| id="S5.T1.41.41.5" style="padding: 1pt 4.0pt"| <math display="inline">75.2 \pm 0.2</math>
| id="S5.T1.42.42.6" style="padding: 1pt 4.0pt"| <math display="inline">1.6</math>K
|- id="S5.T1.48.48"
! id="S5.T1.48.48.7" style="padding: 1pt 4.0pt"| Variational (untied weights, MC)
| id="S5.T1.43.43.1" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.44.44.2" style="padding: 1pt 4.0pt"| <math display="inline">78.6 \pm 0.1</math>
| id="S5.T1.45.45.3" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.46.46.4" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
| id="S5.T1.47.47.5" style="padding: 1pt 4.0pt"| <math display="inline">73.4 \pm 0.0</math>
| id="S5.T1.48.48.6" style="padding: 1pt 4.0pt"| <math display="inline">-</math>
|}

<!--T:105-->
We also see that the tied weights model achieves very close performance to that of the untied weights one when using the dropout approximation. Assessing model run time though (on a Titan X GPU), we see that tying the weights results in a more time-efficient implementation. This is because the single matrix product is implemented as a single GPU kernel, instead of the four smaller matrix products used in the untied weights model (where four GPU kernels are called sequentially). Note though that a low level implementation should give similar run times.

<!--T:106-->
We further experimented with a model averaging experiment following [<nowiki/>[[#bib.bib4|4]]]’s setting, where several large models are trained independently with their outputs averaged. We used Variational LSTMs with MC dropout following the setup above. Using 10 Variational LSTMs we improve [<nowiki/>[[#bib.bib4|4]]]’s test set perplexity from <math display="inline">69.5</math> to <math display="inline">68.7</math> – obtaining identical perplexity to [<nowiki/>[[#bib.bib4|4]]]’s experiment with 38 models.

<!--T:107-->
Lastly, we report validation perplexity with reduced learning rate decay (with the medium model). Learning rate decay is often used for regularisation by setting the optimiser to make smaller steps when the model starts overfitting (as done in [<nowiki/>[[#bib.bib4|4]]]). By removing it we can assess the regularisation effects of dropout alone. As can be seen in fig. [[#S5.F2|2]], even with early stopping, Variational LSTM achieves lower perplexity than naive dropout LSTM and standard LSTM. Note though that a significantly lower perplexity for all models can be achieved with learning rate decay scheduling as seen in table [[#S5.T1|1]]

<!--T:108-->
<div id="S5.F2" class="figure">

<!--T:109-->
[[File:Arxiv_1512_05287_lua_exp8_val.png|598x129px|Refer to caption]]

<!--T:110-->
<div id="S5.SS2" class="section">

<!--T:111-->
<span id="sentiment-analysis"></span>

=== 5.2 Sentiment Analysis === <!--T:112-->

<!--T:113-->
We next evaluate our dropout variant with both LSTM and GRU models on a sentiment analysis task, where labelled data is scarce. We use MC dropout (which we compare to the dropout approximation further in appendix [[#A2|B]]), and untied weights model parametrisations.

<!--T:114-->
We use the raw Cornell film reviews corpus collected by Pang and Lee [<nowiki/>[[#bib.bib27|27]]]. The dataset is composed of 5000 film reviews. We extract consecutive segments of <math display="inline">T</math> words from each review for <math display="inline">T = 200</math>, and use the corresponding film score as the observed output <math display="inline">y</math>. The model is built from one embedding layer (of dimensionality <math display="inline">128</math>), one LSTM layer (with <math display="inline">128</math> network units for each gate; GRU setting is built similarly), and finally a fully connected layer applied to the last output of the LSTM (resulting in a scalar output). We use the Adam optimiser [<nowiki/>[[#bib.bib28|28]]] throughout the experiments, with batch size 128, and MC dropout at test time with 10 samples.

<!--T:115-->
The main results can be seen in fig. [[#S5.F3|3]]. We compared Variational LSTM (with our dropout variant applied with each weight layer) to standard techniques in the field. Training error is shown in fig. [[#S5.F3.sf1|3(a)]] and test error is shown in fig. [[#S5.F3.sf2|3(b)]]. Optimal dropout probabilities and weight decay were used for each model (see appendix [[#A2|B]]). It seems that the only model not to overfit is the Variational LSTM, which achieves lowest test error as well. Variational GRU test error is shown in fig. [[#A2.F15|15]] (with loss plot given in appendix [[#A2|B]]). Optimal dropout probabilities and weight decay were used again for each model. Variational GRU avoids overfitting to the data and converges to the lowest test error. Early stopping in this dataset will result in smaller test error though (lowest test error is obtained by the non-regularised GRU model at the second epoch). It is interesting to note that standard techniques exhibit peculiar behaviour where test error repeatedly decreases and increases. This behaviour is not observed with the Variational GRU. Convergence plots of the loss for each model are given in appendix [[#A2|B]].

<!--T:116-->
<div id="S5.F3" class="figure">

<!--T:117-->
<div id="S5.F3.sf1" class="figure">

<!--T:118-->
[[File:Arxiv_1512_05287_exp1_p_025_1250_rmse_train.png|598x406px|Refer to caption]]

<!--T:119-->
<div id="S5.F3.sf2" class="figure">

<!--T:120-->
[[File:Arxiv_1512_05287_exp1_p_025_1250_rmse.png|598x406px|Refer to caption]]

<!--T:121-->
<div id="S5.F3.sf3" class="figure">

<!--T:122-->
[[File:Arxiv_1512_05287_exp10_rmse_GRU.png|598x399px|Refer to caption]]

<!--T:123-->
<div id="S5.F4" class="figure">

<!--T:124-->
<div id="S5.F4.sf1" class="figure">

<!--T:125-->
[[File:Arxiv_1512_05287_exp4_rmse.png|598x406px|Refer to caption]]

<!--T:126-->
<div id="S5.F4.sf2" class="figure">

<!--T:127-->
[[File:Arxiv_1512_05287_exp7_rmse.png|598x406px|Refer to caption]]

<!--T:128-->
<div id="S5.F4.sf3" class="figure">

<!--T:129-->
[[File:Arxiv_1512_05287_exp6_rmse.png|598x391px|Refer to caption]]

<!--T:130-->
We next explore the effects of dropping-out different parts of the model. We assessed our Variational LSTM with different combinations of dropout over the embeddings (<math display="inline">p_{E} = {0,0.5}</math>) and recurrent layers (<math display="inline">p_{U} = {0,0.5}</math>) on the sentiment analysis task. The convergence plots can be seen in figure [[#S5.F4.sf1|4(a)]]. It seems that without both strong embeddings regularisation and strong regularisation over the recurrent layers the model would overfit rather quickly. The behaviour when <math display="inline">p_{U} = 0.5</math> and <math display="inline">p_{E} = 0</math> is quite interesting: test error decreases and then increases before decreasing again. Also, it seems that when <math display="inline">p_{U} = 0</math> and <math display="inline">p_{E} = 0.5</math> the model becomes very erratic.

<!--T:131-->
Lastly, we tested the performance of Variational LSTM with different recurrent layer dropout probabilities, fixing the embedding dropout probability at either <math display="inline">p_{E} = 0</math> or <math display="inline">p_{E} = 0.5</math> (figs. [[#S5.F4.sf2|4(b)]]-[[#S5.F4.sf3|4(c)]]). These results are rather intriguing. In this experiment all models have converged, with the loss getting near zero (not shown). Yet it seems that with no embedding dropout, a higher dropout probability within the recurrent layers leads to overfitting! This presumably happens because of the large number of parameters in the embedding layer which is not regularised. Regularising the embedding layer with dropout probability <math display="inline">p_{E} = 0.5</math> we see that a higher recurrent layer dropout probability indeed leads to increased robustness to overfitting, as expected. These results suggest that embedding dropout can be of crucial importance in some tasks.

<!--T:132-->
In appendix [[#A2|B]] we assess the importance of weight decay with our dropout variant. Common practice is to remove weight decay with naive dropout. Our results suggest that weight decay plays an important role with our variant (it corresponds to our prior belief of the distribution over the weights).

<!--T:133-->
<div id="S6" class="section">

<!--T:134-->
<span id="conclusions"></span>

== 6 Conclusions == <!--T:135-->

<!--T:136-->
We presented a new technique for recurrent neural network regularisation. Our RNN dropout variant is theoretically motivated and its effectiveness was empirically demonstrated. In future research we aim to assess model uncertainty in Variational LSTMs [<nowiki/>[[#bib.bib17|17]]]. Together with the developments presented here, this will have important implications for modelling language ambiguity and modelling dynamics in control tasks.

<!--T:137-->
<div id="bib" class="section">

<!--T:138-->
<span id="references"></span>

== References == <!--T:139-->

<!--T:140-->
* <span id="bib.bib1">Sundermeyer et al. [2012] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In ''INTERSPEECH'', 2012.</span>
* <span id="bib.bib2">Kalchbrenner and Blunsom [2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In ''EMNLP'', 2013.</span>
* <span id="bib.bib3">Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In ''NIPS'', 2014.</span>
* <span id="bib.bib4">Zaremba et al. [2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. ''arXiv preprint arXiv:1409.2329'', 2014.</span>
* <span id="bib.bib5">Hinton [2012] Geoffrey E others Hinton. Improving neural networks by preventing co-adaptation of feature detectors. ''arXiv preprint arXiv:1207.0580'', 2012.</span>
* <span id="bib.bib6">Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. ''JMLR'', 2014.</span>
* <span id="bib.bib7">Pachitariu and Sahani [2013] Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? ''arXiv preprint arXiv:1301.5650'', 2013.</span>
* <span id="bib.bib8">Bayer et al. [2013] Justin Bayer, Christian Osendorfer, Daniela Korhammer, Nutan Chen, Sebastian Urban, and Patrick van der Smagt. On fast dropout and its applicability to recurrent networks. ''arXiv preprint arXiv:1311.0701'', 2013.</span>
* <span id="bib.bib9">Pham et al. [2014] Vu Pham, Theodore Bluche, Christopher Kermorvant, and Jerome Louradour. Dropout improves recurrent neural networks for handwriting recognition. In ''ICFHR''. IEEE, 2014.</span>
* <span id="bib.bib10">Bluche et al. [2015] Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Where to apply dropout in recurrent neural networks for handwriting recognition? In ''ICDAR''. IEEE, 2015.</span>
* <span id="bib.bib11">Rezende et al. [2014] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ''ICML'', 2014.</span>
* <span id="bib.bib12">Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ''ICML'', 2015.</span>
* <span id="bib.bib13">Hernandez-Lobato and Adams [2015] Jose Miguel Hernandez-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In ''ICML'', 2015.</span>
* <span id="bib.bib14">Gal and Ghahramani [2015a] Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference. ''arXiv:1506.02158'', 2015a.</span>
* <span id="bib.bib15">Kingma et al. [2015] Diederik Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In ''NIPS''. Curran Associates, Inc., 2015.</span>
* <span id="bib.bib16">Korattikara Balan et al. [2015] Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowledge. In ''NIPS''. Curran Associates, Inc., 2015.</span>
* <span id="bib.bib17">Gal and Ghahramani [2015b] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ''arXiv:1506.02142'', 2015b.</span>
* <span id="bib.bib18">Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. ''Neural computation'', 9(8), 1997.</span>
* <span id="bib.bib19">Cho et al. [2014] Kyunghyun Cho et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In ''EMNLP'', Doha, Qatar, October 2014. ACL.</span>
* <span id="bib.bib20">Moon et al. [December 2015] Taesup Moon, Heeyoul Choi, Hoshik Lee, and Inchul Song. RnnDrop: A Novel Dropout for RNNs in ASR. In ''ASRU Workshop'', December 2015.</span>
* <span id="bib.bib21">MacKay [1992] David JC MacKay. A practical Bayesian framework for backpropagation networks. ''Neural computation'', 4(3):448–472, 1992.</span>
* <span id="bib.bib22">Neal [1995] Radford M Neal. ''Bayesian learning for neural networks''. PhD thesis, University of Toronto, 1995.</span>
* <span id="bib.bib23">Hinton and Van Camp [1993] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In ''COLT'', pages 5–13. ACM, 1993.</span>
* <span id="bib.bib24">Barber and Bishop [1998] David Barber and Christopher M Bishop. Ensemble learning in Bayesian neural networks. ''NATO ASI SERIES F COMPUTER AND SYSTEMS SCIENCES'', 168:215–238, 1998.</span>
* <span id="bib.bib25">Graves [2011] Alex Graves. Practical variational inference for neural networks. In ''NIPS'', 2011.</span>
* <span id="bib.bib26">Graves et al. [2013] Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ''ICASSP''. IEEE, 2013.</span>
* <span id="bib.bib27">Pang and Lee [2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ''ACL''. ACL, 2005.</span>
* <span id="bib.bib28">Kingma and Ba [2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ''arXiv preprint arXiv:1412.6980'', 2014.</span>
* <span id="bib.bib29">Bergstra et al. [2010] James Bergstra et al. Theano: a CPU and GPU math expression compiler. In ''Proceedings of the Python for Scientific Computing Conference (SciPy)'', June 2010. Oral Presentation.</span>
* <span id="bib.bib30">fchollet [2015] fchollet. Keras. https://github.com/fchollet/keras, 2015.</span>

<!--T:141-->
<div id="A1" class="section">

<!--T:142-->
<span id="appendix-a-bayesian-versus-ensembling-interpretation-of-dropout"></span>

== Appendix A Bayesian versus ensembling interpretation of dropout == <!--T:143-->

<!--T:144-->
Apart from our Bayesian approximation interpretation, dropout in deep networks can also be seen as following an ensembling interpretation [<nowiki/>[[#bib.bib6|6]]]. This interpretation also leads to MC dropout at test time. But the ensembling interpretation does not determine whether the ensemble should be over the network units or the weights. For example, in an RNN this view will not lead to our dropout variant, unless the ensemble is defined to tie the weights of the network ad hoc. This is in comparison to the Bayesian approximation view where the weight tying is forced by the probabilistic interpretation of the model.

<!--T:145-->
<div id="A2" class="section">

<!--T:146-->
<span id="appendix-b-sentiment-analysis-further-experiments"></span>

== Appendix B Sentiment analysis – further experiments == <!--T:147-->

<!--T:148-->
Sentiment analysis hyper-parameters were obtained by evaluating each model with dropout probabilities <math display="inline">0.25</math> and <math display="inline">0.5</math>, and weight decays ranging from <math display="inline">10^{- 6}</math> to <math display="inline">10^{- 4}</math>. The optimal setting for Variational LSTM is dropout probabilities <math display="inline">0.25</math> and weight decay <math display="inline">10^{- 3}</math>, and for naive dropout LSTM the dropout probabilities are <math display="inline">0.5</math> (no weight decay is used in reference implementations of naive dropout LSTM [<nowiki/>[[#bib.bib4|4]]]).

<!--T:149-->
We assess the dropout approximation in Variational LSTMs. The dropout approximation is often used in deep networks as means of approximating the MC estimate. In the approximation we replace each weight matrix <math display="inline">\mathbf{M}</math> by <math display="inline">p\hspace{0pt}\mathbf{M}</math> where <math display="inline">p</math> is the dropout probability, and perform a deterministic pass through the network (without dropping out units). This can be seen as propagating the mean of the random variables <math display="inline">\mathbf{W}</math> through the network [<nowiki/>[[#bib.bib17|17]]]. The approximation has been shown to work well for deep networks [<nowiki/>[[#bib.bib6|6]]], yet it fails with convolution layers [<nowiki/>[[#bib.bib14|14]]]. We assess the approximation empirically with our Variational LSTM model, repeating the first experiment with the approximation used at test time instead of MC dropout. The results can be seen in fig. [[#A2.F9|9]]. It seems that the approximation gives a good estimate to the test error, similar to the results in figure [[#S5.F4.sf1|4(a)]].

<!--T:150-->
We further tested the Variational LSTM model with different weight decays, observing the effects of different values for these. Note that weight decay is applied to all layers, including the embedding layer. In figure [[#A2.F7|7]] we can see that higher weight decay values result in lower test error, with significant differences for different weight decays. This suggests that weight decay still plays an important role even when using dropout (whereas common practice is to remove weight decay with naive dropout). Note also that the weight decay can be optimised (together with the dropout parameters) as part of the variational approximation. This is not done in practice in the deep learning community, where grid-search or Bayesian optimisation are often used for these instead.

<!--T:151-->
Testing the Variational LSTM with different sequence lengths (with sequences of lengths <math display="inline">T = {20,50,200,400}</math>) we can see that sequence length has a strong effect on model performance as well (fig. [[#A2.F7|7]]). Longer sequences result in much better performance but with the price of longer convergence time. We hypothesised that the diminished performance on shorter sequences is caused by the high dropout probability on the embeddings. But a follow-up experiment with sequence lengths 50 and 200, and different embedding dropout probabilities, shows that lower dropout probabilities result in even worse model performance (figures [[#A2.F9|9]] and [[#A2.F7|7]]).

<!--T:152-->
<div id="A2.F7" class="figure">

<!--T:153-->
<div id="A2.F7.5" class="figure" style="width:130.1pt;">

<!--T:154-->
[[File:Arxiv_1512_05287_exp8_rmse.png|598x400px|Refer to caption]]

<!--T:155-->
<div id="A2.F7.6" class="figure" style="width:130.1pt;">

<!--T:156-->
[[File:Arxiv_1512_05287_exp2_rmse.png|598x391px|Refer to caption]]

<!--T:157-->
<div id="A2.F7.9" class="figure" style="width:130.1pt;">

<!--T:158-->
[[File:Arxiv_1512_05287_exp5_1e-6_rmse.png|598x400px|Refer to caption]]

<!--T:159-->
<div id="A2.F9" class="figure">

<!--T:160-->
<div id="A2.F9.1" class="figure" style="width:130.1pt;">

<!--T:161-->
[[File:Arxiv_1512_05287_exp9_rmse.png|598x400px|Refer to caption]]

<!--T:162-->
<div id="A2.F9.2" class="figure" style="width:130.1pt;">

<!--T:163-->
[[File:Arxiv_1512_05287_exp4_rmse_approx.png|598x406px|Refer to caption]]

<!--T:164-->
In fig. [[#A2.F10.sf1|10(a)]] we see how different dropout probabilities and weight decays affect GRU model performance.

<!--T:165-->
<div id="A2.F10" class="figure">

<!--T:166-->
<div id="A2.F10.sf1" class="figure">

<!--T:167-->
[[File:Arxiv_1512_05287_exp11_rmse_GRU.png|598x399px|Refer to caption]]

<!--T:168-->
We compare naive dropout LSTM to Variational LSTM with dropout probability in the recurrent layers set to zero: <math display="inline">p_{U} = 0</math> (referred to as dropout LSTM). Both models apply dropout to the input and outputs of the LSTM alone, with no dropout applied to the embeddings. Naive dropout LSTM uses different masks at different time steps though, tied across the gates, whereas dropout LSTM uses the same mask at different time steps. The test error for both models can be seen in fig. [[#A2.F12|12]]. It seems that without dropout over the recurrent layers and embeddings both models overfit, and in fact result in identical performance.

<!--T:169-->
Next, we assess the dropout approximation in the GRU model. The approximation seems to give similar results to MC dropout in the GRU model (fig. [[#A2.F12|12]]).

<!--T:170-->
<div id="A2.F12" class="figure">

<!--T:171-->
<div id="A2.F12.1" class="figure" style="width:212.5pt;">

<!--T:172-->
[[File:Arxiv_1512_05287_exp3_rmse.png|598x406px|Refer to caption]]

<!--T:173-->
<div id="A2.F12.2" class="figure" style="width:212.5pt;">

<!--T:174-->
[[File:Arxiv_1512_05287_exp10_rmse_approx_GRU.png|598x399px|Refer to caption]]

<!--T:175-->
Lastly, we plot the train loss for various models from the main body of the paper. All models have converged, with a stable train loss.

<!--T:176-->
<div id="A2.F15" class="figure">

<!--T:177-->
<div id="A2.F15.1" class="figure" style="width:138.8pt;">

<!--T:178-->
[[File:Arxiv_1512_05287_exp1_p_025_1250_loss.png|598x403px|Refer to caption]]

<!--T:179-->
<div id="A2.F15.2" class="figure" style="width:138.8pt;">

<!--T:180-->
[[File:Arxiv_1512_05287_exp10_loss_GRU.png|598x403px|Refer to caption]]

<!--T:181-->
<div id="A2.F15.3" class="figure" style="width:138.8pt;">

<!--T:182-->
[[File:Arxiv_1512_05287_exp4_loss.png|598x403px|Refer to caption]]

<!--T:183-->
<div id="A3" class="section">

<!--T:184-->
<span id="appendix-c-code"></span>

== Appendix C Code == <!--T:185-->

<!--T:186-->
An efficient Theano [<nowiki/>[[#bib.bib29|29]]] implementation of the method above into Keras [<nowiki/>[[#bib.bib30|30]]] is as simple as:

<!--T:187-->
<pre>def get_output(self, train=False):
    X = self.get_input(train)

<!--T:188-->
    retain_prob_W = 1. - self.p_W[0]
    retain_prob_U = 1. - self.p_U[0]
    B_W = self.srng.binomial((4, X.shape[1], self.input_dim),
        p=retain_prob_W, dtype=theano.config.floatX)
    B_U = self.srng.binomial((4, X.shape[1], self.output_dim),
        p=retain_prob_U, dtype=theano.config.floatX)

<!--T:189-->
    xi = T.dot(X * B_W[0], self.W_i) + self.b_i
    xf = T.dot(X * B_W[1], self.W_f) + self.b_f
    xc = T.dot(X * B_W[2], self.W_c) + self.b_c
    xo = T.dot(X * B_W[3], self.W_o) + self.b_o

<!--T:190-->
    [outputs, memories], updates = theano.scan(
        self._step,
        sequences=[xi, xf, xo, xc],
        outputs_info=[
            T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
            T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1)
        ],
        non_sequences=[self.U_i, self.U_f, self.U_o, self.U_c, B_U],
        truncate_gradient=self.truncate_gradient)

<!--T:191-->
    return outputs[-1]

<!--T:192-->
def _step(self,
          xi_t, xf_t, xo_t, xc_t,
          h_tm1, c_tm1,
          u_i, u_f, u_o, u_c, B_U):
    i_t = self.inner_activation(xi_t + T.dot(h_tm1 * B_U[0], u_i))
    f_t = self.inner_activation(xf_t + T.dot(h_tm1 * B_U[1], u_f))
    c_t = f_t * c_tm1 + i_t * self.activation(xc_t + T.dot(h_tm1 * B_U[2], u_c))
    o_t = self.inner_activation(xo_t + T.dot(h_tm1 * B_U[3], u_o))
    h_t = o_t * self.activation(c_t)
    return h_t, c_t</pre>
</translate>

[[Category:Machine Learning]]
[[Category:Research]]
[[Category:Research Papers]]