A Theoretically Grounded Application of Dropout in Recurrent Neural Networks/paper/zh

SummarySource

Research Paper
Authors	Yarin Gal; Zoubin Ghahramani
Year	2015
Topic area	Machine Learning
Difficulty	Research
arXiv	1512.05287
PDF	Download PDF

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

&Yarin Gal &
University of Cambridge
{yg279,zg201}@cam.ac.uk &Zoubin Ghahramani

摘要

循环神经网络(RNN)位于深度学习诸多最新进展的最前沿。然而,这类模型的一大难题是容易过拟合,而 dropout 在应用于循环层时已被证明失败。贝叶斯建模与深度学习交叉领域的近期成果为 dropout 等常见深度学习技术提供了贝叶斯解释。将 dropout 立足于近似贝叶斯推断,意味着可以对其理论结果加以延伸,从而为在 RNN 模型中使用 dropout 提供新见解。我们将这种基于变分推断的新 dropout 技术应用于 LSTM 与 GRU 模型,并在语言建模与情感分析任务上加以评估。新方法优于现有技术,据我们所知,在 Penn Treebank 上的单模型语言建模性能上达到了新的最优水平(测试困惑度 73.4),从而扩充了深度学习中的变分工具箱。

1 引言

循环神经网络(RNN)是基于序列的模型,对自然语言理解、语言生成、视频处理及许多其他任务都至关重要 [1, 2, 3]。该模型的输入是符号序列,在每个时间步,一个简单的神经网络(RNN 单元)同时作用于单个输入符号和上一时间步的网络输出。RNN 是性能很强的模型,在许多任务上表现卓越,但很容易过拟合。RNN 模型缺乏正则化使其难以处理小规模数据,为避免过拟合,研究者通常采用早停或使用较小且欠表达能力的模型 [4]。

dropout 是深度网络中一种流行的正则化技术 [5, 6],在训练过程中随机将网络单元置零(丢弃)。但该技术从未在 RNN 上成功应用。实验结果使许多人相信,在循环层(RNN 单元之间的连接)上添加的噪声会在长序列中被放大并淹没信号 [4]。因此,既有研究得出结论:该技术只应施加在 RNN 的输入和输出上 [4, 7, 8, 9, 10]。但这种做法仍会导致过拟合,我们在实验中证实了这一点。

贝叶斯研究与深度学习交叉领域的近期成果,为常见的深度学习技术提供了贝叶斯视角下的解释 [11, 12, 13, 14, 15, 16]。这种贝叶斯视角使该领域涌现出新的方法,例如从深度学习网络中获得有原则的不确定性估计的方法 [14, 17]。例如,Gal 与 Ghahramani [14] 证明,dropout 可以解释为对贝叶斯神经网络(NN)后验的一种变分近似;他们的变分近似分布是两高斯的混合,方差较小,其中一个高斯的均值固定为零。把 dropout 立足于近似贝叶斯推断这一事实表明,把上述理论结果加以延伸,可能为在 RNN 模型中使用该技术提供新的见解。

本文关注该领域中常见的 RNN 模型(LSTM [18]、GRU [19]),并将它们视为概率模型,即将网络权重视为随机变量、并适当定义似然函数。随后,我们在这些概率贝叶斯模型(我们称之为变分 RNN)中进行近似变分推断。用混合高斯(其中一个分量固定为零、方差较小)来近似权重的后验分布,可得到一个可处理的优化目标。优化该目标等价于在相应 RNN 中执行一种新的 dropout 变体。

[[File:data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjM0Ny4yOSIgaWQ9IlMxLkYxLnNmMS5waWMxIiBvdmVyZmxvdz0idmlzaWJsZSIgdmVyc2lvbj0iMS4xIiB3aWR0aD0iMjYzLjA5Ij48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsMzQ3LjI5KSBtYXRyaXgoMSAwIDAgLTEgMCAwKSB0cmFuc2xhdGUoMTMwLjY4LDApIHRyYW5zbGF0ZSgwLDI4Ljg1KSI+PGcgc3Ryb2tlLXdpZHRoPSIwLjRwdCI+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTExLjcyIC0xMC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuMzIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgOS40NikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTIuMTUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjMuNDUiPlxwZ2ZtYXRocmVzdWx0cHQ8L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgMTguMTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSA2LjM3IDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguMzQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTAuNzEiPjxtYXRoIGFsdHRleHQ9Inhfe3R9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPng8L21pPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+dDwvbWk+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxjaSBpZD0iUzEuRjEuc2YxLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+8J2RoTwvY2k+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dH08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNC42MSA1OS43NiBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAtNC42MSAxNTMuMjYgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC01LjA0IDI1MS4zNykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuNjk1KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDQ1LjMzKSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAwKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjY1IiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEwLjA4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0fSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj55PC9taT48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPnQ8L21pPjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPvCdkaE8L2NpPjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjEucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnlfe3R9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04My4wMiAtMjMuOTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDQzLjU5NSkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC40NSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxNy4xNiI+PG1hdGggYWx0dGV4dD0ieF97dC0xfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+4oiSPC9tbz48bW4gaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PG1pbnVzIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9taW51cz48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIj7wnZGhPC9jaT48Y24gaWQ9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIj4xPC9jbj48L2FwcGx5PjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjEucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnhfe3QtMX08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNzkuMDUgNzMuNDMgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gLTc5LjA1IDE2Ni45MyBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjcgMjY1LjA0KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My42OTUpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguNjUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTYuNTQiPjxtYXRoIGFsdHRleHQ9Inlfe3QtMX0iIGRpc3BsYXk9ImlubGluZSIgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtc3ViIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCI+eTwvbWk+PG1yb3cgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiPnQ8L21pPjxtbyBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiPuKIkjwvbW8+PG1uIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxtaW51cyBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiPjwvbWludXM+PGNpIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjEucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YxLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0LTF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDY1Ljg1IC0yNC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuOTIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjkuMSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIyMC42MSI+PG1hdGggYWx0dGV4dD0ieF97dCsxfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+KzwvbW8+PG1uIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGlPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxwbHVzIGlkPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9wbHVzPjxjaSBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiPvCdkaE8L2NpPjxjbiBpZD0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMiPjE8L2NuPjwvYXBwbHk+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dCsxfTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PC9nPjwvZz48cGF0aCBkPSJNIDcxLjU0IDczLjc1IGggOS4yMiB2IDQ4LjU5IGggLTkuMjIgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDcxLjU0IDE2Ny4yNSBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgNjYuMTcgMjY1LjM2KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My45MikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOS4xIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjE5Ljk4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0KzF9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPnk8L21pPjxtcm93IGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIj50PC9taT48bW8gaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIj4rPC9tbz48bW4gaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaY8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PHBsdXMgaWQ9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIj48L3BsdXM+PGNpIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjEucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YxLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0KzF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwMDBGRiI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMzEuMjkgTCAtNzQuNDMgNjguOTkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAtNzQuNDMgNjguOTkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTcuNjIgTCAwIDU1LjMyIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgMCA1NS4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiM4MDgwODAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMzEuNjEgTCA3Ni4xNiA2OS4zMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIDc2LjE2IDY5LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwRkYwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMTI0Ljc5IEwgLTc0LjQzIDE2Mi40OSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAxNjIuNDkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTExLjEyIEwgMCAxNDguODIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDE0OC44MikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRkZGMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMTI1LjExIEwgNzYuMTYgMTYyLjgxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMTYyLjgxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGMDAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMjE4LjI5IEwgLTc0LjQzIDI1NS45OCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAyNTUuOTgpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDBGRkZGIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMjA0LjYyIEwgMCAyNDIuMzIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDI0Mi4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRjAwRkYiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMjE4LjYxIEwgNzYuMTYgMjU2LjMxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMjU2LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9IjMuMHB0LDMuMHB0IiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiPjxwYXRoIGQ9Ik0gLTEyMS4xOCA5Ny43MyBMIC04Mi40NSA5Ny43MyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjQ1IDk3LjczKSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSAtNjcuMDUgOTYuMzcgTCAtOC4wMSA4NS41MyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU1IC0wLjE4MDYzIDAuMTgwNjMgMC45ODM1NSAtOC4wMSA4NS41MykiPjxwYXRoIGQ9Ik0gLTEuNjYgMi4yMSBDIC0xLjUyIDEuMzggMCAwLjE0IDAuNDIgMCBDIDAgLTAuMTQgLTEuNTIgLTEuMzggLTEuNjYgLTIuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9IjMuMHB0LDMuMHB0IiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiPjxwYXRoIGQ9Ik0gNy4zOCA4NS40MSBMIDY4LjE1IDk2LjU4IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIwLjMycHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTQgMC4xODA3MyAtMC4xODA3MyAwLjk4MzU0IDY4LjE1IDk2LjU4KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSA4My41MyA5OC4wNSBMIDEyMi4yNyA5OC4wNSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTIyLjI3IDk4LjA1KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSAtMTIxLjE4IDE5MS4yMiBMIC04Mi40NSAxOTEuMjIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjAuMzJwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04Mi40NSAxOTEuMjIpIj48cGF0aCBkPSJNIC0xLjY2IDIuMjEgQyAtMS41MiAxLjM4IDAgMC4xNCAwLjQyIDAgQyAwIC0wLjE0IC0xLjUyIC0xLjM4IC0xLjY2IC0yLjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSIzLjBwdCwzLjBwdCIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0Ij48cGF0aCBkPSJNIC02Ny4wNSAxODkuODcgTCAtOC4wMSAxNzkuMDMiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjAuMzJwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC45ODM1NSAtMC4xODA2MyAwLjE4MDYzIDAuOTgzNTUgLTguMDEgMTc5LjAzKSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0iMy4wcHQsMy4wcHQiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCI+PHBhdGggZD0iTSA3LjM4IDE3OC45MSBMIDY4LjE1IDE5MC4wOCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMC4zMnB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU0IDAuMTgwNzMgLTAuMTgwNzMgMC45ODM1NCA2OC4xNSAxOTAuMDgpIj48cGF0aCBkPSJNIC0xLjY2IDIuMjEgQyAtMS41MiAxLjM4IDAgMC4xNCAwLjQyIDAgQyAwIC0wLjE0IC0xLjUyIC0xLjM4IC0xLjY2IC0yLjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSIzLjBwdCwzLjBwdCIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0Ij48cGF0aCBkPSJNIDgzLjUzIDE5MS41NSBMIDEyMi4yNyAxOTEuNTUiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIwLjMycHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAxMjIuMjcgMTkxLjU1KSI+PHBhdGggZD0iTSAtMS42NiAyLjIxIEMgLTEuNTIgMS4zOCAwIDAuMTQgMC40MiAwIEMgMCAtMC4xNCAtMS41MiAtMS4zOCAtMS42NiAtMi4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjwvc3ZnPg==]]

[[File:data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjM0Ny4yOSIgaWQ9IlMxLkYxLnNmMi5waWMxIiBvdmVyZmxvdz0idmlzaWJsZSIgdmVyc2lvbj0iMS4xIiB3aWR0aD0iMjYzLjA5Ij48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMC40cHQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsMzQ3LjI5KSBtYXRyaXgoMSAwIDAgLTEgMCAwKSB0cmFuc2xhdGUoMTMwLjY4LDApIHRyYW5zbGF0ZSgwLDI4Ljg1KSI+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTExLjcyIC0xMC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuMzIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgOS40NikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTIuMTUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjMuNDUiPlxwZ2ZtYXRocmVzdWx0cHQ8L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgMTguMTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSA2LjM3IDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguMzQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTAuNzEiPjxtYXRoIGFsdHRleHQ9Inhfe3R9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPng8L21pPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+dDwvbWk+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxjaSBpZD0iUzEuRjEuc2YyLnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+8J2RoTwvY2k+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMi5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dH08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNC42MSA1OS43NiBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAtNC42MSAxNTMuMjYgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC01LjA0IDI1MS4zNykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuNjk1KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDQ1LjMzKSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAwKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjY1IiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEwLjA4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0fSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj55PC9taT48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPnQ8L21pPjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPvCdkaE8L2NpPjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjIucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnlfe3R9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC04My4wMiAtMjMuOTEpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDQzLjU5NSkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC40NSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxNy4xNiI+PG1hdGggYWx0dGV4dD0ieF97dC0xfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+4oiSPC9tbz48bW4gaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaU8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PG1pbnVzIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9taW51cz48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIj7wnZGhPC9jaT48Y24gaWQ9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIj4xPC9jbj48L2FwcGx5PjwvYXBwbHk+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJTMS5GMS5zZjIucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnhfe3QtMX08L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PC9nPjwvZz48L2c+PHBhdGggZD0iTSAtNzkuMDUgNzMuNDMgaCA5LjIyIHYgNDguNTkgaCAtOS4yMiBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gLTc5LjA1IDE2Ni45MyBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgyLjcgMjY1LjA0KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My42OTUpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguNjUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTYuNTQiPjxtYXRoIGFsdHRleHQ9Inlfe3QtMX0iIGRpc3BsYXk9ImlubGluZSIgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtc3ViIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCI+eTwvbWk+PG1yb3cgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiPjxtaSBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yLmNtbWwiPnQ8L21pPjxtbyBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiPuKIkjwvbW8+PG1uIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGmPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxtaW51cyBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEiPjwvbWludXM+PGNpIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjIucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YyLnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0LTF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDY1Ljg1IC0yNC4yNCkiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgNDMuOTIpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgNDUuMzMpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDApIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjkuMSIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIyMC42MSI+PG1hdGggYWx0dGV4dD0ieF97dCsxfSIgZGlzcGxheT0iaW5saW5lIiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIj54PC9taT48bXJvdyBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuY21tbCI+PG1pIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCI+dDwvbWk+PG1vIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjEuY21tbCI+KzwvbW8+PG1uIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCI+MTwvbW4+PC9tcm93PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNzeW1ib2wgY2Q9ImFtYmlndW91cyIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj5zdWJzY3JpcHQ8L2NzeW1ib2w+PGNpIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yIj7wnZGlPC9jaT48YXBwbHkgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiPjxwbHVzIGlkPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMSI+PC9wbHVzPjxjaSBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIuY21tbCIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjIiPvCdkaE8L2NpPjxjbiBpZD0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLjMiPjE8L2NuPjwvYXBwbHk+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IlMxLkYxLnNmMi5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+eF97dCsxfTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PC9nPjwvZz48cGF0aCBkPSJNIDcxLjU0IDczLjc1IGggOS4yMiB2IDQ4LjU5IGggLTkuMjIgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDcxLjU0IDE2Ny4yNSBoIDkuMjIgdiA0OC41OSBoIC05LjIyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgNjYuMTcgMjY1LjM2KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCA0My45MikiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIDEgMCA0NS4zMykiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOS4xIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjE5Ljk4Ij48bWF0aCBhbHR0ZXh0PSJ5X3t0KzF9IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bXN1YiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMiIgeHJlZj0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4yLmNtbWwiPnk8L21pPjxtcm93IGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy5jbW1sIj48bWkgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4yIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIj50PC9taT48bW8gaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMS5jbW1sIj4rPC9tbz48bW4gaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4zIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIj4xPC9tbj48L21yb3c+PC9tc3ViPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj48Y3N5bWJvbCBjZD0iYW1iaWd1b3VzIiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPnN1YnNjcmlwdDwvY3N5bWJvbD48Y2kgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjIiPvCdkaY8L2NpPjxhcHBseSBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4zLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMyI+PHBsdXMgaWQ9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xLmNtbWwiIHhyZWY9IlMxLkYxLnNmMi5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMy4xIj48L3BsdXM+PGNpIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMi5jbW1sIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMiI+8J2RoTwvY2k+PGNuIGlkPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMy5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJTMS5GMS5zZjIucGljMS42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjMuMyI+MTwvY24+PC9hcHBseT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iUzEuRjEuc2YyLnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj55X3t0KzF9PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjwvZz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwMDBGRiI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMzEuMjkgTCAtNzQuNDMgNjguOTkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAtNzQuNDMgNjguOTkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDAwMEZGIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTcuNjIgTCAwIDU1LjMyIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgMCA1NS4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiMwMDAwRkYiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMzEuNjEgTCA3Ni4xNiA2OS4zMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIDc2LjE2IDY5LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iIzAwRkYwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMTI0Ljc5IEwgLTc0LjQzIDE2Mi40OSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAxNjIuNDkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjMDBGRjAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMTExLjEyIEwgMCAxNDguODIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDE0OC44MikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiMwMEZGMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMTI1LjExIEwgNzYuMTYgMTYyLjgxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMTYyLjgxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGMDAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtNzQuNDMgMjE4LjI5IEwgLTc0LjQzIDI1NS45OCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjAgMS4wIC0xLjAgMC4wIC03NC40MyAyNTUuOTgpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkYwMDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDAgMjA0LjYyIEwgMCAyNDIuMzIiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC4wIDEuMCAtMS4wIDAuMCAwIDI0Mi4zMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNGRjAwMDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gNzYuMTYgMjE4LjYxIEwgNzYuMTYgMjU2LjMxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgNzYuMTYgMjU2LjMxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGODAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSAtMTIxLjE4IDk3LjczIEwgLTgzLjQ5IDk3LjczIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtODMuNDkgOTcuNzMpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIC02Ny4wNSA5Ni4zNyBMIC05LjAzIDg1LjcxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTUgLTAuMTgwNjMgMC4xODA2MyAwLjk4MzU1IC05LjAzIDg1LjcxKSI+PHBhdGggZD0iTSAtMy4xNSA0LjIxIEMgLTIuODkgMi42MyAwIDAuMjYgMC43OSAwIEMgMCAtMC4yNiAtMi44OSAtMi42MyAtMy4xNSAtNC4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PC9nPjxnIHN0cm9rZT0iI0ZGODAwMCI+PGcgc3Ryb2tlLXdpZHRoPSIxLjZwdCI+PHBhdGggZD0iTSA3LjM4IDg1LjQxIEwgNjcuMTMgOTYuMzkiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIHN0cm9rZS1kYXNoYXJyYXk9Im5vbmUiIHN0cm9rZS1kYXNob2Zmc2V0PSIwLjBwdCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBzdHJva2Utd2lkdGg9IjEuMjhwdCIgdHJhbnNmb3JtPSJtYXRyaXgoMC45ODM1NCAwLjE4MDczIC0wLjE4MDczIDAuOTgzNTQgNjcuMTMgOTYuMzkpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjRkY4MDAwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDgzLjUzIDk4LjA1IEwgMTIxLjIzIDk4LjA1IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAxMjEuMjMgOTguMDUpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIC0xMjEuMTggMTkxLjIyIEwgLTgzLjQ5IDE5MS4yMiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTgzLjQ5IDE5MS4yMikiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNCRjAwNDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gLTY3LjA1IDE4OS44NyBMIC05LjAzIDE3OS4yMSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgwLjk4MzU1IC0wLjE4MDYzIDAuMTgwNjMgMC45ODM1NSAtOS4wMyAxNzkuMjEpIj48cGF0aCBkPSJNIC0zLjE1IDQuMjEgQyAtMi44OSAyLjYzIDAgMC4yNiAwLjc5IDAgQyAwIC0wLjI2IC0yLjg5IC0yLjYzIC0zLjE1IC00LjIxIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48L2c+PGcgc3Ryb2tlPSIjQkYwMDQwIj48ZyBzdHJva2Utd2lkdGg9IjEuNnB0Ij48cGF0aCBkPSJNIDcuMzggMTc4LjkxIEwgNjcuMTMgMTg5Ljg5IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBzdHJva2UtZGFzaGFycmF5PSJub25lIiBzdHJva2UtZGFzaG9mZnNldD0iMC4wcHQiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIgc3Ryb2tlLXdpZHRoPSIxLjI4cHQiIHRyYW5zZm9ybT0ibWF0cml4KDAuOTgzNTQgMC4xODA3MyAtMC4xODA3MyAwLjk4MzU0IDY3LjEzIDE4OS44OSkiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48ZyBzdHJva2U9IiNCRjAwNDAiPjxnIHN0cm9rZS13aWR0aD0iMS42cHQiPjxwYXRoIGQ9Ik0gODMuNTMgMTkxLjU1IEwgMTIxLjIzIDE5MS41NSIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgc3Ryb2tlLWRhc2hhcnJheT0ibm9uZSIgc3Ryb2tlLWRhc2hvZmZzZXQ9IjAuMHB0IiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIHN0cm9rZS13aWR0aD0iMS4yOHB0IiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTIxLjIzIDE5MS41NSkiPjxwYXRoIGQ9Ik0gLTMuMTUgNC4yMSBDIC0yLjg5IDIuNjMgMCAwLjI2IDAuNzkgMCBDIDAgLTAuMjYgLTIuODkgLTIuNjMgLTMuMTUgLTQuMjEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjwvZz48L2c+PC9zdmc+]]

在新的 dropout 变体中,我们对输入、输出和循环层在每个时间步都使用同一个 dropout 掩码(即在每个时间步都丢弃相同的网络单元)。这与现有的临时性做法形成对比:后者仅在输入和输出处于每个时间步采样不同的 dropout 掩码,而循环连接上不使用 dropout(因为在这些连接上使用不同的掩码会导致性能下降)。我们的方法及其与现有技术的关系如图 1 所示。在用于离散输入(即词)时,我们也在词嵌入上施加一个分布。基于词的模型上的 dropout 因而对应于在句子中随机丢弃整个词类型,可解释为强制模型不依赖单个词来完成其任务。

接下来我们回顾相关文献和背景材料,然后形式化变分 RNN 的近似推断,从而得到上文提出的 dropout 变体。实验结果将在后文给出。

2 相关工作

近年来,大量工作累积起来,证明朴素地在 RNN 循环连接上施加 dropout 会带来负面影响。Pachitariu 与 Sahani [7] 在语言模型上的研究指出,在 RNN 的循环连接中添加噪声会引起模型不稳定,因此他们仅在模型的解码部分加入噪声。Bayer 等人 [8] 在 RNN 中应用了 dropout 的确定性近似(fast dropout)。他们指出,使用 dropout 后 RNN 的动力学变化剧烈,因此 dropout 应当只用于模型的非动态部分,即从隐藏层到输出层的连接。Pham 等人 [9] 在手写识别任务上评估 dropout,认为在循环层施加 dropout 会破坏 RNN 建模序列的能力,dropout 应只用于前馈连接而非循环连接。Zaremba、Sutskever 和 Vinyals 的工作 [4] 与 Pham 等人 [9] 同期展开。Zaremba 等人 [4] 在大量任务上评估 RNN 中的 dropout,证明仅在非循环连接上施加 dropout 可以提高性能,并在 Penn Treebank 语言建模上取得了(至今尚未被超越的)最先进结果。他们指出,没有 dropout 时,以往为避免过拟合只能使用小模型,而引入 dropout 后则可以使用更大的模型,从而获得更好的结果。这项工作被许多人视为参考实现(下文我们也将其用作基线)。Bluche 等人 [10] 在此基础上拓展,对 dropout 在 RNN 单元之前、内部、之后的表现进行了探索性分析,结果好坏参半,未显示出对既有技术的明显改进。最近,与本工作同期,Moon 等人 [20] 在语音识别社区提出了 RNN 中的一种新 dropout 变体。他们随机丢弃 LSTM 内部单元 $ {\textstyle \mathbf{c}_{t}} $ 中的元素,并在每个时间步都使用同一掩码。这一做法最接近我们提出的方法(尽管在根本上不同,详见 §4.1),我们在下文中也将与此变体作比较。

现有方法都是基于对各种 dropout 变体的经验实验,采取试错过程。这些做法令许多人相信:dropout 无法扩展到循环层中数量庞大的参数,从而使这些参数得不到正则化。与上述结论相反,我们通过将方法立足于近期的理论研究,证明可以推导出一种基于变分推断的 dropout 变体,能成功正则化此类参数。

3 背景

我们回顾贝叶斯神经网络与近似变分推断所需的背景知识。基于这些思想,我们将在下一节中提出在概率 RNN 中进行近似推断的方法,并由此引出一种新的 dropout 变体。

3.1 贝叶斯神经网络

给定训练输入 $ {\textstyle \mathbf{X} = {\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}}} $ 及其对应输出 $ {\textstyle \mathbf{Y} = {\{\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\}}} $,在贝叶斯(参数化)回归中,我们希望推断出某函数 $ {\textstyle \mathbf{y} = {\mathbf{f}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}}} $ 中可能生成我们输出的参数 $ {\textstyle \mathbf{ω}} $。哪些参数最可能产生这些数据?按照贝叶斯方法,我们会在参数空间上设定某个先验分布 $ {\textstyle p\hspace{0pt}{({\mathbf{ω}})}} $,它表示我们对哪些参数可能生成数据的先验信念。我们还需定义似然分布 $ {\textstyle p\hspace{0pt}{(\left. \mathbf{y} \middle| {\mathbf{x},{\mathbf{ω}}} \right.)}} $。对于分类任务,可以假设 softmax 似然,

$ {\textstyle {p\hspace{0pt}\left( {y = \left. d \middle| {\mathbf{x},{\mathbf{ω}}} \right.} \right)} = {\text{Categorical}\hspace{0pt}\left( {{\exp{({f_{d}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}})}}/{\sum\limits_{d^{\prime}}{\exp{({f_{d^{\prime}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x})}})}}}} \right)}} $

或回归任务的高斯似然。给定数据集 $ {\textstyle \mathbf{X},\mathbf{Y}} $,我们随后寻找参数空间上的后验分布:$ {\textstyle p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}} $。该分布刻画了在给定观测数据时,各种函数参数的可能程度。利用它,我们可以通过下式积分来对一个新输入点 $ {\textstyle \mathbf{x}^{\ast}} $ 进行预测:

$ {\textstyle {{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},\mathbf{X},\mathbf{Y}} \right.)}} = {\int{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\mathbf{ω}}} \right.)}\hspace{0pt}p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}\hspace{0pt}{\text{d}{\mathbf{ω}}}}}}.} $

(1)

一种在参数化函数集合上定义分布的方式,是在神经网络的权重上放置先验,从而得到贝叶斯神经网络 [21, 22]。设第 $ {\textstyle i} $ 层的权重矩阵为 $ {\textstyle \mathbf{W}_{i}} $、偏置向量为 $ {\textstyle \mathbf{b}_{i}} $,我们通常对权重矩阵采用标准矩阵高斯先验 $ {\textstyle {p\hspace{0pt}{(\mathbf{W}_{i})}} = {\mathcal{N}\hspace{0pt}{(\mathbf{0},\mathbf{I})}}} $,并且为简单起见,对偏置向量采用点估计。

3.2 贝叶斯神经网络中的近似变分推断

我们关心的是找到能生成数据的权重矩阵(用于参数化我们的函数)的分布,即给定观测 $ {\textstyle \mathbf{X},\mathbf{Y}} $ 时权重的后验:$ {\textstyle p\hspace{0pt}{(\left. {\mathbf{ω}} \middle| {\mathbf{X},\mathbf{Y}} \right.)}} $。一般而言该后验难以处理,可以使用变分推断对其加以近似(如 [23, 24, 25, 12] 所做)。我们需要定义一个近似变分分布 $ {\textstyle q\hspace{0pt}{({\mathbf{ω}})}} $,然后最小化该分布与完整后验之间的 KL 散度:

	$ {\textstyle \text{KL}\left( q{({\mathbf{ω}})}||p{({\mathbf{ω}}|\mathbf{X},\mathbf{Y})} \right)} $	$ {\textstyle \propto - \int q{({\mathbf{ω}})}\log p{(\mathbf{Y}|\mathbf{X},{\mathbf{ω}})}{\text{d}{\mathbf{ω}}} + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}} $
		$ {\textstyle = - \sum\limits_{i = 1}^{N}\int q{({\mathbf{ω}})}\log p{(\mathbf{y}_{i}|\mathbf{f}^{\mathbf{ω}}{(\mathbf{x}_{i})})}{\text{d}{\mathbf{ω}}} + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}.} $		(2)

接下来,我们将这种近似变分推断扩展到概率 RNN,并采用一种 $ {\textstyle q\hspace{0pt}{({\mathbf{ω}})}} $ 分布,从而引出 RNN 中的一种新 dropout 变体。

4 循环神经网络中的变分推断

为简洁起见,本节专注于简单 RNN 模型,LSTM 与 GRU 的推导类似。给定长度为 $ {\textstyle T} $ 的输入序列 $ {\textstyle \mathbf{x} = {\lbrack\mathbf{x}_{1},\ldots,\mathbf{x}_{T}\rbrack}} $,简单 RNN 通过反复应用函数 $ {\textstyle \mathbf{f}_{\mathbf{h}}} $ 形成,从而在每个时间步 $ {\textstyle t} $ 生成隐藏状态 $ {\textstyle \mathbf{h}_{t}} $:

$ {\textstyle \mathbf{h}_{t}} $

$ {\textstyle = {\mathbf{f}_{\mathbf{h}}\hspace{0pt}{(\mathbf{x}_{t},\mathbf{h}_{t - 1})}} = {\sigma\hspace{0pt}{({{\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{\mathbf{h}}} + {\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{\mathbf{h}}} + \mathbf{b}_{\mathbf{h}}})}}} $

其中 $ {\textstyle \sigma} $ 为某种非线性函数。模型输出可以例如定义为 $ {\textstyle {\mathbf{f}_{\mathbf{y}}\hspace{0pt}{(\mathbf{h}_{T})}} = {{\mathbf{h}_{T}\hspace{0pt}\mathbf{W}_{\mathbf{y}}} + \mathbf{b}_{\mathbf{y}}}} $。我们将该 RNN 视为概率模型,把 $ {\textstyle {\mathbf{ω}} = {\{\mathbf{W}_{\mathbf{h}},\mathbf{U}_{\mathbf{h}},\mathbf{b}_{\mathbf{h}},\mathbf{W}_{\mathbf{y}},\mathbf{b}_{\mathbf{y}}\}}} $ 视为随机变量(服从正态先验)。为明确对 $ {\textstyle \mathbf{ω}} $ 的依赖,我们将 $ {\textstyle \mathbf{f}_{\mathbf{y}}} $ 写为 $ {\textstyle \mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}} $,对 $ {\textstyle \mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}} $ 类似处理。我们仍按上文(第 3.1 节)的方式定义概率模型的似然。随机变量 $ {\textstyle \mathbf{ω}} $ 上的后验相当复杂,我们使用近似分布 $ {\textstyle q\hspace{0pt}{({\mathbf{ω}})}} $ 进行变分推断以加以近似。

用我们的 RNN 模型对上式 (3.2) 中每个求和项求值,得到

	$ {\textstyle \int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}{(\left. \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{h}_{T})}} \right.)}\hspace{0pt}{\text{d}{\mathbf{ω}}}}} $	$ {\textstyle = {\int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{T},\mathbf{h}_{T - 1})}} \right)} \right)\hspace{0pt}{\text{d}{\mathbf{ω}}}}}} $
		$ {\textstyle = {\int{q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\mathbf{ω}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{T},{\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{({\ldots\hspace{0pt}\mathbf{f}_{\mathbf{h}}^{\mathbf{ω}}\hspace{0pt}{(\mathbf{x}_{1},\mathbf{h}_{0})}\hspace{0pt}\ldots})}})}} \right)} \right)\hspace{0pt}{\text{d}{\mathbf{ω}}}}}} $

其中 $ {\textstyle \mathbf{h}_{0} = \text{0}} $。我们用单样本 Monte Carlo(MC)积分对其进行近似:

$ {\textstyle {\approx {{\log p}\hspace{0pt}\left( \mathbf{y} \middle| {\mathbf{f}_{\mathbf{y}}^{\hat{\mathbf{ω}}}\hspace{0pt}\left( {\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{(\mathbf{x}_{T},{\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{({\ldots\hspace{0pt}\mathbf{f}_{\mathbf{h}}^{\hat{\mathbf{ω}}}\hspace{0pt}{(\mathbf{x}_{1},\mathbf{h}_{0})}\hspace{0pt}\ldots})}})}} \right)} \right)}},} $

$ {\textstyle \hat{\mathbf{ω}} \sim {q\hspace{0pt}{({\mathbf{ω}})}}} $

由此得到对每个求和项的无偏估计。

将该估计代入式 (3.2),得到我们的最小化目标

$ {\textstyle \mathcal{L}} $

$ {\textstyle \approx - \sum\limits_{i = 1}^{N}\log p\left( \mathbf{y}_{i} \middle| \mathbf{f}_{\mathbf{y}}^{{\hat{\mathbf{ω}}}_{i}}\left( \mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\mathbf{x}_{i,T},\mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\ldots\mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}{(\mathbf{x}_{i,1},\mathbf{h}_{0})}\ldots)})} \right) \right) + \text{KL}{(q{({\mathbf{ω}})}||p{({\mathbf{ω}})})}.} $

(3)

注意,对于每个序列 $ {\textstyle \mathbf{x}_{i}} $,我们采样一组新的实现 $ {\textstyle {\hat{\mathbf{ω}}}_{i} = {\{{\hat{\mathbf{W}}}_{\mathbf{h}}^{i},{\hat{\mathbf{U}}}_{\mathbf{h}}^{i},{\hat{\mathbf{b}}}_{\mathbf{h}}^{i},{\hat{\mathbf{W}}}_{\mathbf{y}}^{i},{\hat{\mathbf{b}}}_{\mathbf{y}}^{i}\}}} $,并且序列 $ {\textstyle \mathbf{x}_{i} = {\lbrack\mathbf{x}_{i,1},\ldots,\mathbf{x}_{i,T}\rbrack}} $ 中的每个符号都由函数 $ {\textstyle \mathbf{f}_{\mathbf{h}}^{{\hat{\mathbf{ω}}}_{i}}} $ 处理,且在每个时间步 $ {\textstyle t \leq T} $ 都使用相同的权重实现 $ {\textstyle {\hat{\mathbf{W}}}_{\mathbf{h}}^{i},{\hat{\mathbf{U}}}_{\mathbf{h}}^{i},{\hat{\mathbf{b}}}_{\mathbf{h}}^{i}} $。

依照 [17],我们定义近似分布在 $ {\textstyle \mathbf{ω}} $ 中按权重矩阵及其各行因式分解。对每个权重矩阵的行 $ {\textstyle \mathbf{w}_{k}} $,近似分布为:

$ {\textstyle q\hspace{0pt}{(\mathbf{w}_{k})}} $

$ {\textstyle = {{p\hspace{0pt}\mathcal{N}\hspace{0pt}{(\mathbf{w}_{k};\text{0},{\sigma^{2}\hspace{0pt}I})}} + {{({1 - p})}\hspace{0pt}\mathcal{N}\hspace{0pt}{(\mathbf{w}_{k};\mathbf{m}_{k},{\sigma^{2}\hspace{0pt}I})}}}} $

其中 $ {\textstyle \mathbf{m}_{k}} $ 是变分参数(行向量),$ {\textstyle p} $ 提前给定(即 dropout 概率),$ {\textstyle \sigma^{2}} $ 取小值。我们对 $ {\textstyle \mathbf{m}_{k}} $ 进行优化,即对随机权重矩阵的变分参数进行优化;在标准视角下,它们对应于 RNN 的权重矩阵¹¹1Graves 等人 [26] 进一步将近似分布在每行的元素上做因式分解,并对每个元素使用高斯近似分布(而非混合);上述近似分布似乎能给出更好的性能,并与 dropout 关系密切 [17]。。式 (3) 中的 KL 项可近似为对变分参数 $ {\textstyle \mathbf{m}_{k}} $ 的 $ {\textstyle L_{2}} $ 正则化 [17]。

用样本 $ {\textstyle \hat{\mathbf{ω}} \sim {q\hspace{0pt}{({\mathbf{ω}})}}} $ 计算模型输出 $ {\textstyle \mathbf{f}_{\mathbf{y}}^{\hat{\mathbf{ω}}}\hspace{0pt}{( \cdot )}} $,等价于在前向传播中随机将每个权重矩阵 $ {\textstyle \mathbf{W}} $ 的若干行置零(掩蔽),即执行 dropout。我们的目标 $ {\textstyle \mathcal{L}} $ 与标准 RNN 完全一致。在我们带序列输入的 RNN 设置中,权重矩阵的每一行只随机掩蔽一次,而且非常关键的是:在所有时间步上都使用同一掩码。²²2附录 A 中,我们讨论了所提 dropout 解释与集成解释之间的关系。

预测可以通过两种方式近似:将每层的均值传播到下一层(即所谓标准 dropout 近似),或用 $ {\textstyle q\hspace{0pt}{({\mathbf{ω}})}} $ 对式 (1) 中的后验加以近似:

$ {\textstyle p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},\mathbf{X},\mathbf{Y}} \right.)}} $

$ {\textstyle \approx {\int{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\mathbf{ω}}} \right.)}\hspace{0pt}q\hspace{0pt}{({\mathbf{ω}})}\hspace{0pt}{\text{d}{\mathbf{ω}}}}} \approx {\frac{1}{K}\hspace{0pt}{\sum\limits_{k = 1}^{K}{p\hspace{0pt}{(\left. \mathbf{y}^{\ast} \middle| {\mathbf{x}^{\ast},{\hat{\mathbf{ω}}}_{k}} \right.)}}}}} $

(4)

其中 $ {\textstyle {\hat{\mathbf{ω}}}_{k} \sim {q\hspace{0pt}{({\mathbf{ω}})}}} $,即在测试时执行 dropout 并对结果取平均(MC dropout)。

4.1 实现及与 RNN 中 dropout 的关系

实现我们的近似推断,等同于在 RNN 中实现这样的 dropout:在每个时间步丢弃同一组网络单元,随机丢弃输入、输出和循环连接。这与既有技术形成对比——后者在不同时间步丢弃不同的网络单元,并且不对循环连接施加 dropout(图 1)。

某些 RNN 模型(例如 LSTM 与 GRU)在 RNN 单元内部使用多个不同的门。例如,LSTM 使用四个门进行定义:输入、遗忘、输出和输入调制(input modulation),

	$ {\textstyle \underset{¯}{\mathbf{i}}} $	$ {\textstyle = {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{i}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{i}}} \right)}} $	$ {\textstyle \underset{¯}{\mathbf{f}} = {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{f}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{f}}} \right)}} $
	$ {\textstyle \underset{¯}{\mathbf{o}}} $	$ {\textstyle = {\text{sigm}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{o}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{o}}} \right)}} $	$ {\textstyle \underset{¯}{\mathbf{g}} = {\text{tanh}\hspace{0pt}\left( {{\mathbf{h}_{t - 1}\hspace{0pt}\mathbf{U}_{g}} + {\mathbf{x}_{t}\hspace{0pt}\mathbf{W}_{g}}} \right)}} $
	$ {\textstyle \mathbf{c}_{t}} $	$ {\textstyle = {{\underset{¯}{\mathbf{f}} \circ \mathbf{c}_{t - 1}} + {\underset{¯}{\mathbf{i}} \circ \underset{¯}{\mathbf{g}}}}} $	$ {\textstyle \mathbf{h}_{t} = {{\underset{¯}{\mathbf{o}} \circ \text{tanh}}\hspace{0pt}{(\mathbf{c}_{t})}}} $		(5)

其中 $ {\textstyle {\mathbf{ω}} = {\{\mathbf{W}_{i},\mathbf{U}_{i},\mathbf{W}_{f},\mathbf{U}_{f},\mathbf{W}_{o},\mathbf{U}_{o},\mathbf{W}_{g},\mathbf{U}_{g}\}}} $ 为权重矩阵,$ {\textstyle \circ} $ 为逐元素乘积。这里内部状态 $ {\textstyle \mathbf{c}_{t}} $(也称为cell)以加性方式更新。

或者,如 [26] 所述,可对模型进行如下重参数化:

$ {\textstyle \begin{pmatrix} \underset{¯}{\mathbf{i}} \\ \underset{¯}{\mathbf{f}} \\ \underset{¯}{\mathbf{o}} \\ \underset{¯}{\mathbf{g}} \end{pmatrix} = {\begin{pmatrix} \text{sigm} \\ \text{sigm} \\ \text{sigm} \\ \text{tanh} \end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix} \mathbf{x}_{t} \\ \mathbf{h}_{t - 1} \end{pmatrix} \cdot \mathbf{W}} \right)}} $

(6)

其中 $ {\textstyle {\mathbf{ω}} = {\{\mathbf{W}\}}} $,$ {\textstyle \mathbf{W}} $ 是 $ {\textstyle 2\hspace{0pt}K} $ × $ {\textstyle 4\hspace{0pt}K} $ 的矩阵($ {\textstyle K} $ 为 $ {\textstyle \mathbf{x}_{t}} $ 的维数)。我们将该参数化称为绑定权重(tied-weights)LSTM(相对于式 (4.1) 中的非绑定权重 LSTM)。

尽管这两种参数化产生相同的确定性模型,但会得到不同的近似分布 $ {\textstyle q\hspace{0pt}{({\mathbf{ω}})}} $。在第一种参数化下,即便使用同一输入 $ {\textstyle \mathbf{x}_{t}} $,我们也可以为不同门使用不同的 dropout 掩码,因为近似分布是放在矩阵上而非输入上的:可以在作用于 $ {\textstyle \mathbf{x}_{t}} $ 的某个权重矩阵 $ {\textstyle \mathbf{W}} $ 中丢弃某些行,而在另一矩阵 $ {\textstyle \mathbf{W}^{\prime}} $ 中丢弃不同的行。在第二种参数化下,我们将分布放在单一矩阵 $ {\textstyle \mathbf{W}} $ 上。这会带来更快的前向传播,但效果略差,正如实验部分所示。

更具体地,使用第二种参数化(式 (6))可将我们的 dropout 变体写为

$ {\textstyle \begin{pmatrix} \underset{¯}{\mathbf{i}} \\ \underset{¯}{\mathbf{f}} \\ \underset{¯}{\mathbf{o}} \\ \underset{¯}{\mathbf{g}} \end{pmatrix} = {\begin{pmatrix} \text{sigm} \\ \text{sigm} \\ \text{sigm} \\ \text{tanh} \end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix} {\mathbf{x}_{t} \circ \mathtt{z}_{\mathbf{x}}} \\ {\mathbf{h}_{t - 1} \circ \mathtt{z}_{\mathbf{h}}} \end{pmatrix} \cdot \mathbf{W}} \right)}} $

(7)

其中 $ {\textstyle \mathtt{z}_{\mathbf{x}},\mathtt{z}_{\mathbf{h}}} $ 是在所有时间步上重复使用的随机掩码(对式 (4.1) 的参数化亦同)。

相比之下,Zaremba 等人 [4] 的 dropout 变体把式 (7) 中的 $ {\textstyle \mathtt{z}_{\mathbf{x}}} $ 替换为时间相关的 $ {\textstyle \mathtt{z}_{\mathbf{x}}^{t}} $,在每个时间步重新采样(而 $ {\textstyle \mathtt{z}_{\mathbf{h}}} $ 被去掉,循环连接 $ {\textstyle \mathbf{h}_{t - 1}} $ 不会被丢弃):

$ {\textstyle {\begin{pmatrix} \underset{¯}{\mathbf{i}} \\ \underset{¯}{\mathbf{f}} \\ \underset{¯}{\mathbf{o}} \\ \underset{¯}{\mathbf{g}} \end{pmatrix} = {\begin{pmatrix} \text{sigm} \\ \text{sigm} \\ \text{sigm} \\ \text{tanh} \end{pmatrix}\hspace{0pt}\left( {\begin{pmatrix} {\mathbf{x}_{t} \circ \mathtt{z}_{\mathbf{x}}^{t}} \\ \mathbf{h}_{t - 1} \end{pmatrix} \cdot \mathbf{W}} \right)}}.} $

(8)

另一方面,Moon 等人 [20] 的 dropout 变体则修改式 (4.1),对内部 cell 进行调整

$ {\textstyle \mathbf{c}_{t} = {\mathbf{c}_{t} \circ \mathtt{z}_{\mathbf{c}}}} $

(9)

在所有时间步上都使用同一掩码 $ {\textstyle \mathtt{z}_{\mathbf{c}}} $。注意,与 [20] 不同,通过将 dropout 视作权重上的操作,我们的方法可平凡地推广到 RNN 与 GRU。

4.2 词嵌入 dropout

在输入连续的数据集上,我们通常对输入层(即输入向量本身)应用 dropout。这等价于在紧随输入之后的权重矩阵上放置一个分布,并对其近似积分(若不如此,该矩阵在被优化的过程中容易过拟合)。

但对于词等离散输入(每个词被映射到一个连续向量——即词嵌入)的模型,这一点很少这样做。对于词嵌入,输入既可以视为词嵌入本身,也可以更方便地视为one-hot编码(在某一位置上为 $ {\textstyle 1} $、其他位置为 0 的向量)。one-hot 向量与嵌入矩阵 $ {\textstyle \mathbf{W}_{E} \in {\mathbb{R}}^{V \times D}} $ 相乘(其中 $ {\textstyle D} $ 为嵌入维度、$ {\textstyle V} $ 为词表大小)即得到词嵌入。耐人寻味的是,在大多数语言应用中,这一参数层是最大的,却往往得不到正则化。由于嵌入矩阵会被优化,它可能导致过拟合,因此对 one-hot 编码向量施加 dropout 是可取的。这事实上等同于在输入句子中随机丢弃词,也可解释为促使模型不要依赖单一词来产生输出。

注意,与之前一样,我们随机将矩阵 $ {\textstyle \mathbf{W}_{E} \in {\mathbb{R}}^{V \times D}} $ 的若干行置零。由于在每个时间步使用相同的掩码,因此整条序列中丢弃的是相同的词——也就是说,我们随机丢弃的是词类型而非词token(例如,句子 "the dog and the cat" 可能变为 "— dog and — cat" 或 "the — and the cat",但绝不会出现 "— dog and the cat")。实现上的一个潜在低效之处是需要采样 $ {\textstyle V} $ 个 Bernoulli 随机变量,而 $ {\textstyle V} $ 可能很大。可以这样解决:对于长度为 $ {\textstyle T} $ 的序列,最多只有 $ {\textstyle T} $ 个 embedding 会被丢弃(其他被丢弃的 embedding 对模型输出毫无影响)。因此,当 $ {\textstyle T \ll V} $ 时,更高效的做法是先把词映射为词嵌入,然后再根据词类型对相应嵌入置零。

5 实验评估

我们首先把所提 dropout 变体接入 Zaremba 等人 [4] 的 Torch 实现——后者已成为该领域许多人采用的参考实现。Zaremba 等人 [4] 在 Penn Treebank 上设立了一个基准,据我们所知近 2 年来无人能超越。我们对 [4] 的结果有所改进,并表明我们的 dropout 变体在性能上优于早停以及使用欠表达能力的模型。随后我们在标注数据稀缺的情感分析任务上,用 LSTM 与 GRU 模型继续评估所提 dropout 变体。最后,我们对所提方法的性质进行深入分析;由于篇幅所限,代码与许多实验放在附录中。

5.1 语言建模

我们复现 Zaremba、Sutskever 和 Vinyals [4] 的语言建模实验。实验使用 Penn Treebank,这是该领域的标准基准。在语言处理社区中该数据集被视为小型数据集,共有 $ {\textstyle 887,521} $ 个 token(词),因此过拟合问题尤其值得关注。整个实验中,我们将按贝叶斯解释提出的 dropout 用于 LSTM 时称为变分 LSTM(Variational LSTM),将既有的 dropout 技术称为朴素 dropout LSTM(naive dropout LSTM)(式 (8),不同时间步使用不同掩码,且仅施加于 LSTM 的输入和输出)。把不使用 dropout 的 LSTM 称为标准 LSTM(standard LSTM)。

我们对 [4] 的中型模型(每层 650 个单元、共 2 层)与大型模型(每层 1500 个单元、共 2 层)分别实现了变分 LSTM。相对于 [4] 的设置,我们仅做了两点改动:1)使用所提 dropout 变体而非朴素 dropout;2)调优权重衰减(在 [4] 中权重衰减取零)。其余所有超参数与 [4] 完全相同:学习率衰减未针对我们的设置调优,而是沿用 [4] 的方案。Dropout 参数通过网格搜索进行优化(将嵌入层的 dropout 概率与循环层的绑在一起,同时把输入/输出的 dropout 概率也绑在一起),据此最小化验证困惑度³³3对于大型模型,最优概率分别为 0.3 与 0.5,相较 [4] 的 0.6;对于中型模型则分别为 0.2 与 0.35,相较 [4] 的 0.5。。我们还对 Moon 等人 [20] 进行了比较;他们只对 LSTM 的内部状态用同一掩码丢弃元素(此外也对输入和输出施加 dropout,见式 (9))。我们针对每种模型规模实现了他们的 dropout 变体,并重复上述流程以寻找最优 dropout 概率(中型模型为 0.3,大型模型为 0.5)。对于大型模型在 [20] 的变体下,因 16 个 epoch 后就开始过拟合,我们不得不采用早停。Moon 等人 [20] 是在语音识别社区提出该变体的,他们无需考虑嵌入过拟合的问题(下文会看到,嵌入过拟合对循环层影响相当大)。因此我们另做了一组实验,把 [20] 的变体与我们的 embedding dropout 结合(称为 Moon et al. [20]+emb dropout)。

我们的结果列于表 1。对于变分 LSTM,我们既给出绑定权重模型(式 (6)–(7),Variational (tied weights))的结果,也给出非绑定权重模型(式 (4.1),Variational (untied weights))的结果。对每个模型,我们都给出两种性能:一种使用标准 dropout 近似(在测试时把权重取平均——将每个近似分布的均值作为下一层的输入进行传播),另一种使用 MC dropout(在测试时执行 1000 次 dropout 并按式 (4) 平均模型输出,记为 MC)。对每个模型我们报告平均困惑度及标准差(每个实验用不同随机种子重复 3 次后取平均)。模型训练时间以每秒处理词数(WPS)给出。

有趣的是,使用 dropout 近似时,绑定权重的模型在验证误差和测试误差上都低于非绑定权重模型;但在使用 MC dropout 时,非绑定权重模型的表现要好得多。对大型模型,验证困惑度由 [4] 的 $ {\textstyle 82.2} $ 改善到 $ {\textstyle 77.3} $(绑定权重),或 $ {\textstyle 77.9} $(非绑定)。测试困惑度由 $ {\textstyle 78.4} $ 降至 $ {\textstyle 73.4} $(使用 MC dropout 且非绑定权重)。据我们所知,这是目前在 Penn Treebank 上最好的单模型困惑度。

看起来,Moon 等人 [20] 的表现甚至不如 [4]。在没有 embedding dropout 的情况下,大型模型出现过拟合,必须采用早停(若不早停,30 个 epoch 内验证困惑度可上升到 131)。加上我们的 embedding dropout 后表现明显改善,但仍不如仅在输入和输出上施加 dropout 的做法。

把我们的结果与未正则化的 LSTM(使用早停评估,性能与 [4] 中的早停实验相近)相比,可以看到无论模型规模大小,使用我们的 dropout 变体都能带来改进。把中型变分模型与大型变分模型比较,可以发现通过使用更大的模型,困惑度能显著降低;而未正则化的 LSTM 无法做到这一点——更大的模型反而导致更差的结果。这表明,降低模型复杂度——一种避免过拟合的常用方法——在使用 dropout 时反而会带来更差的拟合。

	Medium LSTM			Large LSTM
	Validation	Test	WPS	Validation	Test	WPS
Non-regularized (early stopping)	$ {\textstyle 121.1} $	$ {\textstyle 121.7} $	$ {\textstyle 5.5} $K	$ {\textstyle 128.3} $	$ {\textstyle 127.4} $	$ {\textstyle 2.5} $K
Moon et al. [20]	$ {\textstyle 100.7} $	$ {\textstyle 97.0} $	$ {\textstyle 4.8} $K	$ {\textstyle 122.9} $	$ {\textstyle 118.7} $	$ {\textstyle 3} $K
Moon et al. [20] +emb dropout	$ {\textstyle 88.9} $	$ {\textstyle 86.5} $	$ {\textstyle 4.8} $K	$ {\textstyle 88.8} $	$ {\textstyle 86.0} $	$ {\textstyle 3} $K
Zaremba et al. [4]	$ {\textstyle 86.2} $	$ {\textstyle 82.7} $	$ {\textstyle 5.5} $K	$ {\textstyle 82.2} $	$ {\textstyle 78.4} $	$ {\textstyle 2.5} $K
Variational (tied weights)	$ {\textstyle 81.8 \pm 0.2} $	$ {\textstyle 79.7 \pm 0.1} $	$ {\textstyle 4.7} $K	$ {\textstyle 77.3 \pm 0.2} $	$ {\textstyle 75.0 \pm 0.1} $	$ {\textstyle 2.4} $K
Variational (tied weights, MC)	$ {\textstyle -} $	$ {\textstyle 79.0 \pm 0.1} $	$ {\textstyle -} $	$ {\textstyle -} $	$ {\textstyle 74.1 \pm 0.0} $	$ {\textstyle -} $
Variational (untied weights)	$ {\textstyle 81.9 \pm 0.2} $	$ {\textstyle 79.7 \pm 0.1} $	$ {\textstyle 2.7} $K	$ {\textstyle 77.9 \pm 0.3} $	$ {\textstyle 75.2 \pm 0.2} $	$ {\textstyle 1.6} $K
Variational (untied weights, MC)	$ {\textstyle -} $	$ {\textstyle 78.6 \pm 0.1} $	$ {\textstyle -} $	$ {\textstyle -} $	$ {\textstyle 73.4 \pm 0.0} $	$ {\textstyle -} $

我们还看到,使用 dropout 近似时,绑定权重模型与非绑定权重模型性能非常接近。然而,在评估运行时间时(使用 Titan X GPU),发现绑定权重的实现更为时间高效。这是因为单一的矩阵乘法被实现为单个 GPU kernel,而非绑定权重模型中需要顺序调用四个较小的矩阵乘法 kernel。但需要注意,低层实现下两者的运行时间应当相近。

此外,我们按照 [4] 的设置进行了模型平均实验:独立训练多个大型模型,然后平均它们的输出。我们使用上文设置下、配合 MC dropout 的变分 LSTM。使用 10 个变分 LSTM,即可将 [4] 的测试集困惑度从 $ {\textstyle 69.5} $ 降到 $ {\textstyle 68.7} $——与 [4] 使用 38 个模型的实验持平。

最后,我们报告在中型模型上降低学习率衰减时的验证困惑度。学习率衰减常被用作一种正则化手段:当模型开始过拟合时,优化器会采用更小的步长(如 [4] 中所做)。去掉学习率衰减可以让我们单独评估 dropout 的正则化效果。如图 2 所示,即便使用早停,变分 LSTM 仍能取得比朴素 dropout LSTM 与标准 LSTM 更低的困惑度。但需要注意,使用学习率衰减调度时,所有模型的困惑度都能进一步显著降低,见表 1。

5.2 情感分析

接下来我们在标注数据稀缺的情感分析任务上,使用 LSTM 与 GRU 模型评估所提 dropout 变体。我们使用 MC dropout(在附录 B 中进一步与 dropout 近似进行比较),并采用非绑定权重的模型参数化。

我们使用 Pang 与 Lee [27] 收集的 Cornell 电影评论原始语料,共 5000 条影评。我们从每条影评中抽取长度为 $ {\textstyle T} $ 词的连续片段,取 $ {\textstyle T = 200} $,并将对应的影评评分作为观测输出 $ {\textstyle y} $。模型由一个嵌入层(维度 $ {\textstyle 128} $)、一个 LSTM 层(每个门包含 $ {\textstyle 128} $ 个单元;GRU 设置类似)以及对 LSTM 最后输出施加的全连接层(输出标量)构成。整个实验使用 Adam 优化器 [28],batch size 为 128,测试时使用 10 次采样的 MC dropout。

主要结果见图 3。我们将变分 LSTM(在每个权重层都施加所提 dropout 变体)与该领域的标准技术做了对比。训练误差见图 3(a),测试误差见图 3(b)。对每个模型都使用最优的 dropout 概率与权重衰减(见附录 B)。看起来唯一不会过拟合的模型是变分 LSTM,它同时取得最低的测试误差。变分 GRU 的测试误差见图 15(损失曲线见附录 B)。每个模型同样使用最优 dropout 概率与权重衰减。变分 GRU 同样避免了对数据的过拟合,并收敛到最低测试误差。但在该数据集上,采用早停反而能得到更小的测试误差(最低测试误差由非正则化 GRU 在第 2 个 epoch 时取得)。值得一提的是,标准方法表现出测试误差反复下降与上升的奇特现象,而变分 GRU 中并未观察到。各模型损失曲线见附录 B。

接下来我们考察对模型不同部分施加 dropout 的影响。我们在情感分析任务上评估变分 LSTM 在不同的 embedding dropout($ {\textstyle p_{E} = {0,0.5}} $)与循环层 dropout($ {\textstyle p_{U} = {0,0.5}} $)组合下的表现。收敛曲线见图 4(a)。看起来,如果在 embedding 与循环层上都缺乏强正则化,模型很快就会过拟合。$ {\textstyle p_{U} = 0.5} $ 且 $ {\textstyle p_{E} = 0} $ 时的表现相当有趣:测试误差先下降、再上升,然后再次下降。此外,$ {\textstyle p_{U} = 0} $ 且 $ {\textstyle p_{E} = 0.5} $ 时,模型变得非常不稳定。

最后,我们测试变分 LSTM 在不同循环层 dropout 概率下的表现,把 embedding dropout 概率分别固定为 $ {\textstyle p_{E} = 0} $ 或 $ {\textstyle p_{E} = 0.5} $(图 4(b)–4(c))。这些结果颇为耐人寻味。在该实验中所有模型都已收敛,损失接近零(未画出)。然而看起来,在没有 embedding dropout 的情况下,提高循环层 dropout 概率反而会导致过拟合!这大概是因为未正则化的 embedding 层参数量巨大。当用 $ {\textstyle p_{E} = 0.5} $ 对 embedding 层加以正则化后,提高循环层 dropout 概率确实如预期般提升了对过拟合的鲁棒性。这些结果表明,在某些任务中 embedding dropout 可能至关重要。

在附录 B 中,我们评估了在所提 dropout 变体下权重衰减的重要性。通常的做法是在朴素 dropout 中去掉权重衰减;但我们的结果表明,在我们的变体中权重衰减起着重要作用(它对应于我们对权重分布的先验信念)。

6 结论

我们提出了循环神经网络的一种新正则化技术。所提 RNN dropout 变体在理论上有依据,且在实验中得到了有效性验证。在未来工作中,我们打算评估变分 LSTM [17] 中的模型不确定性。结合本文成果,这将对语言歧义建模与控制任务中的动力学建模产生重要意义。

参考文献

• Sundermeyer 等. [2012] Martin Sundermeyer、Ralf Schlüter 与 Hermann Ney. LSTM neural networks for language modeling. 见 INTERSPEECH, 2012.
• Kalchbrenner 与 Blunsom [2013] Nal Kalchbrenner 与 Phil Blunsom. Recurrent continuous translation models. 见 EMNLP, 2013.
• Sutskever 等. [2014] Ilya Sutskever、Oriol Vinyals 与 Quoc VV Le. Sequence to sequence learning with neural networks. 见 NIPS, 2014.
• Zaremba 等. [2014] Wojciech Zaremba、Ilya Sutskever 与 Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
• Hinton [2012] Geoffrey E others Hinton. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
• Srivastava 等. [2014] Nitish Srivastava、Geoffrey Hinton、Alex Krizhevsky、Ilya Sutskever 与 Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014.
• Pachitariu 与 Sahani [2013] Marius Pachitariu 与 Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv preprint arXiv:1301.5650, 2013.
• Bayer 等. [2013] Justin Bayer、Christian Osendorfer、Daniela Korhammer、Nutan Chen、Sebastian Urban 与 Patrick van der Smagt. On fast dropout and its applicability to recurrent networks. arXiv preprint arXiv:1311.0701, 2013.
• Pham 等. [2014] Vu Pham、Theodore Bluche、Christopher Kermorvant 与 Jerome Louradour. Dropout improves recurrent neural networks for handwriting recognition. 见 ICFHR. IEEE, 2014.
• Bluche 等. [2015] Théodore Bluche、Christopher Kermorvant 与 Jérôme Louradour. Where to apply dropout in recurrent neural networks for handwriting recognition? 见 ICDAR. IEEE, 2015.
• Rezende 等. [2014] Danilo Jimenez Rezende、Shakir Mohamed 与 Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. 见 ICML, 2014.
• Blundell 等. [2015] Charles Blundell、Julien Cornebise、Koray Kavukcuoglu 与 Daan Wierstra. Weight uncertainty in neural network. 见 ICML, 2015.
• Hernandez-Lobato 与 Adams [2015] Jose Miguel Hernandez-Lobato 与 Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. 见 ICML, 2015.
• Gal 与 Ghahramani [2015a] Yarin Gal 与 Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv:1506.02158, 2015a.
• Kingma 等. [2015] Diederik Kingma、Tim Salimans 与 Max Welling. Variational dropout and the local reparameterization trick. 见 NIPS. Curran Associates, Inc., 2015.
• Korattikara Balan 等. [2015] Anoop Korattikara Balan、Vivek Rathod、Kevin P Murphy 与 Max Welling. Bayesian dark knowledge. 见 NIPS. Curran Associates, Inc., 2015.
• Gal 与 Ghahramani [2015b] Yarin Gal 与 Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. arXiv:1506.02142, 2015b.
• Hochreiter 与 Schmidhuber [1997] Sepp Hochreiter 与 Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997.
• Cho 等. [2014] Kyunghyun Cho 等. Learning phrase representations using RNN encoder–decoder for statistical machine translation. 见 EMNLP, Doha, Qatar, 2014 年 10 月. ACL.
• Moon 等. [2015 年 12 月] Taesup Moon、Heeyoul Choi、Hoshik Lee 与 Inchul Song. RnnDrop: A Novel Dropout for RNNs in ASR. 见 ASRU Workshop, 2015 年 12 月.
• MacKay [1992] David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural computation, 4(3):448–472, 1992.
• Neal [1995] Radford M Neal. Bayesian learning for neural networks. 博士论文,多伦多大学,1995.
• Hinton 与 Van Camp [1993] Geoffrey E Hinton 与 Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. 见 COLT, 第 5–13 页. ACM, 1993.
• Barber 与 Bishop [1998] David Barber 与 Christopher M Bishop. Ensemble learning in Bayesian neural networks. NATO ASI SERIES F COMPUTER AND SYSTEMS SCIENCES, 168:215–238, 1998.
• Graves [2011] Alex Graves. Practical variational inference for neural networks. 见 NIPS, 2011.
• Graves 等. [2013] Alan Graves、Abdel-rahman Mohamed 与 Geoffrey Hinton. Speech recognition with deep recurrent neural networks. 见 ICASSP. IEEE, 2013.
• Pang 与 Lee [2005] Bo Pang 与 Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. 见 ACL. ACL, 2005.
• Kingma 与 Ba [2014] Diederik Kingma 与 Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
• Bergstra 等. [2010] James Bergstra 等. Theano: a CPU and GPU math expression compiler. 见 Proceedings of the Python for Scientific Computing Conference (SciPy), 2010 年 6 月. 口头报告.
• fchollet [2015] fchollet. Keras. https://github.com/fchollet/keras, 2015.

附录 A dropout 的贝叶斯解释与集成解释对比

除了我们的贝叶斯近似解释,深度网络中的 dropout 也可以从集成解释 [6] 来看。该解释同样得出测试时使用 MC dropout 的结论。但集成解释并未规定集成应建立在网络单元上还是权重上。例如,对于 RNN,该视角不会直接得到我们的 dropout 变体,除非临时规定集成需要绑定网络的权重;而在贝叶斯近似视角下,权重绑定是由模型的概率解释自然强制得到的。

附录 B 情感分析——更多实验

情感分析的超参数通过对每个模型使用 dropout 概率 $ {\textstyle 0.25} $ 与 $ {\textstyle 0.5} $、权重衰减自 $ {\textstyle 10^{- 6}} $ 到 $ {\textstyle 10^{- 4}} $ 的取值范围进行评估而得到。对于变分 LSTM,最优设置为 dropout 概率 $ {\textstyle 0.25} $ 且权重衰减 $ {\textstyle 10^{- 3}} $;对于朴素 dropout LSTM,dropout 概率为 $ {\textstyle 0.5} $(朴素 dropout LSTM 的参考实现 [4] 中不使用权重衰减)。

我们评估变分 LSTM 中的dropout 近似。dropout 近似在深度网络中常被用作 MC 估计的近似手段:把每个权重矩阵 $ {\textstyle \mathbf{M}} $ 替换为 $ {\textstyle p\hspace{0pt}\mathbf{M}} $(其中 $ {\textstyle p} $ 为 dropout 概率),然后对网络做一次确定性前向传播(不丢弃单元)。这可以视为将随机变量 $ {\textstyle \mathbf{W}} $ 的均值在网络中传播 [17]。该近似在深度网络中已被证明效果良好 [6],但在卷积层上会失败 [14]。我们用变分 LSTM 模型对该近似做实证评估:重复第一个实验,但在测试时使用近似而非 MC dropout。结果见图 9。看起来该近似对测试误差给出了良好的估计,与图 4(a) 中的结果相似。

我们还在不同的权重衰减下测试变分 LSTM 模型,以观察不同取值的影响。注意权重衰减会应用于所有层,包括嵌入层。从图 7 可以看出,更大的权重衰减带来更低的测试误差,不同权重衰减之间差异显著。这表明即便使用 dropout,权重衰减仍发挥重要作用(而朴素 dropout 通常会去掉它)。此外注意,权重衰减可以(连同 dropout 参数一起)作为变分近似的一部分进行优化。深度学习社区在实践中并不这样做,而是常常对这些参数使用网格搜索或贝叶斯优化。

用不同的序列长度(取 $ {\textstyle T = {20,50,200,400}} $)测试变分 LSTM,可以看出序列长度对模型性能也有显著影响(图 7)。更长的序列带来明显更好的性能,但代价是更长的收敛时间。我们最初假设较短序列性能下降是由于嵌入层的 dropout 概率过高;但在长度为 50 和 200、不同 embedding dropout 概率下的后续实验显示,较低的 dropout 概率反而使模型性能更差(图 9 与 7)。

图 10(a) 中我们观察到不同的 dropout 概率与权重衰减如何影响 GRU 模型的性能。

我们将朴素 dropout LSTM 与 dropout 概率在循环层为零($ {\textstyle p_{U} = 0} $)的变分 LSTM(称为 dropout LSTM)加以对比。两种模型都仅在 LSTM 的输入和输出上施加 dropout,且不对嵌入施加 dropout。区别在于,朴素 dropout LSTM 在不同时间步使用不同掩码、各门之间共享;而 dropout LSTM 在不同时间步使用同一掩码。两种模型的测试误差见图 12。看起来,在循环层和嵌入层都没有 dropout 的情况下,两种模型都会过拟合,且实际上性能完全相同。

接下来,我们在 GRU 模型上评估 dropout 近似。在 GRU 模型中,该近似似乎得到了与 MC dropout 相似的结果(图 12)。

最后,我们绘出正文部分若干模型的训练损失曲线。所有模型均已收敛,训练损失保持稳定。

附录 C 代码

在 Theano [29] 上、面向 Keras [30] 的上述方法的高效实现可以简洁如下:

def get_output(self, train=False):
    X = self.get_input(train)

    retain_prob_W = 1. - self.p_W[0]
    retain_prob_U = 1. - self.p_U[0]
    B_W = self.srng.binomial((4, X.shape[1], self.input_dim),
        p=retain_prob_W, dtype=theano.config.floatX)
    B_U = self.srng.binomial((4, X.shape[1], self.output_dim),
        p=retain_prob_U, dtype=theano.config.floatX)

    xi = T.dot(X * B_W[0], self.W_i) + self.b_i
    xf = T.dot(X * B_W[1], self.W_f) + self.b_f
    xc = T.dot(X * B_W[2], self.W_c) + self.b_c
    xo = T.dot(X * B_W[3], self.W_o) + self.b_o

    [outputs, memories], updates = theano.scan(
        self._step,
        sequences=[xi, xf, xo, xc],
        outputs_info=[
            T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
            T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1)
        ],
        non_sequences=[self.U_i, self.U_f, self.U_o, self.U_c, B_U],
        truncate_gradient=self.truncate_gradient)

    return outputs[-1]

def _step(self,
          xi_t, xf_t, xo_t, xc_t,
          h_tm1, c_tm1,
          u_i, u_f, u_o, u_c, B_U):
    i_t = self.inner_activation(xi_t + T.dot(h_tm1 * B_U[0], u_i))
    f_t = self.inner_activation(xf_t + T.dot(h_tm1 * B_U[1], u_f))
    c_t = f_t * c_tm1 + i_t * self.activation(xc_t + T.dot(h_tm1 * B_U[2], u_c))
    o_t = self.inner_activation(xo_t + T.dot(h_tm1 * B_U[3], u_o))
    h_t = o_t * self.activation(c_t)
    return h_t, c_t