BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding/paper/zh

SummarySource

Research Paper
Authors	Jacob Devlin; Ming-Wei Chang; Kenton Lee; Kristina Toutanova
Year	2018
Topic area	NLP
Difficulty	Research
arXiv	1810.04805
PDF	Download PDF

<span id="bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding">

BERT:用于语言理解的深度双向Transformerspre-training

Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
Google AI Language
{jacobdevlin,mingweichang,kentonl,kristout}@google.com

摘要

我们提出了一种新的语言表征模型,称为 BERT,即 Bidirectional Encoder Representations from Transformers。与近期的语言表征模型 Peters et al. (2018a); Radford et al. (2018) 不同,BERT 旨在通过在所有层中同时基于左右上下文进行条件建模,从未标注文本中pre-training深度双向表征。因此,经过pre-training的 BERT 模型只需添加一个额外的输出层进行fine-tuning,即可为广泛的任务(如问答和语言推断)创建最先进的模型,而无需对任务特定的架构进行实质性修改。

BERT 在概念上简单,在实证上强大。它在十一项自然语言处理任务上取得了新的最先进成果,包括将 GLUE 分数提升至 80.5%(绝对提升 7.7 个百分点)、MultiNLI 准确率提升至 86.7%(绝对提升 4.6%)、SQuAD v1.1 问答测试集 F1 提升至 93.2(绝对提升 1.5 个百分点)以及 SQuAD v2.0 测试集 F1 提升至 83.1(绝对提升 5.1 个百分点)。

1 引言

语言模型pre-training已被证明能有效改善许多自然语言处理任务 Dai and Le (2015); Peters et al. (2018a); Radford et al. (2018); Howard and Ruder (2018)。这些任务包括句子级任务,如自然语言推断 Bowman et al. (2015); Williams et al. (2018) 和释义 Dolan and Brockett (2005),其目标是通过整体分析句子来预测它们之间的关系;以及词元级任务,如命名实体识别和问答,这些任务要求模型在词元级别产生细粒度输出 Tjong Kim Sang and De Meulder (2003); Rajpurkar et al. (2016)。

将pre-training的语言表征应用到下游任务有两种现有策略:基于特征的方法和fine-tuning方法。基于特征的方法,如 ELMo Peters et al. (2018a),使用任务特定的架构,将pre-training的表征作为附加特征。fine-tuning方法,如 Generative Pre-trained transformer(OpenAI GPT)Radford et al. (2018),引入最少的任务特定参数,并通过对所有pre-training参数进行fine-tuning在下游任务上进行训练。这两种方法在pre-training期间共享相同的目标函数,均使用单向语言模型来学习通用的语言表征。

我们认为当前技术限制了pre-training表征的能力,特别是对于fine-tuning方法。主要局限在于标准语言模型是单向的,这限制了可在pre-training期间使用的架构选择。例如,在 OpenAI GPT 中,作者使用从左到右的架构,其中每个词元只能在 transformer 的自层中关注先前的词元 Vaswani et al. (2017)。此类限制对句子级任务来说是次优的,在将基于的方法应用于词元级任务(如问答)时可能非常有害,因为这类任务必须结合双向上下文。

在本文中,我们通过提出 BERT 改进了基于的方法:Bidirectional Encoder Representations from 。BERT 通过使用受 Cloze 任务 Taylor (1953) 启发的"掩码语言模型"(MLM)目标,缓解了前面提到的单向性约束。掩码语言模型随机掩盖输入中的部分词元,目标是仅根据上下文预测被掩盖词的原始词表 id。与从左到右的语言模型不同,MLM 目标使表征能够融合左右上下文,从而使我们能够一个深度双向。除了掩码语言模型外,我们还使用"下一句预测"任务联合文本对表征。本文的贡献如下:

• •

  我们证明了双向对语言表征的重要性。与使用单向语言模型进行的 Radford et al. (2018) 不同,BERT 使用掩码语言模型来实现的深度双向表征。这也与 Peters et al. (2018a) 形成对比,后者使用独立训练的从左到右和从右到左 LM 的浅层拼接。

• •

  我们表明的表征可减少对许多重度工程化任务特定架构的需求。BERT 是首个基于的表征模型,在大量句子级和词元级任务上实现了最先进的性能,超越了许多任务特定的架构。

• •

  BERT 在十一项 NLP 任务上推进了技术前沿。代码和模型可在 https://github.com/google-research/bert 获取。

2 相关工作

通用语言表征的有着悠久的历史,本节简要回顾最广泛使用的方法。

2.1 基于特征的无监督方法

学习广泛适用的词语表征数十年来一直是活跃的研究领域,包括非神经方法 Brown et al. (1992); Ando and Zhang (2005); Blitzer et al. (2006) 和神经方法 Mikolov et al. (2013); Pennington et al. (2014)。的词是现代 NLP 系统的重要组成部分,相比从零学习的带来显著改进 Turian et al. (2010)。为了词向量,已使用从左到右的语言建模目标 Mnih and Hinton (2009),以及在左右上下文中区分正确词与错误词的目标 Mikolov et al. (2013)。

这些方法已被推广到更粗粒度的对象,如句子 Kiros et al. (2015); Logeswaran and Lee (2018) 或段落 Le and Mikolov (2014)。为训练句子表征,先前工作使用了对候选下一句进行排序的目标 Jernite et al. (2017); Logeswaran and Lee (2018),给定前一句表征从左到右生成下一句词语的目标 Kiros et al. (2015),或源自去噪自编码器的目标 Hill et al. (2016)。

ELMo 及其前身 Peters et al. (2017, 2018a) 在不同维度上推广了传统的词研究。它们从从左到右和从右到左的语言模型中提取上下文敏感特征。每个词元的上下文表征是从左到右和从右到左表征的拼接。当将上下文词与现有任务特定架构结合时,ELMo 在多个主要 NLP 基准上推进了技术前沿 Peters et al. (2018a),包括问答 Rajpurkar et al. (2016)、情感分析 Socher et al. (2013) 和命名实体识别 Tjong Kim Sang and De Meulder (2003)。Melamud et al. (2016) 提出通过使用 从左右上下文中预测单个词的任务来学习上下文表征。与 ELMo 类似,他们的模型基于特征,不是深度双向的。Fedus et al. (2018) 表明 cloze 任务可用于提升文本生成模型的鲁棒性。

<span id="unsupervised--approaches">

2.2 基于的无监督方法

与基于特征的方法类似,该方向的首批工作仅从未标注文本中词参数 Collobert and Weston (2008)。

近期,产生上下文词元表征的句子或文档编码器已从未标注文本中,并针对下游有监督任务进行 Dai and Le (2015); Howard and Ruder (2018); Radford et al. (2018)。这些方法的优势是需要从零学习的参数很少。至少部分由于这一优势,OpenAI GPT Radford et al. (2018) 在 GLUE 基准的许多句子级任务上取得了此前最先进的结果 Wang et al. (2018a)。从左到右的语言建模和自编码器目标已被用于此类模型 Howard and Ruder (2018); Radford et al. (2018); Dai and Le (2015)。

2.3 来自有监督数据的迁移学习

也有研究展示了从带有大型数据集的有监督任务中进行有效迁移,如自然语言推断 Conneau et al. (2017) 和机器翻译 McCann et al. (2017)。计算机视觉研究也证明了从大型模型进行迁移学习的重要性,其中一种有效做法是对 ImageNet 的模型进行 Deng et al. (2009); Yosinski et al. (2014)。

3 BERT

本节我们介绍 BERT 及其详细实现。我们的框架分为两个步骤:和。在阶段,模型在无标注数据上针对不同的任务进行训练。对于,BERT 模型首先用的参数初始化,然后使用下游任务的标注数据对所有参数进行。每个下游任务都有独立的模型,即使它们由相同的参数初始化。图 1 中的问答示例将作为本节的运行示例。

BERT 的一个独特特征是其在不同任务间统一的架构。架构与最终下游架构之间的差异极小。

模型架构

BERT 的模型架构是基于 Vaswani et al. (2017) 中描述的原始实现并发布在 tensor2tensor 库中的多层双向编码器。¹¹1https://github.com/tensorflow/tensor2tensor 由于 的使用已变得普遍且我们的实现几乎与原始实现相同,我们将省略对模型架构的详尽背景描述,读者可参考 Vaswani et al. (2017) 以及"The Annotated "等优秀指南。²²2http://nlp.seas.harvard.edu/2018/04/03/.html

在本文中,我们将层数(即块数)记为 $ {\textstyle L} $,隐藏层大小记为 $ {\textstyle H} $,自头数记为 $ {\textstyle A} $。³³3在所有情况下,我们将前馈/滤波器大小设为 $ {\textstyle 4\hspace{0pt}H} $,即 $ {\textstyle H = 768} $ 时为 3072,$ {\textstyle H = 1024} $ 时为 4096。我们主要报告两种模型规模的结果:BERT$ {\textstyle _{\text{BASE}}} $(L=12, H=768, A=12,总参数=110M)和 BERT$ {\textstyle _{\text{LARGE}}} $(L=24, H=1024, A=16,总参数=340M)。

为便于比较,BERT$ {\textstyle _{\text{BASE}}} $ 被选定为与 OpenAI GPT 相同的模型规模。但关键的是,BERT 使用双向自,而 GPT 使用受约束的自,其中每个词元只能关注其左侧的上下文。⁴⁴4我们注意到在文献中双向常被称为"编码器",而仅左上下文版本被称为"解码器",因为它可用于文本生成。

输入/输出表征

为了让 BERT 处理多种下游任务,我们的输入表征能够在一个词元序列中明确地表示单个句子和句子对(例如 $ {\textstyle \langle} $ 问题, 答案 $ {\textstyle \rangle} $)。在整个工作中,"句子"可以是任意连续文本跨度,而不必是真正语言学意义上的句子。"序列"指 BERT 的输入词元序列,可以是单个句子,也可以是两个打包在一起的句子。

我们使用具有 30,000 个词元词表的 WordPiece Wu et al. (2016)。每个序列的第一个词元始终是一个特殊的分类词元([CLS])。该词元对应的最终隐藏状态用作分类任务的聚合序列表征。句子对被打包到单一序列中。我们以两种方式区分句子。首先,我们用一个特殊词元([SEP])将它们分隔开。其次,我们在每个词元上添加一个习得的,指示它属于句子 A 还是 B。如图 1 所示,我们将输入记为 $ {\textstyle E} $,特殊 [CLS] 词元的最终隐藏向量记为 $ {\textstyle C \in {\mathbb{R}}^{H}} $,第 $ {\textstyle i^{th}} $ 个输入词元的最终隐藏向量记为 $ {\textstyle T_{i} \in {\mathbb{R}}^{H}} $。

对于给定的词元,其输入表征通过将相应的词元、段落和位置相加构建而成。该构造的可视化可见图 2。

<span id="-bert">

3.1 BERT

与 Peters et al. (2018a) 和 Radford et al. (2018) 不同,我们不使用传统的从左到右或从右到左的语言模型来 BERT。相反,我们使用本节描述的两种无监督任务来 BERT。该步骤在图 1 的左侧呈现。

任务 #1:掩码 LM

直观上,有理由相信深度双向模型严格强于从左到右模型或从左到右与从右到左模型的浅层拼接。遗憾的是,标准条件语言模型只能从左到右或从右到左训练,因为双向条件会使每个词间接地"看到自己",模型在多层上下文中就能轻易预测目标词。

为了训练深度双向表征,我们只是随机掩盖一定百分比的输入词元,然后预测这些被掩盖的词元。我们将此过程称为"掩码 LM"(MLM),尽管在文献中常被称为 Cloze 任务 Taylor (1953)。在这种情况下,与掩码词元对应的最终隐藏向量被送入词表上的 输出,这与标准 LM 相同。在所有实验中,我们在每个序列中随机掩盖所有 WordPiece 词元的 15%。与去噪自编码器 Vincent et al. (2008) 不同,我们仅预测被掩盖的词,而不是重建整个输入。

尽管这使我们能够获得双向模型,但缺点在于我们在与之间制造了不匹配,因为 [MASK] 词元在时不会出现。为缓解这一问题,我们不总是用真正的 [MASK] 词元替换"被掩盖"的词。训练数据生成器随机选择 15% 的词元位置进行预测。如果选择第 $ {\textstyle i} $ 个词元,我们以 80% 的概率用 [MASK] 词元替换它,(2) 以 10% 的概率用随机词元替换,(3) 以 10% 的概率保持第 $ {\textstyle i} $ 个词元不变。然后 $ {\textstyle T_{i}} $ 将通过交叉熵损失用于预测原始词元。我们在附录 C.2 中对此过程的变体进行比较。

任务 #2:下一句预测 (NSP)

许多重要下游任务,如问答(QA)和自然语言推断(NLI),都基于对两个句子之间关系的理解,而语言建模并不直接捕获这种关系。为了训练理解句子关系的模型,我们针对一个二值化的下一句预测任务进行,该任务可以从任何单语语料中轻松生成。具体而言,在为每个样例选择句子 A 和 B 时,50% 的情况下 B 是实际跟随 A 的下一句(标注为 IsNext),50% 的情况下它是来自语料库的随机句子(标注为 NotNext)。如图 1 所示,$ {\textstyle C} $ 用于下一句预测(NSP)。⁵⁵5最终模型在 NSP 上达到 97%-98% 的准确率。尽管简单,我们在第 5.1 节中证明针对该任务的对 QA 和 NLI 都非常有益。⁶⁶6没有时向量 $ {\textstyle C} $ 不是有意义的句子表征,因为它是用 NSP 训练的。NSP 任务与 Jernite et al. (2017) 和 Logeswaran and Lee (2018) 中使用的表征学习目标密切相关。然而,在先前工作中只有句子被迁移到下游任务,而 BERT 迁移所有参数以初始化最终任务模型参数。

数据 过程在很大程度上沿用了现有语言模型文献。对于语料库,我们使用 BooksCorpus(800M 词)Zhu et al. (2015) 和英文维基百科(2,500M 词)。对于维基百科,我们只提取文本段落,忽略列表、表格和标题。使用文档级语料库而不是像 Billion Word Benchmark Chelba et al. (2013) 这样打乱的句子级语料库,对于提取长连续序列至关重要。

<span id="-bert">

3.2 BERT

很直接,因为 中的自机制使 BERT 能够通过简单地替换合适的输入和输出来对许多下游任务建模——无论涉及单一文本还是文本对。对于涉及文本对的应用,常见模式是先独立编码文本对,然后应用双向交叉,如 Parikh et al. (2016); Seo et al. (2017)。BERT 改用自机制来统一这两个阶段,因为以自对拼接的文本对进行编码,实际上就包含了两个句子之间的双向交叉。

对于每个任务,我们只是将任务特定的输入和输出接入 BERT,并端到端地所有参数。在输入端,中的句子 A 和 B 类似于(1)释义中的句子对,(2)蕴含中的假设-前提对,(3)问答中的问题-段落对,(4)文本分类或序列标注中退化的文本-$ {\textstyle \varnothing} $ 对。在输出端,词元表征被馈入用于词元级任务的输出层,如序列标注或问答;而 [CLS] 表征被馈入用于分类的输出层,如蕴含或情感分析。

与相比,相对成本较低。本文中所有结果都可以在单个 Cloud TPU 上至多 1 小时,或在 GPU 上数小时内,从完全相同的模型开始复现。⁷⁷7例如,BERT SQuAD 模型可以在单个 Cloud TPU 上大约 30 分钟内训练完成,达到 91.0% 的 Dev F1 分数。我们在第 4 节相应的小节中描述任务特定的细节。更多细节见附录 A.5。

System	MNLI-(m/mm)	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	Average
	392k	363k	108k	67k	8.5k	5.7k	3.5k	2.5k	-
Pre-OpenAI SOTA	80.6/80.1	66.1	82.3	93.2	35.0	81.0	86.0	61.7	74.0
BiLSTM+ELMo+Attn	76.4/76.1	64.8	79.8	90.4	36.0	73.3	84.9	56.8	71.0
OpenAI GPT	82.1/81.4	70.3	87.4	91.3	45.4	80.0	82.3	56.0	75.1
BERT$ {\textstyle _{\text{BASE}}} $	84.6/83.4	71.2	90.5	93.5	52.1	85.8	88.9	66.4	79.6
BERT$ {\textstyle _{\text{LARGE}}} $	86.7/85.9	72.1	92.7	94.9	60.5	86.5	89.3	70.1	82.1

⁹⁹footnotetext: 参见 https://gluebenchmark.com/faq 中的 (10)。

4 实验

本节中,我们呈现 BERT 在 11 项 NLP 任务上的结果。

4.1 GLUE

General Language Understanding Evaluation (GLUE) 基准 Wang et al. (2018a) 是各种自然语言理解任务的集合。GLUE 数据集的详细描述见附录 B.1。

为了在 GLUE 上进行,我们按第 3 节所述表示输入序列(单句或句子对),并使用与第一个输入词元([CLS])对应的最终隐藏向量 $ {\textstyle C \in {\mathbb{R}}^{H}} $ 作为聚合表征。期间引入的唯一新参数是分类层权重 $ {\textstyle W \in {\mathbb{R}}^{K \times H}} $,其中 $ {\textstyle K} $ 是标签数量。我们使用 $ {\textstyle C} $ 和 $ {\textstyle W} $ 计算标准分类损失,即 $ {\textstyle \log{({{softmax}\hspace{0pt}{({C\hspace{0pt}W^{T}})}})}} $。

我们使用批大小 32,并在所有 GLUE 任务的数据上进行 3 个 的。对于每项任务,我们在 Dev 集上选择最佳的 (在 5e-5、4e-5、3e-5 和 2e-5 之间)。此外,对于 BERT$ {\textstyle _{\text{LARGE}}} $ 我们发现,在小数据集上有时不稳定,因此我们进行多次随机重启,并在 Dev 集上选择最佳模型。在随机重启中,我们使用相同的检查点,但执行不同的数据洗牌和分类器层初始化。¹⁰¹⁰10GLUE 数据集分布不包含测试集标签,我们只为 BERT$ {\textstyle _{\text{BASE}}} $ 和 BERT$ {\textstyle _{\text{LARGE}}} $ 各做了一次 GLUE 评估服务器提交。

结果列于表 9。BERT$ {\textstyle _{\text{BASE}}} $ 和 BERT$ {\textstyle _{\text{LARGE}}} $ 在所有任务上都以显著优势超越所有系统,相对于之前最先进水平分别获得 4.5% 和 7.0% 的平均准确率提升。请注意,BERT$ {\textstyle _{\text{BASE}}} $ 和 OpenAI GPT 在模型架构方面几乎完全相同,仅在掩码上有所不同。对于规模最大、报告最广泛的 GLUE 任务 MNLI,BERT 获得了 4.6% 的绝对准确率提升。在官方 GLUE 排行榜¹¹¹¹11https://gluebenchmark.com/leaderboard 上,撰写本文时,BERT$ {\textstyle _{\text{LARGE}}} $ 获得 80.5 分,相比之下 OpenAI GPT 为 72.8 分。

我们发现 BERT$ {\textstyle _{\text{LARGE}}} $ 在所有任务上都显著优于 BERT$ {\textstyle _{\text{BASE}}} $,特别是在训练数据非常少的任务上。模型规模的影响在第 5.2 节中有更深入的探讨。

4.2 SQuAD v1.1

Stanford Question Answering Dataset (SQuAD v1.1) 是一个包含 10 万个众包问答对的集合 Rajpurkar et al. (2016)。给定一个问题和一段包含答案的维基百科文本,任务是预测段落中答案文本的跨度。

如图 1 所示,在问答任务中,我们将输入问题和段落表示为单个打包序列,问题使用 A ,段落使用 B 。在期间,我们仅引入一个起始向量 $ {\textstyle S \in {\mathbb{R}}^{H}} $ 和一个结束向量 $ {\textstyle E \in {\mathbb{R}}^{H}} $。词 $ {\textstyle i} $ 作为答案跨度起点的概率计算为 $ {\textstyle T_{i}} $ 与 $ {\textstyle S} $ 的点积,后跟段落中所有词的 :$ {\textstyle P_{i} = \frac{e^{S \cdot T_{i}}}{\sum_{j}e^{S \cdot T_{j}}}} $。答案跨度终点使用类似公式。从位置 $ {\textstyle i} $ 到位置 $ {\textstyle j} $ 的候选跨度分数定义为 $ {\textstyle {S \cdot T_{i}} + {E \cdot T_{j}}} $,以 $ {\textstyle j \geq i} $ 中得分最高的跨度作为预测。训练目标是正确起始和结束位置对数似然的和。我们以 5e-5 的和 32 的批大小 3 个 。

System	Dev		Test
	EM	F1	EM	F1
Top Leaderboard Systems (Dec 10th, 2018)
Human	-	-	82.3	91.2
#1 Ensemble - nlnet	-	-	86.0	91.7
#2 Ensemble - QANet	-	-	84.5	90.5
Published
BiDAF+ELMo (Single)	-	85.6	-	85.8
R.M. Reader (Ensemble)	81.2	87.9	82.3	88.5
Ours
BERT$ {\textstyle _{\text{BASE}}} $ (Single)	80.8	88.5	-	-
BERT$ {\textstyle _{\text{LARGE}}} $ (Single)	84.1	90.9	-	-
BERT$ {\textstyle _{\text{LARGE}}} $ (Ensemble)	85.8	91.8	-	-
BERT$ {\textstyle _{\text{LARGE}}} $ (Sgl.+TriviaQA)	84.2	91.1	85.1	91.8
BERT$ {\textstyle _{\text{LARGE}}} $ (Ens.+TriviaQA)	86.2	92.2	87.4	93.2

System	Dev		Test
	EM	F1	EM	F1
Top Leaderboard Systems (Dec 10th, 2018)
Human	86.3	89.0	86.9	89.5
#1 Single - MIR-MRC (F-Net)	-	-	74.8	78.0
#2 Single - nlnet	-	-	74.2	77.1
Published
unet (Ensemble)	-	-	71.4	74.9
SLQA+ (Single)	-		71.4	74.4
Ours
BERT$ {\textstyle _{\text{LARGE}}} $ (Single)	78.7	81.9	80.0	83.1

表 2 显示了排行榜上的最高条目以及顶级已发表系统的结果 Seo et al. (2017); Clark and Gardner (2018); Peters et al. (2018a); Hu et al. (2018)。SQuAD 排行榜的最高结果没有可用的最新公开系统描述,¹²¹²12QANet 在 Yu et al. (2018) 中有描述,但发表后系统有了实质性改进。且允许在训练系统时使用任何公开数据。因此,我们在系统中采用适度的数据增强,先在 TriviaQA Joshi et al. (2017) 上进行,然后再在 SQuAD 上进行。

我们表现最佳的系统在集成时比排行榜首位系统高 +1.5 F1,作为单一系统则高 +1.3 F1。事实上,我们的单一 BERT 模型在 F1 分数上超过了排行榜首位的集成系统。在没有 TriviaQA 数据的情况下,我们只损失 0.1-0.4 F1,仍以较大幅度超越所有现有系统。¹³¹³13我们使用的 TriviaQA 数据由来自 TriviaQA-Wiki 的段落组成,由文档前 400 个词元构成,且至少包含一个提供的可能答案。

4.3 SQuAD v2.0

SQuAD 2.0 任务扩展了 SQuAD 1.1 的问题定义,允许提供的段落中可能不存在简短答案,使问题更接近实际。

我们采用一个简单的方法将 SQuAD v1.1 BERT 模型扩展到该任务。我们将无答案问题视为答案跨度的起点和终点都在 [CLS] 词元上。答案起始和结束位置的概率空间扩展为包含 [CLS] 词元的位置。在预测时,我们将无答案跨度的得分:$ {\textstyle s_{\mathtt{n}\mathtt{u}\mathtt{l}\mathtt{l}} = {{S \cdot C} + {E \cdot C}}} $ 与最佳非空跨度得分 $ {\textstyle \hat{s_{i,j}}} $ = $ {\textstyle {{{\mathtt{m}\mathtt{a}\mathtt{x}}_{j \geq i}\hspace{0pt}S} \cdot T_{i}} + {E \cdot T_{j}}} $ 进行比较。当 $ {\textstyle \hat{s_{i,j}} > {s_{\mathtt{n}\mathtt{u}\mathtt{l}\mathtt{l}} + \tau}} $ 时我们预测非空答案,其中阈值 $ {\textstyle \tau} $ 在 dev 集上选择以最大化 F1。该模型未使用 TriviaQA 数据。我们以 5e-5 的和 48 的批大小 2 个 。

与之前排行榜条目和顶级已发表工作 Sun et al. (2018); Wang et al. (2018b) 相比的结果在表 3 中显示,不包括将 BERT 作为其组件之一的系统。我们观察到比之前最佳系统提升了 +5.1 F1。

4.4 SWAG

Situations With Adversarial Generations (SWAG) 数据集包含 113k 个句子对续写示例,用于评估有依据的常识推理 Zellers et al. (2018)。给定一个句子,任务是在四个选项中选择最合理的续写。

在 SWAG 数据集上时,我们构造四个输入序列,每个序列包含给定句子(句子 A)和可能的续写(句子 B)的拼接。引入的唯一任务特定参数是一个向量,它与 [CLS] 词元表征 $ {\textstyle C} $ 的点积表示每个选项的分数,通过 层进行归一化。

我们以 2e-5 的和 16 的批大小模型 3 个 。结果显示在表 4 中。BERT$ {\textstyle _{\text{LARGE}}} $ 比作者的基线 ESIM+ELMo 系统高 +27.1%,比 OpenAI GPT 高 8.3%。

System	Dev	Test
ESIM+GloVe	51.9	52.7
ESIM+ELMo	59.1	59.2
OpenAI GPT	-	78.0
BERT$ {\textstyle _{\text{BASE}}} $	81.6	-
BERT$ {\textstyle _{\text{LARGE}}} $	86.6	86.3
Human (expert)†	-	85.0
Human (5 annotations)†	-	88.0

5 消融研究

本节中,我们对 BERT 的若干方面进行消融实验,以更好地理解它们的相对重要性。其他消融研究见附录 C。

<span id="effect-of--tasks">

5.1 任务的影响

我们通过使用与 BERT$ {\textstyle _{\text{BASE}}} $ 完全相同的数据、方案和评估两种目标,来证明 BERT 深度双向性的重要性:
No NSP:使用"掩码 LM"(MLM)但没有"下一句预测"(NSP)任务训练的双向模型。
LTR & No NSP:使用标准的从左到右(LTR)LM 而不是 MLM 训练的仅左上下文模型。仅左约束也在中应用,因为去除它引入了/不匹配,会降低下游性能。此外,此模型在时没有 NSP 任务。这与 OpenAI GPT 直接可比,但使用了我们更大的训练数据集、我们的输入表征和我们的方案。

	Dev Set
Tasks	MNLI-m	QNLI	MRPC	SST-2	SQuAD
	(Acc)	(Acc)	(Acc)	(Acc)	(F1)
BERT$ {\textstyle _{\text{BASE}}} $	84.4	88.4	86.7	92.7	88.5
No NSP	83.9	84.9	86.5	92.6	87.9
LTR & No NSP	82.1	84.3	77.5	92.1	77.8
+ BiLSTM	82.1	84.1	75.7	91.6	84.9

我们首先检查 NSP 任务带来的影响。在表 5 中,我们表明去除 NSP 会显著损害 QNLI、MNLI 和 SQuAD 1.1 的性能。接下来,我们通过比较"No NSP"与"LTR & No NSP"来评估训练双向表征的影响。LTR 模型在所有任务上都比 MLM 模型表现更差,在 MRPC 和 SQuAD 上下降幅度很大。

对于 SQuAD 来说,LTR 模型在词元预测上表现不佳是直观的,因为词元级隐藏状态没有右侧上下文。为了诚实地尝试加强 LTR 系统,我们在其上添加了一个随机初始化的 BiLSTM。这显著改善了 SQuAD 的结果,但结果仍远逊于的双向模型。BiLSTM 损害了 GLUE 任务的性能。

我们认识到也可以训练独立的 LTR 和 RTL 模型,并将每个词元表示为两个模型的拼接,正如 ELMo 所做。然而:(a) 这比单个双向模型成本高一倍;(b) 这对于 QA 等任务并不直观,因为 RTL 模型无法将答案以问题为条件;(c) 这严格弱于深度双向模型,因为后者可以在每一层同时使用左右上下文。

5.2 模型规模的影响

本节中,我们探讨模型规模对任务准确率的影响。我们训练了多个具有不同层数、隐藏单元数和头数的 BERT 模型,其余使用与前述相同的和训练过程。

所选 GLUE 任务的结果显示在表 6 中。在此表中,我们报告 5 次随机重启的平均 Dev 集准确率。我们可以看到,更大的模型在所有四个数据集上都带来严格的准确率提升,即使对于只有 3,600 个标注训练样本、与任务实质上不同的 MRPC 也是如此。也许令人惊讶的是,我们能够在相对于现有文献已经相当大的模型基础上取得如此显著的提升。例如,Vaswani et al. (2017) 中探索的最大 为 (L=6, H=1024, A=16),编码器有 1 亿参数,而我们在文献中找到的最大 为 (L=64, H=512, A=2),含 2.35 亿参数 Al-Rfou et al. (2018)。相比之下,BERT$ {\textstyle _{\text{BASE}}} $ 包含 1.1 亿参数,BERT$ {\textstyle _{\text{LARGE}}} $ 包含 3.4 亿参数。

Hyperparams				Dev Set Accuracy
#L	#H	#A	LM (ppl)	MNLI-m	MRPC	SST-2
3	768	12	5.84	77.9	79.8	88.4
6	768	3	5.24	80.6	82.2	90.7
6	768	12	4.68	81.9	84.8	91.3
12	768	12	3.99	84.4	86.7	92.9
12	1024	16	3.54	85.7	86.9	93.3
24	1024	16	3.23	86.6	87.8	93.7

长期以来已知增加模型规模会在机器翻译和语言建模等大规模任务上带来持续改进,这一点由表 6 中显示的留出训练数据的 LM 困惑度所证明。然而,我们相信这是首项令人信服地证明,即使在扩展到极端模型规模时,只要模型经过充分,也能在非常小规模的任务上带来大幅提升的工作。Peters et al. (2018b) 在将的双向 LM 规模从两层增加到四层对下游任务影响方面给出了混合结果;Melamud et al. (2016) 顺带提到将隐藏维度从 200 增加到 600 有所帮助,但进一步增加到 1,000 没有带来进一步提升。这两项先前工作都使用基于特征的方法——我们假设,当模型直接在下游任务上并仅使用非常少量的随机初始化附加参数时,即使下游任务数据非常少,任务特定模型也能从更大、更具表达力的表征中受益。

5.3 BERT 的基于特征方法

到目前为止,所有 BERT 结果都采用了方法,即在模型上添加一个简单的分类层,所有参数在下游任务上联合。然而,基于特征的方法,即从模型中提取固定特征,具有一定优势。首先,并非所有任务都能轻易由 编码器架构表示,因此需要添加任务特定的模型架构。其次,只对训练数据进行一次昂贵表征的预计算,然后基于该表征用更便宜的模型运行许多实验,具有显著的计算收益。

本节中,我们通过将 BERT 应用于 CoNLL-2003 命名实体识别(NER)任务 Tjong Kim Sang and De Meulder (2003) 来比较两种方法。在 BERT 的输入中,我们使用保留大小写的 WordPiece 模型,并包含数据所提供的最大文档上下文。按照标准做法,我们将其表述为标注任务,但在输出端不使用 CRF 层。我们使用第一个子词元的表征作为 NER 标签集上词元级分类器的输入。

为了对方法进行消融,我们采用基于特征的方法,提取一层或多层的而不任何 BERT 参数。这些上下文用作分类层之前一个随机初始化的两层 768 维 BiLSTM 的输入。

结果显示在表 7 中。BERT$ {\textstyle _{\text{LARGE}}} $ 与最先进方法表现相当。表现最好的方法是拼接 顶部四层的词元表征,仅比对整个模型进行低 0.3 F1。这证明 BERT 对和基于特征两种方法都有效。

System	Dev F1	Test F1
ELMo Peters et al. (2018a)	95.7	92.2
CVT Clark et al. (2018)	-	92.6
CSE Akbik et al. (2018)	-	93.1
approach
BERT$ {\textstyle _{\text{LARGE}}} $	96.6	92.8
BERT$ {\textstyle _{\text{BASE}}} $	96.4	92.4
Feature-based approach (BERT$ {\textstyle _{\text{BASE}}} $)
	91.0	-
Second-to-Last Hidden	95.6	-
Last Hidden	94.9	-
Weighted Sum Last Four Hidden	95.9	-
Concat Last Four Hidden	96.1	-
Weighted Sum All 12 Layers	95.5	-

6 结论

近期由于使用语言模型进行迁移学习而产生的实证改进已表明,丰富的无监督是许多语言理解系统的重要组成部分。特别是,这些结果使得即使是低资源任务也能从深度单向架构中受益。我们的主要贡献是将这些发现进一步推广到深度双向架构,使同一模型能够成功应对广泛的 NLP 任务。

参考文献

• Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649.
• Al-Rfou et al. (2018) Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-. arXiv preprint arXiv:1808.04444.
• Ando and Zhang (2005) Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817–1853.
• Bentivogli et al. (2009) Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. NIST.
• Blitzer et al. (2006) John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120–128. Association for Computational Linguistics.
• Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. Association for Computational Linguistics.
• Brown et al. (1992) Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479.
• Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
• Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005.
• Chen et al. (2018) Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs.
• Clark and Gardner (2018) Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL.
• Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914–1925.
• Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM.
• Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
• Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087.
• Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.
• Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
• Fedus et al. (2018) William Fedus, Ian Goodfellow, and Andrew M Dai. 2018. Maskgan: Better text generation via filling in the_. arXiv preprint arXiv:1801.07736.
• Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
• Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
• Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. Association for Computational Linguistics.
• Hu et al. (2018) Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In IJCAI.
• Jernite et al. (2017) Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. CoRR, abs/1705.00557.
• Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
• Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302.
• Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188–1196.
• Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47.
• Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations.
• McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS.
• Melamud et al. (2016) Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context with bidirectional . In CoNLL.
• Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc.
• Mnih and Hinton (2009) Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 1081–1088. Curran Associates, Inc.
• Parikh et al. (2016) Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable model for natural language inference. In EMNLP.
• Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
• Peters et al. (2017) Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL.
• Peters et al. (2018a) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL.
• Peters et al. (2018b) Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word : Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509.
• Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI.
• Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
• Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional flow for machine comprehension. In ICLR.
• Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
• Sun et al. (2018) Fu Sun, Linyang Li, Xipeng Qiu, and Yang Liu. 2018. U-net: Machine reading comprehension with unanswerable questions. arXiv preprint arXiv:1810.06638.
• Taylor (1953) Wilson L Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433.
• Tjong Kim Sang and De Meulder (2003) Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In CoNLL.
• Turian et al. (2010) Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 384–394.
• Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010.
• Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103. ACM.
• Wang et al. (2018a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
• Wang et al. (2018b) Wei Wang, Ming Yan, and Chen Wu. 2018b. Multi-granularity hierarchical fusion networks for reading comprehension and question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
• Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.
• Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
• Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
• Yosinski et al. (2014) Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328.
• Yu et al. (2018)  Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining local with global self- for reading comprehension. In ICLR.
• Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27.

"BERT:用于语言理解的深度双向"附录

我们将附录分为三个部分:

• •

  BERT 的其他实现细节见附录 A;

• •

  我们实验的其他细节见附录 B;以及

• •

  其他消融研究见附录 C。

  我们为 BERT 提供其他消融研究,包括:

  • –

    训练步数的影响;以及

  • –

    不同掩码过程的消融。

附录 A BERT 的其他细节

<span id="a.1-illustration-of-the--tasks">

A.1 任务图示

下面我们提供任务的示例。

掩码 LM 与掩码过程

假设未标注的句子是 my dog is hairy,在随机掩码过程中我们选择了第 4 个词元(对应 hairy),我们的掩码过程可以进一步说明为

• •

  80% 的概率:用 [MASK] 词元替换该词,例如 my dog is hairy →→\rightarrow my dog is [MASK]

• •

  10% 的概率:用随机词替换该词,例如 my dog is hairy →→\rightarrow my dog is apple

• •

  10% 的概率:保持该词不变,例如 my dog is hairy →→\rightarrow my dog is hairy。其目的是让表征偏向实际观察到的词。

该过程的优势在于 编码器不知道将要求其预测哪些词或哪些已被随机词替换,因此被迫为每个输入词元保留分布式上下文表征。此外,由于随机替换仅发生在所有词元的 1.5%(即 15% 的 10%),这似乎不会损害模型的语言理解能力。在第 C.2 节中,我们评估此过程的影响。

与标准语言模型训练相比,掩码 LM 在每个批次中仅对 15% 的词元进行预测,这表明模型收敛可能需要更多的步数。在第 C.1 节中,我们证明 MLM 确实收敛略慢于从左到右模型(后者预测每个词元),但 MLM 模型的实证改进远远超过训练成本的增加。

下一句预测

下一句预测任务可以通过以下示例说明。

	Input	$ {\textstyle = \text{[CLS] the man went to [MASK] store [SEP]}} $
		he bought a gallon [MASK] milk [SEP]
	Label	$ {\textstyle = \text{IsNext}} $
	Input	$ {\textstyle = \text{[CLS] the man [MASK] to the store [SEP]}} $
		penguin [MASK] are flight ##less birds [SEP]
	Label	$ {\textstyle = \text{NotNext}} $

<span id="a.2--procedure">

A.2 过程

为了生成每个训练输入序列,我们从语料库中采样两个文本跨度,将其称为"句子",尽管它们通常比单个句子长得多(但也可能更短)。第一句获得 A ,第二句获得 B 。50% 的情况下 B 是实际跟随 A 的下一句,50% 的情况下它是随机句子,这用于"下一句预测"任务。采样使得组合长度 $ {\textstyle \leq} $ 512 个词元。LM 掩码在 WordPiece 之后应用,统一掩码率为 15%,对部分子词片段不做特殊处理。

我们以 256 个序列的批大小(256 个序列 * 512 词元 = 128,000 词元/批)训练 1,000,000 步,约相当于在 33 亿词语料上 40 个 。我们使用 ,为 1e-4,$ {\textstyle \beta_{1} = 0.9} $,$ {\textstyle \beta_{2} = 0.999} $,L2 为 $ {\textstyle 0.01} $,前 10,000 步进行热身,以及线性衰减。我们在所有层使用 0.1 的 概率。沿用 OpenAI GPT,我们使用 gelu Hendrycks and Gimpel (2016) 而非标准 relu。训练损失是平均掩码 LM 似然和平均下一句预测似然之和。

BERT$ {\textstyle _{\text{BASE}}} $ 的训练在 4 个 Pod 配置的 Cloud TPU 上进行(共 16 个 TPU 芯片)。¹⁴¹⁴14https://cloudplatform.googleblog.com/2018/06/Cloud-TPU-now-offers-preemptible-pricing-and-global-availability.html BERT$ {\textstyle _{\text{LARGE}}} $ 的训练在 16 个 Cloud TPU 上进行(共 64 个 TPU 芯片)。每次耗时 4 天完成。

较长序列的成本不成比例地高,因为是序列长度的平方级。为了加速我们实验中的,我们在 90% 的步数中以序列长度 128 模型。然后,我们在剩余 10% 的步数中以序列长度 512 训练,以学习位置。

<span id="a.3--procedure">

A.3 过程

对于,大部分模型与相同,只有批大小、和训练数除外。 概率始终保持在 0.1。最佳值是任务特定的,但我们发现以下可能值的范围在所有任务上都效果良好:

• •

  批大小:16, 32

• •

  ():5e-5, 3e-5, 2e-5

• •

  数:2, 3, 4

我们还观察到,大数据集(例如 100k+ 标注训练样本)对的选择远不如小数据集敏感。通常非常快,因此简单地对上述参数进行穷举搜索并选择在开发集上表现最佳的模型是合理的。

A.4 BERT、ELMo 与 OpenAI GPT 的比较

此处我们研究近期流行的表征学习模型(包括 ELMo、OpenAI GPT 和 BERT)之间的差异。模型架构之间的比较以视觉方式显示在图 3 中。注意,除了架构差异之外,BERT 和 OpenAI GPT 是方法,而 ELMo 是基于特征的方法。

与 BERT 最具可比性的现有方法是 OpenAI GPT,它在大型文本语料上训练一个从左到右的 LM。事实上,BERT 中许多设计决策是有意做得尽可能接近 GPT,以便两种方法能够进行最少差异的比较。本工作的核心论点是双向性以及第 3.1 节中提出的两个任务解释了大部分实证改进,但我们注意到 BERT 和 GPT 训练之间还存在若干其他差异:

• •

  GPT 在 BooksCorpus(800M 词)上训练;BERT 在 BooksCorpus(800M 词)和维基百科(2,500M 词)上训练。

• •

  GPT 使用句子分隔符([SEP])和分类器词元([CLS]),它们仅在时引入;BERT 在期间学习 [SEP]、[CLS] 和句子 A/B 的。

• •

  GPT 训练了 1M 步,批大小为 32,000 词;BERT 训练了 1M 步,批大小为 128,000 词。

• •

  GPT 在所有实验中使用相同的 5e-5 ;BERT 选择在开发集上表现最佳的任务特定 。

为了分离这些差异的影响,我们在第 5.1 节进行消融实验,证明大部分改进实际上来自两个任务及其所启用的双向性。

<span id="a.5-illustrations-of--on-different-tasks">

A.5 不同任务上的图示

在不同任务上 BERT 的图示见图 4。我们的任务特定模型通过将 BERT 与一个附加输出层相结合形成,因此只需从零学习极少数参数。在这些任务中,(a) 和 (b) 是序列级任务,而 (c) 和 (d) 是词元级任务。图中,$ {\textstyle E} $ 代表输入,$ {\textstyle T_{i}} $ 代表词元 $ {\textstyle i} $ 的上下文表征,[CLS] 是用于分类输出的特殊符号,[SEP] 是用于分隔非连续词元序列的特殊符号。

附录 B 详细实验设置

B.1 GLUE 基准实验的详细描述。

我们在表 9 中的 GLUE 结果来自 https://gluebenchmark.com/leaderboard 和 https://blog.openai.com/language-unsupervised。GLUE 基准包括以下数据集,其描述最初在 Wang et al. (2018a) 中作了总结:

MNLI

Multi-Genre Natural Language Inference 是一个大规模、众包的蕴含分类任务 Williams et al. (2018)。给定一对句子,目标是预测第二句相对于第一句是蕴含、矛盾还是中性。

QQP

Quora Question Pairs 是一个二分类任务,目标是判断 Quora 上提出的两个问题在语义上是否等价 Chen et al. (2018)。

QNLI

Question Natural Language Inference 是 Stanford Question Answering Dataset Rajpurkar et al. (2016) 的一个版本,已被转换为二分类任务 Wang et al. (2018a)。正例是确实包含正确答案的(问题, 句子)对,负例是来自同一段落但不包含答案的(问题, 句子)。

SST-2

Stanford Sentiment Treebank 是一个单句二分类任务,由从电影评论中提取的句子及其人工情感标注组成 Socher et al. (2013)。

CoLA

Corpus of Linguistic Acceptability 是一个单句二分类任务,目标是预测一个英语句子在语言学上是否"可接受" Warstadt et al. (2018)。

STS-B

Semantic Textual Similarity Benchmark 是从新闻标题和其他来源中抽取的句子对集合 Cer et al. (2017)。它们被标注 1 到 5 的分数,表示两个句子在语义上的相似程度。

MRPC

Microsoft Research Paraphrase Corpus 由从在线新闻来源自动提取的句子对组成,并附有句子对在语义上是否等价的人工标注 Dolan and Brockett (2005)。

RTE

Recognizing Textual Entailment 是类似于 MNLI 的二元蕴含任务,但训练数据少得多 Bentivogli et al. (2009)。¹⁵¹⁵15请注意,本文中我们仅报告单任务结果。多任务方法可能进一步推动性能。例如,我们确实观察到通过与 MNLI 进行多任务训练,RTE 上获得了大幅提升。

WNLI

Winograd NLI 是一个小型自然语言推断数据集 Levesque et al. (2011)。GLUE 网页指出该数据集的构造存在问题,¹⁶¹⁶16https://gluebenchmark.com/faq 提交到 GLUE 的所有训练系统都表现不及预测多数类的 65.1 基线准确率。因此,为了对 OpenAI GPT 公平,我们排除该数据集。对于我们的 GLUE 提交,我们始终预测多数类。

附录 C 其他消融研究

C.1 训练步数的影响

图 5 呈现了从已经过 $ {\textstyle k} $ 步的检查点后的 MNLI Dev 准确率。这使我们能够回答以下问题:

[[File:data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjQwOC4yNyIgaWQ9IkEzLkY1LnBpYzEiIG92ZXJmbG93PSJ2aXNpYmxlIiB2ZXJzaW9uPSIxLjEiIHdpZHRoPSI1NzAuMTEiPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgc3Ryb2tlLXdpZHRoPSIwLjRwdCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCw0MDguMjcpIG1hdHJpeCgxIDAgMCAtMSAwIDApIHRyYW5zbGF0ZSg0Mi4wMSwwKSB0cmFuc2xhdGUoMCw0MS40NykgbWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtNDIuMDEgLTQxLjQ3KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDApIHRyYW5zbGF0ZSg0Mi4wMSwwKSB0cmFuc2xhdGUoMCw0MS40NykiPjxnIGNvbG9yPSIjQkZCRkJGIiBmaWxsPSIjQkZCRkJGIiBzdHJva2U9IiNCRkJGQkYiIHN0cm9rZS13aWR0aD0iMC40cHQiPjxwYXRoIGQ9Ik0gMTAxLjU1IDAgTCAxMDEuNTUgMzU3LjczIE0gMjAzLjA5IDAgTCAyMDMuMDkgMzU3LjczIE0gMzA0LjY0IDAgTCAzMDQuNjQgMzU3LjczIE0gNDA2LjE5IDAgTCA0MDYuMTkgMzU3LjczIE0gNTA3LjczIDAgTCA1MDcuNzMgMzU3LjczIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBjb2xvcj0iI0JGQkZCRiIgZmlsbD0iI0JGQkZCRiIgc3Ryb2tlPSIjQkZCRkJGIiBzdHJva2Utd2lkdGg9IjAuNHB0Ij48cGF0aCBkPSJNIDAgMCBMIDUwNy43MyAwIE0gMCAzNS43NyBMIDUwNy43MyAzNS43NyBNIDAgNzEuNTUgTCA1MDcuNzMgNzEuNTUgTSAwIDEwNy4zMiBMIDUwNy43MyAxMDcuMzIgTSAwIDE0My4wOSBMIDUwNy43MyAxNDMuMDkgTSAwIDE3OC44NyBMIDUwNy43MyAxNzguODcgTSAwIDIxNC42NCBMIDUwNy43MyAyMTQuNjQgTSAwIDI1MC40MSBMIDUwNy43MyAyNTAuNDEgTSAwIDI4Ni4xOSBMIDUwNy43MyAyODYuMTkgTSAwIDMyMS45NiBMIDUwNy43MyAzMjEuOTYgTSAwIDM1Ny43MyBMIDUwNy43MyAzNTcuNzMiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIGNvbG9yPSIjODA4MDgwIiBmaWxsPSIjODA4MDgwIiBzdHJva2U9IiM4MDgwODAiIHN0cm9rZS13aWR0aD0iMC4ycHQiPjxwYXRoIGQ9Ik0gMTAxLjU1IDAgTCAxMDEuNTUgNS45MSBNIDIwMy4wOSAwIEwgMjAzLjA5IDUuOTEgTSAzMDQuNjQgMCBMIDMwNC42NCA1LjkxIE0gNDA2LjE5IDAgTCA0MDYuMTkgNS45MSBNIDUwNy43MyAwIEwgNTA3LjczIDUuOTEgTSAxMDEuNTUgMzU3LjczIEwgMTAxLjU1IDM1MS44MyBNIDIwMy4wOSAzNTcuNzMgTCAyMDMuMDkgMzUxLjgzIE0gMzA0LjY0IDM1Ny43MyBMIDMwNC42NCAzNTEuODMgTSA0MDYuMTkgMzU3LjczIEwgNDA2LjE5IDM1MS44MyBNIDUwNy43MyAzNTcuNzMgTCA1MDcuNzMgMzUxLjgzIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBjb2xvcj0iIzgwODA4MCIgZmlsbD0iIzgwODA4MCIgc3Ryb2tlPSIjODA4MDgwIiBzdHJva2Utd2lkdGg9IjAuMnB0Ij48cGF0aCBkPSJNIDAgMCBMIDUuOTEgMCBNIDAgMzUuNzcgTCA1LjkxIDM1Ljc3IE0gMCA3MS41NSBMIDUuOTEgNzEuNTUgTSAwIDEwNy4zMiBMIDUuOTEgMTA3LjMyIE0gMCAxNDMuMDkgTCA1LjkxIDE0My4wOSBNIDAgMTc4Ljg3IEwgNS45MSAxNzguODcgTSAwIDIxNC42NCBMIDUuOTEgMjE0LjY0IE0gMCAyNTAuNDEgTCA1LjkxIDI1MC40MSBNIDAgMjg2LjE5IEwgNS45MSAyODYuMTkgTSAwIDMyMS45NiBMIDUuOTEgMzIxLjk2IE0gMCAzNTcuNzMgTCA1LjkxIDM1Ny43MyBNIDUwNy43MyAwIEwgNTAxLjgzIDAgTSA1MDcuNzMgMzUuNzcgTCA1MDEuODMgMzUuNzcgTSA1MDcuNzMgNzEuNTUgTCA1MDEuODMgNzEuNTUgTSA1MDcuNzMgMTA3LjMyIEwgNTAxLjgzIDEwNy4zMiBNIDUwNy43MyAxNDMuMDkgTCA1MDEuODMgMTQzLjA5IE0gNTA3LjczIDE3OC44NyBMIDUwMS44MyAxNzguODcgTSA1MDcuNzMgMjE0LjY0IEwgNTAxLjgzIDIxNC42NCBNIDUwNy43MyAyNTAuNDEgTCA1MDEuODMgMjUwLjQxIE0gNTA3LjczIDI4Ni4xOSBMIDUwMS44MyAyODYuMTkgTSA1MDcuNzMgMzIxLjk2IEwgNTAxLjgzIDMyMS45NiBNIDUwNy43MyAzNTcuNzMgTCA1MDEuODMgMzU3LjczIiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMC40cHQiPjxwYXRoIGQ9Ik0gMCAwIEwgMCAzNTcuNzMgTCA1MDcuNzMgMzU3LjczIEwgNTA3LjczIDAgTCAwIDAgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCA5MS4xNyAtMTMuODEpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjAuNzYiPjxtYXRoIGFsdHRleHQ9IjIwMCIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtbiBpZD0iQTMuRjUucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+MjAwPC9tbj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iQTMuRjUucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxjbiBpZD0iQTMuRjUucGljMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+MjAwPC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj4yMDA8L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTkyLjcyIC0xMy44MSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC45MiIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIyMC43NiI+PG1hdGggYWx0dGV4dD0iNDAwIiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj40MDA8L21uPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYiI+PGNuIGlkPSJBMy5GNS5waWMxLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iQTMuRjUucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj40MDA8L2NuPjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iQTMuRjUucGljMS4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPjQwMDwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAyOTQuMjYgLTEzLjgxKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjkyIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjIwLjc2Ij48bWF0aCBhbHR0ZXh0PSI2MDAiIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IkEzLkY1LnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bW4gaWQ9IkEzLkY1LnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iQTMuRjUucGljMS4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjYwMDwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjYwMDwvY24+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4zLjMuMy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+NjAwPC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDM5NS44MSAtMTMuODEpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMjAuNzYiPjxtYXRoIGFsdHRleHQ9IjgwMCIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtbiBpZD0iQTMuRjUucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+ODAwPC9tbj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iQTMuRjUucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxjbiBpZD0iQTMuRjUucGljMS40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+ODAwPC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjQuNC40LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj44MDA8L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgNDkxLjk3IC0xMy44MSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTEuNjEiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMzEuNTIiPjxtYXRoIGFsdHRleHQ9IjF7LH0wMDAiIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIiPjxzZW1hbnRpY3MgaWQ9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjJhIj48bXJvdyBpZD0iQTMuRjUucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMi4zLjIiIHhyZWY9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIuMy4xLmNtbWwiPjxtbiBpZD0iQTMuRjUucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+MTwvbW4+PG1vIGlkPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4yLjMuMi4xIiB4cmVmPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4yLjMuMS5jbW1sIj4sPC9tbz48bW4gaWQ9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIuMiIgeHJlZj0iQTMuRjUucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMi4yLmNtbWwiPjAwMDwvbW4+PC9tcm93Pjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4yYiI+PGxpc3QgaWQ9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIuMy4xLmNtbWwiIHhyZWY9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIuMy4yIj48Y24gaWQ9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjE8L2NuPjxjbiBpZD0iQTMuRjUucGljMS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMi4yLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjIuMiI+MDAwPC9jbj48L2xpc3Q+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS41LjUuNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4yYyI+MXssfTAwMDwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgLTQuNDYpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTMuODQiPjxtYXRoIGFsdHRleHQ9Ijc1IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj43NTwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjc1PC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjYuNi42LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj43NTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgMzEuMzEpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTMuODQiPjxtYXRoIGFsdHRleHQ9Ijc2IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj43NjwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjc2PC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjcuNy43LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj43NjwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgNjcuMDkpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTMuODQiPjxtYXRoIGFsdHRleHQ9Ijc3IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj43NzwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjc3PC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjguOC44LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj43NzwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgMTAyLjg2KSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjkyIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEzLjg0Ij48bWF0aCBhbHR0ZXh0PSI3OCIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtbiBpZD0iQTMuRjUucGljMS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+Nzg8L21uPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYiI+PGNuIGlkPSJBMy5GNS5waWMxLjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iQTMuRjUucGljMS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj43ODwvY24+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS45LjkuOS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+Nzg8L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTE4LjczIDEzOC42MykiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC45MiIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxMy44NCI+PG1hdGggYWx0dGV4dD0iNzkiIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iQTMuRjUucGljMS4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xMC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+Nzk8L21uPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMTAuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj43OTwvY24+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEwLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj43OTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgMTc0LjQxKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjkyIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEzLjg0Ij48bWF0aCBhbHR0ZXh0PSI4MCIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bW4gaWQ9IkEzLkY1LnBpYzEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjExLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj44MDwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxjbiBpZD0iQTMuRjUucGljMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iQTMuRjUucGljMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjgwPC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMTEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPjgwPC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC0xOC43MyAyMTAuMTgpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjguOTIiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTMuODQiPjxtYXRoIGFsdHRleHQ9IjgxIiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IkEzLkY1LnBpYzEuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtbiBpZD0iQTMuRjUucGljMS4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMTIuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjgxPC9tbj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iQTMuRjUucGljMS4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYiI+PGNuIGlkPSJBMy5GNS5waWMxLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB0eXBlPSJpbnRlZ2VyIiB4cmVmPSJBMy5GNS5waWMxLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEyLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+ODE8L2NuPjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iQTMuRjUucGljMS4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xMi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+ODE8L2Fubm90YXRpb24+PC9zZW1hbnRpY3M+PC9tYXRoPjwvZm9yZWlnbm9iamVjdD48L2c+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgLTE4LjczIDI0NS45NikiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC45MiIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxMy44NCI+PG1hdGggYWx0dGV4dD0iODIiIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iQTMuRjUucGljMS4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xMy4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+ODI8L21uPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMTMuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj44MjwvY24+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEzLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj44MjwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgMjgxLjczKSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjkyIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEzLjg0Ij48bWF0aCBhbHR0ZXh0PSI4MyIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bW4gaWQ9IkEzLkY1LnBpYzEuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjE0LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj44MzwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxjbiBpZD0iQTMuRjUucGljMS4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iQTMuRjUucGljMS4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xNC4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjgzPC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMTQuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPjgzPC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIC0xOC43MyAzMTcuNSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iOC45MiIgb3ZlcmZsb3c9InZpc2libGUiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMTYuNikiIHdpZHRoPSIxMy44NCI+PG1hdGggYWx0dGV4dD0iODQiIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1uIGlkPSJBMy5GNS5waWMxLjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iQTMuRjUucGljMS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xNS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+ODQ8L21uPjxhbm5vdGF0aW9uLXhtbCBlbmNvZGluZz0iTWF0aE1MLUNvbnRlbnQiIGlkPSJBMy5GNS5waWMxLjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFiIj48Y24gaWQ9IkEzLkY1LnBpYzEuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiIHR5cGU9ImludGVnZXIiIHhyZWY9IkEzLkY1LnBpYzEuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMTUuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIj44NDwvY24+PC9hbm5vdGF0aW9uLXhtbD48YW5ub3RhdGlvbiBlbmNvZGluZz0iYXBwbGljYXRpb24veC10ZXgiIGlkPSJBMy5GNS5waWMxLjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjE1LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFjIj44NDwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+PC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDEuMCAwLjAgMC4wIDEuMCAtMTguNzMgMzUzLjI4KSI+PGZvcmVpZ25vYmplY3QgaGVpZ2h0PSI4LjkyIiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjEzLjg0Ij48bWF0aCBhbHR0ZXh0PSI4NSIgZGlzcGxheT0iaW5saW5lIiBpZD0iQTMuRjUucGljMS4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xIj48c2VtYW50aWNzIGlkPSJBMy5GNS5waWMxLjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjFhIj48bW4gaWQ9IkEzLkY1LnBpYzEuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjE2LjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj44NTwvbW4+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxjbiBpZD0iQTMuRjUucGljMS4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgdHlwZT0iaW50ZWdlciIgeHJlZj0iQTMuRjUucGljMS4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xNi4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjg1PC9jbj48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMTYuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPjg1PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD48L2ZvcmVpZ25vYmplY3Q+PC9nPjxjbGlwcGF0aCBpZD0icGdmY3AxIj48cGF0aCBkPSJNIDAgMCBMIDUwNy43MyAwIEwgNTA3LjczIDM1Ny43MyBMIDAgMzU3LjczIFoiIC8+PC9jbGlwcGF0aD48ZyBjbGlwLXBhdGg9InVybCgjcGdmY3AxKSI+PGcgY29sb3I9IiM0Mjg1RjQiIGZpbGw9IiM0Mjg1RjQiIHN0cm9rZT0iIzQyODVGNCI+PHBhdGggZD0iTSAxNS4yMyAxMjguNzggTCAyNS4zOSAxNjQuNTYgTCA1MC43NyAxOTYuNzUgTCAxMDEuNTUgMjU3LjU3IEwgMjAzLjA5IDI5My4zNCBMIDMwNC42NCAzMjEuOTYgTCA0MDYuMTkgMzMyLjY5IEwgNTA3LjczIDMzNi4yNyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGc+PC9nPjxnIGNvbG9yPSIjREI0NDM3IiBmaWxsPSIjREI0NDM3IiBzdHJva2U9IiNEQjQ0MzciPjxwYXRoIGQ9Ik0gMTUuMjMgMTU3LjQgTCAyNS4zOSAxNzEuNzEgTCA1MC43NyAxODkuNiBMIDEwMS41NSAyMTQuNjQgTCAyMDMuMDkgMjM5LjY4IEwgMzA0LjY0IDI0Ni44NCBMIDQwNi4xOSAyNTMuOTkgTCA1MDcuNzMgMjU3LjU3IiBzdHlsZT0iZmlsbDpub25lIiAvPjwvZz48Zz48L2c+PC9nPjxnIGNvbG9yPSIjNDI4NUY0IiBmaWxsPSIjNDI4NUY0IiBzdHJva2U9IiM0Mjg1RjQiPjxwYXRoIGQ9Ik0gMTUuMjMgMTMyLjk0IEwgMTguODMgMTI2LjcxIEwgMTEuNjQgMTI2LjcxIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAyNS4zOSAxNjguNzEgTCAyOC45OCAxNjIuNDggTCAyMS43OSAxNjIuNDggWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDUwLjc3IDIwMC45IEwgNTQuMzcgMTk0LjY4IEwgNDcuMTggMTk0LjY4IFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAxMDEuNTUgMjYxLjcyIEwgMTA1LjE0IDI1NS40OSBMIDk3Ljk1IDI1NS40OSBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gMjAzLjA5IDI5Ny40OSBMIDIwNi42OSAyOTEuMjcgTCAxOTkuNSAyOTEuMjcgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDMwNC42NCAzMjYuMTEgTCAzMDguMjMgMzE5Ljg4IEwgMzAxLjA0IDMxOS44OCBaIiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gNDA2LjE5IDMzNi44NCBMIDQwOS43OCAzMzAuNjIgTCA0MDIuNTkgMzMwLjYyIFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSA1MDcuNzMgMzQwLjQyIEwgNTExLjMzIDMzNC4xOSBMIDUwNC4xNCAzMzQuMTkgWiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgY29sb3I9IiNEQjQ0MzciIGZpbGw9IiNEQjQ0MzciIHN0cm9rZT0iI0RCNDQzNyI+PHBhdGggZD0iTSAxMi4zIDE1NC40NyBMIDE4LjE3IDE2MC4zNCBNIDEyLjMgMTYwLjM0IEwgMTguMTcgMTU0LjQ3IiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gMjIuNDUgMTY4Ljc4IEwgMjguMzIgMTc0LjY1IE0gMjIuNDUgMTc0LjY1IEwgMjguMzIgMTY4Ljc4IiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gNDcuODQgMTg2LjY2IEwgNTMuNzEgMTkyLjUzIE0gNDcuODQgMTkyLjUzIEwgNTMuNzEgMTg2LjY2IiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gOTguNjEgMjExLjcxIEwgMTA0LjQ4IDIxNy41OCBNIDk4LjYxIDIxNy41OCBMIDEwNC40OCAyMTEuNzEiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAyMDAuMTYgMjM2Ljc1IEwgMjA2LjAzIDI0Mi42MiBNIDIwMC4xNiAyNDIuNjIgTCAyMDYuMDMgMjM2Ljc1IiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gMzAxLjcgMjQzLjkgTCAzMDcuNTggMjQ5Ljc3IE0gMzAxLjcgMjQ5Ljc3IEwgMzA3LjU4IDI0My45IiBzdHlsZT0iZmlsbDpub25lIiAvPjxwYXRoIGQ9Ik0gNDAzLjI1IDI1MS4wNiBMIDQwOS4xMiAyNTYuOTMgTSA0MDMuMjUgMjU2LjkzIEwgNDA5LjEyIDI1MS4wNiIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDUwNC44IDI1NC42MyBMIDUxMC42NyAyNjAuNSBNIDUwNC44IDI2MC41IEwgNTEwLjY3IDI1NC42MyIgc3R5bGU9ImZpbGw6bm9uZSIgLz48L2c+PGcgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSIjMDAwMDAwIiB0cmFuc2Zvcm09Im1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMTU4LjMgLTMzLjQpIj48Zm9yZWlnbm9iamVjdCBoZWlnaHQ9IjEzLjg0IiBvdmVyZmxvdz0idmlzaWJsZSIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAxNi42KSIgd2lkdGg9IjE5MS4xNCI+UHJlLXRyYWluaW5nIFN0ZXBzIChUaG91c2FuZHMpPC9mb3JlaWdub2JqZWN0PjwvZz48ZyBmaWxsPSIjMDAwMDAwIiBzdHJva2U9IiMwMDAwMDAiIHRyYW5zZm9ybT0ibWF0cml4KDAuMCAxLjAgLTEuMCAwLjAgLTI3Ljk0IDExNS44MSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTIuMTUiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTI2LjExIj5NTkxJIERldiBBY2N1cmFjeTwvZm9yZWlnbm9iamVjdD48L2c+PGcgZmlsbD0iI0ZGRkZGRiIgc3Ryb2tlPSIjMDAwMDAwIj48cGF0aCBkPSJNIDMxNS4xMSAwLjI4IGggMTkyLjM1IHYgNDEuNTEgaCAtMTkyLjM1IFoiIC8+PC9nPjxnIGZpbGw9IiNGRkZGRkYiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDMxOS4yNiAzLjA0KSI+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAyNi45ODUpIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAxIDAgOC45OSkiPjxnIGNvbG9yPSIjNDI4NUY0IiBmaWxsPSIjNDI4NUY0IiBzdHJva2U9IiM0Mjg1RjQiIHRyYW5zZm9ybT0ibWF0cml4KDEgMCAwIC0xIDAgMCkgdHJhbnNsYXRlKDAuMjgsMCkiPjxwYXRoIGQ9Ik0gMCAwIEwgMTEuODEgMCBMIDIzLjYyIDAiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PHBhdGggZD0iTSAxMS44MSA0LjE1IEwgMTUuNDEgLTIuMDggTCA4LjIyIC0yLjA4IFoiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMjQuNDUgMCkgdHJhbnNsYXRlKC0wLjI4LDApIG1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMy4wNCAtNC4xNSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTMuODQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTQ3Ljc5Ij5CRVJUPG1hdGggYWx0dGV4dD0ie31fe1x0ZXh0c2N7QkFTRX19IiBkaXNwbGF5PSJpbmxpbmUiIGlkPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEiPjxzZW1hbnRpY3MgaWQ9IkEzLkY1LnBpYzEuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWEiPjxtc3ViIGlkPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSIgeHJlZj0iQTMuRjUucGljMS4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PG1pIGlkPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMWEiIHhyZWY9IkEzLkY1LnBpYzEuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLmNtbWwiPjwvbWk+PG10ZXh0IGlkPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xYS5jbW1sIj5CQVNFPC9tdGV4dD48L21zdWI+PGFubm90YXRpb24teG1sIGVuY29kaW5nPSJNYXRoTUwtQ29udGVudCIgaWQ9IkEzLkY1LnBpYzEuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWIiPjxhcHBseSBpZD0iQTMuRjUucGljMS4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCIgeHJlZj0iQTMuRjUucGljMS4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEiPjxjaSBpZD0iQTMuRjUucGljMS4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMWEuY21tbCIgeHJlZj0iQTMuRjUucGljMS4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xNy4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMSI+PG10ZXh0IGlkPSJBMy5GNS5waWMxLjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjE3LjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xLmNtbWwiIG1hdGhzaXplPSI3MCUiIHhyZWY9IkEzLkY1LnBpYzEuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjEiPkJBU0U8L210ZXh0PjwvY2k+PC9hcHBseT48L2Fubm90YXRpb24teG1sPjxhbm5vdGF0aW9uIGVuY29kaW5nPSJhcHBsaWNhdGlvbi94LXRleCIgaWQ9IkEzLkY1LnBpYzEuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMTcuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMWMiPnt9X3tcdGV4dHNje0JBU0V9fTwvYW5ub3RhdGlvbj48L3NlbWFudGljcz48L21hdGg+IChNYXNrZWQgTE0pPC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PGcgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgMSAwIDI2Ljk4KSI+PGcgY29sb3I9IiNEQjQ0MzciIGZpbGw9IiNEQjQ0MzciIHN0cm9rZT0iI0RCNDQzNyIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMCAwKSB0cmFuc2xhdGUoMC4yOCwwKSI+PHBhdGggZD0iTSAwIDAgTCAxMS44MSAwIEwgMjMuNjIgMCIgc3R5bGU9ImZpbGw6bm9uZSIgLz48cGF0aCBkPSJNIDguODggLTIuOTQgTCAxNC43NSAyLjk0IE0gOC44OCAyLjk0IEwgMTQuNzUgLTIuOTQiIHN0eWxlPSJmaWxsOm5vbmUiIC8+PC9nPjxnIGZpbGw9IiMwMDAwMDAiIHN0cm9rZT0iIzAwMDAwMCIgdHJhbnNmb3JtPSJtYXRyaXgoMSAwIDAgLTEgMjQuNDUgMCkgdHJhbnNsYXRlKC0wLjI4LDApIG1hdHJpeCgxLjAgMC4wIDAuMCAxLjAgMy4wNCAtNC4xNSkiPjxmb3JlaWdub2JqZWN0IGhlaWdodD0iMTMuODQiIG92ZXJmbG93PSJ2aXNpYmxlIiB0cmFuc2Zvcm09Im1hdHJpeCgxIDAgMCAtMSAwIDE2LjYpIiB3aWR0aD0iMTU1LjYiPkJFUlQ8bWF0aCBhbHR0ZXh0PSJ7fV97XHRleHRzY3tCQVNFfX0iIGRpc3BsYXk9ImlubGluZSIgaWQ9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMSI+PHNlbWFudGljcyBpZD0iQTMuRjUucGljMS4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYSI+PG1zdWIgaWQ9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xIiB4cmVmPSJBMy5GNS5waWMxLjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIj48bWkgaWQ9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xYSIgeHJlZj0iQTMuRjUucGljMS4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuY21tbCI+PC9taT48bXRleHQgaWQ9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjEiIHhyZWY9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjFhLmNtbWwiPkJBU0U8L210ZXh0PjwvbXN1Yj48YW5ub3RhdGlvbi14bWwgZW5jb2Rpbmc9Ik1hdGhNTC1Db250ZW50IiBpZD0iQTMuRjUucGljMS4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYiI+PGFwcGx5IGlkPSJBMy5GNS5waWMxLjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS5jbW1sIiB4cmVmPSJBMy5GNS5waWMxLjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMSI+PGNpIGlkPSJBMy5GNS5waWMxLjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xYS5jbW1sIiB4cmVmPSJBMy5GNS5waWMxLjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjE4LjIuMi4yLjIuMi4yLjIuMi4yLjIuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLm0xLjEuMS4xIj48bXRleHQgaWQ9IkEzLkY1LnBpYzEuMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMTguMi4yLjIuMi4yLjIuMi4yLjIuMi4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEubTEuMS4xLjEuY21tbCIgbWF0aHNpemU9IjcwJSIgeHJlZj0iQTMuRjUucGljMS4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xLjEuMSI+QkFTRTwvbXRleHQ+PC9jaT48L2FwcGx5PjwvYW5ub3RhdGlvbi14bWw+PGFubm90YXRpb24gZW5jb2Rpbmc9ImFwcGxpY2F0aW9uL3gtdGV4IiBpZD0iQTMuRjUucGljMS4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4xOC4yLjIuMi4yLjIuMi4yLjIuMi4yLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS4xLjEuMS5tMS4xYyI+e31fe1x0ZXh0c2N7QkFTRX19PC9hbm5vdGF0aW9uPjwvc2VtYW50aWNzPjwvbWF0aD4gKExlZnQtdG8tUmlnaHQpPC9mb3JlaWdub2JqZWN0PjwvZz48L2c+PC9nPjwvZz48L2c+PC9nPjwvZz48L3N2Zz4=]]

1. 1.

   问题:BERT 真的需要如此大量的(128,000 词/批 * 1,000,000 步)才能达到高准确率吗?
   回答:是的,BERT$ {\textstyle _{\text{BASE}}} $ 在训练 1M 步相对于 500k 步时,在 MNLI 上几乎获得 1.0% 的额外准确率。

2. 2.

   问题:由于每批中只预测 15% 的词而非每个词,MLM 收敛是否比 LTR 慢?
   回答:MLM 模型确实比 LTR 模型收敛略慢。然而,就绝对准确率而言,MLM 模型几乎立即开始超越 LTR 模型。

C.2 不同掩码过程的消融

在第 3.1 节中我们提到,BERT 在使用掩码语言模型(MLM)目标进行时采用混合策略掩盖目标词元。下面是一个评估不同掩码策略效果的消融研究。

注意,掩码策略的目的是减少与之间的不匹配,因为 [MASK] 符号在阶段从未出现。我们报告 MNLI 和 NER 的 Dev 结果。对于 NER,我们报告和基于特征两种方法,因为我们预期在基于特征的方法中不匹配会被放大,因为模型不会有机会调整表征。

Masking Rates			Dev Set Results
Mask	Same	Rnd	MNLI	NER
			Fine-tune	Fine-tune	Feature-based
80%	10%	10%	84.2	95.4	94.9
100%	0%	0%	84.3	94.9	94.0
80%	0%	20%	84.1	95.2	94.6
80%	20%	0%	84.4	95.2	94.7
0%	20%	80%	83.7	94.8	94.6
0%	0%	100%	83.6	94.9	94.6

结果显示在表 8 中。表中 Mask 表示我们将目标词元替换为 [MASK] 符号用于 MLM;Same 表示保持目标词元不变;Rnd 表示我们将目标词元替换为另一个随机词元。

表格左侧的数字代表 MLM 期间所用特定策略的概率(BERT 使用 80%、10%、10%)。论文的右侧代表 Dev 集结果。对于基于特征的方法,我们将 BERT 的最后 4 层拼接作为特征,这已在第 5.3 节中证明是最佳方法。

从表中可以看出,对不同掩码策略具有出人意料的鲁棒性。然而,正如预期的那样,在将基于特征的方法应用于 NER 时,只使用 Mask 策略会出现问题。有趣的是,只使用 Rnd 策略的表现也比我们的策略差得多。