text #1Transformer-XL: Attentive Language Models

text #2Beyond a Fixed-Length Context

text #3Zihang Dai⇤12, Zhilin Yang⇤12, Yiming Yang1, Jaime Carbonell1,

text #4Quoc V. Le2, Ruslan Salakhutdinov1

text #51Carnegie Mellon University, 2Google Brain

text #6{dzihang,zhiliny,yiming,jgc,rsalakhu}@cs.cmu.edu, qvl@google.com

text #7Abstract

text #50Term Memory (LSTM) networks (Hochreiter and

text #51Schmidhuber, 1997), have been a standard solu-

text #8Transformers have a potential of learning

text #52tion to language modeling and obtained strong

text #9longer-term dependency, but are limited by a

text #53results on multiple benchmarks.

text #54Despite the

text #10fixed-length context in the setting of language

text #55wide adaption, RNNs are difficult to optimize

text #11modeling.

text #12We propose a novel neural ar-

text #13chitecture Transformer-XL that enables learn-

text #56due to gradient vanishing and explosion (Hochre-

text #14ing dependency beyond a fixed length with-

text #57iter et al., 2001), and the introduction of gat-

text #15out disrupting temporal coherence.

text #16It con-

text #58ing in LSTMs and the gradient clipping tech-

text #17sists of a segment-level recurrence mechanism

text #59nique (Graves, 2013) might not be sufficient to

text #18and a novel positional encoding scheme. Our

text #60fully address this issue.

text #61Empirically, previous

text #19method not only enables capturing longer-term

text #62work has found that LSTM language models use

text #20dependency, but also resolves the context frag-

text #63200 context words on average (Khandelwal et al.,

text #21mentation problem. As a result, Transformer-

text #642018), indicating room for further improvement.

text #22XL learns dependency that is 80% longer than

text #23RNNs and 450% longer than vanilla Trans-

text #65On the other hand, the direct connections be-

text #24formers, achieves better performance on both

text #66tween long-distance word pairs baked in atten-

text #25short and long sequences, and is up to 1,800+

text #67tion mechanisms might ease optimization and en-

text #26times faster than vanilla Transformers during

text #68able the learning of long-term dependency (Bah-

text #27evaluation. Notably, we improve the state-of-

text #69danau et al., 2014; Vaswani et al., 2017).

text #70Re-

text #28the-art results of bpc/perplexity to 0.99 on en-

text #71cently, Al-Rfou et al. (2018) designed a set of aux-

text #29wiki8, 1.08 on text8, 18.3 on WikiText-103,

text #72iliary losses to train deep Transformer networks

text #3021.8 on One Billion Word, and 54.5 on Penn

text #73for character-level language modeling, which out-

text #31Treebank (without finetuning). When trained

text #32only on WikiText-103, Transformer-XL man-

text #74perform LSTMs by a large margin. Despite the

text #33ages to generate reasonably coherent, novel

text #75success, the LM training in Al-Rfou et al. (2018)

text #34text articles with thousands of tokens.

text #35Our

text #76is performed on separated fixed-length segments

text #36code, pretrained models, and hyperparameters

text #37are available in both Tensorflow and PyTorch1.

text #77of a few hundred characters, without any informa-

text #78tion flow across segments. As a consequence of

text #381

text #39Introduction

text #79the fixed context length, the model cannot capture

text #80any longer-term dependency beyond the prede-

text #40Language modeling is among the important prob-

text #81fined context length. In addition, the fixed-length

text #41lems that require modeling long-term dependency,

text #82segments are created by selecting a consecutive

text #42with successful applications such as unsupervised

text #83chunk of symbols without respecting the sentence

text #43pretraining (Dai and Le, 2015; Peters et al., 2018;

text #84or any other semantic boundary. Hence, the model

text #44Radford et al., 2018; Devlin et al., 2018). How-

text #85lacks necessary contextual information needed to

text #45ever, it has been a challenge to equip neural

text #86well predict the first few symbols, leading to inef-

text #46networks with the capability to model long-term

text #87ficient optimization and inferior performance. We

text #47dependency in sequential data.

text #48Recurrent neu-

text #88refer to this problem as context fragmentation.

text #49ral networks (RNNs), in particular Long Short-

text #89To address the aforementioned limitations of

text #94⇤Equal contribution. Order determined by swapping the

text #90fixed-length contexts, we propose a new architec-

text #95one in Yang et al. (2017).

text #91ture called Transformer-XL (meaning extra long).

text #961https://github.com/kimiyoung/

text #92We introduce the notion of recurrence into our

text #97transformer-xl

text #982978

text #93Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988

text #99Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics

shape #100