text #1Provided proper attribution is provided, Google hereby grants permission to

text #2reproduce the tables and figures in this paper solely for use in journalistic or

text #3scholarly works.

title #4Attention Is All You Need

title #5arXiv:1706.03762v7 [cs.CL] 2 Aug 2023

text #6Ashish Vaswani∗

text #7Noam Shazeer∗

text #8Niki Parmar∗

text #9Jakob Uszkoreit∗

text #10Google Brain

text #11Google Brain

text #12Google Research

text #13Google Research

text #14avaswani@google.com

text #15noam@google.com

text #16nikip@google.com

text #17usz@google.com

text #18Llion Jones∗

text #19Aidan N. Gomez∗†

text #20Łukasz Kaiser∗

text #21Google Research

text #22University of Toronto

text #23Google Brain

text #24llion@google.com

text #25aidan@cs.toronto.edu

text #26lukaszkaiser@google.com

text #27Illia Polosukhin∗‡

text #28illia.polosukhin@gmail.com

text #29Abstract

text #30The dominant sequence transduction models are based on complex recurrent or

text #31convolutional neural networks that include an encoder and a decoder. The best

text #32performing models also connect the encoder and decoder through an attention

text #33mechanism. We propose a new simple network architecture, the Transformer,

text #34based solely on attention mechanisms, dispensing with recurrence and convolutions

text #35entirely. Experiments on two machine translation tasks show these models to

text #36be superior in quality while being more parallelizable and requiring significantly

text #37less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-

text #38to-German translation task, improving over the existing best results, including

text #39ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,

text #40our model establishes a new single-model state-of-the-art BLEU score of 41.8 after

text #41training for 3.5 days on eight GPUs, a small fraction of the training costs of the

text #42best models from the literature. We show that the Transformer generalizes well to

text #43other tasks by applying it successfully to English constituency parsing both with

text #44large and limited training data.

text #45∗Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started

text #46the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and

text #47has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head

text #48attention and the parameter-free position representation and became the other person involved in nearly every

text #49detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and

text #50tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and

text #51efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and

text #52implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating

text #55our research.

text #53†Work performed while at Google Brain.

text #54‡Work performed while at Google Research.

text #5631st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

shape #57

shape #58

shape #59