Attention Is All You Need

> I love these word-salad exaltations of LLMs - people don't even know what they're writing half the time but it sure is provocative! Well FWIW I have done an implementation of Llama LLM (3 8-70b to be specific) in non-python so I do sort of know what I'm talking about. I'm not the originator of the hash table analogy. I got it from here: ▶ this video Hell, the vectors that are generated are called K/V, so. Yeah it's a hash table. And the idea that first order logic and other facets of intelligence can just be cleverly arranged lookup tables comes from here: link

The Yannic kilcher review is quite good. ▶ this video I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "performing the lookup query" over all other tokens in the attention frame, so every token gets a "contextual semantic understanding" that takes in both all the other stuff in the attention frame and it's relative position. Gpt is impressive because the N is really huge and it has many layers. A big N means you can potentially access information farther away. Each layer gives more opportunities to summarize and integrate long range information in a fractal process. Two key takeaways: - differentiable hash tables - encoding relative position using periodic functions NB: the attention frame tokens are actually K-vectors (so the frame is a KxN matrix) and the query matrix is an NxNxK tensor IIRC but it's easier to describe it this way

YouTube channel that explains the paper in detail: ▶ this video And subsequent follow-ups (ROME, editing transformer arch): link I find the channel amazing explaining super complex topics in simple enough terms for people who have some background in AI.

For accessibility I recommend Yannic Kilcher video review of "Attention Is All You Need" ▶ this video Yannic has been making about 62 other transformer paper reviews since. You can find the usual suspects. link

Everyone has a different way of approaching. I like video walk-throughs of papers. If you are the same, for the theory bit you may find Yannic Kilcher's videos [0,1] helpful. On implementation side, the best project out there is [2] with good documentation. Also spacy has transformer package [3] (I personally haven't tried it), so maybe it will be easier for you to jump in if you have prior experience with spacy. [0] ▶ this video [1] link [2] link [3] link