Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Name: Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Uploaded: 2023-07-05T15:12:38Z
Duration: 39 min 43 s
Channel: Sasha Rush
Description: Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Sasha Rush3 HN recommenders · recommended in 3 months, 2024-2024

YouTube

35K views · 3 years ago · lecture · dated

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

Machine learning & deep learning LLMs & generative AI #ssm #linearrnn #mamba #attention #s4 #nlp

3 comments from Hacker News

@a_wild_dandan2024 · in “Jamba: Production-grade Mamba-based AI model”

To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: ▶ this video

Strongly recommendsView on HN

@srush2024 · in “The Annotated S4”

Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: ▶ this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

3 comments from Hacker News

CS480/680 Lecture 19: Attention and Transformer Networks

Attention Is All You Need

LSTM is dead. Long Live Transformers!

Transformers, the tech behind LLMs | Deep Learning Chapter 5

Attention in transformers, step-by-step | Deep Learning Chapter 6

Deep Dive into LLMs like ChatGPT

A Hackers' Guide to Language Models

[1hr Talk] Intro to Large Language Models

State of GPT | BRK216HFS

Large Language Models from scratch

AI Language Models & Transformers - Computerphile

The spelled-out intro to language modeling: building makemore

StrictlyVC in conversation with Sam Altman, part two (OpenAI)

Let's build GPT: from scratch, in code, spelled out.

CS231n Winter 2016: Lecture1: Introduction and Historical Context

John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

Let's build the GPT Tokenizer

An Observation on Generalization

Richard Sutton – Father of RL thinks LLMs are a dead end

Sparks of AGI: early experiments with GPT-4

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

3 comments from Hacker News

CS480/680 Lecture 19: Attention and Transformer Networks

Attention Is All You Need

LSTM is dead. Long Live Transformers!

Transformers, the tech behind LLMs | Deep Learning Chapter 5

Attention in transformers, step-by-step | Deep Learning Chapter 6

Deep Dive into LLMs like ChatGPT

A Hackers' Guide to Language Models

[1hr Talk] Intro to Large Language Models

State of GPT | BRK216HFS

Large Language Models from scratch

AI Language Models & Transformers - Computerphile

The spelled-out intro to language modeling: building makemore

StrictlyVC in conversation with Sam Altman, part two (OpenAI)

Let's build GPT: from scratch, in code, spelled out.

CS231n Winter 2016: Lecture1: Introduction and Historical Context

John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

Let's build the GPT Tokenizer

An Observation on Generalization

Richard Sutton – Father of RL thinks LLMs are a dead end

Sparks of AGI: early experiments with GPT-4