The spelled-out intro to language modeling: building makemore

Name: The spelled-out intro to language modeling: building makemore
Uploaded: 2022-09-07T19:14:47Z
Duration: 1 h 57 min 45 s
Channel: Andrej Karpathy
Description: Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Andrej Karpathy2 HN recommenders · recommended in 2 months, 2024-2024

YouTube

1.3M views · 4 years ago · tutorial · evergreen

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

HN recommends it as a long but worthwhile way to demystify how language models work technically.

Machine learning & deep learning LLMs & generative AI #makemore #bigram #pytorch #neuralnetworks #karpathy #next-tokenprediction

4 comments from Hacker News

@caprock2024 · in “The n-gram Language Model”

For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * link It appears to be a generalized version of the bi-gram project, which he links to here: * ▶ this video It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.

CitesView on HN

@georgehill2024 · in “Show HN: Next-token prediction in JavaScript”

You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - link - Highly recommend this one ▶ this video - link - link 1: ex: link

Strongly recommendsView on HN

@chrsig2024 · in “GPT in 500 Lines of SQL”

This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] ▶ this video

CitesView on HN

@islewis2024 · in “Markov Chains are the Original Language Models”

I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. ▶ this video

The spelled-out intro to language modeling: building makemore

4 comments from Hacker News

[1hr Talk] Intro to Large Language Models

Deep Dive into LLMs like ChatGPT

Let's build GPT: from scratch, in code, spelled out.

Large Language Models from scratch

State of GPT | BRK216HFS

Let's build the GPT Tokenizer

AI Language Models & Transformers - Computerphile

A Hackers' Guide to Language Models

The spelled-out intro to neural networks and backpropagation: building micrograd

GPT3: An Even Bigger Language Model - Computerphile

Transformers, the tech behind LLMs | Deep Learning Chapter 5

Building the Software 2 0 Stack (Andrej Karpathy)

Andrej Karpathy: Software Is Changing (Again)

A Style-Based Generator Architecture for Generative Adversarial Networks

CS231n Winter 2016: Lecture1: Introduction and Historical Context

LSTM is dead. Long Live Transformers!

Lecture 1 | Natural Language Processing with Deep Learning

Swift for TensorFlow: The Next-Generation Machine Learning Framework (TF Dev Summit '19)

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Attention Is All You Need

The spelled-out intro to language modeling: building makemore

4 comments from Hacker News

[1hr Talk] Intro to Large Language Models

Deep Dive into LLMs like ChatGPT

Let's build GPT: from scratch, in code, spelled out.

Large Language Models from scratch

State of GPT | BRK216HFS

Let's build the GPT Tokenizer

AI Language Models & Transformers - Computerphile

A Hackers' Guide to Language Models

The spelled-out intro to neural networks and backpropagation: building micrograd

GPT3: An Even Bigger Language Model - Computerphile

Transformers, the tech behind LLMs | Deep Learning Chapter 5

Building the Software 2 0 Stack (Andrej Karpathy)

Andrej Karpathy: Software Is Changing (Again)

A Style-Based Generator Architecture for Generative Adversarial Networks

CS231n Winter 2016: Lecture1: Introduction and Historical Context

LSTM is dead. Long Live Transformers!

Lecture 1 | Natural Language Processing with Deep Learning

Swift for TensorFlow: The Next-Generation Machine Learning Framework (TF Dev Summit '19)

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Attention Is All You Need