WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...

Name: WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...
Uploaded: 2024-04-01T12:31:30Z
Duration: 1 h 49 min 11 s
Channel: Machine Learning Street Talk
Description: Machine Learning Street Talk interviews Paul Lessard about Categorical Deep Learning, which uses category theory and monads to make neural network architectures more interpretable and composable. They also discuss neural networks' limits in reasoning and unbounded computation.

Machine Learning Street TalkCited 3× on HN, 2024-2024

YouTube

101K views · 2 years ago · podcast · dated

Machine Learning Street Talk interviews Paul Lessard about Categorical Deep Learning, which uses category theory and monads to make neural network architectures more interpretable and composable. They also discuss neural networks' limits in reasoning and unbounded computation.

Mostly cited in passing, e.g. regarding claims that transformers cannot do recursion and as an example of structured deep learning work.

Machine learning & deep learning Functional programming #categorytheory #neuralnetworks #monads #architectures #machinelearning

3 comments from Hacker News

@taliesinb2024 · in “Ask HN: Who is hiring? (June 2024)”

Symbolica.ai | London, Australia | REMOTE, INTERNS, VISA We're trying to apply the insights of category theory, dependent type theory, and functional programming to deep learning. How do we best equip neural nets with strong inductive biases from these fields to help them reason in a structured way? Our upcoming ICML paper gives some flavor link ; you can also watch ▶ this video ; but there is a lot more to say. If you are fluent in 2 or more of { category theory, Haskell (/Idris/Agda/...), deep learning }, you'll probably have a lot of fun with us! Check out our open positions at link

CitesView on HN

@itchyjunk2024 · in “Energy, not compute, will be the #1 bottleneck to AI progress – Mark Zuckerberg [video]”

Do you have any citation on lack of funding for foundational research? From the looks of things, it seems like there is insane amount of money in AI and while maybe majority of it goes to "projects with high probability of success", whatever that means, fringe ideas seem to generate good amount of capital as well. Even (what I would consider to be) academic projects seems to get good funding [0]. On top of that, it seems like you're not saying that Transformers might be a dead end but Neural Networks itself might be one. At this point, NNs already gives a lot of value. If someone's goal is to just get products out that say diffusion models already helps you achieve, how is that even a dead end? Do you mean dead end for AGI? In which case, there seems to be research on alternatives there as well? [1] [0] ▶ this video [1] link

CitesView on HN

@f_devd2024 · in “Transformer as a general purpose computer”

Interesting, I thought it was proven Transformers cannot do recursion[0] and aren't actually able to be equivalent to a turing machine due to lack of expandable memory, and are rather finite state automata. [0]: ▶ this video

WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...

3 comments from Hacker News

Neural Networks Demystified [Part 1: Data and Architecture]

Category Theory 1.1: Motivation and Philosophy

Lecture 1 | Natural Language Processing with Deep Learning

AI can't cross this line and we don't know why.

Category Theory by Tom LaGatta

Backpropagation, intuitively | Deep Learning Chapter 3

An Observation on Generalization

CS480/680 Lecture 19: Attention and Transformer Networks

The spelled-out intro to neural networks and backpropagation: building micrograd

AI Language Models & Transformers - Computerphile

Gradient descent, how neural networks learn | Deep Learning Chapter 2

The Simple Essence of Automatic Differentiation - Conal Elliott

Attention Is All You Need

CS231n Winter 2016: Lecture1: Introduction and Historical Context

GPT3: An Even Bigger Language Model - Computerphile

But what is a neural network? | Deep learning chapter 1

LSTM is dead. Long Live Transformers!

Lecture 1 | Machine Learning (Stanford)

Tensorflow and deep learning - without a PhD by Martin Görner

Lecture 01 - The Learning Problem

WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...

3 comments from Hacker News

Neural Networks Demystified [Part 1: Data and Architecture]

Category Theory 1.1: Motivation and Philosophy

Lecture 1 | Natural Language Processing with Deep Learning

AI can't cross this line and we don't know why.

Category Theory by Tom LaGatta

Backpropagation, intuitively | Deep Learning Chapter 3

An Observation on Generalization

CS480/680 Lecture 19: Attention and Transformer Networks

The spelled-out intro to neural networks and backpropagation: building micrograd

AI Language Models & Transformers - Computerphile

Gradient descent, how neural networks learn | Deep Learning Chapter 2

The Simple Essence of Automatic Differentiation - Conal Elliott

Attention Is All You Need

CS231n Winter 2016: Lecture1: Introduction and Historical Context

GPT3: An Even Bigger Language Model - Computerphile

But what is a neural network? | Deep learning chapter 1

LSTM is dead. Long Live Transformers!

Lecture 1 | Machine Learning (Stanford)

Tensorflow and deep learning - without a PhD by Martin Görner

Lecture 01 - The Learning Problem