Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70

I didn't make the connection until now, but Jim Keller was on the Lex Fridman podcast twice [1, 2] and it was awesome. In terms of founders this is pretty great team. [1] ▶ this video [2] link

Jim Keller talks about his impression working with Elon - skip to 1:20:50 of this (effing amazing) interview: ▶ this video Edit: Also Jim talks about Elon’s engineering skills at 50 minutes into this interview: link Disclaimer: I’m not an Elon fan at all.

> By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition. Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI. In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2? It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing,

I'm generally against rewrites as they're often shortcuts for new developers that instantly reject a code base as 'too complicated' without understanding it. However, I have had success in rewriting my own code base a number of times. This is a unique situation of a one-man-band that understands the business case, handles support, and every bit that went into every decision of the code base. With this kind of clarity, when you've spent years with the problem domain, you can uniquely rewrite a project to get at the real business case that, now with years of experience, you see what your customers actually wanted you to solve in the first place, or maybe what they evolved to want in the end. Jim Keller advocates rewrites in a similar way as the only way to progress in chip design. I think we could rewrite very basic things in exciting ways with this sort of attitude. For instance, we know a lot more about what we need in a desktop OS, an email client, a search engine, etc. Basic things that have gotten a lot of cruft over the years as they evolved. Taking a fresh look could be rewarding. I guess one could reframe this as 'first principals thinking" but with the caveat that the problem needs to be truly understood. Here's Jim Keller talking about it: ▶ this video

I'm not qualified to truly understand this, but when Lex first interviewed Jim Keller, Jim basically said regarding processor design -- 'yeah, if you just guess the same result as last time, you get a 50% speed increase.' First Interview (where that poorly paraphrased quote resides): ▶ this video Second Interview: link

Jim Keller (cpu designer): "I joke, like, I read books. And people think, 'Oh, you read books'. Well, no, I've read a couple of books a week, for [50] years."[1] Thought you might like. [1] ▶ this video

> I expect within the next 6-15 years that we'll hit this limit You are talking about fundamental physical limits? Um... No, [1] "At about 26-28.5 minutes of the Jim Keller: Jim notes that transistors could reach 10 atoms by 10 atoms by 10 atoms while avoiding quantum effects. People are also working on harnessing quantum effects." We are so far away from fundamental limits. That even if we assume we could somehow double transistor density every two years, which we dont anymore, there are still at least another 15-20 years before we are close to that limit. And that is not accounting any 3D Transistors. Realistically we already have Roadmap up to 2030 from TSMC. The only limit we will hit is that the node is too expensive and market could no longer afford the premium. Which You could expect to happen within next 10-15 years. [1] ▶ this video

Loved the episodes with Jim Keller [0] and Brian Kernighan [1] as well. Lex really does a great show and has amazing guests. [0]: ▶ this video [1]: link

I agree, their problem wasn't lack of resources it was they stopped believing in Moore's law. They started to believe that 5 and 3nm advancements were not coming and didn't make the hundreds/thousands of new inventions needed to get them there. At least that's what I get reading between the lines here... Jim Keller talks about this in this in the Lex Fridman podcast: ▶ this video

Yeah that's how RISC vs CISC is taught in class, I've heard that same thing. I think it's an outdated paradigm though, if it's not been wrong all along. A CISC chip can still officially support instructions, but deprecate them and implement them very slowly in microcode, as is done with some x86 instructions. And a RISC chip manufacturer might just have used this phrase as a marketing line because designing RISC chips is easier than starting with a monster that has tons of instructions, and they needed some catchy phrase to market their alternative. They then get into the gradual process of making their customers happier one by one by adding special, mostly cold, silicon targeted for special workflows, like in this instance. Ultimately, the instruction set isn't that relevant anyways, what's more relevant is how the microcode can handle speculative execution for common workflows. There's a great podcast/interview from the x86 specification author: ▶ this video

Carmack was actually mentioning he asked Jim Keller about Moore's Law. For more details about Jim's thoughts I recommend this interview ▶ this video

Add timestamps to your link instead of messing around with parenthesis. ▶ this video Right click -> "Copy video URL at current time"

I found this to be an excellent explanation for what is going on in these x86 CPUs that makes them so powerful: ▶ this video (Starting from ~6:30)

Yup - and this year's top CPUs have almost 10x the performance per watt of CPUs from even 2-3 years ago [0] Raw computation is only half the story. The other half is: what the hell do we do with all these extra transistors? [1] 0 - link 1 - ▶ this video

Moore's law might be dead but the deeper law is still alive. Moore's law is technically "the number of transistors per unit area doubles every 24 months" [1]. The more important law is that the cost of transistors halves every 18-24 months. That is, Moore's law talks about how many transistors we can pack into a unit area. The deeper issue is how much it costs . If we can only pack in a certain amount transistors per area but the cost drops exponentially, we still see massive gains. There's also Wright's law that comes into play [3] that talks about dropping exponential costs just from institutional knowledge (2x in production leads to (.75-.9)x in cost). [1] link [2] ▶ this video [3] link

My understanding is that architectural improvements (i.e. new approaches to detect more parts in code that can be evaluated at the same time, and then do so) need more transistors, ergo a smaller process. (Jim Keller explains in this interview how CPU designers are making use of the transistor budget: ▶ this video )

If you noticed how insanely Slack freaked out and handled the service outage. The application just got stuck, didn't know what to do, I see a faint 503 error message which looks like a webpage nginx error displayed right in the app. Clearing cache didn't help and the app just quit. Electron apps are a nuisance and should be eradicated from this planet. We've come a full circle -> Bloat the web -> Create javascript frameworks -> Build SPA/PWA apps -> Invade the desktop land -> Gain traction because "its crossplatform" and now try to reimplement GUI elements from some fancy_new framework that is developed by people who are entirely unqualified for it. Throw away all the work that's done by OS developers and designers that are getting paid as professionals to develop guidelines like this: link The modern tech stack needs to be refactored just like they refactor silicon architecture every 10 years: ▶ this video Edit: People talking about would native app handle this better? Sure. I argue that it wouldn't just get rid of all UI elements and drop the entire DOM on the page, the UI just crashed. And everytime I open slack it seems like its doing something else entirely. There is a massive difference in quality of native UI libraries and Bulma for example, and its not just the library, you gotta rely on React to redraw things and it depends on 4 billion npm dependencies. The web truly sucks. And it is now eating into desktop apps.

I just wanted to say this about Intel vs AMD. Back when Intel was the "underdog" it couldn't compete with Athlon/Opeteron CPUs from AMD with its Pentium IV line, fan boys from both sides continually escalated the armchair warfare on Slashdot and Anandtech forums when Intel finally came out with the Conroe architecture that blew the competition away. I've seen this pendulum swing from one side to the other and back. Putting Intel marketing (awful) and AMD's marketing (awful again...), and their fan bases (toxic) aside, can we acknowledge the fact that working on a computer architecture is an extraordinarily complex task and that requires brilliant people all working together? I'll defer this fascinating topic, which most software engineers and tech enthusiasts are completely and utterly unaware of, to this talk by the legendary chip designer (x86 spec coauthor, A4/A5/Zen/Tesla AI ASIC architect) - Jim Keller and his interview with Lex Fridman: ▶ this video Designing & manufacturing computer chips is hard af, having worked in semiconductor manufacturing for over 12 years (primarily on the backend side of the Fab). Edit: Redacting some specifics

I've been watching Lex Fridman's youtube podcast and there is a recent interview with Jim Keller [1]. Keller is a chip designer famous for his involvement in multiple chips at Intel, AMD, Apple and he was co-author of the x86-64 instruction set. He also worked for Tesla. There is a point in the conversation where Lex and Jim clearly disagree about how "easy" self-driving AI should be. Lex is clearly pessimistic and Jim is clearly optimistic. I have to admit I was more swayed by Lex's points than by Jim's, but it is hard to discount someone so clearly (extraordinarily) expert and working directly in the field. 1. ▶ this video

I highly suggest viewing one of the Alpha's original designers, Jim Keller, recent chat with MIT Prof Lex Fridman ▶ this video I was stunned by Jim's total command of Computer Architecture, and his deep insights about the subject. I will never again utter the phrase 'modern computer' without requisite awe.