Trends in Deep Learning Hardware: Bill Dally (NVIDIA)

There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it. Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. ▶ this video Edit 2: about 26:40 he starts talking about energy use in the context of performance.

In a recent talk[1], Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me of the way the Yamaha DX7 works. The lookup tables can be expensive, and the talk contains an idea how to batch them when adding up the results of many many multiplications (as is also often the case for AI workloads). [1]: ▶ this video

Depends what you consider as part of chip production. If you consider new instructions and the related software as new chip production, then the performance improvements have continued quite dramatically. Single chip inference has gone 1000x over the last 10 years according to Bill Dally from Nvidia [1] at 10 minutes (probably not original source). Main gains according to the video have been different numeric types (fp16 for example) and more specialised hardware together with more complex instructions. Note that process improvements have only caused a 2.5x from that 1000x. He also mentions that Google‘s TPU has no fundamental benefit over dedicated instructions in GPUs. Quite an interesting video IMO. [1]: ▶ this video

It looks like both Pat Gelsinger and Lisa Su have no clue about software and can't even delegate this. They expect "the community" to do the software for their extremely complicated hardware. If you see the presentations by Nvidia's Bill Daly [1] it shows they've been evolving hardware and software together. The CUDA programming model was a massive bet a long time ago. And they deservedly won ML/AI. Unless Intel and AMD do a radical change it's game over for them. They will lose to ARM and Nvidia. [1] ▶ this video

> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte. As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization. [1]: ▶ this video