Tesla AI Day 2021

I spent time working with Andrej and the rest of the FSD team back in 2020/2021, and we had plenty of conversations on how human visual processing maps onto our neural network architectures. Our approach—transformer-based attention blocks, multi-scale feature extraction, and temporal fusion—mirrors elements of the biological visual cortex (retina → LGN → V1 → V2 → V4 → IT) which break down raw inputs and integrate them over time. It’s amazing how closely this synthetic perceptual pipeline parallels the way our own brains interpret the world. The key insight we discovered was that explicitly enforcing brain-like topographic organization (as some academic work attempts - such as this one here) isn't necessary - what matters is having the right functional components that parallel biological visual processing. Our experience showed that the key elements of biological visual processing - like hierarchical feature extraction and temporal integration - emerge naturally when you build architectures that have to solve real visual tasks. The brain's organization serves its function, not the other way around. This was validated by the real-world performance of our synthetic visual cortex in the Tesla FSD stack. Link to the 2021 Tesla AI day talk: ▶ this video

I believe Tesla uses synthetic data for FSD/Autopilot training as well. Not sure what's the ratio between synthetic data and real videos, but it seems that they plan to rely more onto the first group in the future. ▶ this video

Machine Learning and Neural Networks are not applicable for solving all kinds of problems. In fact, in Tesla AI Day Elon Musk remarked that "I discourage the use of machine learning... unless you have to use machine learning, don't do it" [1]. Getting machine learning to work right is hard. If it can be avoided it should be. So to cast this as Software 1.0 vs 2.0 doesn't make sense. There is a class of problems where neural networks work better. Everywhere else we will continue to use traditional code. [1] ▶ this video

The Tesla AI day videos [1] go into some detail about this. They use multiple networks that are dedicated to specific tasks. [1] ▶ this video (2021), link (2022)

Waymo has superior performance based on their historical statistics. It makes sense, since their lidar sensors capture more of the environment, and directly in 3D. Their AI also seems better QA’ed. The Tesla AI Day[0] surprised me as it showed they only had a simple architecture for a very long time, simply feeding barely processed camera pixels to a DNN and hoping for the best with little more than supervised learning off human feeds. Their big claim to glory was that they rearchitectured it to produce a 2D map of the environment… which I thought they had years ago, and is still a far cry from the 3D modeling that is needed. After all, sure, we humans only take two video feeds as input… But we can appreciate from it the position, intent, and trajectory of a wealth of elements around us, with reasonable probability estimates for multiple possibilities, sometimes pertaining to things that are invisible, such as kids crossing from nowhere when near a school. Cruise also seems to have better tech; they had a barely-watched 2h30 description of their systems[1] which shows they do create a richer environment, evaluate the routing of many objects, and train their systems on a very realistic simulation, not just supervised training, which means it can learn from very low-probability events. They have a whole segment on including the probability that unseen cars may travel from perpendicular roads; Tesla’s creeping hit-or-miss are well-documented on Youtube. [0]: ▶ this video [1]: link

link This is a rather intuitive mathematical proof that neural nets can emulate all functions in many dimensions. Knowing integrals (Calc 102) would help. With this information it’s either that self-driving is eventually possible or human brains are not mathematically representable (i.e. something akin to a soul). Meanwhile, the data collection/storage is getting better (all Teslas are continually testing FSD and reporting failure cases, synthetic generation of “good and unique” perfectly labeled data[0] is only a year or two “old”). Sensors/cameras are improving (and have a lot more room to improve). The compute hardware is getting better and cheaper (analog computers[1], TSMC 2nm and beyond, Apple/Google/Teslas neural cores, etc). Additionally, given Waymo is already in production without humans in PHX it’s arguable that we already have “autonomous vehicles”. We now just need them to improve slightly and for Waymo or similar to test in the top 25 cities in the US. Which is mostly a matter of capital and time not technology limits. Maybe weather will be an issue for a few more years/decades - but even if Waymo only operated in Summer, I’d be happy to reduce my Lyft costs - and with enough data and maybe better sensors the neural nets will be able to handle weather too. Especially if the car just goes slow. [0] ▶ this video [1] link [2] link

Cruise actually consider both social dynamics and uncertainty (i.e. what can hide behind an obstacle, or where are pedestrians/bikes/cars likely to move to). If you are interested in self-driving cars, I can highly recommend their presentation from November 2021: link For me it felt more convincing than Tesla's (a few months prior); ▶ this video

> The "images" are part information. This. It is a mashup of both. I can clearly visualize many 3D objects in my mind, as well as zoom, pan, scale, rotate, recolor, animate, distort them, etc. But it is not purely visual, because I often "see" opposite side of a cube momentarily, which I should not be able to from this perspective. So, it is not a rendering engine. It is not physically, spatially and temporally stable. Stability is lost especially as your focus shifts (e.g. "seeing" behind objects as your attention slips there) The best way I know to describe this is by old and new Tesla FSD predictions which are sampled here: ▶ this video Improved version is how people talk about visualization, but the old unstable version is how (at least for me) it mostly is.

Here's Tesla's AI day presentation on how they are going to use AI in their planning system because traditional methods aren't scalable enough: ▶ this video (1:20:15). You can see hints of this in presentations from the other self-driving companies. Boston Dynamics was definitely an exception for a long time. Certainly the perception systems that they are adding use ML, but you are right that they use expressly use classical methods for their control system.

This is a poor use case for ML. You'd only have a couple hundred examples for a neighborhood with maybe two dozen variables. The model will also miss key variables that that there aren't datasets for but would be apparent upon a visit. ML only beats humans in very, very narrow scenarios. ▶ this video

I mean in the same event that you list, Elon literally opens with: "Tesla is much more than an automotive company, and we have lots of deep AI activity..." I think it is fair to say that a company that is literally inventing its own AI training supercomputer is probably an AI company as well :) ▶ this video

The Q&A section on their compiler and software that the author links to is very interesting: ▶ this video It sounds like they're going to have write a ton of custom software in order to use this hardware at scale. And, based on the team being speechless when asked a follow up question, it doesn't sound like they know (yet) how they're going to solve this. Nvidia gets a lot of credit for their hardware advances, but what really what their chips work so well for deep learning was the huge software stack they created around CUDA. Underestimating the software investment required has plagued a lot of AI chip startups. It doesn't sound like Tesla is immune to this.

> […] AI has a massive value to every aspect of the company's work. That's also just wrong. During the recent "Tesla AI Day", when asked during Q/A, Elon Musk specifically mentioned that they intentionally use machine learning only for very few cases: Q: "Is Tesla using machine learning within its manufacturing, design or any other engineering processes?" Elon: "I discourage use of machine learning, because it's really difficult. Unless you have to use machine learning, don't do it. It's usually a red flag when somebody is saying 'We wanna use machine learning to solve this task'. I'm like: That sounds like bullshit. 99.9% of the time you don't need it." ▶ this video

They use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!). Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection ( ▶ this video ). You can use the comma and period keys to step through the video frame-by-frame.

Can you provide a more specific timestamp? 54.40 doesn't seem to mention anything about transformers, and "onwards" is two hours. I'd be really surprised if they use transformers due to how computationally expensive they are for anything involving vision. EDIT: Found it. 1h: ▶ this video Fascinating. I guess transformers are efficient.

In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transformers come higher up the stack: ▶ this video Transformers mentioned on the slide at timestamp 1:00:18.

They do. Karpathy spoke about it in on Tesla AI day. They use it for transforming image space to a vector space. See: ▶ this video (timestamp 54.40 onwards)

Elon's been giving some lip service to closer-to-GI capabilities anyways, e.g. when describing how it would be optimal to be able to control Tesla's humanoid robot by saying "please go to the store and get me the following groceries"... ▶ this video Whether Tesla wants to fund serious work towards that may be a different question (but they do seem to be serious about at least building the robot).

The newly announced Tesla humanoid bot is around 2:05:11. Expected prototype in 2022. The demo was an obvious troll but it looks pretty exciting. I'd rather have resources put into actual innovation than learning how to increase the likelihood someone will click a button by 0.01% ▶ this video

I don't know if anyone else caught this, but their "optimal trajectory" planned path is illegally changing lanes in an intersection, at -1:26:00: ▶ this video . Pretty funny.