Thinking outside the 10-dimensional box

There's a great 3blue1brown video that demonstrates this unintuitive result in a relatively intuitive way: ▶ this video

Three Blue, One Brown has a decent video which perhaps helps get a handle on how one can use higher-dimensional spaces, without needing to wrap you head around trying to envision a four-dimensional cube, or similar. ▶ this video

Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: link - Thinking outside the 10-dimensional box - 3blue1brown: ▶ this video - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: link

As you increase the dimensionality of a sphere, an ever-increasing proportion of the sphere is within epsilon of the surface. This 3Blue1Brown video addresses enough related stuff that the rest should become reasonably comprehensible: ▶ this video This particular result is tossed in as a side note around 23:25, so it's not addressed directly, but it'll help.

Yeah, I see now. “times i” is discrete 90 degree rotation itself, not just ‘i’, nor ‘b’. Thanks everyone for making that clear. This though shows that explanations via analogies or non-strict wording may confuse one rather than enlighten. I’m not good at math, but once understood to not search analogies or geometry in things. Instead it is better to “shut up and calculate”. Not sure if imagining something is required to manage it. It’s only our brain’s faulty quirk. ▶ this video

Don't forget 3Blue1Brown's incredible video about visualizing higher dimensions[1]. He's getting to xkcd levels of "Of course there's a relevant video" [1]: ▶ this video

Great article! I've been thinking about this on and off recently, as I wonder if we might be having intuitive issues when it comes to gradient descent optimization. 3 Blue,1 Brown had a video recently that kicked off my head scratching and is a great complement to your article: ▶ this video

I would reiterate that simply because there are problems with naive visualization, we shouldn't discredit visual thinking. There are several key elements to effective visual thinking. The primary importance is to keep it grounded in proofs and theorems, so you know exactly what are your limitations. Often you can use a geometric argument on top of a few theorems and you get a very strong result intuitively, and then use this intuition with a tiny amount of algebra to prove it (which might take you forever to arrive from a purely algebraic perspective). Another key is that there are several ways of visualizing things. You can almost always transform a problem into an equivalent one that is easy to visualize (just need a little bit of care with the transformation, etc). --- For example, you can show functions form a vector space, visualizing some interesting algebraic properties about them, even if it constitutes an infinite-dimensional space. You can show several operators (such as d/dx) are linear, you can give it a norm, internal product, etc. This trick lets you use visual tools (and linear algebra tools) with arbitrary functions. You can visualize projection of a function into a subspace, or into some non-canonical basis -- yielding useful applications -- such as Fourier analysis. Fourier analysis itself is a fertile ground for visual thinking. You'll be finding trivial arguments for seemingly difficult decisions such as "Does this linear system have a bounded output for a

A trick for visualizing higher dimensions: ▶ this video Only 383k subscribers. Hmm. Remember to subscribe, comment, and smash that like button. Real Engineering is another great channel. And of course Veritasium and Numberphile.