Efficient Bayesian inference with Hamiltonian Monte Carlo -- Michael Betancourt (Part 1)

Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: link - Thinking outside the 10-dimensional box - 3blue1brown: link - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: ▶ this video

Recently watched this on Hamiltonian Monte Carlo, and it was a fantastic primer on the basic concepts and motivations of mcmc, and later hmc ▶ this video

This is an interesting take and kudos to the author for relaying a helpful way to think about high dimensional distributions. I really like and often come back to this talk by Michael Betancourt where the theme is quite similar: ▶ this video