"Performance Matters" by Emery Berger

One of my previous struggles has been profilers blaming whatever code is the first bit of code to force some memory into cache. We devs could take turns and claim to have improved our own code by a huge margin simply by rearranging the order of calls so ours was not first. Then pass the baton; global performance utterly unchanged, everyone gets to "optimize" their own code. Something like link might be the way forward. ▶ this video

You're probably thinking of "Performance Matters" by Emery Berger, a Strange Loops talk. ▶ this video

There's a great talk that explains how/why this works for those looking for more info. Also, Go UMass! ▶ this video

Woah, I can't believe it, I've found the discussion that linked the thing I was looking for. [1] It's called Coz, the causal profiler [2]. I've submitted this talk again, because I think it's super interesting. [1] link [2] ▶ this video

I understand where you are coming from, but without comparing the generated assembly, we are comparing an implementation of bounds checks. I think we should have a bounds check instruction and operates concurrently. I should play around this with this using a couple RISC-V cores. > Performance matters! ▶ this video by Emery Berger Another, yesbut, encryption and compression are and will handled by on die accelerators.

Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit. Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2]. Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal. Coz is pretty trivial to set up with Zig too. 1. ▶ this video 2. link

> changing my username breaks $product. ▶ this video Discussing a performance regression due to longer username due to username being in ENVIRONMENT variable which changes memory layout of process.

We really don't know what causes perf differences, layout has more of an effect than -O1 to -O3. "Performance Matters" by Emery Berger ▶ this video

I think the profiling section could benefit from a section describing the sort of profiling noise described in this video: ▶ this video The noise source described is that changes to the code typically alter alignments of code and/or data and that is often the cause of the performance difference measured, not the change to the algorithm. The reason this is noise is because unrelated changes to the code might make it change again. IIRC, the speaker suggests improving compiler/profiler tools to randomise these alignments and benchmark many times.

Relevant talk by Emery Berger: Performance matters ▶ this video I'd recommend everyone interested in the subject to watch it.

Try getting them to watch this talk by Emery Berger: ▶ this video And who knows, maybe they'd even be interested in using coz afterwards: link

I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications. [1] ▶ this video

> I originally missed your point about "multiples of" That wasn't my point, but the claim of the poster I was replying to, and it was exactly this claim that I think is unsupportable. Has some software gotten faster? Sure. But mostly software has a gotten slower and the rarer cases of software getting faster have been outpaced significantly by HW. > You want examples where software has sped up at a rate faster than hardware "several multiples": that in some/many areas software has improved performance by several multiples of the improvement in hardware performance over the last 20 years > [JavaScript] JIT compilers have improved The original JITs were done in the late 80s early 90s. And their practical impact is far less then the claimed impact. link As an example the Cog VM is a JIT for Squeak. They claim a 5x speedup in bytecodes/s. Nice. However the naive bytecode interpreter, in C, on commodity hardware in 1999 (Pentium/400) was 45 times faster than the one microcoded on a Xerox Dorado in 1984, which was a high-end, custom-built ECL machine costing many hundred thousands of dollars. (19m bytecodes/s vs. 400k bytecodes/s). So 5x for software, at least 45x for hardware. And the hardware kept improving afterward, nowadays at least another 10x. > [compilers] Parsing isn't as simple as it seems [..] Parsing is not where the time goes. > 2005-era swift running faster Swift generally has not gotten faster at all. I refer you back to Proebsting's Law and the evidence gathered i

It's mentioned elsewhere in this HN discussion, but I think I'd encourage watching this talk: ▶ this video . It's particularly relevant to what you're seeing.

This[1] talk from Emery Berger at last year's Strange Loop does a good job discussing the impact of alignment and other 'small' changes on performance, and how to estimate the actual impact of changes on performance [1]: ▶ this video

> deleting this one line of code can have a dramatic effect on seemingly unrelated performance micro-benchmarks. Some numbers get better (e.g. +5%), some numbers get worse (e.g. -10%). The same micro-benchmark can get faster on one C compiler but slower on another There's a fundamental disconnect that makes it difficult for humans to reason about performance in computer programs. Because the speed of light is so slow, computer architecture as we know it will always rely on cache and OoO to be fast. The human brain does seem to work out of order, but it's only used to thinking about a world that runs in order. When we use theory of mind, we don't model other people's minds, we use our own as a model for theirs; see mirror neurons[1]. Because of this, standard code benchmarks are not very useful, unless they can demonstrate order-of-magnitude speedups. Even something like a causal profiler[2][3][4], which attempts to control for the volatile aspects of performance, is of limited use; it cannot control for all variables and its results will likely be invalidated by the same architectural variation it tries to control for. Instead (with respect to performance) we should focus on three factors: - Code maintainability - Algorithmic complexity - Cache coherency Everything else is a distraction. 1. link 2. ▶ this video 3. link 4. link

Aside: this is the most amazing optimisation information I have seen: ▶ this video Unfortunate that it is a video... Summary: Modern CPUs have so many hidden causes of performance variation, that you require special tools to actually measure small performance gains. He finds an amazing but unobvious speed up in SQLite using the causal analysis tool they developed. And a surprising difference between -O2 and -O3 implying overfitting.

I haven't had a chance to mess around with it much, but the coz-profiler [1] uses basically the same idea, but the other way around (i.e., slow down everything else to get a "virtual speedup") which helps quantify what the effect of optimizing some code would be. I learnt about this tool from Emery Berger's talk [2] on this (at strangeloop), which I highly recommend. Lots of really nice insight, even outside of this tool. [1] link [2] ▶ this video

"Performance matters" by Emery Berger on StrangeLoop'19 is a great talk - ▶ this video Also, rest of his talks are also fascinating: - link - link

>There was a really interesting talk at Strange Loop this year about how quirks in code layout fool programmers into chasing their tails thinking they’re improving performance when they’re not. I think the author may be referring to this talk: ▶ this video