I just want to serve 5 terabytes.

My other favorite internal internal meme from that era was the Broccoli Man video, "I just want to serve 5 Terabytes." ▶ this video

Broccoli Man! classic ▶ this video (2010) To be fair, a lot of this changed after that video became a meme.. but I'd bet that the broccoli man template is still trending on memegen

Have you ever watched the original? It most certainly did not involve real people talking ( ▶ this video ). In fact, the overtly robotic voices added to the humor of the original, IMO. It was lost in this translation.

This is probably the most accurate statement possible. “I just want to store 5TiB somewhere” “Ha! Did you book multiple bigtable cells” ▶ this video

It's way over my head but aware it already was quite intricate and frustrating [1] [1a] at times. Finding a good stable online file storage area is getting harder, especially the last few years where people have been caught out when fires or other catastrophes [2] occur. [1] ▶ this video [video - 2010] [1a] link *A more universal link to a mp4 (15 megs) or mkv (5 megs) download as per the original youtube link since imposed restrictions youtube likes to impose on some people from time to time. [2] link

A couple of things have struck me about Google as an outsider: - Learning that they had 100K+ employees - Seeing resumes from people who had spent 5+ years at Google and had a project list with what I would expect from <1 year at a hedge fund or <3 years at a bank - Learning about the "I want to just serve 5 Terabytes" video/story [0] - Hiring people away from Google who were, for example, network engineers who turned out to be great SREs with good programming skills - They just kill off greats apps that people enjoy with little to no notice (granted, keeping stuff around forever ala AWS is its own issue) In other words, they became a big corporate behemoth and lost a lot of their "spark". Which is sad b/c I remember the early days of using Google search, Google maps etc where it felt like we were all living in the future. Sure, they are breaking ground in the AI space but 100% agree that it feels like they are on the downward trajectory. (although other firms have been here and come back so let's see...) 0 - ▶ this video

Maybe fear killed engineering. Even in a decently managed company, I can see so many engineering effort gets stalled, compromised, bastardized, or killed. Like your manager asks you to get sign-offs from 10 different orgs. Like you want to do a POC, yet somehow an infra team demanded that you integrate with their system even though your targeted users didn't give a shit. Like you just want to submit a workflow to Temporal Cloud yet a product manager demands that you create an abstraction because he is afraid of X and Y. Like you just want to serve 5TB of data, and you got sucked into this: ▶ this video . I'm not sure how companies battle this kind of fear. It looks Amazon's Working Backwards, Netflix's Freedom and Responsibility, Uber's Let Builders Build more or less can counter such fear, but ultimately it's about finding the right people who have good sense of product and who strive to make progress.

Ridiculously low-quality documentation, a build and development system that puts the local Rube Goldberg competition winner to shame, constant churn, total deafness or willful ignoring of wishes of dev ... and so on have nothing to do with what happened 20 years ago. It's the classic "I would like to serve 5 terabytes"[0] Google problem. (Their engineering culture reeks of this. See the tools they have released. Angular, Bazel, Go, Kubernetes, etc. The last one escaped the usual decline of incompetence because the community showed up. Go has GOPATH. Even in a language designed for "simplicity" initially getting it set up had this roadblock.) And no doubt the roots of the problem are manifold, but it seems all stemming from ongoing management incompetence (creation of shareholder value[1] maaaybe excluded, though I am very curious what kind of inane metrics the Android SDK team is whipped to chase). [0] ▶ this video [1] link

Ha ha ha. Have you seen the Broccoli Man short? And that was a satire of what Google's infrastructure looked like in 2010 . ▶ this video

millions sounds like a lot but it's like just 3 employees. To quote broccoli man, "i've forgotten how to count that low" [1]. [1]: ▶ this video

This video [0] was from 2010s, and seems like they had similar issues even back then. (never worked at Google) [0] ▶ this video

There are only so many TPUs available and they're kept at 100% utilization. Google has had complex "marketplaces" around company resources for years: ▶ this video It's a cultural value, like playing board games.

You might have had 20 years of experience building products that don't need to scale for millions of concurrent users, if you consider all "purely internal architectural changes" to be lower priority than the customer facing work just because it's purely internal. And the Google guy might have tried to spend time scaling a product that will have five concurrent users, because "this is Google" ( ▶ this video ). It's all about perspective, and CEO thought the other perspective was more valuable.

I am a Xoogler. The source document/presentation ("Coordination Headwinds") and idea are many years old, and from what I recall were not aimed to be critical of the company and its organization, rather to give its employee readers a feeling of solace and understanding for why getting seemingly small things done can feel overly hard and slow… a less humorous, and less SWE-specific version of "Broccoliman" if you will. Refer to the red panda that merely wants to serve 5TB of data. - ▶ this video - link I’ll try to continue using the organism-metaphor in my commentary. While the “slime mold” story provides a convenient and charitable explanation of Google's bureaucracy and dysfunctions, another explanation is: - As an organism, Google's executive decision-making functions are severely impaired by lack of accurate information, dysfunction of its middle-management “organs” or tiers, and impedance mismatches arising from genetic encodings or mental models associated with the slime-mold idea. - Despite being on a mission to “organize the world's information and make it universally accessible and useful,” Google is far less successful at doing this with its own information. These characteristics are not unique to Google, but some of Google’s cultural history makes this a particularly challenging problem. It may be convenient to think of Google as a slime-mold organism, and in an individual contributor’s day-to-day engineering activities it may very well be, but as a corporation it is no

Yeah, when I was there I saw plenty of <1% optimizations saving REDACTED gobs of money, and people were rewarded for it. I don't think it's applicable to most teams though. Imagine a foo/bar/widget app that only serves 20B people (obvious exaggeration to illustrate the point) and is only necessary up to a few hundred times per day. You can handle that sort of traffic on a laptop on my home router and still have enough hootzpah left to stream netflix. I mean, you are Google, and you need to do something better than that [0], but the hardware for your project is going to be negligible compared to other concerns unless you're doing FHE or video transcoding or something extraordinarily expensive. Walk that backward to, how many teams have 20B users or are doing extraordinarily expensive things? I don't have any clue, but when you look at public examples of cheap things that never got much traction and probably had a suite of engineers [1], I'd imagine it's not everyone in any case. You're probably mostly looking at people with enough seniority to be able to choose to work on core code affecting most services. [0] ▶ this video [1] link

You mean like: link This is common in the research PA. People don't want to deal with broccoli man [1]. [1] ▶ this video

The first time I heard about Monarch was in discussions about the hilarious "I just want to serve 5 terabytes" video[0]. [0]: ▶ this video

> The worst is, of course, GCL/borgcfg, the (turing complete) configuration language for Google's internal job scheduling software. “Noone has borgmon readability.” — Broccoli Man ( ▶ this video )

> At a different (unnamed) FAANG I'm guessing Google, on the basis of the recently published (to the public) "I just want to serve 5TB"[1] video. If it isn't Google, then the broccoli man video is still a cogent reminder that unyielding multi-region rigor comes with costs. 1. ▶ this video

Regarding the phrase "I don't know how to count that low", here's the video where it originated. Frankly I'm astonished this hasn't been shared before, and I hope I don't get in trouble for posting it (it's like 11 years old, surely nobody cares at this point, right?) ▶ this video