I used to work with an engineer who didn’t want anyone to touch his perfect JavaScript code, so back in 2012 he isolated it into a separate folder in Facebook’s monorepo called static_upstream.
The official reason of course was different: Refactor Facebook’s core JS library code (including React) into an isolated folder so that it can be shared on sites other than the desktop site, such as on mobile. That kinda made sense.
However, it was implemented in a way that static_upstream was then synced into downstream folders – one copy of every core JS file per project. Once we had three projects, the duplication had already reached millions of lines of code.
In 2016, I set out to get rid of this duplication. Get rid of static_upstream, the downstreams and just have a single html/js folder that every site inherits. It turned out the technical change to make that happen was literally one line of code! We should have never even created this elaborate system for syncing the same code across the repo!
The only problem was we had to actually move static_upstream and delete all the downstream folders while thousands of people were committing conflicting code and Mercurial (a Git that nobody uses) was constantly under massive load.
I was in Japan then, and worked with an intern based in the US. We wrote scripts, but given the amount of files it was still a highly manual process that took us both about 10 hours. We deleted more than 100,000 duplicated files – it was beautiful! I had to “hg push” and fake the approval (which at the time was verified by putting “Reviewed By: <username>” into the commit message) because the diffs became so large that our merge infra couldn’t handle it. I watched it ship to production, everything looked good, and then I went to bed.
The next morning I woke up to dozens of messages. The engineer owning web JS infra sent many insults my way, off our work chat: “What the hell are you doing, you should have known better.”
As it turned out, I didn’t know any better. Moving this many files completely wiped away the JavaScript bundling cache – which was based on file paths instead of unique module ids like every other system at Facebook – and it led the ML based bundling system to think all those files were new, resulting in facebook.com sending each file individually. Kind of like how Vite works in development, and it made facebook.com over one second slower for every user. I don’t even wanna think about how many millions of dollars that was.
A three-person team in New York spent 12 hours reverting our changes – using the same hg push method in reverse – and failing constantly because other people kept pushing changes to the files they were reverting.
I’m more of a fix-forward kind of guy, so I shut up that day planning how to do the same thing again - and make it stick. I remember trying to lighten the mood. I said: “No no no this cannot be happening” in the incident channel and when the engineering director for web infra asked what I was talking about I said: “Pokémon Go was just released”.
Anyway, the team reverted the one-million line change, and a week later the same intern and I wrote another set of scripts to fully automate the migration – this time with the correct updates to file paths in a random PHP file to make sure we didn’t wipe away the intricate machine learning bundling cache setup. Every second counts.
