So, this is worthy of a longer blog post, but the short story is, looking under the streetlight for your keys may provide visibility, but it won’t help you find them unless you have reason to think that you dropped them there.
A lot of time in software development, we encounter things that don’t work, and problems like stochastic errors and heisenbugs can lead software developers to try voodoo solutions – rebooting, recompiling from head, running the build again, or restarting the server.
These “solutions” work well enough for us that it’s worthwhile trying them as a first step – several times recently I have encountered issues which simply evaporated when I refreshed the web page, restarted the server, retried the build, or restarted my computer.
But these are not actual fixes. They’re tools to clear away transient errors so we can give our systems the best chance of properly functioning. And if they don’t work, the most important thing to do is to really dig in and try to find the root cause of the problem.
On one of those systems – where I literally had tried refreshing the web page, restarting the server, refreshing the code, reinstalling the packages, and rebooting the computer – the build itself was something that you had to run twice.
Now, this is a common problem with software builds: they take a lot of files and memory, and they can get halfway through their builds and die, leaving enough artifacts for a second pass to successfully solve the problem. I had to do this at Google quite a bit.
But it isn’t a real solution. (Well, it was at Google, for this particular software, because its build was running up against some hard limits that weren’t fixable, but I digress). And so, eventually, my build for this website failed to run at all, even after a reboot.
The solution? A quick search online revealed it’s possible to change the parameters of the build to give it more memory – and once I did that, the build ran smooth as glass. The “run the build again” trick I’d learned at Google – learned where, for one particular project, it was not possible to fix the problem – had led me astray, and overapplying this trick to my new problem – where it had partial success – kept me from finding a real, permanent fix.
So, if a fix feels like voodoo, stop for a minute – and search for a better way.
-the Centaur
Pictured: What do you mean, you haven’t updated the package configuration for THREE YEARS?
