So! I made it. I’m at the hotel where the Nebulas will be happening, and by the time you read this (according to my records this should be post 159, and according to this day-of-the-year calendar that should be Friday, but as I type it’s a hair before midnight Wednesday night) I should have already done the first of my events.
But I also have a day job, and since I was “drawing, on average, every day” and reading Neal Asher’s Shadow of the Scorpion on the flight out, I took advantage of my oops-you-checked-in-in-advance-but-we-didn’t-reserve-your-actual-room drink voucher and got an Old Fashioned at the soon-to-close hotel bar.
(Apologies to the hotel bar staff: I came in 30 minutes before close, just as they were cleaning up and switching over to sidework, but me plopping myself down at the bar apparently opened the floodgates, for something like half a dozen people then showed up right after I did).
At the bar, I cracked open Visual Studio Code, ChatGPT and Stack Overflow in an attempt to find a more parsimonious dataset representation for one of my clients. I’d built a horribly data-inefficient version of a machine learning dataset for them on the principle “get the fucker running so we can see whether it works” but the fucker worked, so we need now to make it at least marginally more efficient as we now turn our attention to “let’s see whether this fucker can fucking scale up.”
It looks like a data representation called HDF5 is worth a first shot (not that it’s the best or the only, but it has C++ and Python bindings and appears simple to integrate into both our custom data set writer and into our custom PyTorch data loader). So, I did a little digging via Bing/Google to verify the best way to install HDF5 for Python (h5py for Conda, in this case) and set down to try out ChatGPT’s recommended test case.
But … the installation locked up.
Restarted the install. No dice. Then I thought it was the janky hotel Wi-Fi. Switched to my own personal hotspot. No dice. Tried a bunch of StackOverflow recommendations to fix the problem. No dice. Fifteen, then thirty, then forty-five minutes stretched by, as I tried to get a simple darn package to load.
This is, as I’ve said before, the problem of “molasses” in computer programming: the gummy gook which makes it impossible to do simple tasks. Another colleague called it “the novice penalty, and it’s real”: people who work in a domain all the time learn the tricks to make it work, but novices don’t know these tricks, and struggle to do things that “experts” think are easy because they’ve forgotten they are difficult.
I almost gave up. But molasses needs to be fought. As I often say, oftentimes, you need to work a little bit harder than you think you need to, and when you do, you’ll find that you’re greatly rewarded by a breakthrough. Molasses can gum us up, but if we push through, we may find that it becomes smooth sailing.
In this case, the solution was actually to use ChatGPT’s suggestion for installing the HDF5 package: ‘pip install h5py’, rather than ‘ conda install anaconda:h5py’. The benefit of doing it the ‘conda’ way is that the installation is in a ‘Python environment’ that corrals the installed software so it doesn’t break anything else; but, for whatever reason, my conda environment was having trouble with that, so pip – which installs the program globally on the computer, across ‘environments’ – was the way to go,
From there I was able to start making progress on my dataset loader problem, and have a clear direction for the project to take tomorrow. Had I accepted the slowdown imposed by the molasses, I would have returned to this problem tomorrow with no real clue of the next steps to take, other than remembering that I had tried a bunch of stuff, got exhausted, and decided to start fresh in the morning.
Sometimes that’s the right thing to do, of course. But if we can push through the molasses through to the other side, we often will be doubly rewarded: not only will we solve the immediate problem we were facing, but also will have a solid foundation to move forward on our next task.
So don’t let the molasses bog you down. Push on through, and leave it behind if you can.
-the Centaur
Pictured: One from the archives – some Mathematica analysis of a problem – while the blog images are down.
