So I wasn't kidding about the long slog: I am still chewing through the classic textbook Pattern Recognition and Machine Learning (PDF) by Christopher Bishop, day and night, even though he's got a newer book out. This is in part because I'm almost done, and in part because his newer book focuses on the foundations of deep learning and is "almost entirely non-Bayesian" - and it's Bayesian theory I'm trying to understand.

This, I think, is part of the discovery I've made recently about "deep learning" - by which I mean learning in depth by people, as opposed to deep learning by machines: hard concepts are by definition tough nuts to crack, and to really understand them, you need to hit them coming and going - to break apart the concept in as many ways as possible to ensure you can take it apart and put it back together again. As Marvin Minsky once said, "You don't understand anything until you learn it more than one way."

To some people, that idea is intuitive; to others, it is easy to dismiss. But if you think about it, when you're learning a subject you don't know, it's like going in blind. And like the parable of the blind men and the elephant - each of whom touched one part of an elephant and assumed they understood the whole - if you dig deeply into a narrow view of a subject, you can get a distorted view, like extrapolating a giant snake from an elephant's trunk, or a tall tree from its leg, or a wide fan from its ear, or a long rope from its tail.

Acting as if those bad assumptions were true could easily get you stomped on - or skewered, by the elephant's tusk, which is sharp like a spear.

So back to Bayesian theory. Now, what the hell is a "Bayes," some of you may ask? (Why are you reviewing the obvious, others of you may snark). Look, we take chances every day, don't we? And we blame ourselves for making a mistake if we know that something is risky, but not so much if we don't know what we don't know - even though we intuitively know that the underlying chances aren't affected by what we know. Well, Thomas Bayes not only understood that, he built a framework to put that on a solid mathematical footing.

Some people think that Bayes' work on probability was trying to refute Hume's argument against miracles, though that connection is disputed (pdf). But the big dispute that arose was between "frequentists" who want to reduce probability to statistics, and "Bayesians" who represent probability as a statement of beliefs. Frequentists incorrectly argued that Bayesian theory was somehow "subjective", and tried to replace Bayesian reasoning with statistical analyses of imaginary projections of existing data out to idealized collections of objects which don't exist. Bayesians, in contrast, recognize that Bayes' Theorem is, well, a theorem, and we can use it to make objective statements of the predictions we can make over different statements of belief - statements which are often hidden in frequentist theory as unstated assumptions.

Now, I snark a bit about frequentist theory there - and not just because the most extreme statements of frequentist theory are objectively wrong, but because some frequentist mathematicians around the first half of the twentieth century engaged in some really shitty behavior which set mathematical progress back decades - but even the arch-Bayesian, E. T. Jaynes, retreated from his dislike of frequentist theory. In his perspective, frequentist methods are how we check the outcome of Bayesian work, and Bayesian theory is how we justify and prove the mathematical structure of frequentist methods. They're a synergy of approaches, and I use frequentism and the tools of frequentists in my research, um, frequently.

But my point, and I did have one, is that even something I thought I understood well is something that I could learn more about. Case in point was not, originally, what I learned about frequentism and Bayesianism a while back; it was what I learned about principal component analysis (PCA) at the session where I took the picture. (I was about to write "last night", but, even though this is a "blogging every day" post, due to me getting interrupted when I was trying to post, this was a few days ago).

PCA is another one of those fancy math terms for a simple idea: you can improve your understanding by figuring out what you should focus on. Imagine you're firing cannon, and you want to figure out where the cannonballs are going to land. There are all sorts of factors that affect this: the direction of the wind, the presence of rain, even thermal noise in the cannon if you wanted to be super precise. But the most important variables in figuring out where the cannonball is going to land is where you're aiming the thing! Unless you're standing on Larry Niven's We Made It in the windy season, you should be far more worried about where the cannon is pointed than the way the wind blows.

PCA is a mathematical tool to help you figure that out by reducing a vast number of variables down to just a small number - usually two or three dimensions so humans can literally visualize it on a graph or in a tank. And PCA has an elegant mathematical formalism in terms of vectors and matrix math which is taught in schools. But it turns out there's an even more elegant Bayesian formalism which models PCA as a process based on "latent" variables, which you can think about as the underlying process behind the variables we observe - using our cannonball example, that process is again "where they're aiming the thing," even if we ultimately just observe where the cannonballs land.

Bayesian PCA is equivalent (you can recover the original PCA formalism from it easily) and elegant (it provides a natural explanation of the dimensions PCA finds as the largest sources of variance) and extensible (you can easily adapt the number of dimensions to the data) and efficient (if you know you just want a few dimensions, you can approximate it with something called the expectation-maximization algorithm, which is way more efficient than the matrix alternative). All that is well and good.

But I don't think I could have even really understood all that if I hadn't already seen PCA in half a dozen other textbooks. The technique is so useful, and demonstrations about it are so illuminating, that I felt I had seen it before - so when Bishop cracked open his Bayesian formulation, I didn't feel like I was just reading line noise. Because, let me tell you, the first time I read a statistical proof, it often feels like line noise.

But this time, I didn't feel that way.

I often try to tackle new problems by digging deep into one book at a time. And I've certainly learned from doing that. But often, after you slog through a whole textbook, it's hard to keep everything you've learned in your head (especially if you don't have several spare weeks to work through all the end-of-chapter exercises, which is a situation I find myself in more often than not).

But more recently I have found going through books in parallel has really helped me. Concepts that one book flies over are dealt with deeply in another. Concepts that another book provides one angle on are tackled from a completely different one in another. Sometimes the meaning and value of concepts are different between different authors. Even intro books sometimes provide crucial perspective that helps you understand some other, deeper text.

So if you're digging into something difficult ... don't try to go it alone. When you reach a tough part, don't give up, search out other references to help you. At first it may seem an impossible nut to crack, but someone, somewhere, may have found the words that will help you understand.

-the Centaur

Pictured: Again Bishop, and again pound cake.

I once told my wife I was patient - and it was indeed four years from our first meeting to our marriage - but the truth of the matter is that I'm terrible at delayed gratification. I have a kazillion things I want to do and I want them all done now, now, now - but if these things I want done are MY creative projects, then I can't really hire anyone else to do them. I've got to do them myself.

This is a big bottleneck if I haven't yet learned the skill to my own satisfaction.

I've talked before about one of the techniques I use - reading the difficult book at the dinner table. I eat out a lot, and do a lot of my reading either in coffeehouses, at dinnertime, or sitting on a rocking chair near my house. But those places are useful for books that can be read in pieces, in any order. At the dinner table, I have one book set aside - usually the most difficult or challenging thing I am reading, a book which I take in a little bit at breakfast, a little bit at late night milk and pound cake, one bite-sized step at a time.

At the dinner table, I have read Wolfram's A New Kind of Science and Davies' Machine Vision and Jayne's Probability Theory: The Logic of Science and even an old calculus textbook from college that I was convinced I had failed to fully understand on the first readthrough (hint: I hadn't; I had inadvertently skipped one part of a chapter which unlocked a lot of calculus for me). And now I'm going through Bishop's Pattern Recognition and Machine Learning, which has taught me much that I missed about deep learning.

Here's the thing: having gone through (most of) two whole probability textbooks and a calculus textbook that I read to help support the probability textbooks, I no longer feel as unexpert about probability as I once did. It was my worst subject in college, hands down, but I have reached the point where I understand what I did not understand and why I didn't understand it, I know how to solve certain problems that I care about, I know where to look to get help on problems that I can't solve, and I have realized the need to be humble about problems that are currently beyond my framework of understanding.

[Whew! I almost said "I have learned to be humble" there. Ha! No, I don't think you can really learn to actually be humble. You can however learn the need to be humble and then try to achieve it, but humility is one thing that it is really difficult to actually have - and if you claim you have it, you probably don't.]

Now, I know this seems obvious. I know, I know, I know, if you read a buncha textbooks on something and are actually trying to learn, you should get better at it. But my experience is that just reading a textbook doesn't actually make you any kind of expert. At best, it can give you a slightly better picture of the subject area. You can't easily train yourself up for something quickly - you've got to build up the framework of knowledge that you can then use to actually learn the skill.

Which can lead you to despair. It feels like you read a buncha textbooks about something and end up more or less where you started, minus the time you spent reading the textbooks.

But that's only because the process of learning something complex can indeed be a really long slog.

If you keep at it, long enough, you can make progress.

You just have to be patient ... with yourself.

-the Centaur

Pictured: Pattern Recognition and Machine Learning by Bishop, sitting next to my breakfast a few days ago.