Good Friday Vigil at Saint Stephen's in-the-Field. We dress down the church and set up a bare wood cross and labyrinth, and encourage people to sign up to stay and pray so we have coverage all night.
I am a night owl, so I signed up for 1 a.m. through 2 a.m. So why am I here with a cough at 2:45 a.m. when I have an early-for-me meeting tomorrow? Someone changed my slot without telling me, to 2 a.m. through 4 a.m.
So I had the double pleasure of waiting fifteen minutes in the cold for the shift change (while I confirmed, via Google Docs history, that I was not misremembering my time), finding out that the person inside was still only partially through their two hour shift, going home to crash, and coming back to wait in the colder cold again while the previous person ran over. (The irony of the sleeping apostles is not lost on me).
This has been my least effective Lent in recent memory. I went to Ash Wednesday service to get ashes, only to get quizzed about it by my favorite server at one of my favorite restaurants, who then to my dismay turned into an insulting, manipulative proselytizer. I have had a surprising share of similar bad reactions with people leaving me more rattled about how I treat and react to people (even though I was never the aggressor) than focused on God or reading the Bible. Visiting the sick has not worked as my friend who is hurt the most is too touch and go for visitors. And giving up alcohol for Lent proved more of an inconvenience than a prompt for reflection.
And yet, like going to church on Sunday, or volunteering for the church Vestry, or reading the Bible, the Vigil is serving its function: to draw my attention back to God.
May God's peace, which passes all understanding, be with you always.
-the Centaur
Posts published by “centaur”
The biggest "current" in my mind is the person I am currently worried about, my good friend and great Game AI developer Dave Mark. Dave is the founder of the GDC AI Summit ... but was struck by a car leaving the last sessions at GDC, and still is in the hospital, seriously injured.
Dave is a really special person. I've been going to GDC longer than Dave, but it was he (along with my friend Neil Kirby) who drew me out of my shell and got me to participate in the Game AI community, which is a super important part of my life even though I don't do Game AI for my day job.
Dave's friends and family have set up a Go Fund Me to help cover his medical expenses and the travel and other expenses of his family while he remains in the hospital in the Bay Area. I encourage you all to help out - especially if you've ever played a game and found the AI especially clever.
Dave, you're in our prayers ...
-the Centaur
Pictured: Dave (on the right) and friends.
SO! Hey! GDC and Clockwork Alchemy are over and I'm not dead! (A joke which actually I don't find that funny given the circumstances, which I'll dig into in just a moment). Strangely enough, hitting two back-to-back conferences, both of which you participate super heavily in, can take something out of your blog. Who knew?
But I need to get better at blogging, so I thought I'd try something new: a "check-in" in which I try to hit all the same points each time - what am I currently writing, editing, programming, etc? For example, I am currently:
- Listening To: Tomb Raider soundtrack (the original).
- Reading: Theoretical Neuroscience (book).
- Writing: "Death is a Game for the Young", a novella in the Jeremiah Willstone multiverse.
- Editing: SPECTRAL IRON, Dakota Frost #4.
- Reviewing: SHATTERED SKY, Lunar Cycle #2 by David Colby.
- Researching: Neural Approaches to Universal Subgoaling.
- Programming: A toy DQN (Deep Q Network) to stretch my knowledge.
- Drawing: Steampunk girls with goggles.
- Planning: Camp Nanowrimo for April, ROOT USER, Cinnamon Frost #3.
- Taking on: Giving up alcohol for Lent.
- Dragging on: Doing my taxes.
- Spring Cleaning: The side office.
- Trying to Ignore: The huge pile of blogposts left over from GDC and CA.
- Caring For: My cat Lenora, suffering from cancer.
- Waiting For: My wife Sandi, returning from a business trip.
As an author, I'm interested in how well my books are doing: not only do I want people reading them, I also want to compare what my publisher and booksellers claim about my books with my actual sales. (Also, I want to know how close to retirement I am.)
In the past, I used to read a bunch of web pages on Amazon (and Barnes and Noble too, before they changed their format) and entered them into an Excel spreadsheet called "Writing Popularity" (but just as easily could have been called "Writing Obscurity", yuk yuk yuk). That was fine when I had one book, but now I have four novels and an anthology out. This could take out half an a hour or more, which I needed for valuable writing time. I needed a better system.
I knew about tools for parsing web pages, like the parsing library Beautiful Soup, but it had been half a decade since I touched that library and I just never had the time to sit down and do it. But, recently, I've realized the value of a great force multiplier for exploratory software development (and I don't mean Stack Exchange): interactive programming notebooks. Pioneered by Mathematica in 1988 and picked up by tools like iPython and its descendent Jupyter, an interactive programming notebook is like a mix of a command line - where you can dynamically enter commands and get answers - and literate programming, where code is written into the documents that document (and produce it). But Mathematica isn't the best tool for either web parsing or for producing code that will one day become a library - it's written in the Wolfram Language, which is optimized for mathematical computations - and Jupyter notebooks require setting up a Jupyter server or otherwise jumping through hoops.
Enter Google's Colaboratory.
Colab is a free service provided by Google that hosts Jupyter notebooks. It's got most of the standard libraries that you might need, it provides its own backends to run the code, and it saves copies of the notebooks to Google Drive, so you don't have to worry about acquiring software or running a server or even saving your data (but do please hit save). Because you can try code out and see the results right away, it's perfect on iterating ideas: no need to re-start a changed program, losing valuable seconds; if something doesn't work, you can tweak the code and try it right away. In this sense Colab has some of the force multiplier effects of a debugger, but it's far more powerful. Heck, in this version of the system you can ask a question on Stack Overflow right from the Help menu. How cool is that?
My prototyping session got a bit long, so rather than try to insert it inline here, I wrote this blog post in Colab! To read more, go take a look at the Colaboratory notebook itself, "A Sip of the Tracking Soup", available at: https://goo.gl/Mihf1n
-the Centaur
When I was a kid (well, a teenager) I'd read puzzle books for pure enjoyment. I'd gotten started with Martin Gardner's mathematical recreation books, but the ones I really liked were Raymond Smullyan's books of logic puzzles. I'd go to Wendy's on my lunch break at Francis Produce, with a little notepad and a book, and chew my way through a few puzzles. I'll admit I often skipped ahead if they got too hard, but I did my best most of the time.
I read more of these as an adult, moving back to the Martin Gardner books. But sometime, about twenty-five years ago (when I was in the thick of grad school) my reading needs completely overwhelmed my reading ability. I'd always carried huge stacks of books home from the library, never finishing all of them, frequently paying late fees, but there was one book in particular - The Emotions by Nico Frijda - which I finished but never followed up on.
Over the intervening years, I did finish books, but read most of them scattershot, picking up what I needed for my creative writing or scientific research. Eventually I started using the tiny little notetabs you see in some books to mark the stuff that I'd written, a "levels of processing" trick to ensure that I was mindfully reading what I wrote.
A few years ago, I admitted that wasn't enough, and consciously began trying to read ahead of what I needed to for work. I chewed through C++ manuals and planning books and was always rewarded a few months later when I'd already read what I needed to to solve my problems. I began focusing on fewer books in depth, finishing more books than I had in years.
Even that wasn't enough, and I began - at last - the re-reading project I'd hoped to do with The Emotions. Recently I did that with Dedekind's Essays on the Theory of Numbers, but now I'm doing it with the Deep Learning. But some of that math is frickin' beyond where I am now, man. Maybe one day I'll get it, but sometimes I've spent weeks tackling a problem I just couldn't get.
Enter puzzles. As it turns out, it's really useful for a scientist to also be a science fiction writer who writes stories about a teenaged mathematical genius! I've had to simulate Cinnamon Frost's staggering intellect for the purpose of writing the Dakota Frost stories, but the further I go, the more I want her to be doing real math. How did I get into math? Puzzles!
So I gave her puzzles. And I decided to return to my old puzzle books, some of the ones I got later but never fully finished, and to give them the deep reading treatment. It's going much slower than I like - I find myself falling victim to the "rule of threes" (you can do a third of what you want to do, often in three times as much time as you expect) - but then I noticed something interesting.
Some of Smullyan's books in particular are thinly disguised math books. In some parts, they're even the same math I have to tackle in my own work. But unlike the other books, these problems are designed to be solved, rather than a reflection of some chunk of reality which may be stubborn; and unlike the other books, these have solutions along with each problem.
So, I've been solving puzzles ... with careful note of how I have been failing to solve puzzles. I've hinted at this before, but understanding how you, personally, usually fail is a powerful technique for debugging your own stuck points. I get sloppy, I drop terms from equations, I misunderstand conditions, I overcomplicate solutions, I grind against problems where I should ask for help, I rabbithole on analytical exploration, and I always underestimate the time it will take for me to make the most basic progress.
Know your weaknesses. Then you can work those weak mental muscles, or work around them to build complementary strengths - the way Richard Feynman would always check over an equation when he was done, looking for those places where he had flipped a sign.
Back to work!
-the Centaur
Pictured: my "stack" at a typical lunch. I'll usually get to one out of three of the things I bring for myself to do. Never can predict which one though.
Here's what was in the rabbit hole from last time (I had been almost there):
I had way too much data to exploit, so I started to think about culling it out, using the length of the "mumbers" to cut off all the items too big to care about. That led to the key missing insight: my method of mapping mumbers mapped the first digit of each item to the same position - that is, 9, 90, 900, 9000 all had the same angle, just further out. This distance was already a logarithm of the number, but once I dropped my resistance to taking the logarithm twice...
... then I could create a transition plot function which worked for almost any mumber in the sets of mumbers I was playing with ...
Then I could easily visualize the small set of transitions - "mumbers" with 3 digits - that yielded the graph above; for reference these are:
The actual samples I wanted to play with were larger, like this up to 4 digits:
This yields a still visible graph:
And this, while it doesn't let me visualize the whole space that I wanted, does provide the insight I wanted. The "mumbers" up to 10000 do indeed "produce" most of the space of the smaller "mumbers" (not surprising, as the "mumber" rule 2XYZ produces XYZ, and 52XY produces XYXY ... meaning most numbers in the first 10,000 will be produced by one in that first set). But this shows that sequences of 52 rule transitions on the left produce a few very, very large mumbers - probably because 552552 produces 552552552552 which produces 552552552552552552552552552552552552 which quickly zooms away to the "mumberOverflow" value at the top of my chart.
And now the next lesson: finishing up this insight, which more or less closes out what I wanted to explore here, took 45 minutes. I had 15 allotted to do various computer tasks before leaving Aqui, and I'm already 30 minutes over that ... which suggests again that you be careful going down rabbit holes; unlike leprechaun trails, there isn't likely to be a pot of gold down there, and who knows how far down it can go?
-the Centaur
P.S. I am not suggesting this time spent was not worthwhile; I'm just trying to understand the option cost of various different problem solving strategies so I can become more efficient.
SO! There I was, trying to solve the mysteries of the universe, learn about deep learning, and teach myself enough puzzle logic to create credible puzzles for the Cinnamon Frost books, and I find myself debugging the fine details of a visualization system I've developed in Mathematica to analyze the distribution of problems in an odd middle chapter of Raymond Smullyan's The Lady or the Tiger.
I meant well! Really I did. I was going to write a post about how finding a solution is just a little bit harder than you normally think, and how insight sometimes comes after letting things sit.
But the tools I was creating didn't do what I wanted, so I went deeper and deeper down the rabbit hole trying to visualize them.
The short answer seems to be that there's no "there" there and that further pursuit of this sub-problem will take me further and further away from the real problem: writing great puzzles!
I learned a lot - about numbers, about how things could combinatorially explode, about Ulam Spirals and how to code them algorithmically. I even learned something about how I, particularly, fail in these cases.
But it didn't provide the insights I wanted. Feynman warned about this: he called it "the computer disease", worrying about the formatting of the printout so much you forget about the answer you're trying to produce, and it can strike anyone in my line of work.
Back to that work.
-the Centaur
SO RECENTLY I had a very vivid dream in which my veterinarian said to me "There are some cats you save … and some you make comfortable." I think the context behind that dream is worth a little unpacking, don't you?
Loki the Loquacious is a cat that we saved. I came home one day to find him yowling and lethargic, sensitive to the touch yet unwilling to move, with a bloated feeling to the touch, and after a brief search online we rushed him to the nearby animal hospital who quickly diagnosed him with a urinary tract blockage, put him on a catheter, and nursed him back to health.
Now, he hates the urinary tract pet food we feed him and the occasional water droppers when he’s not drinking, but unless this outdoor cat gets too adventurous, he’s probably got a long life ahead of him.
Caesar the Conqueror is a cat that we made comfortable. He’d been made frail by a long battle with a thyroid condition when he decided to start peeing inappropriately indoors, so we had to make him an outdoors cat; but we were able to set up a relatively nice outdoor area for him. But then some nasal obstruction began interfering with his breathing, and he ultimately wheezed himself to death.
We kept him comfortable, of course, until he took a rapid turn downhill, and then we had him peacefully put to sleep in my arms.
As for Lenora the Cat … the jury is still out.
She’s a healthy-looking, happy-looking, active cat, and even though she from time to time got pencil-eraser sized moles, and once even a larger lump on a back leg, they were always benign … until a month ago. Then a new mole appeared, and another, and another, until she had dozens of the tiny, not-itchy, not-bleeding, not-discolored bumps all over her body. We took her to the doctor, who found two more walnut-sized lumps in her abdomen; biopsies revealed these to be mast cell tumors (MCT or mastocytoma).
Our doctor’s recommended regimen - a cortisone shot, followed by predisone and possibly other medications - tracks with what I’ve been able to research. Cortisone and similar drugs are recommended, and sometimes even can cure it, especially if it’s on the skin; but prognosis for lumps in the internals are more guarded - and she’s gotten another lump since the biopsy.
So now we’re researching, weighing the options of continuing treatment vs seeing an oncologist now (our vet is of the opinion that we’d have to wait a few weeks for an oncologist to get good readings on bloodwork because of the cortisone shot, but if I was an oncologist I’d want to see that third lump right now). Cats with this condition can last three years with surgery, a year with palliative care … or can die within weeks if it’s serious.
We don’t yet know if Lenora’s a cat we must make comfortable … or that we can save.
Here’s hoping.
-The Centaur
The following was written just before I left on Christmas vacation. The fact that I’m posting it three weeks later I think says something about the very point I was making in the article … so I’m going to let it stand as I wrote it the day that it happened. Here goes …
—
So, my cat died in my lap today, and while I didn’t kill it, I made it happen.
I’d love to say I have a lot of feelings about that.
The truth is, for me, departures leave a void. I don’t know what to feel, or don’t feel anything. Our precious little fraidy cat Caesar is gone, just gone, and the event passed without the reactions that movies and literature tell me happen when people go through life-changing events.
And this is a change, make no mistake. Almost twelve years ago, I and my wife agreed to adopt two rescue cats, Nero (the big black butch one) and Caesar (the skinny Holstein-cow one afraid of crinkling paper). They’d been turned out onto the street by a couple who got on drugs, and were being fostered by one of our bridesmaids, who already had three tiny, frail, elderly cats, and was forced to keep both cats in a bathroom. We had Nero and Caesar shipped from the East Coast to the West, and made them a part of our lives.
Nero’s long gone, victim of coyotes, but Caesar, with a different behavioral inheritance, survived and thrived, until a few years ago thyroid problems caused him to start to lose weight. He wasted away from twelve pounds to seven over the years, but we were mostly able to control it with medication, even when we ultimately had to put Caesar outside when, in his old age, he decided it was just fine to pee, like, wherever, because he’d reached the age where he didn’t have to give a damn anymore.
Bay Area winters are, of course, as brutal as cream puffs, but we nonetheless set up a huge gazebo enclosure in the back yard, where a tarp, pillows, heating pads and collection of chairs, tables and cat condos gave him a comfy throne for over a year.
But then he started wheezing. At first it was a cute little cooing-dove purr, and we thought he was just becoming more vocal. But it developed into a whistling, ticking sound as he labored for breath. Never comfortable on trips to the vet—always scared and panting, frequently pooping in the carrier even when in the best of health—on his last trip he was so freaked out they had to put him on oxygen. Tweaks to his medication and a cortisone shot helped for a while, but soon he was back where he started, with the recommendation of the vet that we make him comfortable.
And we did—or, mostly, my wife did.
She constantly reworked the outer area to make it a luxurious throne. A night owl herself, she fed him at all hours as, despite his decreasing weight of six and a half pounds, he became our most ravenous cat. And she stayed with him to brush him or sit with him or make him happy.
And me? I’m the one who dragged us out to the Bay Area to work for a search engine company, and I’m the one who has to work long hours keeping the lights on now that I’ve transitioned from search to robotics. I’m the one who chose to take on a huge writing project at which I’m barely started, and I’m the one who chose to take on helping found a small press. I seemingly can’t say no to projects, not because I want to do so many projects, but because that’s the only way I have found to make the projects that I do work on into successes—constantly seeking other avenues, other points of connections that make the work that I do more valuable. So now I find myself with an enormous stack of responsibilities that I can’t easily unwind.
For a variety of reasons, this has become even worse in the last six months, right when Caesar began his decline. Weekend after weekend I planned to spend time sitting in the back yard with the cats, and weekend after weekend I found myself working late at work or putting out fires at the small press. And week after week, I saw Caesar continue to decline.
I even knew this was likely to happen, and took a picture intending to blog about caring for elderly cats. But life intervened, and Caesar has now passed without me ever posting that post about his decline. I can’t look at those pictures without thinking about dereliction of duty.
Finally, I had enough, and started to arrange time to spend more time with Caesar. But it was too late. He’d grown too frail to clean himself, but no longer enjoyed brushing, pulling away from me when I tried to clean out his fur. He’d grown too scattershot to properly drink from poured water, but no longer enjoyed suckling my knuckle, making a few halfhearted attempts at the gesture that had calmed him so much as a young cat before wobbling away. I’d sit in the Adirondack chair in the back yard, hoping he’d come up and sit in my lap, and for a while he did, scrabbling his way up on me, getting a scratch, then shakily hop down and walk away. I eventually tried picking him up to put him in my lap, but he just wanted down. By the end, he barely tolerated a scratch behind the ears, and would quickly give up or walk away.
As Christmas approached, I worried that he wouldn’t be here when I got back from visiting my folks—but last night, we noticed vomit on his pillow. Today he wasn’t sitting in his throne, and I found him lying against the fence in the back yard, muzzle covered in vomit, drooling on his paws, unable to muster the energy to eat and unwilling to tolerate my touch.
I called in at work, woke up my wife, and we started calling for home pet euthanasia services. After half a dozen calls, we had an appointment arranged, and in the mid afternoon, a kindly veterinarian came by. Caesar had slid even further, with a soft, plaintive mew, and the vet gave him a sedative to help him sleep, and soon he was breathing easy for the first time in weeks.
Five minutes later, I was sitting on the porch, with Caesar in my lap. The vet shaved a small patch of fur on his leg to get to his vein, and injected the final shot. I put my hand on his chest as he breathed his last, and the vet listened until his tiny heart stopped. The vet left us an impressed paw print in clay and a tiny bundle of fur, and took our cat, wrapped up in a basket, looking more comfortable than he had in six months. Then Caesar was gone.
I wasn’t there when my dad died. I knew he was going, I even quit work so that I could be there for him while he was dying in Greenville, South Carolina, but for some reason at the time I felt like I had to periodically go back to my home town, Atlanta, Georgia, for what, I don’t remember now, to keep up the condo, or for my karate classes, or whatever, and on one of my returns to Greenville Dad passed while I was finding a parking space in the Greenville Memorial parking lot. Mom stood straight, but was in tears, and I knew what had happened; Dad’s body lay there, his eyes open, half lidded, his head turned partially aside, not rightable, the human body’s unconscious processes of self-stabilization and homeostasis finally ceased. So Dad was gone.
I wasn’t there when my grandmother died. She’d been in the nursing home for a while, and the doctors warned us that she’d had a sharp slide. We came out to see her. Mom, strangely, didn’t want to go into the room, seeming somehow semi-estranged from her, despite being about as good to her as she could have been. I went see Grandma; she was holding her hands tight, her eyes half-lidded, barely registering my presence. We waited a long time, then returned the next day, and waited again. Finally we went for a late lunch, and when we returned, it was over. And Grandma was gone.
I wasn’t there when my Aunt Kitty died. She’d been in decent health, despite a painful hip problem, and was jogging at the gym one day when she had a heart attack and fell off the treadmill. I was already on my way to Greenville for other reasons, but when I arrived, she again was barely holding on, each of her organs struggling to keep up, offloading their problems onto another. I parsed the jargon the doctors were saying and re-uttered the words to the family in words they understood, and they seemed comforted. She lay there, writhing a little; once her eyes, half-lidded, seemed to recognize me. But the family told me to leave, and after a few days, I flew back to the Bay Area. She passed the next day, and I flew back for the funeral. But Aunt Kitty was gone.
I wasn’t there when Gray Cat died. He was a feral who stayed in the yard, and we slowly started the process of trying to tame him. I was the only one who could feed him. I was the only one who could pet him, and I did it with gloves. But we had started to play together, and he started to warm—then got in through the cat door and attacked my wife. She had to fight him off with a broom, and we ultimately decided that he was dangerous enough that we had to put him to sleep. But it was my wife who took him to the pound. And Gray Cat was gone.
I wasn’t there when Caesar’s brother Nero died; as I said, he was taken by coyotes. He was an active outdoor cat, and we could even take him on walks without a leash. But that expanded his range, and he loved hunting on the watershed hill near our home. One night went out late at night, shortly before we heard the coyotes howl. He never came back. We posted flyers and walked the neighborhood, and checked shelters, but none of that mattered; we knew what happened the very next morning. And Nero was gone.
Nero's death came without warning. I knew Caesar’s end was coming. I was determined to not let him die alone and afraid the way Nero did. So I kept close watch on him. I thought through the scenarios he might encounter and decided what I was and was not willing to put him through. The ultimate criteria, I decided, was if he could not breathe, if he could not eat, or if he could not get up; today, two of those three happened. So we acted.
I was there when Caesar died. We let him lie where he had chosen until the drugs put him into a peaceful sleep, and then I held him in my lap until he passed. And after he was gone, I asked my wife to go for a walk, and I unloaded to her about how I wanted to have been there more.
“No,” she said. “We are a team, and I was there for him, several times, every day, while you worked. While you spent your love on the cats that still wanted affection, I focused instead on Ceasar and gave him all the attention he needed. We gave him everything we could.”
I still don’t know what I feel about this. I must feel something: I’ve been prompted to write two thousand words on it. But the feeling is that of a void. An uncertainty of how I should react or how I should feel. The only thing I know is that I made sure I was there when Caesar died.
—
Epilogue: Caesar is gone. Now one of our other cats, Lenora, has erupted in tiny bumps and larger lesions, along with two big lumps in her abdomen. Is it cancer, and she’s soon to be gone? Is it simply cowpox, and she’ll be fine in a month or two? I don’t know. But I do know I am making a special effort to be with her, and with my wife, and my friends and family, while they are alive.
-the Centaur
I often say "I teach robots to learn," but what does that mean, exactly? Well, now that one of the projects that I've worked on has been announced - and I mean, not just on arXiv, the public access scientific repository where all the hottest reinforcement learning papers are shared, but actually, accepted into the ICRA 2018 conference - I can tell you all about it!
When I'm not roaming the corridors hammering infrastructure bugs, I'm trying to teach robots to roam those corridors - a problem we call robot navigation. Our team's latest idea combines "traditional planning," where the robot tries to navigate based on an explicit model of its surroundings, with "reinforcement learning," where the robot learns from feedback on its performance.
For those not in the know, "traditional" robotic planners use structures like graphs to plan routes, much in the same way that a GPS uses a roadmap. One of the more popular methods for long-range planning are probabilistic roadmaps, which build a long-range graph by picking random points and attempting to connect them by a simpler "local planner" that knows how to navigate shorter distances. It's a little like how you learn to drive in your neighborhood - starting from landmarks you know, you navigate to nearby points, gradually building up a map in your head of what connects to what.
But for that to work, you have to know how to drive, and that's where the local planner comes in. Building a local planner is simple in theory - you can write one for a toy world in a few dozen lines of code - but difficult in practice, and making one that works on a real robot is quite the challenge. These software systems are called "navigation stacks" and can contain dozens of components - and in my experience they're hard to get working and even when you do, they're often brittle, requiring many engineer-months to transfer to new domains or even just to new buildings.
People are much more flexible, learning from their mistakes, and the science of making robots learn from their mistakes is reinforcement learning, in which an agent learns a policy for choosing actions by simply trying them, favoring actions that lead to success and suppressing ones that lead to failure. Our team built a deep reinforcement learning approach to local planning, using a state-of-the art algorithm called DDPG (Deep Deterministic Policy Gradients) pioneered by DeepMind to learn a navigation system that could successfully travel several meters in office-like environments.
But there's a further wrinkle: the so-called "reality gap". By necessity, the local planner used by a probablistic roadmap is simulated - attempting to connect points on a map. That simulated local planner isn't identical to the real-world navigation stack running on the robot, so sometimes the robot thinks it can go somewhere on a map which it can't navigate safely in the real world. This can have disastrous consequences - causing robots to tumble down stairs, or, worse, when people follow their GPSes too closely without looking where they're going, causing cars to tumble off the end of a bridge.
Our approach, PRM-RL, directly combats the reality gap by combining probabilistic roadmaps with deep reinforcement learning. By necessity, reinforcement learning navigation systems are trained in simulation and tested in the real world. PRM-RL uses a deep reinforcement learning system as both the probabilistic roadmap's local planner and the robot's navigation system. Because links are added to the roadmap only if the reinforcement learning local controller can traverse them, the agent has a better chance of attempting to execute its plans in the real world.
In simulation, our agent could traverse hundreds of meters using the PRM-RL approach, doing much better than a "straight-line" local planner which was our default alternative. While I didn't happen to have in my back pocket a hundred-meter-wide building instrumented with a mocap rig for our experiments, we were able to test a real robot on a smaller rig and showed that it worked well (no pictures, but you can see the map and the actual trajectories below; while the robot's behavior wasn't as good as we hoped, we debugged that to a networking issue that was adding a delay to commands sent to the robot, and not in our code itself; we'll fix this in a subsequent round).
This work includes both our group working on office robot navigation - including Alexandra Faust, Oscar Ramirez, Marek Fiser, Kenneth Oslund, me, and James Davidson - and Alexandra's collaborator Lydia Tapia, with whom she worked on the aerial navigation also reported in the paper. Until the ICRA version comes out, you can find the preliminary version on arXiv:
So, when I say "I teach robots to learn" ... that's what I do. -the Centaurhttps://arxiv.org/abs/1710.03937 PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning
We present PRM-RL, a hierarchical method for long-range navigation task completion that combines sampling-based path planning with reinforcement learning (RL) agents. The RL agents learn short-range, point-to-point navigation policies that capture robot dynamics and task constraints without knowledge of the large-scale topology, while the sampling-based planners provide an approximate map of the space of possible configurations of the robot from which collision-free trajectories feasible for the RL agents can be identified. The same RL agents are used to control the robot under the direction of the planning, enabling long-range navigation. We use the Probabilistic Roadmaps (PRMs) for the sampling-based planner. The RL agents are constructed using feature-based and deep neural net policies in continuous state and action spaces. We evaluate PRM-RL on two navigation tasks with non-trivial robot dynamics: end-to-end differential drive indoor navigation in office environments, and aerial cargo delivery in urban environments with load displacement constraints. These evaluations included both simulated environments and on-robot tests. Our results show improvement in navigation task completion over both RL agents on their own and traditional sampling-based planners. In the indoor navigation task, PRM-RL successfully completes up to 215 meters long trajectories under noisy sensor conditions, and the aerial cargo delivery completes flights over 1000 meters without violating the task constraints in an environment 63 million times larger than used in training.
So one of the things I like to do each year, as part of my traditional visit to family over the holidays, is to drop in on a Panera Bread, pull out my notebook, review my plans for the previous year, and make plans for the new one.
As of the 7th of January, I still haven't done this yet.
Shit happened last year. Good shit, such as really getting serious about teaching robots to learn; bad shit, such as serious illnesses in the pets in our family; and ugly shit which I'm not going to talk about until the final contracts are signed and everyone agrees everything is hunky and dory. And much of this went down just before the holidays, and once the holidays started, I cared a lot more about spending time with family and friends than sitting by myself in a Panera. (In all fairness, the holidays were easier when I lived in Atlanta and came up to see family many times a year, as opposed to only occasionally).
But I can recommend trying to do a yearly review. One year I decided to list what I wanted to do, both in the immediate future, in the coming year, in the coming 5 years, and in my life; and the next year, almost by chance, I sat down in the same Panera to review it. That served me well for more than a decade, and I find that even trying to do it helps me feel more focused and refreshed.
And so that's precisely what I tried to do yesterday. I didn't accomplish it - I still haven't managed to "clear the thickets" of my TODO lists to get to the actual yearly plan, and I miss being able to take a whole afternoon at Panera doing this - but I did the next best thing, sitting myself down to a nice "reboot" dinner and treating myself to a showing of Star Wars: The Last Jedi.
As someone said (a reference I read recently, but have been unable to find) the very act of doing something daily centers the mind.
Here's to that.
-Anthony
Well, one more orbit around the Sun. Here's to a better 2018! Onward, friends!
What makes you hang on the edge of your seat? I call that a favorite, and I talk about some of my current faves over at the Speculative Chic blog!
[embed]http://speculativechic.com/2017/12/18/my-favorite-things-with-anthony-francis/[/embed]
Go check it out!
I was going to write "And from his labors, he rested" but that's entirely to uncomfortably Messianic for me, so here's the scoop: on the last day of Nano, I have stopped at 75,282 words.
This somehow all magically happened because I never lost my momentum after the Night of Writing Dangerously, oh, and because this is Cinnamon Frost, and she's awesome!
This is the most I've ever written in Nano, by a long shot - almost 10,000 words more. Not quite, and I'm not super motivated to make it exactly 10,000 words more. If I think of more words tonight, eh maybe.
Oh yes, the traditional excerpt:
The first challenge was easy—spirit. Awareness. Being aware of faerie. The second challenge was harder—mind. Intellect. Learnin’ the logic of faerie. The third challenge was the hardest of all. Body. Emotion. Feeling faerie in your bones. A huge cacklin’ thing bursts out of the water. Its head is as big as Krishna’s, a huge green dripping thing under a mass of hair, its wide smooth but mottled nose remindin’ me of a diseased muppet. We can’t see the thing’s eyes, but its arms loom around us. Ben and Surrey screams. “Do you care?” it screams, openin’ a maw filled with giant teeth the size of playing cards. I think it could swallow any of us whole. “Do you care if you diieie?” “Aaaah!” Benjamin and Surrey screams. “We care! We care!” The thing looms further forward. “Then flee, mortals, or you may perish here!” “Don’t flee,” I murmurs. “Or you may perish elsewhere—” “We—we will not flee,” Surrey cries. “For we may perish elsewhere,” Benjamin says with sudden insight. Did he hear me? “But stay here, and death will be certain, mortals!” the thing cries, loomin’ over them. “Stay anywhere, and death is certain, for mortals!” Benjamin cries. “And you don’t care if you die,” I murmurs into Surrey’s ear. “And we don’t care if we die,” Surrey says. “What? Ci—” “Surely death comes to all mortals,” Benjamin says. “Why should we care?” “I could make death hurt,” the thing cries, stretchin’ its arms out like a giant Muppet. “Or we could die in our sleep,” I murmurs. “But I can make death hurt him more.”I actually have practically finished BOT NET, so next up is Cinnamon Frost #3, ROOT USER! Oh, and editing Dakota Frost #4, SPECTRAL IRON! Due in about 4-5 months. Aaaaaaa! Onward! -the Centaur
If I keep up the pace that I've been keeping ...
I won't just beat my best record ever (which I already have) ...
... I'll hit the somewhat ludicrous amount of 75,000 words in a month, beyond the 70,000 I've already hit.
4,648 words to hit that goal ... less than I did yesterday or even today. Let's get cracking.
-the Centaur
If I write 11,293 words by the end of the month ...
~2900 words a day, not counting today ...
I will beat my all time Nanowrimo record of 65,995 words:
Sounds like a worthier goal than spending the same words responding to everyone who's wrong on the Internet.
Onward!
-the Centaur
Hoisted from Facebook:
Q. How should I start looking into getting published? A. The most important thing is writing. The next most important thing is finishing. The next next most important thing is researching markets and sending things out. If you are doing all that, there are two other great force multipliers: not waiting to start your next piece while the previous ones are out, and networking - going where other authors are: cons, writing conferences, writing workshops. If you are doing all that, get yourself an agent - it is the next big multiplier.And that's it, in 95 words! Paraphrasing Robert Heinlein, to get published, you've got to write, you've got to finish what you write, you've got to send out what you write until it gets sold. But if you really want to get published, you can't wait on that first piece to succeed; you need to go ahead and start the next one. And you can't rely on your own ability to find opportunities and markets; you've got to find other writers and editors to help you find the right home for your work. And if you're doing all that, you're on the path to having interesting enough work to attract an agent, so you might as well start looking. -the Centaur
Hey gang, now that I've succeeded at National Novel Writing Month nineteen times, I thought I'd take a little time out to tell you that my secret to National Novel Writing Month success is to put Nano first.
Now, that seems obvious - almost, like, too obvious to be advice - but I want to put it into perspective by first asking you a few questions.
- Do you care about finishing what you write? If you don't, don't worry about the above advice. Write what you want, how you want it, when you want it. Again, this seems obvious, but I want to make sure you understand whether this advice applies to you. I started to write "Do you care about writing success?" but then I realized everyone has their own definition of writing success, whether they realize it or not. A beloved friend, Jan, just died, and I will never again hear her read her stories at our local writing group - and you're never likely to read them, as she was more into having fun writing than she ever was about getting her stories published. She was a success at writing without ever needing publication, or even necessarily to to finish everything she wrote. We'll miss you, Jan, but we'll never worry about whether you were having fun: that was obvious.
- Do you care about completing Nanowrimo? Again, I almost wrote "Do you care about succeeding at Nano" as I did in the title of this essay, but every word is a victory in National Novel Writing Month. It doesn't really matter whether you wrote one word or a hundred thousand if Nano helped you write it. But, again, if you don't care whether you hit that magic 50K, then don't worry about the advice in this article.
- Do you care more about finishing Nanowrimo than doing the comfortable things that you've always done? This is the most important part. Some people need to wait for their muse. Some need to plan ahead. Some need to do other things that seem so important to them. Maybe they are. But given my writing style, those things don't seem so important to me, and given my obsessive-compulsive personality, I'm not going to take on a challenge unless I intend to finish it. But maybe that's not you: you need your outline or your muse or your whatever in your life to make writing possible. That's OK! I know great writers like that. They don't generally take on Nanowrimo, that's all, as Nanowrimo doesn't work for them.
- Turn off your Internet. Frequently when writing, I hit a speedbump, go look something up, and dig into Wikipedia or TV Tropes or (no joke) find myself reading the city planning documents of a harbor community in Newfoundland to carefully craft some details that will only appear in a paragraph or two of a whole novel. That's my process, and it's produced a number of well-received novels of which I'm proud. During National Novel Writing Month, however, I frequently turn off the Internet on my laptop until I've gotten my magic 1,666 words per day.
- Put off your Blogging. I like to blog, but it seems I don't do a lot of it. The reason for this is that I've started doing Nano-like challenges three times per year - Camp Nano in April and July, plus Nano in November - and the rest of the time I focus on finishing manuscripts, editing them, or publishing other people's manuscripts. I have dozens upon dozens of notes for blogposts written on scraps, stored in Scrivener files, even half-finished in Ecto; but during Nano, I put Nano first.
- Research after you Write. So many times this month, a pile like the below was on my table, awaiting my perusal, only to be put back into the bookbag or tote bag after I finish my writing. There's a problem in deep learning I'm trying to crack, and some mathematics I need to know to do it, and research for the Cinnamon Frost puzzle books; but I know from experience I can take my whole lunch break or even afternoon diving into mathematics or programming or research. During Nano, I put the writing first. That forces me to go out into space in the story, even if sometimes I need reference when I edit the text.
- Take Time off to Write. I am always an antisocial loner, having to force myself to go to lunch with my coworkers (rarely) or to dinner with my friends (equally rarely) or my writing peeps (less rarely, since I can excuse it to my brain as writing related). This isn't because I don't like the people that I like, but because I want to finish my books before I die. (I have a lot of books planned). During Nano, I'm even more jealous of my schedule, taking lunch breaks to write, sometimes bailing on group writing sessions to be alone, and, the big one, taking off Thanksgiving week to write. This year, for Night of Writing Dangerously, I got a hotel room, holed up in San Francisco after the Night, and wrote like mad. I almost finished BOT NET during this writing jag, and ended up finishing six days early because I took this time.
- Reject your Comforting Illusions. This last one is the most dangerous advice which may not work for you or even apply to you, and it can be the hardest, but it is this: put getting the 1,666 words a day ahead of your imagination that you need to wait for your muse, or plot your story, or be in the mood, or anything else. Write crap if you have to. It counts (1 word). Write bla bla bla if you have to. It counts (3 words). 1,662 words to go. Feel that this isn't working for you? Write "Nanowrimo isn't working for me!" (5 words) or, even better, "National Novel Writing Month is not working for me!" (9 words). Feel this is a cheat and a sham? Write that down! (I guarantee explaining your feelings will get you more than 9 words). Even better, write down what's wrong with your story and why you can't write any further and what you wish you were writing. That explanation in text will count as words ... and, more importantly, will probably start turning into text. Put another way, feel free to work out your frustrations and even to outline in your manuscript. It will become true words on the next pass ... and will put you ever closer to the end of your story. Once you have a path from beginning to end, believe me, you can revise it into a story that you will truly love.
So, I just succeeded the 19th time at National Novel Writing Month!
This year, I was working on BOT NET, the second Cinnamon Frost novel. I'm writing these three books in one huge manuscript, which I successfully took from 179591 to 229911 words as of today!
This year, the combination of participating in the Night of Writing Dangerously, plus having the luxury of taking off the week of Thanksgiving to write, really pushed me over the edge:
Interesting, the hole at Thanksgiving. I wonder if that's true every year? That's not something you can readily see when you look at the yearly charts since it moves (stay tuned, these charts are going to come back later):
There was a time when almost every post about Nanowrimo I'd include an excerpt. Frankly, that's gotten harder to do as I've switched from doing Nano once per year to three times per year; the Nano material has become more inchoate as I blaze new paths out into story space, requiring more work to turn it into final material. But, occasionally, I can indeed include some material that gives you a flavor ...
“I … I gotta be honest here. I needs help.” “Cinnamon,” Nri says gently. “I know that. I’ve had many, many students before.” “Another damn teacher,” I rollin’ my eyes. Then I realizes—“Did I say that out loud?” “Yes, you did,” Nri says, smiling sardonically. “I don’t even think that was Tourette’s.” “It-it wasn’t,” I says. “I’m sorry, sir, but …” I grimaces. I genuinely don’t know what tone to set here. Act like Mom’s world, use Southern politeness, act like the werekindred, use growls and barks … or, maybe, just be me? Who’s that then? “I, uh, don’t, ah, know how to say this but I wasn’t tryin’ to insult you before or to butter you up now but we gots a real situation and if we leaves it up to my Mom there’s a very good chance that the D of the W. A. will spirit my boyfriend and my alt-crush off to the wilds of nowhevers, and if the elders of the werehold finds out where they are they may go and do somethin’ stupid right on the doorsteps of people totally prepared to do somethin’ stupid, so I’m guessin’ the smart thing is for the people who are smart and wizardly to do somethin’ smart and wizardly, but I can’t do this alone, because I am, like, thirteen, and why in godsname does everybody think I can do everythin?” Nri stares, blinks, shakes his head, like he’s comin’ out of a trance. “God, I’d wish I’d timed that,” he says. “I think you talk faster than JFK—” “Who?” I asks. “Nevermind," Nri says. "I’m sold.”Ah, Cinnamon, you and your wacky hijinks. Thanks for coming into my writing life, wherever the hell you came from. And now, on to all the things I've been putting off blogging while I've been working on Nano, including ... how to succeed at Nano! (I hope you'll agree I have some credentials in that area). Onward, fellow adventurers! -the Centaur
Milestones are coming. And the first of these is catching up on my wordcount for my Nanowrimo project this November, BOT NET!
Winning at Nano always feels like climbing a hill, but for me in particular it almost always feels like I start out sliding back down, Sisyphus-like, as I struggle to get a handle on the story.
But then there comes that magic point where I need to write 1,666 words in a day and I. Got. Nothing. Then I'm forced to be creative, and the real fun stuff happens, an event I call "going off the rails". Hey, let's try to embed a tweet!
So now things are back on track for the month, and I'm smack in the middle of where I normally am this time of Nano ... Actually, it appears I'm ahead. Checking the stats ... yep. At this point, I'm normally just shy of 6,000 words behind ( -5,984, though that estimate is numerically precise, it is not likely to be meaningfully accurate ) but today I am 169 words ahead of the Nano wordcount: I'm one more thing too: 200,000 words into the Cinnamon Frost trilogy. There are 3 published Dakota Frost novels: FROST MOON, BLOOD ROCK and LIQUID FIRE, and three more finished rough drafts: SPECTRAL IRON, PHANTOM SILVER, and SPIRITUAL GOLD. By my count, I've written about 900,000 words about Dakota Frost, Skindancer, the woman who can bring her tattoos to life. But in one sense, that's expected: I planned Dakota. I wanted to write a character that other people who can relate to. Cinnamon Frost, as I've said before, is a character I never expected. She shoved her way into the Dakota Frost universe, in one of those "step off into space moments", and she shows no signs of leaving. Cinnamon might say 200,000 seems significant because of how humans process patterns - how we love all those zeroes - but it's just a number: 2*10*10*10*10*10. But somehow, it feels right to take it this far, and I look forward to writing the next 100,000 to 150,000 words that will finish her trilogy and give her a chance to live her own literary life. Time to get back to it. -the Centaur P. S. I said milestones are coming. If you've read closely in this post, you'll realize another milestone is coming soon. Stay tuned ...It's all too easy for stories to fall in predictable ruts - but in #NaNoWriMo , when you've got to produce 50,000 words in a month, sometimes your story "goes off the rails" into territory your conscious mind never expected ... and that's gold! https://t.co/Qmtuzb1XKE
— Anthony Francis (@xenotaur) November 13, 2017