April 01, 2011
The Falling Frame
Jill answered the knock at the door to find Al behind it. “Oh, hey Al, come on in. Tim’s in the living room.”
Al went back to where his friend Tim was squatting on the floor, inserting a bit into a variable-speed drill he held in his hands, fiddling with the chuck. “Oh, hey Al. What brings you by?”
“Oh, I just wanted to borrow your... hey, what are you doing?”
“Well Al, I’m going to mount this frame on the wall. It keeps falling down but Jill really likes it here.”
“By driving screws through the frame? Seems a little extreme.”
“Well, I have this putty, I think I can make it match...”
“Care if I take a look?” Al inspected the nail currently in the wall and saw the multiple holes there, along with one that looked like the nail had been pulled down on. “What’s the story here? Looks like you’ve tried a few things.”
“Well, I used to have it hanging just on a brad, but every few weeks I’d come through here on my way for morning coffee and see the picture on the floor. So I switched to a bigger nail and it’s come down again. I figure I’ll fix it for good.”
“Wait, hold on. You say you always find it this way in the morning.”
“As far as I remember.”
“And you found it on the floor *this* morning?”
“Why didn’t you see it last night?”
“Well, I went up to bed through here, sure, but it was dark. I had watched an old war movie on the tube and then went up to Jill.” He fired up the drill to check the bit. “Ug ug ugh.”
“Just a second, there, maniacal dentist. Isn’t this wall shared with the family room?”
“Hold on just a second.” Al walked around through the threshold to the room beyond. “Tim, I think you ought to come in here for a second.”
Tim followed his friend through. “This is the wall in question, right Tim?”
“And this is where you were watching that old movie last night?”
“Probably one with lots of explosions and such?”
“Yep. The Dirty Dozen, in fact.”
“And would you say that the subwoofer was thumping along pretty good amongst all the noise?”
“Yeah, I guess so.”
“This subwoofer here. The one touching the wall. The wall on which a somewhat heavy frame keeps falling off?” Al dragged the subwoofer an inch away from the wall. “Why don’t you pop another nail in that wall and rehang the frame on it, see what happens.”
“Well, I’ll be darned...”
Today’s little parable comes to remind us of one fundamental fact about programming, or more specifically, about debugging: you haven’t fixed a bug until you understand its cause.
Over the course of my career, I’ve run into dozens of types of bugs, and have probably committed quite a fair number of them. There are off-by-one errors, underflow errors, uninitialized data errors, constructor argument dependency errors (rare, but they happen), plain old logic mistakes, simple typos, and any number of issues which compilers have grown better at catching over the years, such as unintentional assignments inside of conditionals.
New hardware has brought all manner of different, fun new bugs to learn to diagnose and discover, and new opportunities to foul up, like DMA chaining issues or the host of hard-to-find Heisenbugs that occur when you’re working with multiple processors or threads sharing resources, such as unusual race conditions.
Now, I’ve seen lots of techniques to deal with bugs, some quite... eccentric. I think my favorite was a guy who, encountering unusual data inside of his local variables, would literally rearrange the order in which functions appeared in his source file until the bug disappeared -- certainly the least likely to be effective method I’ve ever heard of for what were most likely uninitialized variables. Almost all of these methods proceeded from the same flaw; that the disappearance of the symptom indicated a solution to the problem.
In the end, I’ve come to believe that there’s literally only one thing that is guaranteed to be the same in any successful solution to a bug: you have to understand the cause of the bug first. Even as hardware changes and our approaches to developing also change, this is one I think that will remain constant throughout my engineering career: to solve a bug, you need to comprehend its root cause. In this case, as so often in engineering, it is better to proceed from knowledge, than from hope.
To do otherwise... well, to do otherwise you risk being a fool any day of the year.
March 05, 2011
Jacques tied the apron around his waist as he finished surveying the ingredients in front of him.
“You know, Dad, I’ve always wondered why you keep so many knives.” His daughter Claudine had come up behind him while he was collecting materials for dinner. “Can’t you just have one or two? I mean, you have several knives even at the same length!”
“Oh, hi, Claudine. Pour yourself a glass of wine and a little more for me, and I’ll answer that as best I can.”
As Claudine poured Bordeaux into two glasses, Jacques picked out several short-bladed knives and moved some raw foodstuffs to a nearby cutting board.
“Now, see here, Claudine. I’ve taken many of my three-inch blades and laid them out here before you. Can you see any obvious differences?”
“Sure. That one there has a serrated edge, that one has a weird curve to the blade, and that one looks a little like your big chef’s knife.”
“Very good, Claudine. Each is different.”
“But why do you need all of those?”
“Well, each of these is better at a different sort of job. That serrated one you mentioned is particularly good for cutting tomatoes, or perhaps sausage. Its wavy edge allows it to slice cleanly into something that is a little bit soft in the center but has an outer skin. You see?” He sliced cleanly into a tomato, slicing it very thinly indeed.
“What about this curved one?”
“That’s a great tool for peeling fruits and vegetables -- the curve of the blade fits to the curve of the fruit.”
“And this one with the wedge point?”
“For removing the eyes of potatoes.” He demonstrated deftly, with a flick of his wrist.
“But couldn’t you do all of these tasks with any old knife? I’ve seen you make full meals with just your chef’s knife, especially when we’ve been vacationing.”
“Oh, certainly I could, Claudine. Most knives are good at doing many things, but these other knives are particularly good at doing one thing. And that lets me move a little more quickly in the kitchen, or make something just a little bit better. For example, when I use the serrated knife to cut tomatoes, it’s much easier to do so without bruising the fruit where the knife goes in.”
“Oh! Didn’t it take you time to learn to use all of these?”
“Well, of course, Claudine. It’s taken you a long time to learn the proper glass to use for each type of wine you drink, and why, n’est-ce pas? With knives, it is much the same thing -- I have some basic skills I use again and again, but I have to tell you, getting familiar with a new blade is such a pleasure.”
“A pleasure? I know you love to cook, but why such a pleasure from this specifically?”
“Well, each knife I’ve come to know has changed how I look at the ingredients I use for my cooking. I might realize that there are ways to look at preparing foods I’ve prepared before just a little bit better, or come up with a new way of thinking about a dish. Perhaps I can slice something thinner, and get a subtler flavor.”
“Does it make you a better cook?”
“Having more equipment doesn’t make me better in the kitchen, I don’t think. But it does let me use my skills in more ways, which might make me capable of doing more things, or doing some of them a little better. Being a good chef is in the soul and the hands and the senses. Now shoo, if we’re to eat tonight I’d better focus.”
“May I help?”
“Absolument. Take this serrated bread knife and cut that baguette on a diagonal...”
Today I want to talk a little bit about the very fundamental tools of programming: the languages themselves and some ways in which I’ve come to view these languages over the past few years.
When I began programming professionally, I spent a great deal of effort becoming proficient with C++¹, reading as much as I could and attempting to apply new techniques to problems as I encountered them. I learned from experience and the wisdom of those around me, and I would say that I had learned many of the fundamentals of programming. In the little parable above, this is like learning to use a single knife² proficiently -- one learns many techniques that will allow you to prepare ingredients in the service of your meals.
I spent a lot of years coding exclusively in C++. I read numberless books on the subject, designed systems in it, coded tools in it. It became the lens through which I viewed many problems in game development³. I refined my thinking about when certain types of approaches were appropriate, discarded that thinking in the face of new evidence, and rethought it all again and again.
But in the end, I started to become like a chef with only a single knife. It wasn’t the best tool for many jobs I would approach, but I didn’t have anything more specialized for the different tasks I would perform.
The thing about learning new tools is that not only does it give you the ability to solve new problems in new ways, but it also changes what sort of problems you find yourself able to think about. Just as getting knives that have certain features, additional languages force you to rethink how you solve different problems, and being able to solve different problems grows your thinking altogether.
I can’t recommend enough trying some new languages, for any professional programmer. A few things have been helpful for me:
- Learning by doing. Rather than attempt to read a bunch and apply what I know later, I try and sit down with a text editor and go -- referring constantly to the web. When learning Python, I put a book on my desk, pointed my browser at python.org, and fired up an interpreter.
I think that’s about all I have on this subject for now. I hope you’ll go out and get yourself some spectacular new knives... and keep them sharp.
¹This is, I note, a process that continues to this day. I hope to be learning new things in whatever languages I use professionally to the day I die.
²Typically a chef’s knife, if one cooks in the Western tradition, or perhaps a butcher’s knife in the Eastern tradition. The Eastern version of a butcher’s knife is a lot more flexible and of greater utility than the one you see hacking away at cuts at your local butcher’s shop.
³Development process was the other lens I would apply, but that’s a whole different can of worms.
4Also a little bit of SQL, and yes, I’ve played around with C# from time to time. The former doesn’t really feel like a programming language like the others, its domain being so specific, and I’m not much of a fan of C#, though I think it also has its uses. In the case of C#, it’s enough like C++ to me to not feel like a new language at all. I’m also not fond of it being tied to a single platform.
February 15, 2011
The sun beat down on poor Brett¹ as he surveyed the pieces around him. When he had taken the job to erect a pre-fab shed in a neighbor’s backyard, he had thought it wouldn’t take him more than an hour. Having just failed in his fifth attempt, he was about ready to pack it in.
“Hey, how’s it going?” Gary² called from his car window as he drove up.
Brett walked down to the sidewalk. “Not so good. I’ve tried at least three different ways of putting this shed up, and I just can’t figure it. I need like six extra pairs of hands just to hold the walls in place so I can add these braces to hold the whole thing together. ‘Easy assembly,’ says the instruction sheets, but I’d have to be The Flash to be able to do some of these steps before the weight pulled it down.”
“Hmmm, let me take a look.” Gary got out of the car and then quickly paged through the instructions. “Yeah, this looks a bit tricky. Care for a suggestion?”
“Are you kidding me? If I don’t figure this out soon, I’ll still be here come midnight. Heck, I’ll still be here come October.”
“Walk with me for a minute.” Brett turned to follow as Gary walked down to his house, just a few doors away.
“I think what you need here is an extra pair of hands, and I’d be glad to help you if I didn’t have some other errands this afternoon. But I think I can give you the next best thing,” Gary said, as he turned up his driveway towards a small detached garage.
Beside the garage was a stack of warped 1” and 2” by 4”s. “Now, you should take a few of those over and see if you can’t build yourself some supports; just enough to hold each wall steady while you do what you need to.”
Brett examined the stack of wood, leftovers from some repairs that had been done in the garage a few months previously. It was all junk wood, warped beyond usefulness and destined for kindling when winter came. But certainly it’d be strong enough to hold a wall or two in place, if he framed it right.
“Can I borrow a hammer and some nails, too?”
“Sure, you know where they are, just put back whatever you don’t use when you’re done.”
“See you around dinnertime?”
“I think *now* you will.”
One of the hallmarks of coding experience is knowing when to build something completely throwaway, in the service of the work you’re doing. Indeed, I’ve heard more junior engineers remark of how a senior programmer really knew how to approach problems by building the right tools to understand the problems they were facing. Due to a work experience from long before I was a professional coder, I like to think of the code we write to figure out the problems we face as scaffolding.
Scaffolding is the work you do to help you deliver the work you’ve agreed upon. I’ve employed, and seen employed, many forms of scaffolding over the years, but each and every time I see a bit it follows a few general rules:
- It’s cheap. Scaffolding shouldn’t take you long to do³ -- it’s simple work that doesn’t respect the needs of shipping code, from memory consumption, performance, or any other work. You build it quickly and put it to use immediately. Often, it doesn’t even survive long enough to get checked in.
- It’s dirty. Really, this goes hand-in-hand with it being cheap, but if you find yourself thinking too much about your scaffolding, you’re probably working too hard on the scaffolding itself. If you’re trying to put up a shed and need a temporary wall to hold together a couple of pre-fab pieces, you don’t expect those pieces to look as good as the finished product you’re making. In fact, no one will likely ever see it, it may not even survive the project. So, feel free to write against all the rules you’ve been taught and whatever coding methodology to which you subscribe -- go in and remove encapsulation from objects so you can access members directly, use terribly abusive algorithms, write reams of data to disk, access memory randomly, hold raw pointers that just happen to be safe... just don’t overthink it.
- It’s useful. There is one measure, and one measure only when it comes to good scaffolding: utility. Scaffolding code is built only to be useful to solve other problems, not to present problems which need to be solved in itself. Once you’ve written the bits you need and answered the questions you needed answered, it’s time to move on.
- It doesn't ship. Take it out before you check-in (whether through pre-processor macros or brute force). Just be sure not to leave any traces which might slow down or bloat your end-product or leave extra rusty nails lying around where someone might step on them -- in particular, replacing any guards that were in place, such as objects' access protections and the like. You want to leave a clean workspace behind.
When approaching a problem that’s a little bit beyond your ability, or just unfamiliar enough to require serious thought, consider building a little scaffolding to help you through it. If you find yourself wondering what approach to take, ask yourself questions like "What information would I need to approach this problem in the right way?" and "What sort of code would help me address this problem more easily?" and build yourself a little scaffolding to give yourself good answers to those questions. It might seem like it takes you off-task for a while, but it’s often helpful enough to generate a much better solution in the long run.
This post was cross-posted with AltDevBlogADay, where I post every couple of weeks. Be sure to check it out for lots of great posts from my fellow authors, on lots of different aspects of game development.
¹Names changed to protect the innocent.
²Oh, who am I kidding. Although I'm a second generation computer programmer, the best lessons my dad taught me about coding had nothing to do with programming at all.
³Though it might give you ideas for future work, given time to properly build for your engine. Today's scaffolding might point you in the right direction for tools that are currently missing from your codebase.
January 31, 2011
Hank answered the door to find his friend Ralph waiting outside. “Oh, hi, Ralph, come on in, the wind is blowing cold off the pond out there! I’m just working on a little project, come on back.”
The two shook hands and went back to Hank’s workroom, where bundles of sticks lay strewn about and on a worktable, a small house fashioned of sticks was beginning to take shape.
“What’s this, Hank? You thinking of starting a fire in here?”
“What? Oh, the sticks. No, that’s all for this little project I’m working on in here. What you see before you is the beginnings of a birdhouse I’m putting together to hang outside come springtime.”
“From sticks? I wouldn’t have thought that even possible. How are you holding it all together?”
“Well, I found that if I carefully heated some green pine, I could get some nice warm sap to use as a bit of glue at the joints here in the corners. It took some experimenting to get just right, but that’s one of the things I came up with.”
“But that must take forever! Why not just go buy a couple of pieces and some lumber, bang ‘em together with some nails, you’d be done in an afternoon.”
“Well, Ralph, as you know I’ve been trying to keep myself from taking odd jobs -- I find when I work for others, my conscience makes me use a lot of my mental power on the work, and I find I have little energy left for my writing when I come to take pen to paper. Lumber is cheap, but it’s not free, and neither are nails. These sticks dropped down in that Nor’easter we had a couple of months back.”
“Sure, I recall.”
“Anyways, that birch by the front porch came down in that storm and I found myself thinking the birds who nested there would need a place to nest come spring, and I do so like hearing them in the mornings when I write before breakfast. They’ve done me a lot of good, and I decided I’d repay the favor to have them back again. I plan to hang this house under the eaves.”
“I can understand that.” Ralph sighed. “It just seems as if you’re putting in a lot of work, and having to figure out a whole lot. It just seems unusual to me.”
“Well, sure. It’s not what most would do. But this works for me: not only do I not have to take a job to pay for the lumber, but it gives me something that I can write about. This project is a birdhouse now, but in a week from now it might be an essay I can sell. Now, what do you say you help me get this roof on, and then we can go smoke our pipes on the porch? Hand me that little pot of pitch off the wood stove...”
One of the things I’m aiming for with this series of posts I’m doing with the AltDevBlogADay group is to find a constraint that helps me improve my writing, and I’ve chosen to do so by restricting myself to telling a parable and then spending a little time explaining what I mean by that parable or anecdote. I don’t often have time for creative writing, although I enjoy it, and so I’ve put myself under a constraint that forces me to write at least a little bit every couple of weeks in hopes that it’ll inspire creativity. In this parable, I recall an author who most elegantly introduced me to the topic under discussion.¹
One would tend to think that having as much time and money and resources of all kinds will lead to the very best games and products. In fact, often engineers will say or think to themselves that if they just had more time or more people, they could really make this library or module sing.² It could do x, y, and z. Indeed, many programmers will start out with a lot of architecture to solve a large domain of related problems, and end up leaving many loose ends when it turns out they don’t have the time to fill out all the functionality, which I’ve seen both in novice code and commercially licensed engines.
But, as it turns out, this is exactly the opposite of what really works, in engineering, in design, in craftsmanship of any kind. What really works is to identify constraints and work within them. Constraints inspire creativity.
In the parable above, what Hank really wanted to do was to have time and material for writing, not money for goods and services. He had identified a constraint he wished to place on his own efforts -- to work as little as possible for others, because that work cost him things that were more precious to him. Indeed, reminding himself constantly of that constraint made him able to creatively solve problems -- in choosing to build a birdhouse by hand, he found he needed to manufacture glue, which lent itself to a creative solution. There were a few elements at work here:
- He didn’t shirk the problem. Sure, Hank could have said good-bye to the birds once and for all, but he decided that this was a problem he wanted to solve, even if it meant difficult work. It’s easy to avoid problems altogether, rather than trying to solve them within your constraints.
- He kept his constraints always in mind. He didn’t run to the hardware store at the first sign of trouble, which left him problems that he needed to find solutions to within his constraints.
- He chose his constraints. What he wanted was time and space to write and think, not money he would likely not even have the time to fully spend. It was a conscious decision not forced upon him, in this case, though external constraints themselves can have value.
Time and again I’ve seen the value of constraints in game development, or heard wonderful anecdotes about how some problem was creatively solved despite the lack of engineering time to fix it “properly”. Here are a couple of memorable ones.
- Dark Forces had a simple scripting language but no variables -- it was a simple command language, where certain types of world state could be set, but it didn’t have variable support. One of the things that could be controlled from script was moving elevators. So, unable to get direct language support for variables, level designers instead would place elevators in the world, unreachable by the player and never rendered, and move them up and down to represent variables.³
- Star Wars: Jedi Starfighter had inherited a very simple scripting environment from its predecessor, and although we extended it in some simple ways, it wasn’t a language that had a lot of state. When level designer John Drake approached me and said he wanted to implement a tug-of-war between the player’s forces and the AI, moving resources back and forth through a space in the level, I thought it couldn’t be done without code support (which we didn’t have to spare). But John pushed that language to its very limits to produce one of the most interesting and tense battles of the game.4
Those are a couple of areas where craft had to respond to constraint with creativity. There are ample examples in narrative and design as well. Consider:
- The storyline behind both Ico and Shadow of the Colossus, two of my favorite games, boils down to “young man rescues the princess”. Each use a pared down design as well, constrained to few player actions in compelling environments.
- Take your favorite indie game. Indies are severely constrained in resources, limiting all aspects of their process, from development to marketing. Yet each successful indie finds creative ways through the morass, from new business models to clever marketing to innovative gameplay to simply crazy fiction.
Constraints don’t just apply to your development or design environment, such as coding languages or scripting, or how you can spend your polygon budget, but also to development itself. As a parent with a couple of grammar-school-aged sons, I have quite a number of constraints on my time that I’ve chosen. When I started at my current employer, I let them know about those constraints -- that I have kids, that there are nights of the week where I am their sole guardian, that I’m very involved in their sports and activities, and that I simply wasn’t available to be at work during those times, even if the team was crunching. This was definitely a choice as some of these constraints are more flexible than others; while I must be around when they are in my care, I’m not obligated to coach their teams.
I try to keep this constraint in mind throughout the life of the project; early in the project, when it’s easy to lose focus and meander through your tasks, both those that you choose and those that are chosen for you. But keeping in mind at all times that the time you are spending is all the time you’ll have helps focus you on remaining efficient, removing as much waste from your workday as possible.5 Early on, I invested in projects that would pay off not only immediately, but in keeping me able to maintain that constraint when the ship date was looming. It’s far better to choose that constraint early than to have a different set of constraints forced on you at the end.
So, I turn it on you, fellow developers. Pull up a chair, grab a pipe, sit a spell, and tell me: how would you comport yourself if you had that constraint? How would you act today if you knew you could only work the forty hours a week we theoretically sign up for? And what other constraints would help inspire your creativity?
¹Bonus points if you can identify the characters here and where they are.
²I’ve been very guilty of this myself. It’s an easy trap to fall into. I call it "worshiping at the altar of Yagni".
³We had a similar limitation in the first Star Wars: Starfighter game, where although there was no variable support, an LD used the fact that he could react to changing properties on an object to set the radius of a waypoint up and down and fire off scripts based on those, depending on where the player was in the level. It was crazy, but it worked.
4The level as originally designed and balanced was made for and by expert Starfighter players and it really gave me a run for my money when I played it during our beta period. We ultimately made it easier, but beating that level on its original difficulty was some of the most fun I had building that game and testing it during development.
5I often jest with a colleague, “Hey, we gotta get going, we only have three years to build this thing!”