July 22, 2014

How I deal with data, part one

A Maze of Stupid Little Languages, All Alike

Advance notice: Nothing here is terribly new or revolutionary. Someone out there may find it useful. This is the sort of stuff that often comes up in game development but that people rarely write about, at least, in my experience.

I'm in the process of designing and building a game with attention-management, simulation, and Sim elements. In it, the player will directly and indirectly affect the happiness of Little Computer People through decisions made in the attention management part of the game, backed by a richer simulation of an audience for that performance. That's all kind of hopelessly abstract, I realize, but it's kind of the level at which I'm comfortable discussing it in a public Internet forum at the moment. (I do discuss more specifics in person with people. So ask me when you see me.)

Lately¹ I've been taking my ugly, rough, hard-coded test data prototype to a series of data files that can help me more fully flesh out the game play and create content that is more reflective of how I expect the final game to actually work and feel, both in terms of quantity and quality. My intent is to have a "campaign" of fully authored data with a procedural approach for people who want to continue playing beyond that, and these articles address the hand-authored content specifically, though I expect the description of procedural stuff to follow along similar lines. After I feel like I have the data thing nailed down enough to build, balance, and play a few (again, super-ugly) levels that have the right feedback loops to the player, I'll be looking into the actual visuals that will support the final product (characters, environment, etc).

Anyway, because I'm a programmer, I tend to go straight to text for data. It's easy to manipulate (by hand if need be, which is where I'm starting from right now) and easy to read. In so doing, I tend to come up with very simple little context-specific languages to deal with my data. But for performance and sanity reasons, I work with binary when I'm actually loading this stuff into my game², mostly because I hate parsing stuff in C++. Granted, I could use a well-known format like JSON or what-have-you, but I haven't at that point really solved anything, I've just offloaded parsing the text into a library (with its own run-time performance concerns, usually having to do with memory allocation), and then I need to walk the built-up data structure to look for stuff, and handle failure conditions. Bleh. Instead, I prefer to build a custom format that can be directly mapped simply to data structures in memory, allowing for a single allocation to read in the file once, a little walking over that data to map structure pointers (typically by converting offsets to pointers). The data can be then deleted in a single deallocation later and the structure cleared out.

It's also worth noting that I could, at a later date, write tools that sit on top of that simple format, and I may well do so to support post-ship modding of the game or the Steam workshop (should I be on Steam), or whatever. For some of the data I expect to author, it'll be useful to be able to visualize the data in terms of frequency of certain occurrences, but I could certainly do that by writing additional simple tools that spit out different data (images, comma-delimited text files, whatever) from

I would almost certainly choose differently if I were working on data that I expected many people to have their hands in at any given time. At that point, I'd probably want a database of some kind and things like that. Solo development makes this sort of thing much easier. In this case, I'm designing development tools for myself -- a person who is super-comfortable with text files and actually can visualize stuff from text files pretty well in his head. Whenever you're developing, you should think first of the client for your tools. Also: if I worked against a database, later steps of this process would be largely the same, I just wouldn't be parsing the data in the same way.

Here's what my languages tend to look like, with some real text that I've moderately fudged to obscure actual subject matter (in other words, not a game about outlaws). Just to give the flavor. There are no lawyers, guns, or butter in my game. I may add butter. ;)

## Josie Wails
:littlecomputerperson JWails "Josie Wails"
:lcpshort JWails "Former outlaw, now reformed"
:lcplong JWails "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et"
:lcplong JWails " dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
:lcparrives JWails 3.3
:lcpskill JWails lawyers 16
:lcpskill JWails guns 32
:lcpskill JWails money 16
:lcpskill JWails butter -8
:lcpappears JWails initial
:lcpinitialHappy JWails 32
:lcpHappy JWails and gte ammunition .3 lte butter 0 3
:lcpHappy JWails gte ammunition .6 3
:lcparrives default 0.0

Stuff to note:


  • Simple command structure. Basically, every line is a command, and they all follow a pretty simple structure that adds a specific bit of data to a particular main container item. I declare the LCPs with friendly names (which will be displayed in-game) but then use identifiers from that point on to prevent the need for global search and replace later.

    This has some great benefits, in particular that I can separate out data however I like -- I might want to keep all the skills descriptions in one place so that I can see at a glance that I don't have anyone who has a positive "butter" skill or I might put all the descriptions in one place at the bottom since I tend to write them once and then not think about them again. If I were working with a format like JSON, my natural inclination would be to keep all the data for a particular LCP together in one place. (It would probably also be significantly more verbose.)

  • Minimal data per line. As I'm not going to ever ship this stuff, I just go ahead and put only one bit of data per line. It's all "this command tells you this about this data object".
  • No multi-line stuff. As you can see in the :lcplong command, it would sometimes be easier to deal with this stuff on multiple lines, but that just makes parsing complicated. Instead, whenever I see a long description, I look for the LCP id in the appropriate Python dictionary, and if it's there, I append.
  • Prepended : for commands. On occasion I have need to rename a command because I've changed what it means and already have data. It's helpful for find-and-replace to have the : there and it doesn't add appreciably to typing. Originally I had named the command "lcplong" as "long" but ended up using the same command in multiple data files. In case I want to merge them all together later, it's helpful to have distinct commands and searching and replacing ":long" with ":lcplong" meant I didn't have to worry about stepping on the word "long" in descriptive text.
  • Not shown: documentation in the file. At the top of each file I explain the syntax to myself for each command, because I expect I'll forget things. Often, it's enough to see the commands already in use, but it's helpful to have something to refer back to in case it's not clear. In a pinch, I can always examine the Python parser, but I'd prefer not to have to.
  • Also not shown: extensibility. Adding an "include" directive is super easy and doesn't complicate the resulting Python all that much, it just means encapsulating the parser loop in a function that can be called in a re-entrant fashion. Because commands are all basically additive, the "state" will be just held in Python data structures and anything that needs to be shared and included in multiple files can be easily handled.
  • Handling default behavior. Ultimately, I'm going to want to have reasonable defaults for things to fall back on, so there's a dedicated keyword that can't be used as the id of any specific LCP, which is "default". All of my parsing commands will notice this, as you'll see tomorrow.
  • Comments. Comments are, like Python, begun with the octothorpe³ and are single-line only.

Tomorrow I'll come back to this and show how simple languages like these are easily handled in Python and output to a binary.




¹Since shipping Sixty Second Shooter about a month ago, which had occasioned a six month break from my game or so.

²Note that I say game and not game engine. My game is currently hand-coded C++ on top of some cross-platform libraries.

³How great is that word.

Posted by Brett Douville at 02:30 PM | Comments (0)

May 06, 2014

Working at 3 miles per hour

A few years ago I started reading reports about the damaging effects of sitting down to work eight hours a day. As a computer programmer working in the games industry, I was obviously at risk, particularly when you consider that I might enjoy sitting down at the end of the day for a few more hours to read, watch a movie, or play video games.

So, I started standing up at work. I happened upon a crate that was just the right height to lift my monitor and keyboard to a comfortable height and sort of jury-rigged something together. I had worked with folks who stood all day in the past at LucasArts, and I had read about Noel Llopis's experiment with it, but as I recall I was maybe the first person to do it at Bethesda Game Studios, and I got a lot of quizzical looks and questions.¹ Standing and coding was no big deal; after a week or so it was completely natural for me to code standing up. A few folks followed suit, too, which was nice. I credit standing for the last year and a half or so of Skyrim with preventing me from gaining any more weight over that period, something I've especially struggled with since reaching my late thirties.

In early 2013, I happened upon a new trend, which was walking while working, and so I investigated. There were quite a fair number of DIY approaches to it online, people discussing how they had cadged together a set-up and how much it had cost, that sort of thing. However, there had started to be a few on the market as well, and I considered those more seriously -- I'm reasonably handy, but if I'm going to be working in a space for hours and hours every day with expensive equipment sitting atop it, I'd maybe prefer to trust someone else's engineering.

Ultimately I settled on the LifeSpan TR-1200 DT5 treadmill desk, which I bought on Amazon and had shipped to BGS. You can see with a quick search that it's not super-cheap and a DIY solution might better fit your budget. The company declined to let me install it in my office, so I brought it home and set it up in my basement. The hardest part of installation was fitting it in the back of my Subaru Forester; I made two trips one hot July afternoon. The combined weight of the boxes was something like 250 lbs, and the weight wasn't evenly distributed, so one box weighed 150 lbs on its own. Still, I managed to lug it down through my basement entrance with the help of a sturdy tiltable dolly.

Assembling the thing was really easy -- less than an hour of work, I'd guess, and I'd say it actually took more effort to wall mount a TV in front of it on the wall. It has caused me no problems, though I never did get the Bluetooth support working to automatically record my data every day; I didn't work super-hard on that, but for those who want all that superautomated be aware. No biggie for me-- I just throw the mileage, number of steps, distance, and calories burned in a spreadsheet when I'm done, and I've only forgotten a handful of times.

I started on it at fairly slow pace, around 2 miles per hour, and only played games or watched movies and TV on it, browsed the web, that sort of thing². On occasion I would find myself so engrossed in a game that I might misstep or drop my hand in frustration, triggering the interrupt -- there's a clip-on thing on a string that connects to a thing that plugs into the digital readout that will cause the belt to stop if you fall down. Sadly, I often would forget about it and pause the treadmill to go get more water³ or whatever, and would pull it away as I surfed off the back of the thing. (I've since given up on that clip altogether, and it just rests atop the desk while I walk.)

In September I ended my time with BGS, and in October I started walking while actually coding -- only a bit at a time at first, still just dipping my toes in, and gradually more and more until I was working about six hours at a time while walking. Originally I was walking around 2.3 mph, which I upped to 2.5 towards December, and as of late February or March I've been walking at 2.8 mph while I work, and as much as 3.5 mph if I play games.

I find it completely natural to walk and code at the same time -- it doesn't hamper me even the least bit. I'm writing this blog post at 2.8 mph and it has flowed as well or better than had I been sitting down.

I do, however, find it impossible to do any sort of handwriting this way, which is how I think through a lot of design issues, either for game design or thinking about how to structure the solution to an engineering problem. Typically, I just step away for an hour, sit down at a table somewhere and sketch out my solutions with pencil and paper, and then return to implement. I have a pretty good sense these days of how much stuff on paper will equate to six hours of coding, too.

The numbers have been pretty good, too. In eight months I've walked about 1500 miles, in a little over 500 hours. I've worn through a third pair of sneakers4, changing those out every five hundred miles, and I'll start in with another pair when they arrive. I've found that I'm pretty hungry and probably eat more than I should, but I've lost 20 lbs this way and I'm hoping to lose another 20 this year. When I walk my breathing is super clear and not really elevated too much; I think my head is actually quite clearer when I code, for the most part. My resting heart rate these days is in the low to mid 60s, which is quite good for a guy in his early 40s.

A few pointers I picked up that may be useful:


  • Make sure your shoes fit pretty well -- although I'm perfectly happy with the shoes I wear, they have left me with some good sized calluses and blisters at time from hours of friction.
  • After the first time I walked quite a long while (more than four hours), I ordered myself up some bicycle shorts to wear under my lightweight workout pants. There were... uh... chafing issues. Enough said.
  • I wear a sweatshirt with the cuffs all the way down. I tend to rest my wrists on the wrist rest the desk has on its user-facing edge, and my wrists often get a bit sweaty as a result.
  • Stay hydrated. I generally drink between half a gallon and a gallon of water every day I'm walking/working. I am walking at a comfortable pace, but I'm still definitely sweating a little bit, particularly in my legs, wrists, and chest -- not enough exertion to pop a sweat from my forehead most of the time, and the perspiration sort of sneaks up on you after half an hour or so. Good to have a quart of water handy.
  • If you try it and find yourself having a hard time, try a different speed for a few hours -- and faster might actually be better. I watched some videos of folks at a tech magazine trying it out, and they were moving far too slowly for it to be natural or comfortable.
  • Take breaks! If you're working on a particularly thorny problem and can't concentrate on it while walking, just stand and continue. I do this at times; mostly I'm more comfortable just walking, but there are times where I need to stop and take notes or just change how I'm working to make my brain think differently.
  • Desk height is super-important. If I'm just standing on it with my belly to the desk, elbows at my sides, I'm maybe at a 100-degree angle for my elbow bend to rest comfortably on the keyboard. I look slightly but comfortably down to look at my monitor, while I can look basically straight ahead to look at the TV. I occasionally put a day game on in the background on the TV while I work at the monitor, and I do all my gaming on the TV.
  • The device is quiet but not at all silent. I'm listening to some Spanish guitar while I type, and I can still faintly hear the hiss and roll of the thing under me. It's probably too loud for a cubicle farm, but not for a private office. Whenever I Skype, I pause it and stand to take the call.

As I said, I'm about eight months into this particular experiment and I can't really see myself going back. I actually find it kind of hard to program while standing around anymore, and it's even worse to try it sitting down. This is just how I work now.

Questions? Shoot me email at brett_douville on yahoo dot com or brettdouville on gmail. Or hit me up on Twitter where I'm @brett_douville5.

¹Eventually I found an infographic about the facts surrounding the damage sitting is doing you, and I just hung that up next to the entrance to my office.
²The first game I played on it was Metal Gear Solid 3: Snake Eater. Throughout the rest of the summer I'd watch Red Sox play in the evenings, sometimes with a beer in hand, which probably counteracted any benefit from the walking. The Red Sox won the pennant and ultimately the World Series, but maybe that wasn't due to my walking. Maybe it was the beer.
³Hey, there's water in beer!
4I wear lightweight Nike running shoes, the Nike Free v4, I think. 5You can probably guess my user names on PSN and Xbox Live now, too :)

Posted by Brett Douville at 12:29 PM