Saturday, February 26, 2011

Limits, pt. 2

The Ultima 6 game engine, as used in its original form for Ultima 6 and slightly modified for The Savage Empire and Martian Dreams, is a classic example of a tile-based renderer, with the main view into the world (the map) and many of the user interface elements composed of reusable graphic tiles. The engine was designed with some specific hard-coded limits for performance and space reasons, and what follows is an explanation of the structure of the engine and its inherent limits, and the potential of relaxing them in an engine recreation.

Each U6 tile is a 16x16 pixel bitmap, each pixel of the bitmap is a one-byte color index into a common 256-color palette. An 11-bit integer is used for indexing into the complete set of possible tiles, so the maximum possible number of tiles is 2048.  The bitmap data is stored in two files: maptiles.vga for the first 256 tiles, and objtiles.vga for the remaining 1792 tiles, and with a few additional files to provide the engine additional data about each tile:
tileindx.vga provides a data offset into the combined maptiles.vga + objtiles.vga decompressed data buffer for the start of each tile’s data.
masktype.vga provides an encoding type indicator for each tile, used to tell the engine how to decode the individual tile’s data. There are three types of encoding:
  1. Plain, where the data is a 16x16 array of color indices, with no transparent color. All map tiles use this encoding, since the map layer must be opaque.
  2. Transparent, where the data is also a 16x16 array of color indices, but where index 255 indicates a transparent pixel. Many object tiles use this encoding, since they are commonly composited on top of the map layer.
  3. PixelBlock, where color pixel data is plain color indices, but transparent pixels use a Run-Length Encoding scheme to save space. This is also used for many object tiles, since transparency is typically a contiguous area surrounding object color pixels.
tileflag contains a set of flags indicating to the game engine what each tile actually represents for use in the various game systems. Boolean flags for each tile include: Passable, Water, OnTop, DoubleWidth, DoubleHeight, and more, with many of them used in the tilemap renderer to control how they are drawn.
animdata contains a set of info for the small subset of tiles that use frame-based animation. In The Savage Empire, display of the animated waterfall tiles is controlled with this data, and in Ultima 6, animated tiles include things like the drawbridge mechanism.
look.lzc contains text descriptions for the tiles (for The Savage Empire and Martian Dreams). The corresponding file for Ultima 6 is look.lzd.
basetile is used in the object system to map object indices to tile indices, since there are fewer actual objects than tiles. There are 1024 possible objects, so an object index can be stored as a 10-bit integer.

The world map for the Ultima 6 engine is stored in a pretty interesting way. First, the world map consists of only the bottom layer of tiles that eventually make up a rendered world view – actors and objects are rendered on top of the map layer.

Only the first 256 tiles are used in the map layer, so they can be indexed by a single byte, a tradeoff against allowing a full 11-bit index so that any of the 2048 possible tiles could be used. This is also why the map layer tiles are stored in a separate file from the rest of the tiles – they were likely kept in memory constantly, while object tile data could be kept in a limited-size cache instead of reading it all in.

At a high level, the world map is not an array of individually-indexed map tiles, but is instead an array of larger chunks of tiles. Each chunk index is a 10-bit integer, allowing for a total of 1024 unique chunks. Each chunk is an 8x8 array of map tiles with a total size of 64 bytes. Put those two facts together, and you get the source data file, chunks, which is 64k bytes in size.

The chunk indices making up the world and dungeon maps are additionally subdivided into ‘superchunks’ for indexing purposes. Each superchunk contains a 16x16 array of chunk indices. The full top-level world map consists of an 8x8 array of superchunks, and each of the five ‘dungeon’ maps allowed by the engine consist of a 2x2 superchunk total area, although the dungeon maps are directly indexed as 32x32 chunks. From my reading of documentation left by the original developers, I believe that only 2x2 superchunks were in-memory at a time at the world map level, another optimization for the limited total memory available.

It’s interesting that this tile / chunk / superchunk scheme, while almost certainly developed as a compression scheme, probably also resulted in actually making the world map easier to build, since world components could be laid out at a higher level. For instance, chunks included tile layouts for roads such as north-south-east T-junction surrounded by grass, which could be next to an east-west road chunk, and so on and so forth. The drawback to this scheme, of course, is the visible repetition of chunk patterns. But since the map window was only drawn as 11x11 tiles anyway, the 8x8 chunk patterns were not always obvious.

It’s also interesting to consider the memory and disk space savings of the scheme. With 8x8 superchunks, 16x16 chunks per superchunk, and 8x8 tiles per chunk, the full world map is 1024x1024 tiles. A directly indexable array of map tiles (1 byte each) would therefore take 1 megabyte of space. Even limiting the tiles loaded into memory to 2x2 superchunks (256x256 tiles) would result in a a 64k array, with 32k discarded and another 32k loaded every time a load boundary was crossed – a considerable amount for a single-threaded engine designed to be runnable off floppy disks.

Using the scheme, the world map consists of a total of 128x128 10-bit chunk indices, for a total footprint of about 20k. With 2x2 superchunks in-memory at a time, that’s 32x32 chunks, 10,240 bits, 1280 bytes. With 640 bytes discarded and 640 loaded at a load boundary, I’d imagine load times were reasonable even when loading map data directly off a floppy disk.

The map data is actually stored in a file called map, and consists of the 16x16 chunk indices for each of the 64 world map superchunks, followed by 32x32 chunk indices for each of the five dungeon maps. The total size of the file is an amazing 32KB, which combined with the 64KB chunks file means that the underlying layout of a 1,048,576 tile world map and five 65,536 tile dungeon maps (1,376,256 tiles total) was stored in 98,304 bytes of disk space,  a 14:1 compression ratio.

So that’s the map, which is rendered for the game as the base layer of tiles in the 11x11 world view window. In additional to the map, there are two other main systems that are eventually rendered into a composite view: the object system and the actor system, each of which has a visual representation derived from the tile system, specifically the 1792 tiles from objtiles.vga.

The dynamic object and actor data is loaded from (and saved to) a set of files in the savegame subdirectory of the main game directory. Actor data, and some general world-level data like world time and plot flags, is stored in objlist. Object data is stored in a series of files corresponding to their containing superblock: objblk[a-h][a-h] for the top-level world map superblocks, and objblk[a-e]i for the five dungeon maps.

Actors are uniquely indexed with a one-byte id, so there are a total of 256 possible actors tracked by the engine at a time. Most of the actor slots are taken up by the unique personalities in each of the three U6 game worlds, but some of them are reserved for temporary actors (such as combat enemies) that are typically created dynamically and returned to the pool when out of the active range of the player.

Dynamic objects in the game are indexed with a 10-bit id, so they may be one of 1024 different object types, but there may be many copies of each type of object in existence in the world. Each objblknn file for a superblock contains a list of objects, each consisting of an object type index, a location,  a set of status flags, an animation frame index, and 1-byte quantity and quality values, the use of which is overloaded for different object types. Location consists of a 10-bit X value, a 10-bit Y value, and a 4-bit Z value, packed into three bytes. The Z value is 0 for the world map, and 1-5 for the five dungeon maps.

Throughout the design of the data structures used to render the world and keep track of the various aspects of it, element sizes were specifically limited by the original engine programmers. To recap:
  • 8-bit tile pixel color indices, limiting total colors to 256.
  • 8-bit map tile indices in each chunk, limiting available map layer tiles to 256.
  • 10-bit chunk indices, limiting total chunks to 1024.
  • 10-bit X and Y coordinates for objects, limiting location to a 1024x1024 tile area (and thus, total world map superblocks to 8x8 superblocks with 16x16 chunks per superblock and 8x8 tiles per chunk).
  • 8-bit actor indices, limiting total actors to 256.
Additionally, although there may or may not have been room in the in-engine pointers for larger total collection sizes, the authored data on disk was limited to specific collection sizes for tiles and objects:
  • 2048 unique tiles (11-bit indices).
  • 1024 object types (10-bit indices).

In a modern recreations of the U6 engine, such as my own project, it is not actually necessary to stick with the original index size limitations, given the amount of memory and storage space and bandwidth available, as long as interoperability with the original engine is not required as part of the design. That is, one could allow for importing the original data with the original index limitations, but actually store it in data structures with extended index sizes.

For example, chunk indices, as stored in the maps file in the original engine, could be extended to 16 bits, to allow for 65,536 possible chunks, which would allow more unique chunks than would probably ever be needed or authored. Similarly, tile indices in each chunk could be doubled from 8 to 16 bits, which would double the chunk total size from 64 to 128 bytes, but result in a vastly expanded palette of map base tiles to choose from for chunk design.

If object / actor location values were extended from the current 10/10/4 bit X/Y/Z values, the potential world map could be vastly larger. A 16-bit X and Y would result in an addressable tile area of 64k x 64k tiles, quite an increase from the current 1k x 1k tiles. I suspect that actually having a world that large would mean a hand-authored world design would take too much time to really be feasible, but it might be workable when combined with procedural world map generation.

Sunday, February 13, 2011

Limits, pt. 1

“A man’s gotta know his limitations.” – Harry Callahan
It’s a bit of a trite observation, but the average personal computer of today has more processing power than the supercomputers of decades past.

The PC I played the Ultima games on originally had a 386-class CPU that ran at 40Mhz, with a game display of 320x200 pixels, each of which might be one of a palette of 256 colors, drawn from a pool of 262,144 possible colors. It had a total of four megabytes of system RAM, but because of the limitations of the legacy memory models supported by MS-DOS, the average program had a maximum of 640Kb of memory actually available to it. It also had a huge 80Mb hard drive, quite a luxury for the time.

The games of the era had to be ruthlessly optimized to keep memory footprints down, and that optimization led directly to quite a few design limitations in the U6 game engine, which I’ll discuss in the future.

The PC I’m developing with, on the other hand, has a four-core CPU, each of which can run at over 3Ghz. It has 4Gb of system memory, with programs typically able to access 2GB of it each. It has multiple terabytes of disk storage space available. The video subsystem is capable of displaying 24-bit color at my full monitor resolution of 1920x200, at frame rates so high that the human eye can’t even detect individual frame drawing.

In short, it’s capable of displaying an amazing illusion of life, as demonstrated in games like Just Cause 2, Crysis, Far Cry 2, Fallout 3 and Dragon Age: Origins, to name a few I have installed. Each of those games is a real-time 3D simulation of a world that looks and feels remarkably like the real one. And each of those games was built by a team of hundreds of people, and consists of a few (well, under a hundred, anyway) megabytes of program code, but multiple gigabytes of data files. The budgets for creating those games are in the tens of millions of dollars range.  Like the saying goes, ‘Content is king.’

No indie lone-wolf or small team of developers can compete in the same arena. So, the most successful indie games of today – those that actually make enough money to support their creators – are very abstracted versions of reality. The obvious examples of this are Dwarf Fortress and Minecraft.

Dwarf Fortress is developed by Tarn and Zach Adams, with Tarn doing all the programming, and both brothers sharing the design. By default, the Dwarf Fortress interface and game view, in the Roguelike tradition, is 100% text, with terrain and actors represented symbolically by the letters and simple glyphs of the IBM Codepage 437 character set. Instead of spending time on art for the visual verisimilitude of their worlds, the brothers have focused on the actual underlying simulation of their worlds, and the end result, while not exactly appealing to the mass-market, has drawn in enough players and financial donations to let Tarn continue working on it full-time. (And I personally think it is the most amazing game ever created.)

Minecraft is developed almost single-handedly by Markus Persson. It focuses on a procedurally-generated dynamic world composed of cubes, giving its players the ability to build and destroy with a simple, accessible interface. Its art assets are low-resolution texture maps and simple, blocky models. It makes no attempt to match the visual realism of the AAA games noted earlier. It hasn’t cost much more than Markus’ time to develop to its current beta stage. But the wide appeal and pure fun it offers has led to (currently) more than 1.2 million purchases of the alpha and beta versions.

To sum up, a game using an abstracted world representation can still gather a significant player base in today’s gamer ecosystem, and even potentially gain enough financial support for its creator to make a living from it. In my own area of interest, CRPGs, Jeff Vogel of Spiderweb Software has been doing it for years, by limiting his art investment and development time for each of his titles and then focusing on what can be done given his chosen limits – which is to say, he writes an interesting storyline and wraps it in a large world full of unique characters and combat situations. I don’t know that I’ll ever release my project commercially (definitely not while it’s using Savage Empire assets) but it’s nice to know that it is at least possible that I could get more back than my personal satisfaction with doing something interesting.

Next up: Limits pt. 2, focused on the specific design of the U6 games engine.

Hello, world

Hello, there.

I'm Brian, and this shall be a record of my thoughts on game development, focusing on my design and implementation of a relatively simple 2D role playing game.

I'll be using Microsoft's XNA platform, but the game design will be largely based on Origin Systems' trio of Ultima games released from 1990-1992: Ultima VI and the two Worlds of Ultima games, The Savage Empire and Martian Dreams. My hope is that by limiting the technology complexity, I'll have a higher chance of actually getting the game into a playable state without worrying about spending too much time on art assets.

In fact, as I write this, I already have an engine capable of rendering Savage Empire's world map using the original game data. Reading through Nuvie's source code has been indispensable in building my understanding of how the original games work. The technical documention built up by better hackers before me and available via the project archives at Ultima Aiera has also been of great help.

Next up is a bit on the limits of the design, I think.