Sign-up for Email Updates:

Help Survive Development:

Development Blog
August 27, 2015 By: David Tse

This last week I was mostly moving and setting up all my stuff at my new place, but the week before that I made some significant progress on the world generation pipeline. I missed the last weekly because of the moving so I’ll talk about the progress I made the week before. I’ll also be doing the daily updates on twitter again starting today now that I’m done moving and back to full time work on the world generation pipeline.

Initial Test
Picture of my new room/office.



Road Scale Preparations

If you remember in the last video I posted the cities were being generated as if they were the size of continents. That was just for testing the generation to get it to an ok point, but now this week was focused on squeezing all that down and getting them to generate at their proper scale.

Initial Test
Screen of terrain tiles all jumbled.

The first thing I had to do to get that working was split the terrain generation into tiled chunks so I could actually render them at the proper scale because opengl generally frowns upon anything over 16k and the world size right now just for testing is 1,000,000 meters by 1,000,000 meters. I will probably scale it up in the future after everything is working, but for now a million feels like a good number.

Initial Test
Same tiles now correctly placed.

Once the terrain was tiled and rendering I had to spend a little time on the debug rendering. Being able to move around the map and zoom all the way down to one city is vitally important when working on the generation algorithms. After all that was finished I shifted to moving all the city road generation code into its own stage and getting it to render at the proper scale.

Road Scale and Interstate Integration

The first step in scaling the roads down was integrating the interstates and the previous stages of the generation pipeline. So just like the rest of the work I’m doing right now I created a temporary, but working, population map. Right now it is just a simple circle around every city. In the future they will be randomized with noise like everything else.

Initial Test
Zoomed in screenshot of city road generation. Green Circles are temporary population, connecting lines are temporary interstates, and yellow lines are the roads.

In addition to exporting the population map I also exported the temporary high level interstates which need integrating with all the city roads. For now the high level interstates are just lines connecting cities. In the future they will wind through the map avoiding mountains, elevation change, and other terrain factors.

Initial Test
Same generated world as previous screenshot, but with view of the whole world instead of really zoomed in.

After that I just had to move over the system to its new stage in the generation pipeline and add a catalyst road at every city. There were some bugs and system issues that came up when changing the scale, but it was designed with this massive scale in mind so it wasn’t too hard. The only issue is generation speed. When we need 10s of millions of road segments generation efficiency starts to become a problem. That’s what I’m going too focused on next, moving the generation to multiple threads and optimizing the gen algorithms.



That’s pretty much what I did the week before last. It was a shorter week because of the move, but now I’m all set up and back at it 100%. So this next week I’m continuing the work on the pipeline integration and I should be moving back into the actual game world testing pretty soon now. Like I said at the beginning of the update I’ll be starting the daily updates again on twitter so be sure to head over and follow me!

August 13, 2015 By: David Tse

This week I focused on formatting the generation pipeline so each stage could be run on its own and save ALL of its output needed by other stages to disk. This is really important when I’m working on the generation algorithms because it allows me to only run the stage I’m actually working on. When some stages like continent generation take a few minutes to complete saving out that data and reusing it for future stages is a massive improvement to iteration times.

Right now all the stages are correctly loading and saving their inputs and outputs respectively and the next step is to do the final offline stage that takes all that data and efficiently packages it for steaming in-game. Like I said before all the stages are doing the bare minimum generation so don’t expect any huge improvements in generation over what I’ve already shown off. Right now it’s all about getting the data from start to finish and working in-game.

Initial Test
Very exciting screenshot of world generation pipeline intermediate files!

Again this week I spent most of my time on getting the pipeline flowing with all the correct inputs and outputs so all my progress is internal and there is no fancy stuff to show off yet. While this stuff is much less glamorous, getting the iteration time really low and setting up this infrastructure is vitally important to generating the realistic world the game needs.

So that’s it for this small weekly update. I don’t really like doing these small updates, but that’s just the reality of game development. Some weeks you need to do internal stuff and there just isn’t that much to talk about or show off. I did a little bit of the streaming work so I should hopefully be able to show off some of this world generation stuff in-game next week.

August 10, 2015 By: David Tse

This is just going to be a very quick update on what I’ve been working on, what’s been going on and what the plans are for the immediate future. So for the last 11 days I was out of town and that’s why I missed the last weekly and this update is a few days later than I would have liked to post it. I don’t really like to leave when the next update is so close and you guys have been waiting so long for it, but I still ended up getting a good amount of work done and it is a really good refresher.

My family usually likes to go places during the summer and that’s why I’ve gone out of town a few times recently, but now that summer is over I’ll be back to normal 100% time spent on development. That being said I am going to be moving in two weeks so there will probably be a week or so where I’m just moving and getting set up at my new place.

Most of what I’ve been working on is setting up and organizing the generation pipeline like I described in the last weekly update. All of this stuff is internal so there are no fancy screenshots to show any progress and I already described the entire pipeline in the last post and there is not much left to say about it. So like I said this is just a short update on what’s going on and I’ll probably have some more concrete stuff to show off in the next weekly update.

July 25, 2015 By: David Tse

This week I worked on improving high level interstate generation and then the reorganization of the generation pipeline. I made an important shift on the world generation development plan. Previously my focus was on trying to get each step to be as realistic as possible before moving on to other steps. This made me really focus and agonize over generation algorithms before I can actually see how they look/feel in game. The new, correct, approach is to get this pipeline fully functional from start to finish. From generating content all the way to loading and walking through it in-game.

This means that many of the gen algorithms will be simplistic and very far from generating realistic worlds. I sort of knew this was the right approach from the beginning, but found it hard to move on to other steps when the gen algorithms were in the lowest level of functionality like I should have. With this reorganization and the distinct definition of the different stages of the pipeline, their inputs, and their outputs I've decided to fully commit to getting everything flowing with the simplest algorithms.

Once the pipeline is functioning I can release the build to you guys with simple algorithms then improve and iterate on them much more quickly with real time feedback. So you guys will get a bare bones update with simple gen algorithms and I can do updates to the generation fairly frequently as aspects of it are improved. Instead of you getting a semi-decent generated world later down the road.

For this weekly update I’m just going to lay out the generation pipeline by showing the inputs/outputs for the stages, the purpose of each stage, and the current state of each stage. Writing this weekly has also really helped me to fully flesh out the pipeline instead of it all just existing in my head and random notes. So first let me quickly define some terms that will make the pipeline a little easier to read:

GP: Generation Parameters – Inputs to the stage usually defined in the .ini files. Stuff you all will be able to tweak for mods or custom worlds in the future.

HL: High Level – I use this a lot talking about some of the different stages. It just means the scale of the data is very large and in the kilometer range (mountain ranges) while the low level stuff is in the meter range (small hills).

World Generation Pipeline



High Level Continent Generation Stage:

Description:

Uses plate tectonics simulation to generate continents and large terrain features like mountain ranges, lakes, and high level elevation. This stage also distributes moisture and temperature to generate different biomes.

Current State:

High level elevation generation is in a decent state that generates good looking continents, but loads more work still needs to be done. Moisture/biome classification is very basic at the moment.

Inputs:

  • GP(number of continents, ocean to land ratio, elevation collision spread ratios, etc.)

Outputs:

  • HL Continent Heightmap split into chunks
  • HL Biome Map with moisture and temperature split into chunks

Initial Test
Screenshot of basic continent generation with mountain ranges, lakes, and high level elevation.

City Placement Stage:

Description:

Uses biome data to distribute cities across the land. It favors locations near water and that are relatively flat, but will create sparse cities in less than optimal locations like in real life. This stage also decides what cities need to be connected to each other.

Current State:

Right now it just does semi random city locations that favor high moisture and does a good job of determining which cities need to be connected.

Inputs:

  • HL Continent Heightmap
  • HL Biome Map
  • GP(biome distribution factors, desired large to small city distribution, population based prune distance, etc.)

Outputs:

  • City Data(Positions, List of Neighboring Cities, Population Count)
  • HL Population Map, created using noise based on city locations/population

Initial Test
Shows example of early city distribution. Ignore terrain its on that is from very old build.

High Level Interstate Generation Stage:

Description:

Goes through city data and creates interstates that make all the necessary connections between cities. It uses the high level continent elevation and the algorithm for low level elevation so it can make accurate decisions about where it can and cannot pass through (hills, rivers, etc.). It also makes sure interstates naturally pass through cities instead of all going through the same city center point.

Current State:

This is one of the stages that needs the most work. It currently tries to do some pathfinding between cities and generally it works, but roads frequently overlap due to the lack of pathfinding resolution. This system also ignores low level elevation. I did a little work on it this week before deciding to focus on the pipeline as a whole, but this system still needs to be heavily revised once I get the whole pipeline functioning. The plan is to use a growth system similar to the city roads instead of high level pathfinding.

Inputs:

  • HL Continent Heightmap
  • HL Population Map
  • City Data
  • Low Level Terrain Gen Algorithm/Sampler, needed to generate local obstacles like small (few hundred meters) hills that need to be avoided.
  • GP(terrain change tolerances, max allowed slope/turn, interstates sharing tolerance, etc.)

Outputs:

  • Interstate Splines.

Initial Test
Screenshot of the current state of high level interstate generation.

Major City Road Generation Stage:

Description:

Uses population map to grow major city roads based on desired city shape and type. Then take those roads and classify them into city blocks.

Current State:

I'm pretty happy with this stage so far. Right now it only has one city shape which just follows population with not high level target shape, but in the future there will be different city types/shapes. The city block classification is fully functional. The city generation video blow is basically the current state with a much larger scale.

Inputs:

  • HL Interstates
  • HL Population Map
  • City Data
  • GP(all the growth parameters for roads, city shape, etc.)

Outputs:

  • Major City Roads
  • City Blocks created by major roads


City Block Generation Stage:

Description:

Takes the city blocks enclosed by the major city roads and fills them with the various buildings and roads needed in a city. There are many different city block types that are filled with different types of buildings. Think retail, residential, schools, business, etc. There can also be a mix of different types like a neighborhood with a school in it or a business park with a large retail store.

Current State:

This is the stage I have probably put the least amount of work into so far. Right now it generates a very simple grid like neighborhood. It also only generates roads at the moment, but adding simple house generation will not take too long. This is one of the stages I am leaving in its current very unfinished state until the pipeline is fully functional. The current state is shown at the end of the city generation video above.

Inputs:

  • HL Interstates
  • HL Population Map
  • Major City Roads
  • City Blocks
  • GP(some controlling params on frequency of different block types and their tendency to be near population, type of city based on population(rural, medium pop, high pop with skyscrapers, etc.), desired population density, etc.)

Outputs:

  • Low Level city roads
  • List of all builds with their position, orientation, size, income/wealth, and type
  • List of city entities(light poles, telephone poles, traffic lights, mail boxes, etc.)


Real Time/On Load:



Building Generation Stage:

Description:

This stage and all subsequent stages are done when you load the game and in the background as the world is streamed. This stage takes the boundaries for the buildings from the block generation stage and creates procedural building layouts. It then fills those houses with all the base entities they need (furniture, doors, windows, trimming, ect.).

Current State:

This is currently version 1.0 of basic house generation. It is in the same state as the build generation video below that talks about it. This algorithm has many problems and gets some fundamental things about how buildings are constructed wrong. The biggest is that they are largely rectangular and don’t waste space. Anyway there will be a new version of house generation once the pipeline is set up. For now those houses will be put in a box and thrown into the world as is.

Inputs:

  • Building Data
  • GP(List of params for each building type being generated, will be a large data file for min/max values for room types and sizes depending on house income/wealth)

Outputs:

  • Custom building layouts for every building needed, determines walls, doors and windows.
  • List of entity data needed per house, location of sinks, couches, beds, door frames, etc.


Geometry Creation Stage:

Description:

This stage just takes all the building layouts and road data then turns them into actual geometry and textures the game can render.

Current State:

It is sort of good enough for now. It can create roads from splines and generate most of the walls for buildings. There are some holes like above the door ways, but like everything else its good enough to get the pipeline set up.

Inputs:

  • Building Data
  • All Road Segments

Outputs:

  • Generated geometry needed to render all the roads and building walls.

Initial Test
Shows early interior of a house with some test balls.

Final In-Game Stream and Render Stage:

Description:

Takes the current position of the player and streams in/generates the needed data in the background. It aggressively culls said data and then renders it. This is basically the real time component that takes all the generated data and makes use of it. The above two stages actually happen inside this stage for the most part. As the generated stuff is streamed in the player save games files are loaded and merged with the generated data in this stage.

Current State:

I have spent a lot of time on this stage so far. All the terrain streaming is done and the backend data structure for streaming chunks of world gen data is all planned and saved out. The only work that needs to be done is on the actual loading part. Once this stage is done and we can see generated stuff in-game then the gen pipeline will be fully functional. Then we can go back and make the world more dense, realistic and interesting.

Inputs:

  • Chunked Building Data
  • Chunked Road Segment Data
  • Chunked Entity Data

Outputs:

  • The World

Initial Test
Only screenshot I have of a building with terrain in-game. Its pretty old and just shows building without its exterior sitting in the terrain.



So that is the general plan for world generation and the steps involved. Like I said before my focus is now on getting this pipeline functional with the absolute bare minimum generation algorithms. This should mean the next Pre-Alpha update is only a few weeks out, but we will see.

That's all I have for this weekly. Hopefully it gave you a better picture of the current state of world generation and where it's headed. If you have any questions about the process or anything else please let me know and don't forget if you want to see my daily development updates you can follow me on twitter!

July 17, 2015 By: David Tse

You may have noticed that I missed the last two weekly updates. There are a couple of different reasons for that. Primarily my focus on getting the mac build of Atlas to the new artist and the fact that I was out of town for a week. All that really matters is we are back on track and updates will return to the normal weekly schedule. This update is a day late from the target release day of Thursday, but that’s because my 21st birthday was on Wednesday and my capacity to work on the following Thursday was greatly diminished...

Anyway on to the actual update. These last 3 weeks were focused primarily on getting a working build of Atlas fully useable on Mac so the new artist could start previewing his work in-engine. On top of that I did get some work done on city generation, but nothing of note. Also keep in mind that one of these weeks I was out of town and only got minimal work done.



The Fall of QT

One thing I wasted a lot of time on was trying to get QT to function correctly with OpenGL contexts on Mac. After trying so long to get that to work in addition to dealing with QT lag spikes and uncontrollable crashes I decided to just completely ditch it.

Initial Test
Screenshot of Atlas running on Mac with new custom menu bar.

In the amount of time I wasted getting QT to work with my engine on Windows, learn how to use all the different QT widgets, and then trying to get it to work again on Mac I could have created a robust QUI system myself. So I decided to just cut my losses and invest future Atlas development time into my own system.

This will take a little more time to get up to the same functionality as QT, but I can build it up one part at a time only creating the GUI widgets I actually need. Having full control of the GUI system and not having to worry about figuring out how to create/get working exactly what I want in QT will allow for much faster development in the long run.

Initial Test
Render of new Front door created by our awesome new artist Erik!

I actually already created a pretty strong GUI foundation in the engine for all the menus in the game. So getting the editors functionality up will not take too long. To start I created a custom menu bar using these foundations that is used to load models, load materials, and toggle different things in the engine. Like I said before I can do the editor GUI one step at a time and currently that menu bar is all I need to implement right now. When Atlas needs more functionality I can easily implement the widgets it needs.

Initial Test
Render of new Front door created by our awesome new artist Erik!

Another big advantage of doing the GUI myself is that it helps to advance the engine’s GUI system which is going to be used for all the in-game GUI for things like inventories and other menus. This means work on Atlas is also directly improving the game itself. Something that definitely cannot be said about using QT which usually results in me wasting my time trying to get it to not suck.

Mac Port

Once I ditched all QT dependencies in Atlas finishing up the port to Mac was more straight forward, but there were a few more challenges to tackle. In addition to getting rid of QT I also had to stop using Xcode because every time I would run Atlas it would just crash Xcode which is completely unacceptable. So I was off to look for a reliable command line build system. Something I previously had little experience with.

After a little while looking and researching I ended up going for Meson with the Ninja backend. Meson is an open source build system created by Jussi Pakkanen that is super fast and really easy to set up. I really love it and if you are looking for a C++ build system there really is nothing better.

Initial Test
Render of new backyard fence created by our awesome new artist Erik!

Once I had the app building and running fine on OSX I had to package it correctly and get it running on the artist’s non development machine. I’ve never deployed a Mac app before so there was a little bit of a learning curve to figure out how OSX deals with dependencies, but I eventually got it packaged in a .app and running perfectly on the new artist’s computer. Meson actually comes with a nice utility to do the packaging which is another big plus.



On top of getting Atlas to the artist I also did some significant restructuring to the perforce server. I moved all the game data from the main depot with the main working directory and the code into another depot so artists can focus completely on the assets. This is also where new builds of the game will be staged for release. This all just means development and deploying builds to you all will get much easier meaning releases can happen more often.

Then like I mentioned I also did some more work on city generation, but there is nothing of note to mention. All my focus is now back on City generation. That is all for this catch-up update and you can start to expect consistent weekly updates starting next Thursday. Don’t forget I’m doing daily updates on Twitter and continued to do them even when missing the last two weeklies so if for some reason I miss a week in the future there will still be news over on the twitter: @Subsurfacegames.

Previous Page