My Take on an Infinite Voxel Engine

[reiti.net]

Hello folks.

This is some project of mine, where I tinker around with implementing my own “Infinite” Voxel Engine (this means, that it’s not limited by some internal borders but only by addressing space, 32bit currently).

Finding and trying to resolve problems is just as fun and demanding as playing a hard puzzle game, so I have fun with that challenge so far and maybe I will make a game out of it later on.

The engine is aimed to be top/down/iso view and mainly for RTS style games or something like that. It runs on a deferred rendering pipeline. I regulary do some sort of progress videos on my channel, so if you like, you can just watch some videos, where I show the features and progress.

Feel free to ask questions.

[Charles_Humphrey]

Sweet! I have a voxel engine I wrote in Unity, been meaning to port it to MG

[KakCAT]

I love the slice idea! I built a voxel engine a few years ago with monogame and I wish I had thought of it, it really simplifies editing.

[Hasindu]

Seriously? How did you made hardware instancing possible in a GameObject oriented engine? Is it performance efficient?

[Hasindu]

I have made one when I was 15y old.
Take a look https://github.com/HasinduLanka/BlockEngine
But this is originally written in VB.NET and Open Sourced in C#
So, it contains a handful of bugs. Ask me if you want that original VB.NET version

plus, I don’t develop it anymore…

It had features like Run-time ground generation, Building, Breaking, saving, lights (Ambient,Diffuse,Specular with Day-night cycle), NPCs (Path finding, reacting, fighting, following, commanding, defending), Collision detections, In-Code key-frame animation system, Sword fights with real sword-to-body collision detections, Collision node hierarchies, HLSL shaders with GPU instancing and much more… 2 years back… While doing GCE O/Ls, Alone…

[Charles_Humphrey]

I do it by chunking. If you are instancing with voxels are you instancing the entire voxel? I only create the faces I need, so instancing would be difficult to do as the vertex structure is so varied…

Here is an old version I did, don’t have any vids of the new one.

Even had a little go at making a simple game with it, never really went anywhere lol

I might look at re creating it in MG at some point. might not be as good as your version though

[Hasindu]

Wow!
I have no idea of visual effects like God rays, Sun Light “visuals”…
(Whether they are models/shader illustrations …)
Can you help me on what to follow?

[Charles_Humphrey]

I have put a load of post processing (including my god ray’s) into my MG repo, I have a thread on it here

Hope it helps

[reiti.net]

Time for some follow up

My next big thing to do was actually doing “ramps” … and I am tinkering with this now for a very long time. As I use Baked AO - which is simple if a world only consists of cubes - but having ramps and corners heavily complicates that stuff.

I am actually not really working with cubes, but with Quads, and my geometry creation is basically a very big state machine, building the faces dependend on moore neighbourhood. The same neighbourhood is used to determine current light exposure and baked AO. Until before, the neighbour was either a block or not - now with ramps, things get more complicated, because the decision if a pixel is in AO or not depends not only if there is a block, but what type of block it is (ramp cube, corner ramp) and what direction that ramp is facing. So that’s a LOT of things to code in a state machine …

In the process I found that my old geometry builder was not suited well and so I refactored it from ground - it’s now faster and easier to handle new sorts of block types. As you can see in the screenshot, I am not finished with the AO and there is still AO where the ramp meets the floor, where there shouldn’t be AO. I am currently still thinking about different solutions to make this process less tedious than adding just another check in the creation and redo this for at least 8 other types … uuuhm.

In an earlier prototype of this engine, I played around with SSAO - but the results were not satisfying - but I may give it another try, because it may solve other issues (like AO at trees, avaters, assets etc) - I am not sure if Baked AO is the way to go here. Baked AO on a 4vert Quad also has some visual drawbacks which are impossible to solve without increasing vertex count.

So it may take some time until I come up with a new video I guess, let’s see if I can get a better SSAO to work, it would be pretty cool to have actually

So long - happy tinkering

[reiti.net]

I actually sacked the baked AO - it was just not working out for what I’d imagine. So I turned over to SSAO again and fight with all the quirks associated with it. Main problem (for me) was mainly that I am overwhelmed with several things like handness, view space and position recreation, TBN multiplication order, halfpixels and some hard to spot errors in available tutorials.

Anyway - I am still not definitely finished with getting everything right, but at least I got a result which I found pretty pleasing and reassemles the baked AO pretty well!

I’ve also completely redesigned my geometry builder (again) to make it more easy to expand it with different Block Forms (and still work together with per-block-lighting, selections, pathfinding etc) - as a bonus I was able to squeeze a bit more speed out of it as well (the geo builder runs single threaded for several reasons) - I now have also built in ramps and ramp corners and already have several other blocktypes in mind

Here is a quick screenshot of how it looks now (I’ve forgotten to add some corners as well… ) - I may do another progress video soon to show some current material, so stay tuned

I have to say it’s already funny to build stuff … anyone knows some artist which is good in making fitting textures for me (paid, free, doesn’t matter), I’ve just made mine up and they don’t tile very good, normals are a bit odd because generated from color) but I like the pixel-alike-but-not-really-pixel look of it

[reiti.net]

The recent day, I’ve worked on several blocktypes. This involved making a mechanism of easily adding them (where I used a trick with T4 functionality because C# lacks the ability to define macros and reuse code snippets without having to waste function calls) which was need to keep the geo generation algorithms fast.

The next thing to do was actually having the engine create all the new ramps with different orientations from a voxel-procedural map (basically combined noise maps). I heavily work with bitfields, bit-wise operators and predefined bitpatterns to keep everything fast

As a result of that work (which was a lot!) everything looks quite smooth now and the new block types also gives a lot of possiblities to actually “build” in the world. What do you think?

I’ve already adapted to water simulation (automata) to correctly handle the ramp blocks and - beside some remaining bugs - the light automata as well (for per-bock-lighting)

Another problem occuring is now, that my current textures (nothing fancy) do not blend well in triangles and other non-voxel-situations, because the borders do not align. I am currently thinking of having the voxel texture consist of several sub textextures for different types of sides (4sided, 3sided, top texture) - this puts more time into working on textures (postponing this for much later, because I don’t enjoy doing textures)

I am also trying out to post on tumbler - so if you are interested in smaller but more frequent update chunks, just find me there and follow

[reiti.net]

Finally I’ve done a new presentation video to show the current engine in moving action. Let me know how you like it

next video will feature something neat - stay tuned

[reiti.net]

The last days I was very busy with doing more optimizations. The new ramps added a bit more special faces which are now properly not generated when occluded by neighbouring blocks (all combinations).

I also improved the pathfinding even further and incooperated diagonal movement over flat surfaces - still open to do the same for diagonal ramps (lots of checks)

another not-yet-to-be-told-feature is waiting to be presented in a video

had my first tries with actual job handling. As this engine shall be a builder/rpg sort of engine, there has to be jobs assigned. Much work to do for this one, as it’s quite complex. i.e: Mark a block for digging, agent picks up the job. No to actually dig a block, the agent have to find a place beside that block to do the work - the ideal way would be to check every possible “standing position” with pathfinding and get the nearest one. But that’s to heavy on the Path Solver - I may give it a try with using heuristics instead, let’s see how that’ll work out

[reiti.net]

I’ve finally made a new video, showing the not-to-be-told-feature which is basically (just basically) working now: It’s the Ingame Asset Editor - it mainly was the reason why I wanted the different Ramp Blocks:

I am currently working on the actual Job/Order System, which is a bit more tricky than I actually supposed. I first tried a naiv approach like so, that each agent, when idle, will search the job queue for the next job. Each job first finds general possible places where the agent can stand to do the job, then it’s handed to the agent and the agent tries each of them if it can reach on of those spots. If not the job goes back as failed and will be picked up later on.

While this approach somehow works it’s not ideal and can lead to weird things and has no sense of which agent is most appropriate to use, beside the fact, that it clumps up the update loop a bit (pathfinding is multithreaded, but all the handling takes a bit) - so what I am now trying to do is turn things around and let the jobhandler reside in its own thread, regulary checking new jobs, sorting the orders by priority and distance to the nearest agent. This all will take a significant bit of performance, so it can only rely in a separate thread in order to not harm the FPS.

Let’s see how that’ll turn out

[reiti.net]

Looks like I finally ended up with something which does work not so bad. I am using an asyncronous runnign “Job Manager” which basically cares for assigning jobs to idle agents. To do there is some job queue for manually issued jobs and if this is empty, the Job Manager queries different zones, if they have job to do.

A stockpile is such a zone, and if queried it will determine if there are any ressources lying around which need to be hauled and if so, they will generate a job which is then assigned to the next idle agent. This should basically keep the queue of “open jobs” relatively small. There is several object locking and caching arrays involved for everything to work well with the renderer and keeping the impact on the framerate as low as possible, but under some circumstances (i.e. a fully locked in agent) will still clog everything up a bit. I do need to make a lot of compromisses between object locking and performance and was quite busy with a lot of bugs evolving around that. In the video some of those bugs a re still visible but most of them are already fixed.

Another issue with this system currently is, that the items in the world (which ressources are) are currently a “global list” together with partioned references for them on every chunk. While this has benefits in some situations I am not enterily sure if this will be viable in a highly populated world and so this may change to a fully partioned system with the drawback of having any lookups being slower in some cases.

Current tests with a lot of agents worked out pretty well, beside the fact, the more agents are assigned a job, the longer it will take until everyone has finished it’s pathfinding, but there is no way around it and with complexer environments this will slow down even further.

In the current system, there are sort of Job-Chains. So each job can basically have follow-up-jobs defined during creation of the job itself. This is in order to lock/reserve any involved resources. In the example of hauling, the stockpile creates a job, reserves a space for it and locks the item on the map. An agent will now have the order to pick up that item and a follow up job, to drop down the item to a specific spot.

Problems arises, when such a job chain needs to be canceled for whatever reason (agent died, way blocked) as now the full chain needs to be processed in order to free up any reserved spots or items. With follow up items that is pretty straight forward, but there may be situations where this is not enough. Need to see if it’s sustainable.

Anway, still a lot of things to play around with and I am eager to actually see the agents building something, felling trees or seeing stacked items displayed differently or having storage containers.

[reiti.net]

Progress:

Alright - so some weeks ago I stumbled over mutliple performance issues, which basically dropped the framerate heavily to a point where I didn’t consider it to be playable. It was frustrating and I put it away for some time.

After some time I decided to displace some code into an externall native c++ DLL to speed things related to noise generation up - this worked a bit but I also had to do a lot of result chaching in order to really eliminate that bottleneck.

The engine now also works with 4 concurrent threads to do several things asyncronoulsy, like path finding, order/job handling and such things - which brought HUGE problems in terms of syncronizing and weird errors which were hard to find the initial issue for. Most work on this engine is heavily related to performance, this really is the main work.

Regular “locking” will just not work out, as it’s slowing down everything up to a point where the framedrops get noticeable, so what the engine actually does is working a lot with chaches multiple objects and syncrinizing those on demand from the threads themselves.

Anyway - I think the basic thing in zoning/logistics is done in the engine. Agent’s are able to get a job/order assigned an able to fulfill whatever they are supposed to do - there are still some problems and hickups in terms of “unexpected behaviour” of the agents but they will be tackled when they arise

I did another Progress Video, which is mainly related to zoning and order/job fulfillment. The engine can (chained) issue jobs/orders and add/remove zones and building tasks now, what do you think?

[reiti.net]

I actually missed to report a new progress video - in the last video I actually showed the newly farming feature and also let the agents build a house by only using orders (the supposed way to do it). I’ve also felt the current lights look a bit odd and decided to rapidly code some fire and light jitter as well

Most of this is actually mocking up a new feature, making it work and then followed by some days refactoring smaller or larger parts of the engine to get better performance or readability (ideally both, but that’s not always possible here)

Anyway - I am currently tackling the cumbersome approach of “saving the gamestate” - yes I could just use binary serialization and use one of the benefits of .Net … but honestly - I fear it will be to slow and the result would be just too big files. I also may be adding some sort of different saving approaches (in terms of speed for loading/unloading inactive but modified chunks, etc) - so I jumped the train and did a fully manual(!) approach of saving/loading the current game state.

This engine is a bit different from regular voxel engines - being mainly for building games. So there is a lot of references handed around over several threads and - oh boy - this is not funny to code a serialization for

At least I am currently at a spot where saving loading basically works and I am just searching/fixing several bugs which appear during testing. There is no video for that, because there is basically nothing to show with loading/saving (it’s one of those things, everyone expects to just be there).

As soon as this is in a complete state, I really want to push production chains, like chopping a tree to get wood (already working), making planks and using planks to make a chair (for example).

I’ll keep you informed

[reiti.net]

Another week, another progress

I basically had the idea of having shadows in my game. The problem with that was actually, that my current rendering pipeline consisted of just a very big function plotting out all the needed data to the shaders - which was not feasible when having to do it twice because of the shadow map

So I used quite some time to completely redesign my render pipeline which is now much more managed, collecting mesh and rendering data in advance into several buckets (batching by shader, texture etc) and rendering that buckets afterwards. I implemented Exponential Shadow Mapping as I do have most data in the defered renderer anyway.

One issue in respect to my engine is still the problem of underground chunks and that’s messing a bit with my spacial partitioning (as currently I basically render the chunks visible for the sun … ) - so I am still in the process to split this up a bit more in the hope the impact on rendertimes isn’t that hard. (but rendertimes are not really a problem in my engine, it’s the stuff around it)

Anyway. in terms of mechanic I finally implemented the “item building” mechanic into the engine (building items, the player can actually design themselves in an ingame item editor). Some other features were distinct names for the thingys and rendering a shoreline (which was one of the few easy things to do)

Also Loading/Saving is now fully working without issues, I just have to track any changes to game data to be reflected in the save/load definitions/interfaces

If you are interested, here is the current progress video:

[reiti.net]

Over a year later … time for some sitrep.

the engine turned out to had some serious faults at a pretty low level with some aspectes of the engine … so first I basically rebuilt the deferred engine from scratch and learned a lot about internal stuff in the process (DeHexTD actually started out as a Test Project for that engine)

Once that engine was done I started refactoring the whole voxel-related stuff into the new engine framework.

Bad things first: It’s no longer “infinte”. I still can make worlds as big as the memory allows, but it will no longer expand infinite dynamically. In the end it was a useless feature anyway with more drawbacks than benefits. This allowed for a whole new range of optimizations and still runs fluid with worlds as big as 7x7km² (fully dynamic!)

I dropped the former Asset-Editor - the new one now supports custom meshes and actual game properties (like materials are defined as wood, and will be rendered depending on with which type of wood that asset was actually created ingame) - very cool. Even allows chaining meshes and stuff

Former Trees were dropped - the new trees are actually living procedurally on the GPU - they grow in realtime (not stepped) now and every tree on the map is actually totally unique (while still looking like a tree).

The cellular automata (doing water and light simulation) was completely removed from CPU and is now processed on the GPU - this is a HUGE benefit to performance and the cellular automata can now run permantly without the need to settle in a static state. Geometry now uses that information directly on the GPU for rendering purposes, so there is no longer need to bake anything into actual geometry.

PathFinding in the meanwhile was improved and is now featureing interconnected areas thruout the world - this massively sped up prior checks to order handling - of course, pathfinding and (structured) order handling all happening multithreaded.

And now the best of all: It’s not longer just an engine. It’s going to get an actual game. The plan is to be something inbetween gnomoria/banished/rimworld but in an actuall full 3D in a fully dynamic environment. The total amount of features is undecided yet. I plan to do a more in detailed look on the current state in some video soon.

Let me know if you have any questions, I may be happy to answer - what started out as a simple hobby project turned into something pretty complex so far

PS: follow me on twitter for some teasing GIFs in the meanwhile

[boot]

Looking good mate, I loved the non cubed look.

What algorithm are you using for pathfinding? And if I read correctly your queueing up pathfinding jobs on a separate thread?

Looking forward to your next video