I have put a load of post processing (including my god ray’s) into my MG repo, I have a thread on it here 
Hope it helps 
Time for some follow up
My next big thing to do was actually doing “ramps” … and I am tinkering with this now for a very long time. As I use Baked AO - which is simple if a world only consists of cubes - but having ramps and corners heavily complicates that stuff.
I am actually not really working with cubes, but with Quads, and my geometry creation is basically a very big state machine, building the faces dependend on moore neighbourhood. The same neighbourhood is used to determine current light exposure and baked AO. Until before, the neighbour was either a block or not - now with ramps, things get more complicated, because the decision if a pixel is in AO or not depends not only if there is a block, but what type of block it is (ramp cube, corner ramp) and what direction that ramp is facing. So that’s a LOT of things to code in a state machine …
In the process I found that my old geometry builder was not suited well and so I refactored it from ground - it’s now faster and easier to handle new sorts of block types. As you can see in the screenshot, I am not finished with the AO and there is still AO where the ramp meets the floor, where there shouldn’t be AO. I am currently still thinking about different solutions to make this process less tedious than adding just another check in the creation and redo this for at least 8 other types … uuuhm.
In an earlier prototype of this engine, I played around with SSAO - but the results were not satisfying - but I may give it another try, because it may solve other issues (like AO at trees, avaters, assets etc) - I am not sure if Baked AO is the way to go here. Baked AO on a 4vert Quad also has some visual drawbacks which are impossible to solve without increasing vertex count.
So it may take some time until I come up with a new video I guess, let’s see if I can get a better SSAO to work, it would be pretty cool to have actually
So long - happy tinkering
3 Likes
I actually sacked the baked AO - it was just not working out for what I’d imagine. So I turned over to SSAO again and fight with all the quirks associated with it. Main problem (for me) was mainly that I am overwhelmed with several things like handness, view space and position recreation, TBN multiplication order, halfpixels and some hard to spot errors in available tutorials.
Anyway - I am still not definitely finished with getting everything right, but at least I got a result which I found pretty pleasing and reassemles the baked AO pretty well!
I’ve also completely redesigned my geometry builder (again) to make it more easy to expand it with different Block Forms (and still work together with per-block-lighting, selections, pathfinding etc) - as a bonus I was able to squeeze a bit more speed out of it as well (the geo builder runs single threaded for several reasons) - I now have also built in ramps and ramp corners and already have several other blocktypes in mind
Here is a quick screenshot of how it looks now (I’ve forgotten to add some corners as well… ) - I may do another progress video soon to show some current material, so stay tuned
I have to say it’s already funny to build stuff … anyone knows some artist which is good in making fitting textures for me (paid, free, doesn’t matter), I’ve just made mine up and they don’t tile very good, normals are a bit odd because generated from color) but I like the pixel-alike-but-not-really-pixel look of it
5 Likes
The recent day, I’ve worked on several blocktypes. This involved making a mechanism of easily adding them (where I used a trick with T4 functionality because C# lacks the ability to define macros and reuse code snippets without having to waste function calls) which was need to keep the geo generation algorithms fast.
The next thing to do was actually having the engine create all the new ramps with different orientations from a voxel-procedural map (basically combined noise maps). I heavily work with bitfields, bit-wise operators and predefined bitpatterns to keep everything fast
As a result of that work (which was a lot!) everything looks quite smooth now and the new block types also gives a lot of possiblities to actually “build” in the world. What do you think?
I’ve already adapted to water simulation (automata) to correctly handle the ramp blocks and - beside some remaining bugs - the light automata as well (for per-bock-lighting)
Another problem occuring is now, that my current textures (nothing fancy) do not blend well in triangles and other non-voxel-situations, because the borders do not align. I am currently thinking of having the voxel texture consist of several sub textextures for different types of sides (4sided, 3sided, top texture) - this puts more time into working on textures (postponing this for much later, because I don’t enjoy doing textures)
I am also trying out to post on tumbler - so if you are interested in smaller but more frequent update chunks, just find me there and follow
3 Likes
Finally I’ve done a new presentation video to show the current engine in moving action. Let me know how you like it
next video will feature something neat - stay tuned
1 Like
The last days I was very busy with doing more optimizations. The new ramps added a bit more special faces which are now properly not generated when occluded by neighbouring blocks (all combinations).
I also improved the pathfinding even further and incooperated diagonal movement over flat surfaces - still open to do the same for diagonal ramps (lots of checks)
another not-yet-to-be-told-feature is waiting to be presented in a video
had my first tries with actual job handling. As this engine shall be a builder/rpg sort of engine, there has to be jobs assigned. Much work to do for this one, as it’s quite complex. i.e: Mark a block for digging, agent picks up the job. No to actually dig a block, the agent have to find a place beside that block to do the work - the ideal way would be to check every possible “standing position” with pathfinding and get the nearest one. But that’s to heavy on the Path Solver - I may give it a try with using heuristics instead, let’s see how that’ll work out
1 Like
I’ve finally made a new video, showing the not-to-be-told-feature which is basically (just basically) working now: It’s the Ingame Asset Editor - it mainly was the reason why I wanted the different Ramp Blocks:
I am currently working on the actual Job/Order System, which is a bit more tricky than I actually supposed. I first tried a naiv approach like so, that each agent, when idle, will search the job queue for the next job. Each job first finds general possible places where the agent can stand to do the job, then it’s handed to the agent and the agent tries each of them if it can reach on of those spots. If not the job goes back as failed and will be picked up later on.
While this approach somehow works it’s not ideal and can lead to weird things and has no sense of which agent is most appropriate to use, beside the fact, that it clumps up the update loop a bit (pathfinding is multithreaded, but all the handling takes a bit) - so what I am now trying to do is turn things around and let the jobhandler reside in its own thread, regulary checking new jobs, sorting the orders by priority and distance to the nearest agent. This all will take a significant bit of performance, so it can only rely in a separate thread in order to not harm the FPS.
Let’s see how that’ll turn out
1 Like
Looks like I finally ended up with something which does work not so bad. I am using an asyncronous runnign “Job Manager” which basically cares for assigning jobs to idle agents. To do there is some job queue for manually issued jobs and if this is empty, the Job Manager queries different zones, if they have job to do.
A stockpile is such a zone, and if queried it will determine if there are any ressources lying around which need to be hauled and if so, they will generate a job which is then assigned to the next idle agent. This should basically keep the queue of “open jobs” relatively small. There is several object locking and caching arrays involved for everything to work well with the renderer and keeping the impact on the framerate as low as possible, but under some circumstances (i.e. a fully locked in agent) will still clog everything up a bit. I do need to make a lot of compromisses between object locking and performance and was quite busy with a lot of bugs evolving around that. In the video some of those bugs a re still visible but most of them are already fixed.
Another issue with this system currently is, that the items in the world (which ressources are) are currently a “global list” together with partioned references for them on every chunk. While this has benefits in some situations I am not enterily sure if this will be viable in a highly populated world and so this may change to a fully partioned system with the drawback of having any lookups being slower in some cases.
Current tests with a lot of agents worked out pretty well, beside the fact, the more agents are assigned a job, the longer it will take until everyone has finished it’s pathfinding, but there is no way around it and with complexer environments this will slow down even further.
In the current system, there are sort of Job-Chains. So each job can basically have follow-up-jobs defined during creation of the job itself. This is in order to lock/reserve any involved resources. In the example of hauling, the stockpile creates a job, reserves a space for it and locks the item on the map. An agent will now have the order to pick up that item and a follow up job, to drop down the item to a specific spot.
Problems arises, when such a job chain needs to be canceled for whatever reason (agent died, way blocked) as now the full chain needs to be processed in order to free up any reserved spots or items. With follow up items that is pretty straight forward, but there may be situations where this is not enough. Need to see if it’s sustainable.
Anway, still a lot of things to play around with and I am eager to actually see the agents building something, felling trees or seeing stacked items displayed differently or having storage containers.
7 Likes
Progress:
Alright - so some weeks ago I stumbled over mutliple performance issues, which basically dropped the framerate heavily to a point where I didn’t consider it to be playable. It was frustrating and I put it away for some time.
After some time I decided to displace some code into an externall native c++ DLL to speed things related to noise generation up - this worked a bit but I also had to do a lot of result chaching in order to really eliminate that bottleneck.
The engine now also works with 4 concurrent threads to do several things asyncronoulsy, like path finding, order/job handling and such things - which brought HUGE problems in terms of syncronizing and weird errors which were hard to find the initial issue for. Most work on this engine is heavily related to performance, this really is the main work.
Regular “locking” will just not work out, as it’s slowing down everything up to a point where the framedrops get noticeable, so what the engine actually does is working a lot with chaches multiple objects and syncrinizing those on demand from the threads themselves.
Anyway - I think the basic thing in zoning/logistics is done in the engine. Agent’s are able to get a job/order assigned an able to fulfill whatever they are supposed to do - there are still some problems and hickups in terms of “unexpected behaviour” of the agents but they will be tackled when they arise 
I did another Progress Video, which is mainly related to zoning and order/job fulfillment. The engine can (chained) issue jobs/orders and add/remove zones and building tasks now, what do you think?
1 Like
I actually missed to report a new progress video - in the last video I actually showed the newly farming feature and also let the agents build a house by only using orders (the supposed way to do it). I’ve also felt the current lights look a bit odd and decided to rapidly code some fire and light jitter as well
Most of this is actually mocking up a new feature, making it work and then followed by some days refactoring smaller or larger parts of the engine to get better performance or readability (ideally both, but that’s not always possible here)
Anyway - I am currently tackling the cumbersome approach of “saving the gamestate” - yes I could just use binary serialization and use one of the benefits of .Net … but honestly - I fear it will be to slow and the result would be just too big files. I also may be adding some sort of different saving approaches (in terms of speed for loading/unloading inactive but modified chunks, etc) - so I jumped the train and did a fully manual(!) approach of saving/loading the current game state.
This engine is a bit different from regular voxel engines - being mainly for building games. So there is a lot of references handed around over several threads and - oh boy - this is not funny to code a serialization for
At least I am currently at a spot where saving loading basically works and I am just searching/fixing several bugs which appear during testing. There is no video for that, because there is basically nothing to show with loading/saving (it’s one of those things, everyone expects to just be there).
As soon as this is in a complete state, I really want to push production chains, like chopping a tree to get wood (already working), making planks and using planks to make a chair (for example).
I’ll keep you informed
1 Like
Another week, another progress
I basically had the idea of having shadows in my game. The problem with that was actually, that my current rendering pipeline consisted of just a very big function plotting out all the needed data to the shaders - which was not feasible when having to do it twice because of the shadow map
So I used quite some time to completely redesign my render pipeline which is now much more managed, collecting mesh and rendering data in advance into several buckets (batching by shader, texture etc) and rendering that buckets afterwards. I implemented Exponential Shadow Mapping as I do have most data in the defered renderer anyway.
One issue in respect to my engine is still the problem of underground chunks and that’s messing a bit with my spacial partitioning (as currently I basically render the chunks visible for the sun … ) - so I am still in the process to split this up a bit more in the hope the impact on rendertimes isn’t that hard. (but rendertimes are not really a problem in my engine, it’s the stuff around it)
Anyway. in terms of mechanic I finally implemented the “item building” mechanic into the engine (building items, the player can actually design themselves in an ingame item editor). Some other features were distinct names for the thingys and rendering a shoreline (which was one of the few easy things to do)
Also Loading/Saving is now fully working without issues, I just have to track any changes to game data to be reflected in the save/load definitions/interfaces
If you are interested, here is the current progress video:
3 Likes
Over a year later … time for some sitrep.
the engine turned out to had some serious faults at a pretty low level with some aspectes of the engine … so first I basically rebuilt the deferred engine from scratch and learned a lot about internal stuff in the process (DeHexTD actually started out as a Test Project for that engine)
Once that engine was done I started refactoring the whole voxel-related stuff into the new engine framework.
Bad things first: It’s no longer “infinte”. I still can make worlds as big as the memory allows, but it will no longer expand infinite dynamically. In the end it was a useless feature anyway with more drawbacks than benefits. This allowed for a whole new range of optimizations and still runs fluid with worlds as big as 7x7km² (fully dynamic!)
I dropped the former Asset-Editor - the new one now supports custom meshes and actual game properties (like materials are defined as wood, and will be rendered depending on with which type of wood that asset was actually created ingame) - very cool. Even allows chaining meshes and stuff
Former Trees were dropped - the new trees are actually living procedurally on the GPU - they grow in realtime (not stepped) now and every tree on the map is actually totally unique (while still looking like a tree).
The cellular automata (doing water and light simulation) was completely removed from CPU and is now processed on the GPU - this is a HUGE benefit to performance and the cellular automata can now run permantly without the need to settle in a static state. Geometry now uses that information directly on the GPU for rendering purposes, so there is no longer need to bake anything into actual geometry.
PathFinding in the meanwhile was improved and is now featureing interconnected areas thruout the world - this massively sped up prior checks to order handling - of course, pathfinding and (structured) order handling all happening multithreaded.
And now the best of all: It’s not longer just an engine. It’s going to get an actual game. The plan is to be something inbetween gnomoria/banished/rimworld but in an actuall full 3D in a fully dynamic environment. The total amount of features is undecided yet. I plan to do a more in detailed look on the current state in some video soon.
Let me know if you have any questions, I may be happy to answer - what started out as a simple hobby project turned into something pretty complex so far 
PS: follow me on twitter for some teasing GIFs in the meanwhile
2 Likes
Looking good mate, I loved the non cubed look.
What algorithm are you using for pathfinding? And if I read correctly your queueing up pathfinding jobs on a separate thread?
Looking forward to your next video
I am using some sort of navigation structure, which splits the world into areas and maintains links to every connected area - this results in a smaller graph and works as abstraction layer for the world (and even allows for detecting rooms and dynamic doors) - this graph is used for early pathing checks mainly. For actual pathfinding a modified A* is used on block-level which runs in a separate thread.
Currently the graph does not find the shortest path as it does not provide any heursistics over its nodes. But the early checks help a lot
The way this is all connected to the main thread is basically some “messaging queue”. For example, when an entity get’s an order to build something, it first issues a search (message to thread) which then finds the closest item of the specified type which has a path to the target and then returns the actual path - the agent then walks this path and so on - so the other thread is basically a task/msg handler. And because all this happens in the other thread I can mostly eliminate syncronization issues and don’t need to “lock” everything.
there is a LOT of optimization in there
Thats very clever. When I implemented A* on larger worlds (4000x4000 tiles), i ran it on a separate thread using a queue system. It generated large amounts of garbage and gc spikes when moving a couple hundred units at once. I ended up trying to get the algorithm as memory efficient as possible and doing jump searches and did get rid of the GC spikes
But I think I might have to try your idea of dividing the world into smaller areas. Im guessing the pathing won’t be as ‘best possible’ but will be a lot faster and easier on the garbage collector.
Are you going to be making some kind of needs based ai tree for your agents to do things?
Love your work mate
I basically try to avoid dynamic lists whereever possible. My a* information sits directly in the datagrid portions and for finding out if a block was already visited I just set an integer which is incremented each run. Of course that means I cannot do concurrent path searches but that doesn’t matter for me.
It’s basically optimized for performance on every possible location, I am actually quite impressed how fast you can make .Net
I’ve always been curious about this Minecraft style voxel. Is it related to old MSDOS games that also used voxels. Is this the same technology?
the second one does not relate to the first screenshot - but no, that was a bit differently. In the first screenshot (which was originally called “voxel”) - you actually plotted lines from back to front to make the illusion of 3D - this was mainly used for heighmaps - that was appears to be pixels are alway axis align to the monitor.
in the second screenshot, the “pixels” are not screen aligned, so I guess that’s something else.
The Voxels in Exipelago … are actually a representation of big cubes (in Exipelago - which is what this engine transformed into) there is all sorts of shapes, each representing a differnt type of “voxel” - ultimately this all is generated into an optimzed mesh for performance reasons and then sent to the GPU as geometry.
A “true” Voxel engine nowadays does more send data to the GPU and builds geometrical information there on the fly - which is different again but allows for all kinds of cool stuff.
In my game, I do need proper information about the geometry created (for game mechanics) so that is all stored and created on the CPU side
The second screen shot is from a Build Engine game. During the 90’s when it was created, they said it supported voxels. That cross was an example of voxels in their engine. It seems the term is used very loosely.
Look into The Making of Desert Strike, I believe it was called, will explain that technique in detail as it was what it was known for…