Fixing Billboarded Sprites Clipping into 3D Geometry

I’m working on an isometric tactical RPG (think Final Fantasy Tactics). My world is full 3D Geometry, while my units are all billboarded sprites. Generally things work fine regarding depth sorting, but when a unit is very close to geometry, it will clip into it due to “leaning back” to orient itself properly to the camera. Here is an example (please forgive some of the awesome 3D programmer art):

As you can see, three of the billboards are clipping into geometry. I’ve actually had pretty decent success fixing this by simply applying a small bias to the value written to the depth buffer by using the SV_DEPTH semantic in my pixel shader output (seen below). However, this is just a magic number arbitrarily applied and I would prefer to have a proper solution.

I cannot do something as simple as always rendering sprites on top or I lose the proper depth sorting that can be seen by the guy who is properly behind the wall. I also considered trying to do manual sorting of everything by hand and then rendering in that order to utilize the “painter’s algorithm” but it doesn’t really work if the level geometry has “holes” units can walk through since it’s all one model.

Let me know if anyone is interested in seeing the shader code. Not including it by default because I can never get the code blocks to work correctly without a massive amount of time fighting with it. Aside from a fair amount of lighting code it’s a pretty straightforward shader that simply renders a billboarded quad and changes the ZEnable and ZWriteenable flags in its technique.

Any thoughts on how this should be handled are greatly appreciated! I think the proper solution may be something along the lines of calculating the Z positions for each vertex as if the quad is not billboarded and using those in the depth buffer, but then in the shader actually billboarding for the rendering. I’ve not had any success with this approach so far however.

1 Like

Just wanted to chime in and say your sprites look great. I actually have the same problem coming up, haven’t thought of a solution yet. The depth bias is not a clean solution since the sprites will then not handle being behind a wall gracefully

Maybe you can have your characters be 3D boxes with correct collision and the sprite texture simply Drawn on these (should look the same)

The billboards are leaning backwards, but you need them to be vertical for proper depth values. I think that’s doable. You can move the vertices into the vertical plane, without affecting the rendering.

As long as you move a vertex only towards the camera, or away from it, the rendering won’t change, only depth will be affected. So it comes down to a line-plane intersection, where the line is btw. vertex position and camera position, and the plane is from the billboard without the back-leaning. Does that make sense?

Thanks for the ideas! I’ll be testing out both solutions (as well as one I got via a private message) over the next couple of days, so will post up my findings.

And thanks for the compliment on the sprites, @kosmonautgames I will pass that along to our awesome artist. He’ll be very glad to hear it!

So I’ve had some limited success and several drawbacks with this.

First off, I unfortunately cannot do your suggestion with the 3D box, Kosmo. Since we are using Spine to draw our animated units instead of a more traditional static sprite sheet, I would have basically have to render all our Spine characters to a texture in order to be able to then apply the texture to the box. Since we have a fair number of units it would end up causing us some memory issues unless I were to handle render them all to specific locations of the same texture and then choosing particular texture sections to draw.

@markus I’ve not been able to get your suggestion to work yet but I’m still working on it. Think I may just be doing my math incorrectly. If I can’t get it working tomorrow I’ll post up the code I have to see if there’s anything I’m missing.

Finally I did have some decent success with another method I was playing around with. I just pass two different World matrices down to the shader - one for rendering (the normal billboarded matrix) and one for depth( this one is oriented to the camera on X and Z but retains Up as its Y axis). This actually get me exactly the results I wanted with units sorting correctly against each other and the world, but all my units started flickering and looking like they were Z-fighting with themselves. I’m talking to the Spine team to see if there is a problem with this approach or if there’s a bug somewhere in the Spine code.

Will keep everyone posted!

The box can be invisible and only used to provide collision data with your environment.

You’d basically just make sure the units are on a grid or use the 3d box to keep them far enough off the walls.

I honestly don’t see a solution that would not require hard coding a limit to the vertical billboard. I would apply a horizontal billboard and then find out the maximum value of your vertical billboard and never let that angle go above that amount.

Doing that will ensure the sprite stays outside of the wall

Make your tiles wider and longer, make the sprites less wider and longer then the tile / squares = more leverage.
You can tilt them more that way if you really want.
(besides it looks like a tile can barely fit a sprite as it is ackk)
Keep your camera low don’t allow it to go to high or rather don’t allow the angle of the camera to go high.
A high camera will cause other perspective problems anyways.

You can manually align the sprite or use some wacky combo of billboard and others. But…

I think Matrix.CreateWorld(pos, target, up) is your best friend here.

Annoyingly i always forget which takes a position and which takes a direction CreateWorld or LookAt for the forward target parameter, either way You want to Use CreateWorld here…

The target is the camera position (or the direction from the player to the camera)
The position is the sprite
The up is the upward direction of your map.

if createworld takes a direction not a position then give it camera.position - sprite.position, instead of just the cameras position.

If you insist on leaning then create world is still nice here.

Being able to assign the upward vector with it. You can test that sprites up vector with a dot product against the up vector of the map. This will give you a resulting cosine. Multiplying that cosine by itself will give you an Acosine result. Takeing that and multiplying it by 90 degrees or PI/2 will give you the Angle you are leaning at in Degrees or in Radians in difference from the sprite pointing straight up…
You will reason the taller the sprite the more it will move outward of your tile at its top for sprites with the same angle and differing hieghts and risk clipping. I.E. you can define limits for how far back each sprite can lean.

Sorry for the late reply - I got pulled away helping our artist with an unrelated matter. Thanks for the additional replies!

@oblivion165 - I am fearful of adding collision / adjusting their in-world positions. Your approach would definitely solve this issue, but by moving sprites around or away from always being in the exact center of a grid cell, I lose the certainty of units being exactly 1 away from each other. This causes problems with some of our more advanced animations for attacking / blocking / etc that rely on this.

@willmotil - Normally the sprites are actually a little smaller that the screenshot I provided. I scaled them a little larger for testing to give sort of a worst-case for any really large creatures. I figured while I’m working on this I might as well find a solution that works regardless of the actual size of the rendered sprite since they’re not necessarily all the same size. What you’re saying makes perfect sense for limiting the billboard leaning, but I’m seeing some strange skewing when using Matrix.CreateWorld when the sprites are near the edge of the screen. I think perhaps the direction vector to the camera position isn’t playing nicely with the camera being orthographic?

The solution I’m currently using is the one I mentioned in an earlier post - where I simply send a second world matrix I use when calculating depth but otherwise leave the current rendering alone. It is the same world matrix except it uses world up as its Up vector, essentially constraining it not to lean. This seems to be working perfectly when rendering simple textured quads. The below image shows the comparison - the left is without doing this and the right is utilizing this depth world matrix.

Here is the code for how I’m doing it. Any feedback is appreciated, especially if there are any possible concerns about this working. I am still running into issues with Spine using this method, but I’m thinking this may just be an issue with Spine. Basically as soon as I output my own depth value from my pixel shaders it starts acting up, but a normal textured quad has no issues at all.

C# code for setting the shader variables and rendering (with Spine):

Matrix billboardWorld = Matrix.Invert( camera.View );
billboardWorld.Translation = _WorldPosition;

Matrix constrainedBillboardWorld = Matrix.Invert( camera.View );
constrainedBillboardWorld.Up = Vector3.Up;
constrainedBillboardWorld.Translation = _WorldPosition;

skelRender.Effect.Parameters["World"].SetValue( worldScale* billboardWorld );
skelRender.Effect.Parameters["Projection"].SetValue( camera.Projection );
skelRender.Effect.Parameters["View"].SetValue( camera.View );
skelRender.Effect.Parameters["DepthWorld"].SetValue( worldScale* constrainedBillboardWorld );

skelRender.Begin();
skelRender.Draw(skel );
skelRender.End();

And the shader code. I’ve tried to remove all the light code to make it easier to read so if anything looks like it’s missing that is why:

matrix World;
matrix DepthWorld;
matrix View;
matrix Projection;

struct VSInput
{
	float4 position		: POSITION0;
	float4 color		: COLOR0;
	float2 texCoord		: TEXCOORD0;

	//Passed down but unused
	//float4 color2		: COLOR1;
};

struct VSOutput
{
	float4 position		: SV_Position;
	float4 color		: COLOR0;
	float2 texCoord		: TEXCOORD0;

	float depthValue : TEXCOORD1;
	float3 WorldPos : TEXCOORD2;
};

sampler TextureSampler : register( s0 );

VSOutput VSQuad(VSInput input)
{
	VSOutput output;

	float4 worldPosition = mul(input.position, World);
	float4 viewPosition = mul(worldPosition, View);
	output.position = mul(viewPosition, Projection);

	float4 depthWorldPosition = mul(input.position, DepthWorld);
	float4 depthViewPosition = mul(depthWorldPosition, View);
	output.depthValue = mul(depthViewPosition, Projection).z;

	output.texCoord = input.texCoord;
	output.color = input.color;

	// Need the world position for lighting effects in Pixel Shader
	output.WorldPos = mul(input.position, World).xyz;

	return output;
}

struct PSOutput
{
	float4 color : SV_Target;
	float depth : SV_Depth;
};

//float4 PSQuad(VSOutput input) : SV_TARGET
PSOutput PSQuad(VSOutput input)
{
	float4 baseColor = tex2D( TextureSampler, input.texCoord );
	baseColor *= input.color;
	clip(baseColor.a < 0.8f ? -1 : 1);

	// Lighting Code

	PSOutput output;
	output.color = baseColor * float4(totalLight, 1.0f);
	output.depth = input.depthValue;
	return output;
}

float4 PSQuadAlpha(VSOutput input) : SV_TARGET
{
	float4 baseColor = tex2D(TextureSampler, input.texCoord);
	baseColor *= input.color;
	clip(baseColor.a >= 0.8f ? -1 : 1);

	// Lighting Code

	return baseColor * float4(totalLight, 1.0f);
}


technique CharacterLit
{
	pass P0
	{
		AlphaBlendEnable = false;
		ZEnable = true;
		ZWriteEnable = true;

		VertexShader = compile vs_4_0 VSQuad();
		PixelShader = compile ps_4_0 PSQuad();
	}
	pass P1
	{
		AlphaBlendEnable = true;
		ZEnable = true;
		ZWriteEnable = false;

		PixelShader = compile ps_4_0 PSQuadAlpha();
	}
}

Oh, it’s an orthographic camera. The method I proposed is for perspective cameras. If you change the direction vector for the line it should also work for orthographic cameras. The line still originates at the world position of the vertex, but it’s direction isn’t towards the camera position, instead it’s just the camera’s direction vector.

A small problem with your current method is that the vertices from the constrainedBillboard will not end up in the exact same screen location as the vertices from the regular billboard. So you are effectively calculating the depth value at the wrong place. As long as the camera angle is not too steep, that error will be small though.

Having thought about it a little more, it seems there’s an even simpler solution, if the billboards are guaranteed to always face the camera, which I think they are.

For orthographic cameras the backwards leaning of the billboard is equivalent to a vertical stretch, nothing more should be happening. So you shouldn’t even need a back-leaning world matrix at all. Just use an upright matrix with the up vector scaled properly to account for this vertical stretch. Visually this should be indistinguishable from a backwards rotation.

I’m pretty sure the scale factor is simply 1/camUp.Y (assuming the camera doesn’t roll/bank), so if you just change this line in your code

constrainedBillboardWorld.Up = Vector3.Up;

to

constrainedBillboardWorld.Up = Vector3.Up / constrainedBillboardWorld.Up.Y;

and then use this matrix as your only world matrix for rendering, you should get the desired effect. At least in my brain it works perfectly, it is a little late though :slight_smile: