Decal Shader Help

Graphics
1

I have been trying to wrap my head around getting decals to work with a deferred rendering system but I don’t understand the matrices enough to know what I am doing wrong. I have looked through many different resources and always end up with the same result where the entire cube gets culled when I run the clip portion of the code.

If anyone can help me point out what I might be doing wrong, that would be awesome! I am building this effect with DesktopGL so it is using 3.0 when compiling the effect.

When I build my depth texture for my environment, I am doing with position.w rather than position.z/position.w but I have tried both approaches and both have the same results.

Effect Source: #if OPENGL #define VS_SHADERMODEL vs_3_0 #define PS_SHADERMODEL ps_3_0 - Pastebin.com
Relevant Monogame C# Source: public void Draw(Matrix world, Matrix view, Matrix projection, float far - Pastebin.com

Side Note: I have commented out the portion in the shader that will actually draw the texture for the decal, and substituted it with just a red color. I believe this should still work and show a red texture over the area where it clips, but instead shows nothing.

1 Like
2

Could still use some help on this. I’m basing my code on GitHub - Kosmonaut3d/DeferredEngine: A deferred engine created with monogame

The only difference is I’m using SM 3.0 rather than something newer. The results I’m getting are confusing to say the least.

At this point I’ve basically copied the code but there are some differences due to the shader model version. I can’t use the input.Position from the vertex shader so I’m passing it through PositionCS, and this appears to be correct.

The portion of the shader where it seems to be incorrect is below. Before this I am about 99.999% certain the depth map is being sampled correctly, as it passes a test I ran where it will clip the portion of the cube that overlaps an area with no object behind it. It still doesn’t clip the other areas to make it a flat decal though.

	float3 cameraDirVS = input.PositionVS.xyz * (FarClip / -input.PositionVS.z);

    //compute ViewSpace position
	float3 positionVS = depth * cameraDirVS;

	//Transform to box space
	float3 positionOS = mul(float4(positionVS, 1), InverseWorldView).xyz;

	clip(1 - abs(positionOS.xyz));

Another peculiar thing is that when I change the 1 value in the clip function, it will clip the outer bounds where the cube overlaps nothing at different values, and won’t ever clip the cube if it’s overlapping anything with depth. I am using linear depth calculated with in the vertex shader for the objects it can collide with which is later written to a depth rendertarget in the pixel shader:

Output.Depth = mul(input.Position, WorldView).z / -FarClip;

This seems to work as it does correctly show different depths with my other shaders that use the depth map.

Is the FarClip not the same thing as the FarPlane distance? Could that be where I’m doing something wrong? Or is it something with my matrix calculations? Like I said, everything seems to line up correctly until the point in the code I posted.

Just in case it’s something related to it, here are the parameters I’m passing to the shader as well

Matrix local = Matrix.CreateScale(1) * brushRotation * Matrix.CreateTranslation((Vector3)brushLocation);

Assets.decalEffect.Parameters["DecalTexture"].SetValue(Assets.snow);
Assets.decalEffect.Parameters["Depth"].SetValue(depthTexture);
Assets.decalEffect.Parameters["WorldView"].SetValue(local * camera.ViewMatrix);
Assets.decalEffect.Parameters["WorldViewProj"].SetValue(local * camera.ViewMatrix * camera.ProjectionMatrix);
Assets.decalEffect.Parameters["InverseWorldView"].SetValue(Matrix.Invert(camera.ViewMatrix) * Matrix.Invert(local));
Assets.decalEffect.Parameters["FarClip"].SetValue(camera.FarPlane);
3

I feel like it’s going to be difficult to help you with this one without spending a lot of time on it.

Maybe it helps if you quickly describe how this decal method is supposed to work. I see 3 matrix parameters: WorldView, WorldViewProj, InverseWorldView. None of these has anything to do with decal projection, right? Those are just standard matrices for rendering an object. I would have expected some matrix that defines how the decal is projected onto the object.

So why not make it work with SM 4 first, and then move back to 3 if you need to.

4

Okay so I was able to figure out what is wrong, but still not able to find a solution. My error lies in the below code in the shader.

//vertex shader
output.PositionVS = mul(input.Position, WorldView);
output.Position = mul(input.Position, WorldViewProj);
output.PositionCS = output.Position;

//pixel shader
float2 texCoord;
texCoord.x = (input.PositionCS.x/input.PositionCS.w + 1) / 2;
texCoord.y = (-input.PositionCS.y/input.PositionCS.w + 1) / 2;
float depth = tex2D(DepthSampler, texCoord).x;

The way it’s written in the example I’m looking at is has the Load function that isn’t available in SM 3.0

float depth = DepthMap.Load(int3(input.Position.xy, 0)).r;

I’m not sure how to translate this and thought what I was doing was correct, but apparently not. So, for now I might test building my project with the compute_shader fork of monogame and see how that works. I was able to get the shader compiled on linux with SM 4.0 which is awesome, but now I just need to test that everything else builds and works as expected, because I would prefer to have SM4 available anyway since that is what I can find for most examples. I have a windows partition I tested the shader in with the Load function and it worked as expected, so hopefully it will work for me on this fork for building in linux.

5

So I have attempted recreating this effect with the compute_shader fork that has shader conductor in it. Everything appears to be working correctly except the rendertarget for the depth map appears to get flipped. From what I have read, this appears to be the expected behavior with OpenGL rendertargets, but I can’t find a way to sample it correctly.

When I sample it with the below code, the decal does work, but it’s grabbing the depth from the opposite Y value.

float depth = DepthMap.Load(int3(input.Position.xy, 0)).r;

However, when I attempt to sample it with the following code, the decal won’t show but I’m not sure what else there would be to invert

float depth = DepthMap.Load(int3(input.Position.x, 1 - input.Position.y, 0)).r;

6

Texture.Load() doesn’t use a 0-1 range. The coordinates are in pixels, so it’s 0 to textureSize-1.

1 Like
7

Yeah I completely overlooked that when I was reading the documentation at: Load (DirectX HLSL Texture Object) - Win32 apps | Microsoft Docs

This works perfectly with that taken into consideration. Thanks for you help and for implementing shader conductor into your forks!

image
image1920×1109 65.3 KB

2 Likes