Could still use some help on this. I’m basing my code on GitHub - Kosmonaut3d/DeferredEngine: A deferred engine created with monogame
The only difference is I’m using SM 3.0 rather than something newer. The results I’m getting are confusing to say the least.
At this point I’ve basically copied the code but there are some differences due to the shader model version. I can’t use the input.Position from the vertex shader so I’m passing it through PositionCS, and this appears to be correct.
The portion of the shader where it seems to be incorrect is below. Before this I am about 99.999% certain the depth map is being sampled correctly, as it passes a test I ran where it will clip the portion of the cube that overlaps an area with no object behind it. It still doesn’t clip the other areas to make it a flat decal though.
float3 cameraDirVS = input.PositionVS.xyz * (FarClip / -input.PositionVS.z);
//compute ViewSpace position
float3 positionVS = depth * cameraDirVS;
//Transform to box space
float3 positionOS = mul(float4(positionVS, 1), InverseWorldView).xyz;
clip(1 - abs(positionOS.xyz));
Another peculiar thing is that when I change the 1 value in the clip function, it will clip the outer bounds where the cube overlaps nothing at different values, and won’t ever clip the cube if it’s overlapping anything with depth. I am using linear depth calculated with in the vertex shader for the objects it can collide with which is later written to a depth rendertarget in the pixel shader:
Output.Depth = mul(input.Position, WorldView).z / -FarClip;
This seems to work as it does correctly show different depths with my other shaders that use the depth map.
Is the FarClip not the same thing as the FarPlane distance? Could that be where I’m doing something wrong? Or is it something with my matrix calculations? Like I said, everything seems to line up correctly until the point in the code I posted.
Just in case it’s something related to it, here are the parameters I’m passing to the shader as well
Matrix local = Matrix.CreateScale(1) * brushRotation * Matrix.CreateTranslation((Vector3)brushLocation);
Assets.decalEffect.Parameters["WorldView"].SetValue(local * camera.ViewMatrix);
Assets.decalEffect.Parameters["WorldViewProj"].SetValue(local * camera.ViewMatrix * camera.ProjectionMatrix);
Assets.decalEffect.Parameters["InverseWorldView"].SetValue(Matrix.Invert(camera.ViewMatrix) * Matrix.Invert(local));