[SOLVED] Issue with stencil plus shader [CODE PROVIDED]

[UPDATE] I figured it out. Mask shader and code on my last post for others to use.

Hi,

Below is a basic test example of what is happening in my game with just the barebones build. It happens in my game as well as this barebones build which tells me something is happening. My guess is the shader doesn’t run on transparent pixels.

Break down of the steps:

  • Generating stencil on RenderTarget2D
  • Generate a screenwide NormalMap on RenderTarget2D
  • Pass both RenderTarget2Ds into a displacement shader. I want the transparent pixels also ran because I want the displacement to bleed into the transparent pixels.

Issue as seen in the video: Why is the NormalMap is being stenciled too?!? My thoughts are the pixel shader is ignoring the stenciled RenderTarget which are transparent.

Game1 Code:
https://paste.ofcode.org/b9NWShgaHJZ7kAqW8PV4iY

Shader:
Standard XNA displacement shader.

Video:
https://drive.google.com/file/d/11GJyq6hbeA6R1zWeb_pLgwuF-j0c0bN4/view?usp=sharing

Thank you in advance!!!

It also works perfectly when rendering just a water square on black. Below is the effect I am trying to achieve. It just won’t work with a RenderTarget2D that was stenciled which makes no sense.

Shows it works perfectly when not stenciled: https://drive.google.com/file/d/14YVt9CS2RQO9pUyG3jLwwY5xvB4NrM92/view?usp=sharing

I swear it must be something with how the stencil is setting the color data. I did try to force the background of the stenciled image to be (0,0,0,10) but it did the same thing. I am just out of ideas at this point.

Thanks

I was tired of the stencil limitations and causing oddities with alpha. So I created a masking shader that does what the stencil does but now with feathering. It solved the water effect issues. As I always try to do, here is the code to help others.

Mask.fx

#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

sampler2D MaskSampler : register(s0);

Texture2D SpriteTexture;
sampler2D SpriteTextureSampler = sampler_state
{
Texture = <SpriteTexture>;
};

float4 MainPS(float4 pos : SV_POSITION, float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
float4 mcolor = tex2D(MaskSampler, texCoord);
float4 tcolor = tex2D(SpriteTextureSampler, texCoord) * color;

tcolor.rgb *= mcolor.a;
tcolor.a = mcolor.a;

return tcolor;

}

technique Mask
{
pass P0
{
PixelShader = compile PS_SHADERMODEL MainPS();
}
};

C# Code

Texture2D Mask = Content.Load(“MaskStencil”);
Texture2D Water= Content.Load(“Water”);
Effect MaskEffect = Content.Load(“Mask.fx”);

GraphicsDevice.SetRenderTarget(null);
GraphicsDevice.Clear(Color.Transparent);
spriteBatch.Begin(
blendState: BlendState.AlphaBlend,
effect: MaskEffect
);
MaskEffect.Parameters[“SpriteTexture”].SetValue(Water);
spriteBatch.Draw(Mask, new Vector2(0, 0), Mask.Bounds, Color.White);
spriteBatch.End();

Mask.png (Stencil Replacement which allows feathering)
Mask

Example of output:
example

1 Like