[SOLVED] Issue with stencil plus shader [CODE PROVIDED]

[UPDATE] I figured it out. Mask shader and code on my last post for others to use.


Below is a basic test example of what is happening in my game with just the barebones build. It happens in my game as well as this barebones build which tells me something is happening. My guess is the shader doesn’t run on transparent pixels.

Break down of the steps:

  • Generating stencil on RenderTarget2D
  • Generate a screenwide NormalMap on RenderTarget2D
  • Pass both RenderTarget2Ds into a displacement shader. I want the transparent pixels also ran because I want the displacement to bleed into the transparent pixels.

Issue as seen in the video: Why is the NormalMap is being stenciled too?!? My thoughts are the pixel shader is ignoring the stenciled RenderTarget which are transparent.

Game1 Code:

Standard XNA displacement shader.


Thank you in advance!!!

It also works perfectly when rendering just a water square on black. Below is the effect I am trying to achieve. It just won’t work with a RenderTarget2D that was stenciled which makes no sense.

Shows it works perfectly when not stenciled: https://drive.google.com/file/d/14YVt9CS2RQO9pUyG3jLwwY5xvB4NrM92/view?usp=sharing

I swear it must be something with how the stencil is setting the color data. I did try to force the background of the stenciled image to be (0,0,0,10) but it did the same thing. I am just out of ideas at this point.


I was tired of the stencil limitations and causing oddities with alpha. So I created a masking shader that does what the stencil does but now with feathering. It solved the water effect issues. As I always try to do, here is the code to help others.


#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1

sampler2D MaskSampler : register(s0);

Texture2D SpriteTexture;
sampler2D SpriteTextureSampler = sampler_state
Texture = <SpriteTexture>;

float4 MainPS(float4 pos : SV_POSITION, float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
float4 mcolor = tex2D(MaskSampler, texCoord);
float4 tcolor = tex2D(SpriteTextureSampler, texCoord) * color;

tcolor.rgb *= mcolor.a;
tcolor.a = mcolor.a;

return tcolor;


technique Mask
pass P0
PixelShader = compile PS_SHADERMODEL MainPS();

C# Code

Texture2D Mask = Content.Load(“MaskStencil”);
Texture2D Water= Content.Load(“Water”);
Effect MaskEffect = Content.Load(“Mask.fx”);

blendState: BlendState.AlphaBlend,
effect: MaskEffect
spriteBatch.Draw(Mask, new Vector2(0, 0), Mask.Bounds, Color.White);

Mask.png (Stencil Replacement which allows feathering)

Example of output:

1 Like