Pixel shader help, order of tex2D matters?

[Laurie]

Hello,

I am starting out with pixel shaders and wondering if anyone can help explain a behavior I am failing to understand.

I have a basic shader that draws a sprite, using another texture’s alpha value as a mask.

float4 MainPS(VertexShaderOutput input) : COLOR
{
float4 Color = tex2D(SpriteSampler, input.texCoord) * input.Color;
float4 maskColor = tex2D(MaskSampler, input.texCoord);
Color.a = maskColor.a;
return Color;
}

However this result completely changes if I swap the order of the two lines…

float4 maskColor = tex2D(MaskSampler, input.texCoord);
float4 Color = tex2D(SpriteSampler, input.texCoord) * input.Color;

I was wondering why this order matters? The full code is below.
Cheers!

#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

sampler2D SpriteSampler;

Texture2D Mask; 
sampler2D MaskSampler
{
	Texture = <Mask>;
};


struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 texCoord : TEXCOORD0;
};

float4 MainPS(VertexShaderOutput input) : COLOR
{
	float4 Color = tex2D(SpriteSampler, input.texCoord) * input.Color;
	float4 maskColor = tex2D(MaskSampler, input.texCoord);
	Color.a = maskColor.a;
	return Color;
}

technique SpriteDrawing
{
	pass P0
	{
		PixelShader = compile PS_SHADERMODEL MainPS();
	}
};

////////////////////////////

frag_shaders.AlphaShader.Parameters["Mask"].SetValue(glow_img);

spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.NonPremultiplied, effect: frag_shaders.AlphaShader);
     spriteBatch.Draw(wall_img, cursor_pos, null, Color.White, 0, Vector2.Zero, 4, SpriteEffects.None, 0);
spriteBatch.End();

[willmotil]

First is this a dx or gl project ?
Second are you saying all you are doing is swapping the order of the calls in the shader ?

float4 MainPS(VertexShaderOutput input) : COLOR
{
	float4 Color = tex2D(SpriteSampler, input.texCoord) * input.Color;
	float4 maskColor = tex2D(MaskSampler, input.texCoord);
	Color.a = maskColor.a;
	return Color;
}

to

float4 MainPS(VertexShaderOutput input) : COLOR
{
    float4 maskColor = tex2D(MaskSampler, input.texCoord);
	float4 Color = tex2D(SpriteSampler, input.texCoord) * input.Color;
	Color.a = maskColor.a;
	return Color;
}

[Laurie]

Thanks, it is dx.
Yeah the only change I make is swapping around the tex2d order exactly as you posted.
The result changes from the sprite being masked correctly to just drawing the sprite with all black pixels.

[vermiliontower]

That is pretty weird. I can think of a couple things to try:

• I don’t see where you’ve defined your sprite texture input. Maybe try doing that and defining your sampler in the same way you’re doing the MaskSampler.
• You can tag your textures and samplers with register values to make sure they get matched to fixed positions. For textures, you do Texture2D Mask : register(t1); and samplers do sampler : register(s1) (the number after the t and s specify the register index).

[Stainless]

Sounds like the texture register is changing

SInce you are not defining the texture for the spritesampler the compiler is free to change the assignment order

try adding
Texture2D Mask : register t1;

[Laurie]

Thanks for all the replies everyone!

I’ve tried defining registers and a sprite texture but I still have issues I am afraid.

Using the register setup below alters the image being drawn. Instead of the image passed in spritebatch.draw() being draw, it now draws the Mask image (passed as a texture parameter) in all black. The order of the Tex2d calls no longer effect the result as shown in the images above.

Texture2D SpriteTex: register (t0);
sampler2D SpriteSampler:register (s0)
{
	Texture = <SpriteTex>;
};

Texture2D Mask : register (t1);
sampler2D MaskSampler :register (s1)
{
	Texture = <Mask>;
};

However, now that the sprite texture has its own definintion “SpriteTex” I decided to see what happens if I set that parameter in the code passing a wedge shaped image.

        frag_shaders.AlphaShader.CurrentTechnique = frag_shaders.AlphaShader.Techniques["SpriteDrawing"];
        frag_shaders.AlphaShader.Parameters["SpriteTex"].SetValue(wedge_img);
        frag_shaders.AlphaShader.Parameters["Mask"].SetValue(glow_img);
        spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.NonPremultiplied, effect: frag_shaders.AlphaShader);
            spriteBatch.Draw(wall_img, cursor_pos, null, Color.White, 0, Vector2.Zero, 4, SpriteEffects.None, 0);
        spriteBatch.End();

Which produces this results (reguardless of the Tex2d orders).

Now the shader is actually working correctly, but instead of using the texture passed to spritebatch.draw it is stuck to using the texture from .Parameters[“SpriteTex”].SetValue(wedge_img); even when drawing multiple sprites (right image).

I thought spritebatch.draw would overwrite t0,s0 but maybe this is not the case?

I then tried only setting registers for the mask’s texture and sampler
(while still performing “.Parameters[“SpriteTex”].SetValue(wedge_img)”)

Texture2D SpriteTex;
sampler2D SpriteSampler
{
	Texture = <SpriteTex>;
};

Texture2D Mask : register (t1);
sampler2D MaskSampler :register (s1)
{
	Texture = <Mask>;
};

This then resulted in the below images where once again the order of the Tex2d calls changes the result.

One order (Left below) correctly masking the image from the spritebatch.draw call.

The other order (right below) seems to use the image from spritebatch.draw (wall_img) as the Mask in the shader instead of that from “Parameters[“Mask”].SetValue(glow_img);” and draws the image from .Parameters[“SpriteTex”].SetValue(wedge_img);

Sorry for the long post but I am still somewhat mythed by what is going wrong.

[AlienScribble]

I’ve seen a few ppl come across this before. I could be wrong but this is my interpretation:
If you’re using spriteBatch (sets Texture[0]):
As a rule, as long as the first thing you sample in the pixel shader is from the first sampler, everything should work ok in the shader. So you may even be forced to do a dummy-sample on the first line in the pixel shader if it won’t be used at first. I never seem to run into the problem because I always end up sampling that first anyway. (lucky so far). ;p
I’m guessing it has something to do with registers - maybe even some weird optimization glitch too (involving order) - and I associate it with setting Texture[0] which I believe SpriteBatch does. (maybe unrelated) [same thing happens if I do it in my own custom SpriteBlast.
However I otherwise don’t notice this problem in other shading stuff so far - however in those I always use Set Param for every “texture”

[willmotil]

You can use samplerstate instead of sampler2d to get around this.
It’s less intuitive and a bit uglier.

color in the shader below equates with the texture passed to spritebatch.Draw( texture, … )
the other textures are used with the sampler but called with t.Sample( texturesampler, …

// instead of something like this.

//sampler2D SpriteTextureSampler : register(s0)
//{
//	Texture = (Texture);
//};
//sampler2D SpriteMultiTextureSampler = sampler_state
//{
//	Texture = (SpriteMultiTexture);
//};
//sampler2D SpriteStencilTextureSampler = sampler_state
//{
//	Texture = (SpriteStencilTexture);
//};


// you can use samplerstate instead of sampler2D
// in this case i use just one for all the textures.

texture2D SpriteMultiTexture : register (t1);
texture2D SpriteStencilTexture  : register (t2);

SamplerState TextureSampler = sampler_state
{
	Texture = (Texture);
};

// this requires you to use a different sampling call in the pixel shader.
// sampling from the texture then using the defined samplerstate.
// that is in addition to using the tex2D(..) call to grab the passed draw texture.

float4 MaskAndOverlayPS(VertexShaderOutput input) : COLOR
{
	//float4 color = tex2D(SpriteTextureSampler, input.TextureCoordinates);
	//float4 overlayColor = tex2D(SpriteMultiTextureSampler, input.TextureCoordinates);
	//float4 stencilColor = tex2D(SpriteStencilTextureSampler, input.TextureCoordinates);

	float4 overlayColor = SpriteMultiTexture.Sample(TextureSampler, input.TextureCoordinates);
	float4 stencilColor = SpriteStencilTexture.Sample(TextureSampler, input.TextureCoordinates);
	float4 color = tex2D(TextureSampler, input.TextureCoordinates);

	...

Yes spritebatch moves stuff around i can’t say off the top of my head what exactly it is doing though to mess that up.

[Laurie]

Sorry for the super late reply!
Thanks this was all super useful information
I finally got around to doing more shader work and bearing everything you said in mind I have not run into any more issues and things are working as expected.
Thanks Again