Alpha Mask Effect

Hey guys!

I am trying to do a simple black/white alpha mask on a spritebatch. I have a spritebatch only for particles, and I want to limit them to a square on the screen. Simplest way to do it would be just to draw something over them, but I cant do that, since I draw other stuff aswell.

Basically what I want to achieve is to create an Effect, that will check an alpha mask and a pixel that is currently being drawn, then based on if the mask on current location is white or black draw the pixel or draw a transparent pixel.

I have found some stuff, but nothing seems to be working for me. Example from doesnt do anything, just makes everything transparent. The most I got it to work is to draw white boxes instead of textures.
Is there any examples of how to do this, or can any of you give me some advice?

Oh and one more question… Is the position (in the PixelShaderFunction (float2 inCoord: TEXCOORD0)) relative or based from the top left corner of the screen?

I’m kinda new to MonoGame, so this is probably really easy :stuck_out_tongue:

Thanks in advance!!

You can just multiply the color of the pixel with the value (any component) of the B/W alpha texture. That way black will multiply by 0 (transparent) and white by 1 (no change).
So in the pixelshader that would mean something like

return tex2D(Texture, position) * tex2D(Mask, position).x;

Thanks!! but it still doesnt want to work, nothing gets drawn (or all transparent) :frowning:

Here is my code right now:

uniform extern texture ScreenTexture;
sampler tex = sampler_state
	Texture = <ScreenTexture>;

uniform extern texture MaskTexture;
sampler mask = sampler_state
	Texture = <MaskTexture>;

float4 PixelShaderFunction(float2 pos: TEXCOORD0) : COLOR
	return tex2D(tex, pos) * tex2D(mask, pos).x;

	pass P0
    		PixelShader = compile ps_4_0_level_9_1 PixelShaderFunction();

And I call it with:


spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.Additive, null, null, null, alphaMapEffect);

currentScene.DrawMasked(spriteBatch); // draws all of the particles


The MaskTexture is a texture (1920x1080), with white where I want the particles to be drawn and black where I want them to be hidden.

Any ideas? :disappointed_relieved:

Maybe because you have an alpha channel ?
float4 res =tex2D(tex, pos).rgb * tex2D(mask, pos).r;
res.a =0; //if you want to test without alpha at all
return res;

When you have built the effect the tool should have warned you that you were using implicit values.
Morreover if you use additive the things that should be black will be… As they were before. A b&w mask is rarely used with additive state.
Try with opaque state instead.

Have you tried using a tecture witbout alpha channel like jpg insteaf of png ?

Ps: i m using my smartphone so i am sorry if there are capitals lurking around in the code example

Thanks Alkher! :slight_smile:
Still no luck though…
This is my code now, the PixelShader function:

float4 res = tex2D(tex, pos);
res.r *= tex2D(mask, pos).r;
res.g *= tex2D(mask, pos).r;
res.b *= tex2D(mask, pos).r;
res.a = 0;

return res;

I also changed to opaque and jpg instead of png.
With this code, only black boxes are drawn on the screen.
Even if i just do return tex2D(tex, pos); I get black boxes… so I think there is something else wrong?
Maybe it’s the positioning of the mask, since the mask starts on (0, 0) on the screen and the particles are at like (500, 300). But I have no idea how the “float2 pos: TEXCOORD0” is formatted… relative to the texture it is drawing (so the particle) or to the whole screen?

This is more and more confusing to me…

The following effect draws everything transparent:

sampler s0;
float4 PixelShaderFunction(float2 coords: TEXCOORD0) : COLOR0
	float4 color = tex2D(s0, coords);
	return color;  //return float4(1, 0, 0, 1); creates red boxes

technique Technique1
    pass Pass1
        PixelShader = compile ps_4_0_level_9_1 PixelShaderFunction();

In Game1:

spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend); 
//BlendState.Opaque = black box for player, so I guess it gets drawn, just transparent
    //draws the player texture at (500, 500) on 1080p screen


Any ideas?

Textures are mapped from 0,0 to 1,1 :wink:
One question: what do you want to do with the black zones ? is it only b&w or grayscale ? display them ? your goal is not clear for me.
If no, you can do something like this to totally avoid drawing mask (ie which should not be visible):
if(tex2D(mask, pos).r > sometresholdbetween0and1) clip(-1); //or discard(); or simpler: clip(tex2D(mask, pos).r - sometresholdbetween0and1);

If you want to make black zones still visible like being a little darker:
res.a = tex2D(mask, pos).r; //Maybe.

In your code did you test:
return color; //return float4(1, 0, 0, 0.5f); should create red boxes half transparent/opaque

If you multiply by .r, then you should use additive state if no alpha is used. (but to me a black and white mask is not meant to be added)

Okay, it’s getting somewhere.

What i want to have is textures present only in one rectangle on the screen. Everything else should have no textures.

I tried avoiding the mask…

float4 tex = tex2D(s0, texCoord);
float coordX = texCoord.x * 1920;
float coordY = texCoord.y * 1080;

if (coordX >= 755 && coordX <= 1154 && coordY >= 125 && coordY <= 1032)
	return tex;
	return float4(0, 0, 0, 0);

this works, but the rectangle is applied to each texture instead of the screen…

like this: (on the right image it should say: “… to each texture, not to the screen”)

Any idea on how do I get screen coordinates for each pixel?


matrix	MatrixTransform;
struct VertexShaderOutput
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 TexCoord : TEXCOORD0;
	float4 Screen : TEXCOORD1;
VertexShaderOutput MainVS(float4 position:SV_POSITION, float2 texcoord : TEXCOORD0, float4 color : COLOR0)
	VertexShaderOutput output;
	output.Position = mul(position, MatrixTransform);
	output.Screen = output.Position;
	output.Color = color;
	output.TexCoord = texcoord;
	return output;
float4 MainPS(VertexShaderOutput input) : COLOR
	float2 screen = input.Screen.xy / input.Screen.w;
	screen = (screen + 1.0) / 2;
	screen.y = 1.0 - screen.y;
	float4 re = tex2D(spriteSampler, input.TexCoord);
	float4 mask = tex2D(maskSampler, screen);
	if (mask.r <= 0)//black
	return re;
technique BasicColorDrawing
	pass P0
		VertexShader = compile VS_SHADERMODEL MainVS();
		PixelShader = compile PS_SHADERMODEL MainPS();


 Matrix projection;
 Matrix.CreateOrthographicOffCenter(0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, 0, 0, -1, out projection);
 Matrix _matrix = Matrix.Identity;//matrix use in spriteBatch.Draw
 Matrix.Multiply(ref _matrix, ref projection, out projection);



float2 Sprite;
float2 SSize;
float4 AltPS(VertexShaderOutput input) : COLOR0
	float2 screen = Sprite + SSize*input.TexCoord;
	float4 re = tex2D(spriteSampler, input.TexCoord)*input.Color;
	float4 mask = tex2D(maskSampler, screen);
	if (mask.r <= 0)
	return re;
technique AltDrawing
	pass P0
		PixelShader = compile PS_SHADERMODEL AltPS();


Vector2 Screen = new Vector2(GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height);
Vector2 Rat = new Vector2(100, 100) / Screen; //100,100 is size of sprite
Vector2 Sprite = pos / Screen;//position of sprite "TOPLEFT"

@PumpkinPudding I tried the 2.hsls method and got it to work!

After getting it to work for one sprite I saw the “float4 Position : SV_POSITION;”, so I checked what it is… its basically the X and Y coordinates of the screen (from 0, 0 to 1920, 1080).
I had to change to ps_4_0, so it let me read from the position, and then I just wrote an if sentence, which checks those coordinates and draws based on them. This is much less code for the same effect, i think…

Does changing from “ps_4_0_level_9_1” to “ps_4_0” have any major differences? From what I Googled, _level_9_1 is DX9.1 and ps_4_0 is DX10? Based on the Steam Hardware Survey 99+% have a DX10+ graphics card…

Thanks for all of you who helped me! I’m starting to get the hang of this :slightly_smiling:

You don’t need a shader for this. Just set the GraphicsDevice.ScissorRectangle before you draw what you want clipped to the box. When done, reset the ScissorRectangle to the full viewport.

@prime31 Yup the ScissorRectangle works, and it is much less code than the shader :smiley:
I didn’t even know this existed!
Thank you!!