Alpha Mask Effect

bambucha · July 6, 2016, 5:07pm

Hey guys!

I am trying to do a simple black/white alpha mask on a spritebatch. I have a spritebatch only for particles, and I want to limit them to a square on the screen. Simplest way to do it would be just to draw something over them, but I cant do that, since I draw other stuff aswell.

Basically what I want to achieve is to create an Effect, that will check an alpha mask and a pixel that is currently being drawn, then based on if the mask on current location is white or black draw the pixel or draw a transparent pixel.

I have found some stuff, but nothing seems to be working for me. Example from http://joshuasmyth.maglevstudios.com/post/XNA-and-Monogame-Introduction-to-Pixel-Shaders1 doesnt do anything, just makes everything transparent. The most I got it to work is to draw white boxes instead of textures.
Is there any examples of how to do this, or can any of you give me some advice?

Oh and one more question… Is the position (in the PixelShaderFunction (float2 inCoord: TEXCOORD0)) relative or based from the top left corner of the screen?

I’m kinda new to MonoGame, so this is probably really easy

Thanks in advance!!

Jjagg · July 6, 2016, 11:09pm

You can just multiply the color of the pixel with the value (any component) of the B/W alpha texture. That way black will multiply by 0 (transparent) and white by 1 (no change).
So in the pixelshader that would mean something like

return tex2D(Texture, position) * tex2D(Mask, position).x;

bambucha · July 7, 2016, 12:25am

Thanks!! but it still doesnt want to work, nothing gets drawn (or all transparent)

Here is my code right now:

uniform extern texture ScreenTexture;
sampler tex = sampler_state
{
	Texture = <ScreenTexture>;
};

uniform extern texture MaskTexture;
sampler mask = sampler_state
{
	Texture = <MaskTexture>;
};

float4 PixelShaderFunction(float2 pos: TEXCOORD0) : COLOR
{
	return tex2D(tex, pos) * tex2D(mask, pos).x;
}

technique
{
	pass P0
	{
    		PixelShader = compile ps_4_0_level_9_1 PixelShaderFunction();
    	}
    }
}

And I call it with:

alphaMapEffect.Parameters["MaskTexture"].SetValue(Art.UI.equipAlphaMap);

spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.Additive, null, null, null, alphaMapEffect);

currentScene.DrawMasked(spriteBatch); // draws all of the particles

spriteBatch.End();

The MaskTexture is a texture (1920x1080), with white where I want the particles to be drawn and black where I want them to be hidden.

Any ideas?

Alkher · July 7, 2016, 6:28am

Maybe because you have an alpha channel ?
float4 res =tex2D(tex, pos).rgb * tex2D(mask, pos).r;
res.a =0; //if you want to test without alpha at all
return res;

When you have built the effect the tool should have warned you that you were using implicit values.
Morreover if you use additive the things that should be black will be… As they were before. A b&w mask is rarely used with additive state.
Try with opaque state instead.

Have you tried using a tecture witbout alpha channel like jpg insteaf of png ?

Ps: i m using my smartphone so i am sorry if there are capitals lurking around in the code example

bambucha · July 7, 2016, 7:30am

Thanks Alkher!
Still no luck though…
This is my code now, the PixelShader function:

float4 res = tex2D(tex, pos);
res.r *= tex2D(mask, pos).r;
res.g *= tex2D(mask, pos).r;
res.b *= tex2D(mask, pos).r;
res.a = 0;

return res;

I also changed to opaque and jpg instead of png.
With this code, only black boxes are drawn on the screen.
Even if i just do return tex2D(tex, pos); I get black boxes… so I think there is something else wrong?
Maybe it’s the positioning of the mask, since the mask starts on (0, 0) on the screen and the particles are at like (500, 300). But I have no idea how the “float2 pos: TEXCOORD0” is formatted… relative to the texture it is drawing (so the particle) or to the whole screen?

bambucha · July 7, 2016, 11:00am

This is more and more confusing to me…

The following effect draws everything transparent:

sampler s0;
float4 PixelShaderFunction(float2 coords: TEXCOORD0) : COLOR0
{
	float4 color = tex2D(s0, coords);
	return color;  //return float4(1, 0, 0, 1); creates red boxes
}

technique Technique1
{
    pass Pass1
    {
        PixelShader = compile ps_4_0_level_9_1 PixelShaderFunction();
    }
}

In Game1:

spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend); 
//BlendState.Opaque = black box for player, so I guess it gets drawn, just transparent
    alphaMapEffect.CurrentTechnique.Passes[0].Apply();
    player.Draw(spriteBatch); 
    //draws the player texture at (500, 500) on 1080p screen
spriteBatch.End();

base.Draw(gameTime);

Any ideas?

Alkher · July 7, 2016, 1:36pm

Textures are mapped from 0,0 to 1,1
One question: what do you want to do with the black zones ? is it only b&w or grayscale ? display them ? your goal is not clear for me.
If no, you can do something like this to totally avoid drawing mask (ie which should not be visible):
if(tex2D(mask, pos).r > sometresholdbetween0and1) clip(-1); //or discard(); or simpler: clip(tex2D(mask, pos).r - sometresholdbetween0and1);

If you want to make black zones still visible like being a little darker:
res.a = tex2D(mask, pos).r; //Maybe.

In your code did you test:
return color; //return float4(1, 0, 0, 0.5f); should create red boxes half transparent/opaque

If you multiply by .r, then you should use additive state if no alpha is used. (but to me a black and white mask is not meant to be added)

bambucha · July 7, 2016, 1:57pm

Okay, it’s getting somewhere.

What i want to have is textures present only in one rectangle on the screen. Everything else should have no textures.

I tried avoiding the mask…

float4 tex = tex2D(s0, texCoord);
float coordX = texCoord.x * 1920;
float coordY = texCoord.y * 1080;

if (coordX >= 755 && coordX <= 1154 && coordY >= 125 && coordY <= 1032)
	return tex;
else
	return float4(0, 0, 0, 0);

this works, but the rectangle is applied to each texture instead of the screen…

like this: (on the right image it should say: “… to each texture, not to the screen”)

Any idea on how do I get screen coordinates for each pixel?

PumpkinPudding · July 7, 2016, 1:59pm

#1:
.fx

matrix	MatrixTransform;

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 TexCoord : TEXCOORD0;
	float4 Screen : TEXCOORD1;
};

VertexShaderOutput MainVS(float4 position:SV_POSITION, float2 texcoord : TEXCOORD0, float4 color : COLOR0)
{
	VertexShaderOutput output;

	output.Position = mul(position, MatrixTransform);
	output.Screen = output.Position;
	output.Color = color;
	output.TexCoord = texcoord;

	return output;
}

float4 MainPS(VertexShaderOutput input) : COLOR
{
	float2 screen = input.Screen.xy / input.Screen.w;
	screen = (screen + 1.0) / 2;
	screen.y = 1.0 - screen.y;

	float4 re = tex2D(spriteSampler, input.TexCoord);
	float4 mask = tex2D(maskSampler, screen);

	if (mask.r <= 0)//black
	{
		clip(-1);
	}
	return re;
}
technique BasicColorDrawing
{
	pass P0
	{
		VertexShader = compile VS_SHADERMODEL MainVS();
		PixelShader = compile PS_SHADERMODEL MainPS();
	}
};

draw

 Matrix projection;
 Matrix.CreateOrthographicOffCenter(0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, 0, 0, -1, out projection);
 Matrix _matrix = Matrix.Identity;//matrix use in spriteBatch.Draw

 Matrix.Multiply(ref _matrix, ref projection, out projection);

 effect.Parameters["MatrixTransform"].SetValue(projection);

#2

hlsl

float2 Sprite;
float2 SSize;

float4 AltPS(VertexShaderOutput input) : COLOR0
{
	float2 screen = Sprite + SSize*input.TexCoord;
	float4 re = tex2D(spriteSampler, input.TexCoord)*input.Color;
	float4 mask = tex2D(maskSampler, screen);
	if (mask.r <= 0)
	{
		clip(-1);
	}
	return re;
}

technique AltDrawing
{
	pass P0
	{
		PixelShader = compile PS_SHADERMODEL AltPS();
	}
};

draw

Vector2 Screen = new Vector2(GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height);
Vector2 Rat = new Vector2(100, 100) / Screen; //100,100 is size of sprite
Vector2 Sprite = pos / Screen;//position of sprite "TOPLEFT"
effect.Parameters["Sprite"].SetValue(Sprite);
effect.Parameters["SSize"].SetValue(Rat);

bambucha · July 7, 2016, 2:56pm

@PumpkinPudding I tried the 2.hsls method and got it to work!

After getting it to work for one sprite I saw the “float4 Position : SV_POSITION;”, so I checked what it is… its basically the X and Y coordinates of the screen (from 0, 0 to 1920, 1080).
I had to change to ps_4_0, so it let me read from the position, and then I just wrote an if sentence, which checks those coordinates and draws based on them. This is much less code for the same effect, i think…

Does changing from “ps_4_0_level_9_1” to “ps_4_0” have any major differences? From what I Googled, _level_9_1 is DX9.1 and ps_4_0 is DX10? Based on the Steam Hardware Survey 99+% have a DX10+ graphics card…

Thanks for all of you who helped me! I’m starting to get the hang of this

prime31 · July 9, 2016, 7:52am

You don’t need a shader for this. Just set the GraphicsDevice.ScissorRectangle before you draw what you want clipped to the box. When done, reset the ScissorRectangle to the full viewport.

bambucha · July 9, 2016, 8:25am

@prime31 Yup the ScissorRectangle works, and it is much less code than the shader
I didn’t even know this existed!
Thank you!!