How to mask 2D tile sprites

I want to use masking technique to be able to reuse tiles. Basically I have the base texture and an array of masking textures for different corners and positions and want to combine the two to get a new texture.

enter image description here

I encountered this technique while browsing and want to know how to implement it in Monogame. Any help is appreciated.

There are many ways to do it, one way would be to write a mask shader. Here is an example of how I do it in my game.

I have a grid texture:

A mask texture:

I pass both to my shader and I get:

I draw that on top of another piece for fun:

The shader is pretty simple if both textures are the same size:

#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#define VS_SHADERMODEL vs_4_0
#define PS_SHADERMODEL ps_4_0

sampler TextureSampler : register(s0);
Texture2D Mask;
sampler MaskSampler {
    Texture = ( Mask );
    MagFilter = LINEAR;
    MinFilter = LINEAR;
    Mipfilter = LINEAR;
    AddressU = CLAMP;
    AddressV = CLAMP;

struct PixelInput {
    float4 Position : SV_Position0;
    float4 Color : COLOR0;
    float4 TexCoord : TEXCOORD0;

float4 SpritePixelShader(PixelInput p): COLOR0 {
    float4 diffuse = tex2D(TextureSampler, p.TexCoord.xy);
    float4 mask = tex2D(MaskSampler, p.TexCoord.xy);

    return diffuse * float4(mask.r, mask.r, mask.r, mask.r);

technique SpriteBatch {
    pass {
        PixelShader = compile PS_SHADERMODEL SpritePixelShader();

The important part is this line: return diffuse * float4(mask.r, mask.r, mask.r, mask.r); It takes your premultiplied texture and uses the red channel of the mask. If the mask’s value is below 1, then you get transparency.

To use this, you can use the SpriteBatch:

s.Begin(effect: MaskShader);
s.Draw(SuperTexture, Vector2.Zero, Color.White);

You can read more on shaders here:

Ok so I have tried this out and this is what I achieve:

Currently I’m facing 2 problems

  1. I am unable to add an outline like the example I posted before
  2. I can only pass in a full texture instead of part of a texture since I plan on using an atlas of masks for different corners and sides like this:

How do recommend I solve these problems? Thanks.

The simplest solution I can think of would be to just draw the border on top. Otherwise, I think it would be possible to apply an edge detection in the shader.

Hell yeah I got it to work.

What I did was change the mask to use white for areas I want to keep, black for the outline and transparent for what I don’t want to keep.

Then in the shader code I multiply the mask and base textures:

float4 SpritePixelShader(PixelInput p) : COLOR0 {
    float4 diffuse = tex2D(TextureSampler, p.TexCoord.xy);
    float4 mask = tex2D(MaskSampler, p.TexCoord.xy);

    return diffuse * float4(mask.r, mask.g, mask.b, mask.a);

The only issue now is how to pass a texture rect instead of a whole texture…


That’s smart!

To do the texture rect, you’ll have to modify the UVs. Essentially, you modify TexCoord in a vertex shader so that you grab the right pixels. You’ll need two TexCoord, one for the texture and one for the mask. I have this tutorial which does some UV stuff, but it’s not quite what you want: Infinite background shader - Learn MonoGame

I’m a bit tired so not sure if this math is right, but given a texture atlas, on the X axis, 0 means the left side of the texture and 1 means the right side. On the Y axis, 0 means the top side of the texture and 1 means the bottoms side. So for a given mask, you could setup a transform matrix that will take the UV of texture you’re about to draw and convert that into the mask’s UVs.

private Matrix GetUVTransform(Texture2D t1, Texture t2, Vector2 v1, Vector2 v2) {
        Matrix.CreateScale(t1.Width, t1.Height, 1f) *
        Matrix.CreateTranslation(v2.X - v1.X, v2.Y - v1.Y, 0f) *
        Matrix.CreateScale(1f / t2.Width, 1f / t2.Height, 1f);

This assumes the texture region and mask region are the same size. v1 and v2 are the top left of each regions.

Maybe this helps to visualize:,

Pass that into the shader.

The problem with that though is that for each texture and mask pair, you have to call Begin and End on the SpriteBatch. It would probably be easier to not use the SpriteBatch and setup the vertices yourself so you can pass both UVs yourself right away. Or maybe there’s a more clever way to apply this solution, perhaps using the color parameter in the draw call to pass the mask index? I don’t see the exact solution for the batching to work well with the SpriteBatch right now ;(.

Actually, I just remembered, the solution I use in my game is to setup a render target where I draw all the textures by themselves. I setup another render target where I draw all the masks by themselves. Then I draw both together using that shader. Using your transparent, black, white system above, I think it will work quite well.

Edit: I just noticed, this line return diffuse * float4(mask.r, mask.g, mask.b, mask.a); can be simplified to return diffuse * mask; since mask is already a float4 and you’re using every field.