I want to use masking technique to be able to reuse tiles. Basically I have the base texture and an array of masking textures for different corners and positions and want to combine the two to get a new texture.
The important part is this line: return diffuse * float4(mask.r, mask.r, mask.r, mask.r); It takes your premultiplied texture and uses the red channel of the mask. If the mask’s value is below 1, then you get transparency.
The simplest solution I can think of would be to just draw the border on top. Otherwise, I think it would be possible to apply an edge detection in the shader.
To do the texture rect, you’ll have to modify the UVs. Essentially, you modify TexCoord in a vertex shader so that you grab the right pixels. You’ll need two TexCoord, one for the texture and one for the mask. I have this tutorial which does some UV stuff, but it’s not quite what you want: Infinite background shader - Learn MonoGame
I’m a bit tired so not sure if this math is right, but given a texture atlas, on the X axis, 0 means the left side of the texture and 1 means the right side. On the Y axis, 0 means the top side of the texture and 1 means the bottoms side. So for a given mask, you could setup a transform matrix that will take the UV of texture you’re about to draw and convert that into the mask’s UVs.
The problem with that though is that for each texture and mask pair, you have to call Begin and End on the SpriteBatch. It would probably be easier to not use the SpriteBatch and setup the vertices yourself so you can pass both UVs yourself right away. Or maybe there’s a more clever way to apply this solution, perhaps using the color parameter in the draw call to pass the mask index? I don’t see the exact solution for the batching to work well with the SpriteBatch right now ;(.
Actually, I just remembered, the solution I use in my game is to setup a render target where I draw all the textures by themselves. I setup another render target where I draw all the masks by themselves. Then I draw both together using that shader. Using your transparent, black, white system above, I think it will work quite well.
Edit: I just noticed, this line return diffuse * float4(mask.r, mask.g, mask.b, mask.a); can be simplified to return diffuse * mask; since mask is already a float4 and you’re using every field.