How to batch many sprites with the same shader but different parameters

I’m trying to implement the Marching Squares Algorithm to reproduce the Oxygen Not Included style terrain bumps. In order to do this, I am using a tilemap where each tile has a sprite that has its TextureCoordinates adjusted depending on where it is in the world, so it will look like one continuous object (as shown in ONI). Then, to apply the bumpy edges I am using a mask and multiplying the color of each pixel by the mask. To make a black edge the mask has black color, to remove the texture the mask is transparent, to keep the texture the mask is white.

There are several ways to apply a mask though, and I’m nervous about performance problems with all of them and trying to pick the best one before spending the time to implement them completely.

The Blend State/Function only works if the render target has nothing else on it, otherwise I will subtract out previously drawn stuff. This means I would have to switch render targets for every texture that I want to do this for. 10-15 target switches seems bad.

The other way to do this would be a pixel shader so the mask can get applied to the sprite before it gets drawn. While I can accomplish this with a single shader, I don’t know how to switch the mask TextureCoordinate that the shader is using for each sprite while batched. If I switch to SpriteSortMode.Immediate, then this is easy, but it would be a big performance hit.

Is there a performant way of doing either of those two solutions?
Are there other solutions?

Oxygen Not Included Tiling

Oxygen Not Included. Notice how they have a continuous texture that they have subtracted from, and then for different textures, just render on top of each other.

My testing mask:

Hey Bob, nice problem you have there and you are overthinking a bit. Blend State/Function is a no no for this kind of task, so with that out of the way lets look at your options.

1 - Pixel Shader solution. This is straightforward and I can see you know how it would work. This is the most robust solution by the way. The problem as you guested is TextureCoordinate. With default SpriteBatch I believe it is not possible to pass two TextureCoordinates which you need. So for this to work you would have to make your custom sprite batching and rendering solution. If there are not many tiles like that (lets say up to 1k) then you don’t even need batching. Sprite rendering is pretty easy and there are many sources on how you would go about that.

2 - More straightforward solution without implementing new Sprite rendering is to have sprites with masks applied. This is what I use for my project and what we use 90% of the time for Factorio. This is the fastest solution overall, the only drawback is little VRam and more manual labor for the sprite making.

Second solution in action:

Oxygen Not Included definitely uses the first option. On top of that they use big textures from which they sample tiles based on position.

One more bonus for having a custom sprite rendering solution is that if your game needs 100k+ sprites rendered and sorted then it won’t be a problem. With standard SpriteBatch its unrealistic.

I hope that answers your question.

I decided to go with the custom batching because I did need to render over 2k of these, and I’ve done custom batching in another project.

Are you one of the Factorio devs/artists?