[SOLVED] Transparent textures in 3D have an outline

[Milun]

Hello,

I’ve been working on a project for quite a few months now, and I think it’s time I looked into fixing a bug I’ve been having for quite a while.

Due to the visual style of my game, I make very heavy use of planes with transparent .pngs applied to them. As you can see in the image, if I don’t sort them correctly I get that ugly outline. The problem is, it’s very common in my game for me to have 3+ transparent textures all overlapping each other, so sorting manually becomes very difficult.

I know that if I set my SamplerState to PointWrap, it fixes this issue. However, I don’t really like the way it looks (I prefer the textures to look blurry).

Would there be a solution to this? Unfortunately, I know little about custom shaders.

Thanks in advance!

[Jjagg]

@Milun

I know that if I set my SamplerState to PointWrap, it fixes this issue

I don’t think so. When geometry is drawn the depth is written to the depth buffer, even if it is transparant the value is stored if it’s closer than the previous value. Changing the sampler state won’t do.

I think you can solve this using the alpha test effect which is built in, so you don’t have to write a custom effect. It can clip pixels if they’re transparant. Though I’m not sure how exactly to implement that off the top of my head.

[Milun]

Unfortunately, what you’re seeing in the picture IS AlphaTestEffect

If it helps, I’m ok with having only two forms of transparency (either 100% solid, or 100% transparent).

I read somewhere of “Super Sampling”. Is that possible to do in MonoGame? My game isn’t very performance intensive, so I think I can afford it.

[Alkher]

You need 2 passes. On with alpha test on to draw opaque pixel. Then on top of that one another one with blending alpha. This works only if the alpha is smooth: the alpha is not only 0 and 1 in the channel

[Milun]

Alright, that works. Is there a guide on how to do that? I can’t really find anything good with Google : /

[GeonBit]

Maybe I misunderstood your issue but wouldn’t adding if (color.a < 1) discard; to your pixel shader solve this (color is the output pixel color before you return it)?

I think it should work for stuff like the grass above (sharp edges) but not for textures with alpha channels like a see-through plane.

[Milun]

That sounds like a good idea too! I suppose it’s high time I learned how to make a shader anyways. Something tells me I’ll need a lot of them.

[Milun]

@GeonBit

So, I did some research into using Shaders, and I added your code, and this is what I got:

One note, setting it to discard anything less than 1 causes the texture to sort of “lose quality” as the camera moves closer to it. Setting 1 to 0.95 got rid of that effect without letting any of that outline through.

Thanks for the help! I’m not a big fan of those hard shadows though, so I’m gonna go look up how to add directional lights to a custom shader.

[Alkher]

It would be better to use clip, discard executes the rest of the shader without drawing the pixel if i remember the doc

[Milun]

Alrighty, let’s give that a shot:

Works pretty good too (though I don’t really see much difference visually, I guess this would be the better alternative if I decide to add to the shader down the line).

To anyone wondering, the code was:

clip(color.a < 0.75f ? -1 : 1);

[Jjagg]

This is exactly what AlphaTestEffect does if you set the ReferenceValue to 0.75 and the CompareFunction to Greater (which is the default).

AlphaTestEffect.fx on GitHub
AlphaTestEffect.cs (the class that wraps the effect) on GitHub

[GeonBit]

Just dropping some extra info you should know - using discard / clip might have impact on performance in some cases. For example see here: http://gamedev.stackexchange.com/questions/40301/do-i-lose-gain-performance-for-discarding-pixels-even-if-i-dont-use-depth-testi

I haven’t tested it personally but it’s recommended to only use discard / clip on meshes that actually need it, eg for the terrain and other meshes that you know don’t need any alpha test use a different shader that don’t use discard / clip.

But like I said, I haven’t tested it personally

[Milun]

@GeonBit Will do! I guess we’ll find out if it slows down or not down the line (about half the textures in my game use some form of alpha).

@Jjagg …haaaa… well, I coulda saved myself some time. Oh well, I’d say I’m richer for it now anyways (at least now I have a basic understanding of how a custom shader works). Thanks for that link by the way. I’ve been meaning to modify SkinnedEffect to have that “clip” code in it too.

[Alkher]

Just add a parameter “useAlphaTest”
and a

[branch]
if(useAlphaTest)
    clip(color.a < 0.75f ? -1 : 1);

This way you only call clip when you know the model (or mesh if you set per mesh) needs alphatest.

But as Jjagg said, this is alphatest, not a 2 passes (nor a single blend pass) render to blend nicely with a gradient

[GeonBit]

@Alkher From what I understand from here and here the existence of the discard / clip command in the shader changes the flow in the GPU (more specifically, drop the early z-rejection optimization), regardless if its actually called or not. But its stuff I just read and could be completely wrong

Also, @Milun, I read that normally discard continue the shader program until the end and only drop the output when done (like @Alkher said) while clip normally stops, but in some places people reported that clip continues the flow as well, so regardless if you use clip or discard you should try adding a return right after.

In general what I’d do if I were you is:

1. look at MonoGame implemented their alpha test shader (I suspect they know a thing or two )
2. create a test scene with a heavy load of objects so you can experiment with performance.

And with everything said above, apparently there are more pitfalls in mobile devices. See what Unity say about Alpha Testing:

• Generally you gain a small advantage when using it to remove totally transparent pixels on most platforms.
• However, on PowerVR GPUs found in iOS and some Android devices, alpha testing is resource-intensive. Do not try to use it for performance optimization on these platforms, as it causes the game to run slower than usual.

And maybe your game is just not that heavy on graphics and you can alpha test everything and it will still be OK. Anyway take everything I said above with a grain of salt, its just some reading I did after the discussion in this thread.

[Milun]

Gotcha, but that’s ok. I ain’t making my game for mobile (but I’ll keep that in mind if that changes down the line).

Thank you all!