Stencil issues [SOLVED]

I don’t think I am quite understanding how stencils work… I have been trying to get it to work, but to no avail.

I created a new monogame app, and have 2 test images. One is the stencil (a green circle), and the other is the image (a red square).

What I hope to end up with is a red circle. However, I always seem to be ending up with a red square… or just a green circle if I tweak the stencil parameters. I can never seem to get a red circle.

I am rendering both the stencil and the image to a rendertarget and then drawing the resulting texture to the screen.

Here is my stencil code:

    spriteBatch = new SpriteBatch(GraphicsDevice);
    Texture2D stencil = Content.Load<Texture2D>("stencil");
    Texture2D image = Content.Load<Texture2D>("image");
    var renderTarget = new RenderTarget2D(GraphicsDevice, 320, 320, false, GraphicsDevice.PresentationParameters.BackBufferFormat, DepthFormat.Depth24Stencil8);
    var s1 = new DepthStencilState()
        StencilEnable = true,
        StencilFunction = CompareFunction.Greater,
        StencilPass = StencilOperation.Replace,
        ReferenceStencil = 1,
        DepthBufferEnable = false,
    GraphicsDevice.Clear(ClearOptions.Target | ClearOptions.DepthBuffer | ClearOptions.Stencil, Color.Transparent, 0, 0);
    spriteBatch.Begin(SpriteSortMode.Deferred, null, null, s1);
    spriteBatch.Draw(stencil, new Vector2(0, 0), Color.White);
    var s2 = new DepthStencilState()
        StencilEnable = true,
        StencilFunction = CompareFunction.Equal,
        StencilPass = StencilOperation.Keep,
        ReferenceStencil = 1,
        DepthBufferEnable = false,
    spriteBatch.Begin(SpriteSortMode.Deferred, null, null, s2);
    spriteBatch.Draw(image, new Vector2(0, 0), Color.White);
    stenciledImage = renderTarget;

I am also uploading the 2 images I am using (the green circle is the stencil, and the red square is the “image” that I want stenciled)…

Can someone point me in the right direction to get this working?


How did you create GraphicsDeviceManager in your Game class constructor?
Did you set PreferredDepthStencilFormat to DepthFormat.Depth24Stencil8?
As it is set DepthFormat.Depth24 by default(without stencil).

I did have it on the default settings, but I changed it to Depth24Stencil8, and it made no difference when I tested the change.

Does it matter though, since I am rendering to a RenderTarget, and I am setting that to Depth24Stencil8?

Oh, true. Didnt see you were using render target. Shouldn’t matter then. Sorry.

I would try changing from deferred to immediate, I can’t see anything else wrong

I am not able to try it until this evening, but I am pretty sure I had it on immediate first and switched it to deferred later as I was trying out all variations.

I will test it again later tonight.

Thanks for the suggestion!

Tried changing it to immediate, and no difference.

I suspect I am not using stencils correctly. Could someone with experience with stencils give it a look over and let me know what might be wrong?


@YTN Right now drawing your circle sprite fills a square in the stencil buffer. You can use the AlphaTestEffect to clip transparent pixels so they aren’t drawn to the stencil buffer.

Here’s a great example of using stencil for masking:

That was it!

By using an AlphaTestEffect when rendering the stencil, I got the red circle to display correctly.

Thanks @Jjagg!

1 Like