Stencil issues [SOLVED]

1

I don’t think I am quite understanding how stencils work… I have been trying to get it to work, but to no avail.

I created a new monogame app, and have 2 test images. One is the stencil (a green circle), and the other is the image (a red square).

What I hope to end up with is a red circle. However, I always seem to be ending up with a red square… or just a green circle if I tweak the stencil parameters. I can never seem to get a red circle.

I am rendering both the stencil and the image to a rendertarget and then drawing the resulting texture to the screen.

Here is my stencil code:

    spriteBatch = new SpriteBatch(GraphicsDevice);

    Texture2D stencil = Content.Load<Texture2D>("stencil");
    Texture2D image = Content.Load<Texture2D>("image");

    var renderTarget = new RenderTarget2D(GraphicsDevice, 320, 320, false, GraphicsDevice.PresentationParameters.BackBufferFormat, DepthFormat.Depth24Stencil8);

    GraphicsDevice.SetRenderTarget(renderTarget);

    var s1 = new DepthStencilState()
    {
        StencilEnable = true,
        StencilFunction = CompareFunction.Greater,
        StencilPass = StencilOperation.Replace,
        ReferenceStencil = 1,
        DepthBufferEnable = false,
    };

    GraphicsDevice.Clear(ClearOptions.Target | ClearOptions.DepthBuffer | ClearOptions.Stencil, Color.Transparent, 0, 0);

    spriteBatch.Begin(SpriteSortMode.Deferred, null, null, s1);
    spriteBatch.Draw(stencil, new Vector2(0, 0), Color.White);
    spriteBatch.End();

    var s2 = new DepthStencilState()
    {
        StencilEnable = true,
        StencilFunction = CompareFunction.Equal,
        StencilPass = StencilOperation.Keep,
        ReferenceStencil = 1,
        DepthBufferEnable = false,
    };

    spriteBatch.Begin(SpriteSortMode.Deferred, null, null, s2);
    spriteBatch.Draw(image, new Vector2(0, 0), Color.White);
    spriteBatch.End();

    GraphicsDevice.SetRenderTarget(null);

    stenciledImage = renderTarget;

I am also uploading the 2 images I am using (the green circle is the stencil, and the red square is the “image” that I want stenciled)…

Can someone point me in the right direction to get this working?

Thanks!

2

How did you create GraphicsDeviceManager in your Game class constructor?
Did you set PreferredDepthStencilFormat to DepthFormat.Depth24Stencil8?
As it is set DepthFormat.Depth24 by default(without stencil).

3

I did have it on the default settings, but I changed it to Depth24Stencil8, and it made no difference when I tested the change.

Does it matter though, since I am rendering to a RenderTarget, and I am setting that to Depth24Stencil8?

4

Oh, true. Didnt see you were using render target. Shouldn’t matter then. Sorry.

5

I would try changing from deferred to immediate, I can’t see anything else wrong

6

I am not able to try it until this evening, but I am pretty sure I had it on immediate first and switched it to deferred later as I was trying out all variations.

I will test it again later tonight.

Thanks for the suggestion!

7

Tried changing it to immediate, and no difference.

I suspect I am not using stencils correctly. Could someone with experience with stencils give it a look over and let me know what might be wrong?

Thanks!

8

@YTN Right now drawing your circle sprite fills a square in the stencil buffer. You can use the AlphaTestEffect to clip transparent pixels so they aren’t drawn to the stencil buffer.

Here’s a great example of using stencil for masking: https://gamedev.stackexchange.com/a/38150

9

That was it!

By using an AlphaTestEffect when rendering the stencil, I got the red circle to display correctly.

Thanks @Jjagg!

1 Like