Stencil Alpha Problem [solved .. sort of]

Phoenix_Feder · August 17, 2016, 9:57pm

English isn’t my mother tongue so i hope i can explain my problem:

I’ve read this article about stencil:
http://scott-franks.com/xna-masking-sprites-without-shaders/

And I copied that into a new project to test it.
It doesn’t work like i wanted and I dont know why.

Those are the two pictures I used:

I want to mask the “B” with the lines in so that I have a “B” with lines ^^ …

But i get this (second with reversed stencil):

I want this:

Can anybody help me?

nkast · August 18, 2016, 2:03am

Does it work with the original pictures?

Do you scale the images?

Can you confirm that the mask has transparent pixels?
Also check the importer, I think it should be set to premultiply-alpha.

Phoenix_Feder · August 18, 2016, 10:47am

there are no original pictures to download there
i made my own

no scale. pictures are 200200 px and rendered 200200 px

yes I can confirm it. If i draw the pictures with monogame without stencil functions, every alpha pixels are shown as intended. (used photoshop to make the pictures without background)

I found an other example at github

github.com

CartBlanche/MonoGame-Samples/blob/master/StencilsCratersTutorial/Game1.cs

using System;
using System.Collections.Generic;
using System.IO;

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Audio;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using Microsoft.Xna.Framework.Media;

namespace StencilCratersTutorial
{
  public class Game1 : Microsoft.Xna.Framework.Game
  {
    const int PlanetDataSize = 256;

    bool firstTime = true;
    Vector2 craterPosition;
    Vector2 planetPosition;
    Texture2D drawingTexture;
    KeyboardState keyboardState;
    KeyboardState previousKeyboardState;

    GraphicsDeviceManager graphics;
    SpriteBatch spriteBatch;
    Random random;

    Texture2D planetTexture;
    Texture2D craterTexture;

    RenderTarget2D renderTargetA;
    RenderTarget2D renderTargetB;
    RenderTarget2D activeRenderTarget;
    RenderTarget2D textureRenderTarget;

    AlphaTestEffect alphaTestEffect;
    DepthStencilState stencilAlways;
    DepthStencilState stencilKeepIfZero;



    public Game1()
    {
      graphics = new GraphicsDeviceManager(this);
      Content.RootDirectory = "Content";

      random = new Random();
    }


    protected override void Initialize()
    {
      base.Initialize();

      // initialize keyboard state
      keyboardState = Keyboard.GetState();
      previousKeyboardState = keyboardState;
    }


    protected override void LoadContent()
    {
      spriteBatch = new SpriteBatch(GraphicsDevice);

      // load assets
      planetTexture = Content.Load<Texture2D>(@"planet");
      craterTexture = Content.Load<Texture2D>(@"crater");

      // set up alpha test effect
      Matrix projection = Matrix.CreateOrthographicOffCenter(0, PlanetDataSize, PlanetDataSize, 0, 0, 1);
      Matrix halfPixelOffset = Matrix.CreateTranslation(-0.5f, -0.5f, 0);

      alphaTestEffect = new AlphaTestEffect(GraphicsDevice);
      alphaTestEffect.VertexColorEnabled = true;
      alphaTestEffect.DiffuseColor = Color.White.ToVector3();
      alphaTestEffect.AlphaFunction = CompareFunction.Equal;
      alphaTestEffect.ReferenceAlpha = 0;
      alphaTestEffect.World = Matrix.Identity;
      alphaTestEffect.View = Matrix.Identity;
      alphaTestEffect.Projection = halfPixelOffset * projection;


      // set up stencil state to always replace stencil buffer with 1
      stencilAlways = new DepthStencilState();
      stencilAlways.StencilEnable = true;
      stencilAlways.StencilFunction = CompareFunction.Always;
      stencilAlways.StencilPass = StencilOperation.Replace;
      stencilAlways.ReferenceStencil = 1;
      stencilAlways.DepthBufferEnable = false;


      // set up stencil state to pass if the stencil value is 0
      stencilKeepIfZero = new DepthStencilState();
      stencilKeepIfZero.StencilEnable = true;
      stencilKeepIfZero.StencilFunction = CompareFunction.Equal;
      stencilKeepIfZero.StencilPass = StencilOperation.Keep;
      stencilKeepIfZero.ReferenceStencil = 0;
      stencilKeepIfZero.DepthBufferEnable = false;


      // create render targets
      renderTargetA = new RenderTarget2D(GraphicsDevice, PlanetDataSize, PlanetDataSize,
                                         false, SurfaceFormat.Color, DepthFormat.Depth24Stencil8,
                                         0, RenderTargetUsage.DiscardContents);

      renderTargetB = new RenderTarget2D(GraphicsDevice, PlanetDataSize, PlanetDataSize,
                                         false, SurfaceFormat.Color, DepthFormat.Depth24Stencil8,
                                         0, RenderTargetUsage.DiscardContents);


      // set up the ping-pong texture pointers
      activeRenderTarget = renderTargetA;
      textureRenderTarget = renderTargetB;
      drawingTexture = planetTexture;
    }


    protected override void UnloadContent()
    {
    }



    protected override void Update(GameTime gameTime)
    {
      // check for exit
      if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
        this.Exit();



      // update keyboard state
      previousKeyboardState = keyboardState;
      keyboardState = Keyboard.GetState();


      // calculate planet position, centered in screen
      planetPosition = new Vector2(GraphicsDevice.PresentationParameters.BackBufferWidth * 0.5f - PlanetDataSize * 0.4f,
                                   GraphicsDevice.PresentationParameters.BackBufferHeight * 0.5f - PlanetDataSize * 0.4f);




      // add a random crater if space bar is pressed
      if ((keyboardState.IsKeyDown(Keys.Space) && previousKeyboardState.IsKeyUp(Keys.Space)))
      {
        craterPosition = new Vector2(random.Next(PlanetDataSize), random.Next(PlanetDataSize));
      }


      base.Update(gameTime);
    }


    public void AddCrater(Vector2 position)
    {
      // set up rendering to the active render target
      GraphicsDevice.SetRenderTarget(activeRenderTarget);

      // cl

This file has been truncated. show original

This one works with my pictures:

I have to compare what this one does right and the other wrong.

Oh, i had to tweak the code a little bit (mainly deleted the halfoffset pixels for the alphatest) and deleted the stuff about the key input

Edit: Got another problem:
Is it normal that this stencil distinguish only between alpha 0 (no pixel) and 1 (draw pixel) and nothing in between like 50% alpha (0.5)?

Phoenix_Feder · August 18, 2016, 3:10pm

For understanding
I want something like this:

what I now get is something like that:

Can anybody help me with a solution with the stencil option?

Ravendarke · August 18, 2016, 4:47pm

Stencil on that matter has binary behavior, either it will display given pixel or cut it off completely. As you can seen in your case - every pixel on map with value under exact 1.0 is being displayed.

Why don´t you use shader and alpha blend?

Phoenix_Feder · August 18, 2016, 5:30pm

Ok thanks for that response
So it isnt possible with stencil =)
I will try a pixel shader

maybe this is what i want … I will try during the weekend
http://www.syntaxwarriors.com/2012/xna-alpha-mapping-with-pixel-shaders/

just need to write a shader with coordinate input because the mask has to move in runtime ^^
maybe this can help too

do you know some good tutorial or example I can read?

nkast · August 18, 2016, 6:52pm

The build-in DualTextureEffect will be a good starting point for something like that.
It depends on what you want to achieve. There are complete tutorials & libraries on 2D lighting if that’s what you are after.

Phoenix_Feder · August 19, 2016, 12:13am

I want to have a movable mask for a texture in my game. For example for blend in effects with a gradient or variable overlay effects. The best sollution for what i Plan is to use texture masks. And i think now pixel shader is the right way… will try during weekend.

nkast · August 19, 2016, 10:09am

If you want to move the mask interdependently, you need to pass a second texture UV. Something similar to https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Graphics/Vertices/VertexPosition.cs but in your case it would be ‘VertexPositionPosition’.
In your shader you can use something like

struct VSInputTx2
{
float4 Position : POSITION0;
float2 TexCoord : TEXCOORD0;
float2 TexCoord2 : TEXCOORD1;
};