HLSL - Clip Pixel based on Screen Position

Hello all, I have an HLSL question for a DesktopGL project. I’m making a dialogue box like in the screenshot below:

<img src=https://yngkyg.dm.files.1drv.com/y4mLdxvLMTeYyl4uI6QgvCbvS0Vc45Wo5fztXPI1QiA0H_zffqni3fOTIZjBWDslekY-VIIAjjMtb-looNPwKTcjeckyKf-nBFnikdU6DTkreEeQDEgxCtKl3W9ABfrLMVO2pfgMJyNvoO38upcw4-hhg-KRNVRFbOCIJ_5Cll6oESrvf2s6QeozwuEPGQuOm-Ki2Vkl0Z0g3xXRNpynM4XIg?width=800&height=600&cropmode=none" width=“400” height=“300”>

All the text in black is associated with the box.

Text prints itself on the box, and when told to, has a prompt for progressing to the next set of text (the star on the lower right).

When that happens, all previous text is moved up (the “Hello World!”), and the next set of text gets printed. If it has already been printed - you can go back to previous text - it’s simply shown by moving all the text down.

Basically, I need any text not in the box to be invisible. I decided that using a shader for this would work well, as I eventually need to use it to render the text with other special effects. My approach is to check the position of the pixels, and if they’re out of the textbox bounds, clip them. However, I can’t seem to get the position in the pixel shader.

Here’s my shader:

#if OPENGL
    #define SV_POSITION POSITION
    #define VS_SHADERMODEL vs_3_0
    #define PS_SHADERMODEL ps_3_0
#else
    #define VS_SHADERMODEL vs_5_0
    #define PS_SHADERMODEL ps_5_0
#endif

matrix WorldViewProjection;

struct VertexShaderInput
{
    float4 Position : POSITION0;
    float4 Color : COLOR0;
};

struct VertexShaderOutput
{
    float4 Position : SV_POSITION;
    float4 Color : COLOR0;
    float2 TextureCoordinates : TEXCOORD0;
};

float YMin;
float YMax;

VertexShaderOutput MainVS(in VertexShaderInput input)
{
    VertexShaderOutput output = (VertexShaderOutput)0;

    output.Position = mul(input.Position, WorldViewProjection);
    output.Color = input.Color;

    return output;
}

float4 MainPS(VertexShaderOutput input) : COLOR
{
    //This is what I want to do
    //if (input.Position.y <= YMin || input.Position.y >= YMax)
    //    clip(-1);

    return input.Color;
}

technique BasicColorDrawing
{
    pass P0
    {
        VertexShader = compile VS_SHADERMODEL MainVS();
        PixelShader = compile PS_SHADERMODEL MainPS();
    }
};

I’ve read online that specifying the position with the VPOS semantic would work for shader model 3, but I get errors whenever I try using it for either the VertexShaderInput, VertexShaderOutput, or both. I even tried defining another float4 in the VertexShaderOutput and setting the Position to that, but it ended up clipping all of the pixels even when I set YMin and YMax to impossibly low/high values (-10,000 and 10,000, respectively).

I would greatly appreciate it if anyone can point me in the right direction for handling this. Thanks in advance!

Those coordinates need to be multiplied by the projection matrix. Roughly divided by half the screen Width and Height and subtracted by 1f;

The gpu positional coordinates range from - 1 to + 1 for both x and y.
The gpu texture coordinates range from 0 to 1 for u and v.

1 Like

I tried that and am getting the same results - all text is being clipped. Am I doing this correctly?

Shader:

struct VertexShaderOutput
{
    float4 Position : SV_POSITION;
    float4 Color : COLOR0;
    float2 TextureCoordinates : TEXCOORD0;
    float4 ScreenCoords : TEXCOORD1;
};

VertexShaderOutput MainVS(in VertexShaderInput input)
{
    VertexShaderOutput output = (VertexShaderOutput)0;

    output.Position = mul(input.Position, WorldViewProjection);
    output.Color = input.Color;
    output.ScreenCoords = output.Position;

    return output;
}

float4 MainPS(VertexShaderOutput input) : COLOR
{
    if (input.ScreenCoords.y <= YMin || input.ScreenCoords.y >= YMax)
        clip(-1);

    return input.Color;
}

And here’s what I’m passing to it:

Vector2 screenSize = SpriteRenderer.Instance.WindowSize / 2;

float yMin = (bubble.Position.Y / screenSize.Y) - 1f;
float yMax = ((bubble.Position.Y + bubble.YMoveAmount) / screenSize.Y) - 1f;

effect.Parameters["YMin"].SetValue(yMin);
effect.Parameters["YMax"].SetValue(yMax);

SpriteRenderer.Instance.WindowSize returns (800, 600), which is my screen size. bubble.Position is (100, 100) and YMoveAmount is 88.

Ya i was going to write a example pixel shader and i wrote the test in gl. Dunno what im seeing atm. Getting some extremely disturbing results Which just hosed my example.

Well im trying to get it to work atm ill post back if i figure it out.

Why do you draw the text outside the box? The best solution would be to just not draw that text.

If there’s a good reason you’re drawing all the text, I’d solve this by using GraphicsDevice.ScissorRectangle. That let’s you clip pixels outside the set rectangle. You need to enable the scissor rectangle in the RasterizerState.

I thought he was trying to get a scrolling into the box effect.
I didn’t know about about the scissorRectangle though wish i had before.

Either way im playing around with gl on this and when i tried to get the position x y in a pixel shader from basic effects vertex shader all im getting is garbage basically all the pixel position values are the same value and its not zero either.
I tested that by turning all the positions into red and green from x and y all the text appears the same color.

Im guessing there is just something i misunderstand about the position symantic.

Meh i was just about to give up i was wrong about what i said.
Well at least if you just use a pixel shader.
To make a long story short here we go.

The pixel shader.

// FxClipText
//
#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

//matrix WorldViewProjection;
float4 visibleBounds;

Texture2D TextureA;
sampler2D TextureSamplerA = sampler_state
{
    Texture = <TextureA>;
};

// pixel shader
float4 MyPixelShader(float4 position : SV_Position, float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
    float4 col = tex2D(TextureSamplerA, texCoord) *color;
    float2 pos = position.xy;
    // i could probably mathematically boolean this down get rid of the ifs in a few ways but im tired.
    float dx = (pos.x - visibleBounds.x) * (visibleBounds.z - pos.x);
    float dy = (pos.y - visibleBounds.y) * (visibleBounds.w - pos.y);
    if (dx < 0.0f) 
        clip(-1);
    if (dy < 0.0f)
        clip(-1);
    return col;
}

technique BasicColorDrawing
{
    pass P0
    {
        PixelShader = compile PS_SHADERMODEL MyPixelShader();
    }
}

Implementation Game1

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;

namespace GLClipTextShader
{
    public class Game1 : Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;
        Effect effect;
        SpriteFont font;
        string message = "";
        Rectangle boundryRectangle = new Rectangle(100, 0, 200, 400);
        public static Texture2D dotTexture;

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";
        }
        protected override void Initialize()
        {
            graphics.PreferredBackBufferWidth = 800;
            graphics.PreferredBackBufferHeight = 600;
            graphics.ApplyChanges();
            base.Initialize();
        }
        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            // load up your stuff
            font = Content.Load<SpriteFont>("MgFont");

            // load up your effect;
            effect = Content.Load<Effect>("FxClipText");
            effect.CurrentTechnique = effect.Techniques["BasicColorDrawing"];

            // make a message
            message =
                " Ok so i had a bit of trouble figuring this out it turns out that it was working after all. " +
                "\n I was wrong when spritebatch passes the pos to the pixel shader. " +
                "\n It passes the untransformed coordinates so it does a bit of magic under the hood. " +
                "\n However its own vertex shader is still lining them up properly even if i pass matrix identity." +
                "\n So there we go if you intend however to pass a projection matrix ill bet youll need to do the transform on the boundry" +
                "\n +\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+"
                ;

            dotTexture = TextureDotCreate(GraphicsDevice);
        }

        public Texture2D TextureDotCreate(GraphicsDevice device)
        {
            Color[] data = new Color[1];
            data[0] = new Color(255, 255, 255, 255);
            Texture2D tex = new Texture2D(device, 1, 1);
            tex.SetData<Color>(data);
            return tex;
        }

        protected override void UnloadContent()
        {
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
                Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.Moccasin);

            Matrix m = Matrix.Identity;
            effect.CurrentTechnique = effect.Techniques["BasicColorDrawing"];
            effect.Parameters["visibleBounds"].SetValue(GetVector4Rectangle(boundryRectangle));
            // Im cheating here cause i dont know the samplers name or if it can be used in the function.
            // So im re-splicing the same texture thats will be sent in a second time.
            // Maybe someone else can chime in on what the name of the texture sampler is, that basic effects, vertex shader uses.
            effect.Parameters["TextureA"].SetValue(font.Texture);

            spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend, SamplerState.PointClamp, DepthStencilState.Default, RasterizerState.CullNone, effect, m);
            spriteBatch.Draw(dotTexture, boundryRectangle, Color.White);
            spriteBatch.Draw(font.Texture, new Vector2(200,320), Color.LightGray);
            spriteBatch.DrawString(font, message, new Vector2(100f, 100f), Color.MonoGameOrange);
            spriteBatch.End();

            base.Draw(gameTime);
        }

        public Vector4 GetVector4Rectangle(Rectangle r)
        {
            // woops you gotta flip the y should do this with the matrixes later on i think.
            // return new Vector4(r.Left, r.Top, r.Right, r.Bottom); 
            int h = graphics.GraphicsDevice.Viewport.Height;
            return new Vector4(r.Left, (h - r.Top), r.Right, (h - r.Bottom));
        }
        public Vector4 GetVector4GpuRectangle(Rectangle r, Matrix vp)
        {
            var lt = Vector2.Transform(new Vector2(r.Left, r.Top), vp);
            var rb = Vector2.Transform(new Vector2(r.Right, r.Bottom), vp);
            return new Vector4(lt.X, lt.Y, rb.X, rb.Y);
        }

        
    }
}

The text scrolls in and out of the box so I still need to render it. Thanks a lot, I’ll try the scissor rectangle and shader and see which one ends up working better for my purposes. Much thanks to both of you!

EDIT: I realized I can use both, so I will likely end up using the scissor rectangle for clipping and dedicate the shader to the various text effects.

Hi! I try to use your sample code, but get a error:
Shader model ps_4_0_level_9_1 doesn’t allow reading from position semantics.
How far I understand, an error occured in line
float2 pos = position.xy;
which contain a position semantics

I googled this problem and guys writes "Yep, you are not allowed this in current model, use a ps_4_0 instead.

But this model in this shader raise new exception: {“HRESULT: [0x80070057], Module: [General], ApiCode: [E_INVALIDARG/Invalid Arguments], Message: The parameter is not set correctly\r\n”}

Any suggestions please?

So, with couple hours of pain, I accidentally found solution:
graphicsManager.GraphicsProfile = GraphicsProfile.HiDef;
And this example works great.

If you don’t want to limit yourself to whichever devices support the HiDef profile, I think there are other ways around this.

In 3D, I would just add another position_copy field to the pixel shader input, and set it in the vertex shader to be the same as the position you tried to use. Maybe someone else can chime in if there’s a more appropriate way to do this with a SpriteBatch since you don’t need to define your vertex shader. You could also transform your texture coordinate to get the position, but that would require passing in other parameters and seems even more difficult.

I saw this last night but i was too tired to respond.

Yes If you wish to bypass the hidef profile, you would need to make a couple minor alterations.
First you will need to add a vertex shader.
You will need to pass a spritebatch like projection matrix to that vertex shader.

Here is a altered example with the vertex shader added.

FxClipTextShader.Fx

// FxClipTextShader.  with vertex shader
//
// This is can be used with spritebatch.
// Normally you can use spritebatches vertex shader and just use a pixel shader.
// Here we use the vertex shader too.
//
float4 VisibleBounds;
float4x4 WorldViewProjection;

Texture2D Texture : register(t0);
sampler TextureSampler : register(s0)
{
    Texture = (Texture);
};

struct VertexShaderInput
{
    float4 Position : POSITION0;
    float4 Color : COLOR0;
    float2 TexureCoordinate : TEXCOORD0;
};

struct VertexShaderOutput
{
    float4 Position : SV_Position;
    float4 Color : COLOR0;
    float2 TexureCoordinate : TEXCOORD0;
    float3 Position3D : TEXCOORD1;
};

struct PixelShaderOutput
{
    float4 Color : COLOR0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;
    output.Position3D = input.Position;
    output.Position = mul(input.Position, WorldViewProjection);
    output.Color = input.Color;
    output.TexureCoordinate = input.TexureCoordinate;
    return output;
}

PixelShaderOutput PixelShaderFunction(VertexShaderOutput input)
{
    PixelShaderOutput output;
    float4 col = tex2D(TextureSampler, input.TexureCoordinate) * input.Color;
    float2 pos = input.Position3D.xy;
    float dx = (pos.x - VisibleBounds.x) * (VisibleBounds.z - pos.x);
    float dy = (pos.y - VisibleBounds.y) * (VisibleBounds.w - pos.y);
    clip((sign(dx) + sign(dy)) - 0.001f);// change the if into a expression, clip is really a if else anyways
    output.Color =  col;
    return output;
}

technique ClipTextBounds
{
    pass
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

Game1 Implementation.

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;

namespace ClipTextShader
{
    /// <summary>
    /// This is a gl project should work for dx too.
    /// </summary>
    public class Game1 : Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;
        Effect effect;
        SpriteFont font;
        string message = "";
        Rectangle boundryRectangle;
        public static Texture2D dotTexture;
        Matrix projection;

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";
        }
        protected override void Initialize()
        {
            graphics.PreferredBackBufferWidth = 800;
            graphics.PreferredBackBufferHeight = 600;
            graphics.ApplyChanges();
            base.Initialize();
        }
        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            // load up your stuff
            font = Content.Load<SpriteFont>("MgFont");

            // load up your effect;
            effect = Content.Load<Effect>("FxClipTextShader");
            // set the current tequnique in your shader
            effect.CurrentTechnique = effect.Techniques["ClipTextBounds"];

            // create your projection matrix that is the same as spritebatches.
            projection = CreateSpiteBatchProjection(true);
            // set that to your effect as well
            effect.Parameters["WorldViewProjection"].SetValue(projection);

            // make a message
            message =
                " Ok so well need a projection matrix like spritebatch well create and set it probably just once to the shader so we just do that in load. " +
                "\n We add a method that handles makeing that spritebatch like matrix so we dont have to worry about it later CreateSpiteBatchProjection(...). " +
                "\n In my case i did this test on open gl so i had to set the bool for it to true in that method. " +
                "\n We make a couple minor alterations to the pixel shader primarily we use that projection matrix and pass the world positions thru the vertex shader." +
                "\n We also set the matrix in begin to null that matrix is really for the world and view but were bypassing it with our own vertex shader." +
                "\n +\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+\n+"
                ;

            dotTexture = TextureDotCreate(GraphicsDevice);
        }

        /// <summary>
        /// When using your own vertex shader with spritebatch things will get a little trickier because you have to handle the projection and viewport changes.
        /// https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Graphics/SpriteBatch.cs
        /// </summary>
        public Matrix CreateSpiteBatchProjection(bool NeedsHalfPixelOffset)
        {
            var projection = Matrix.CreateOrthographicOffCenter(0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, 0, 0, -1f);
            if (NeedsHalfPixelOffset)
            {
                projection.M41 += -0.5f * projection.M11;
                projection.M42 += -0.5f * projection.M22;
            }
            return projection;
        }

        public Texture2D TextureDotCreate(GraphicsDevice device)
        {
            Color[] data = new Color[1];
            data[0] = new Color(255, 255, 255, 255);
            Texture2D tex = new Texture2D(device, 1, 1);
            tex.SetData<Color>(data);
            return tex;
        }

        protected override void UnloadContent()
        {
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
                Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.Moccasin);
            // set the cliping bounds
            boundryRectangle = new Rectangle(100, 100, 200, 200);
            effect.Parameters["VisibleBounds"].SetValue(GetVector4SbRectangle(boundryRectangle));

            spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.NonPremultiplied, SamplerState.PointClamp, DepthStencilState.Default, RasterizerState.CullNone, effect, null);
            effect.CurrentTechnique.Passes[0].Apply();
            spriteBatch.Draw(dotTexture, boundryRectangle, Color.White);
            spriteBatch.Draw(font.Texture, new Vector2(200, 250), Color.LightGray);
            spriteBatch.DrawString(font, message, new Vector2(100f, 100f), Color.MonoGameOrange);
            spriteBatch.End();

            base.Draw(gameTime);
        }
        // we dont need that flip now i think.
        public Vector4 GetVector4SbRectangle(Rectangle r)
        {
            return new Vector4(r.Left, (r.Top), r.Right, (r.Bottom));
        }
    }
}

I suppose you could do all sorts of little effects as well like fade based on distance from the edge or recolor pixels based on distance or whatever. But for just clipping the scissors thing is really easy.