Shader that works in XNA 4.0 is broken in MonoGame

[nbiefeld]

There was an old XNA sample from Microsoft that discussed Sprite Effects. There is a displacement effect that I used in the past with XNA and it’s failing in MonoGame. I found a ported version of this effect as I lost my copy:

CartBlanche/MonoGame-Samples

I ran this effect two days ago in Visual Studio 2010 and XNA 4.0 Refresh. This works. However the shader is failing in MonoGame.

1). If I am using DirectX, MonoGame says it needs Pixel Shader 4.0 or higher. Shader will not work.

2). When using OpenGL, Pixel Shader 2.0 is fine, yet the shader still fails.

Could someone take a look at this and see what needs changing? I have been struggling with this basic shader for 4 days now.

This is the specific shader - https://github.com/CartBlanche/MonoGame-Samples/blob/master/SpriteEffects/Content/refraction.fx

[Charles_Humphrey]

I don’t think ps_2_0 is supported in MG now, try ps_3_0 or ps_4_0

use #defines like this.

[nbiefeld]

Yeah the shader will not work if you just update the ps_2_0 line. Are there some changes to the HLSL syntax that prevent this from working?

[willmotil]

Do you mean it wont compile and run or it is broken ?

Edit nvm ok i guess it works i had two textures that were about the same in there.
It compiles and runs with some minor changes.

If you need to make it run with ps_2_0 then look to the link at the bottom you would have to make further additional changes probably also including a basic vertex shader as well as shown in the post example.

//
// refract
//
#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

//matrix WorldViewProjection;
float2 DisplacementScroll;

Texture2D TextureA;
sampler2D TextureSamplerA = sampler_state
{
    Texture = <TextureA>;
};

Texture2D TextureB;
sampler2D DisplacementSampler = sampler_state
{
    Texture = <TextureB>;
};

float4 main(float4 position : SV_Position, float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
    // Look up the displacement amount.
    float2 displacement = tex2D(DisplacementSampler, DisplacementScroll + texCoord / 3);
    
    // Offset the main texture coordinates.
    texCoord += displacement * 0.2 - 0.15;
    
    // Look up into the main texture.
    return tex2D(TextureSamplerA, texCoord) * color;
}


technique Refraction
{
    pass Pass0
    {
        PixelShader = compile PS_SHADERMODEL main(); // ps_2_0  doesn't error either.
    }
}

the relevant part of the game1 pretty obvious what my textures are and calls are that you need to change.
other then that i dunno what its supposed to be doing.

        protected override void Draw(GameTime gameTime)
        {
            Gu.device.Clear(Color.CornflowerBlue);
            //GraphicsDevice.SetRenderTarget(rt);
            //GraphicsDevice.Clear(ClearOptions.Target | ClearOptions.DepthBuffer, Color.CornflowerBlue , 1f, 0); //new Vector4(1.0f, .99f, .99f, 1.0f)

            DrawRefractCat(gameTime);

            base.Draw(gameTime);
        }

        void DrawRefractCat(GameTime gameTime)
        {
            // Draw the background image.
            Gu.spriteBatch.Begin();
            Gu.spriteBatch.Draw(BasicTextures.checkerBoard, GraphicsDevice.Viewport.Bounds, Color.White);
            Gu.spriteBatch.End();

            // Set an effect parameter to make the
            // displacement texture scroll in a giant circle.
            refractionEffect.Parameters["DisplacementScroll"].SetValue(MoveInCircle(gameTime, new Vector2(100, 100), 0.1f));
            refractionEffect.Parameters["TextureA"].SetValue(BasicTextures.grid);
            refractionEffect.Parameters["TextureB"].SetValue(BasicTextures.checkerBoard);

            // Set the displacement texture.
            //Gu.graphics.GraphicsDevice.Textures[1] = BasicTextures.grid;

            // Begin the sprite batch.
            Gu.spriteBatch.Begin(0, null, null, null, null, refractionEffect);

            // Draw the sprite.
            Gu.spriteBatch.Draw(BasicTextures.checkerBoardRed, MoveInCircle(gameTime,new Vector2(100,100), 1),Color.White);

            // End the sprite batch.
            Gu.spriteBatch.End();
        }
        /// <summary>
        /// Helper for moving a value around in a circle.
        /// </summary>
        static Vector2 MoveInCircle(GameTime gameTime, Vector2 position, float speed)
        {
            double time = gameTime.TotalGameTime.TotalSeconds * speed;

            float x = (float)Math.Cos(time);
            float y = (float)Math.Sin(time);

            return new Vector2(x, y) + position;
        }

        protected override void QuickDrawSpriteBatch(GameTime gameTime)
        {
            itemList01.Draw(gameTime);

            // DrawTest1(gameTime);

            DrawTest2_bound(gameTime);
        }

Look to the bottom example were i got the ps_2_0 shader to run thru spritebatch also look at game1 's draw its a little hacky but it worked.

HLSL - Clip Pixel based on Screen Position

If you don’t want to limit yourself to whichever devices support the HiDef profile, I think there are other ways around this. In 3D, I would just add another position_copy field to the pixel shader input, and set it in the vertex shader to be the same as the position you tried to use. Maybe someone else can chime in if there’s a more appropriate way to do this with a SpriteBatch since you don’t need to define your vertex shader. You could also transform your texture coordinate to get the positi…

[nbiefeld]

It would not work properly when I was able to get it to compile. I will give your changes a try. Quick question though, why did you add two textures in the shader file instead of using the auto-texture? Is that something that changed with new HLSL/PixelShader?

sampler TextureSampler : register(s0);

That should use the texture from the SpriteBatch.Draw call right? But you set it as a parameter instead. Any reason why? That might be what the issue was.

I used this shader 10 years ago roughly when XNA was around.

I was able to find this video of it: https://www.youtube.com/watch?v=49zWFTGsfsw

Go to 15 seconds in and see the cat being displaced. That is the effect I want in my game but I was getting strange results like this:

[willmotil]

Well im not sure this code is a faithful reconstruction maybe but the guy was passing a texture into the move function parameter that expected a float. I added a offset as well because it was drawing in the top left corner.

why not sampler TextureSampler : register(s0);

Habit superstition i do it the way i posted normally with no problems i also like to set the texture in game1 by name feel like it keeps everything cleaner and straight though i didn’t name them very good here.

     refractionEffect.Parameters["TextureA"].SetValue(BasicTextures.grid);
     refractionEffect.Parameters["TextureB"].SetValue(BasicTextures.checkerBoard);

That probably doesn’t matter i think it was the position semantic in the function causing the problem.

In the clip shader link you can see how to use a vertex shader with spritebatch as well with the default structs that line up with the default BasicEffect that took a minute to figure out. In the same post also if you scroll back up is the first version which is for hidef profile.

Anyways might be easier to just search the monogame forum for “Displacement Shader” im sure there are a few on here that regular people wrote.

[Charles_Humphrey]

Ill grab a copy of the shader today and have a look too.

[Charles_Humphrey]

OK,

There are a few issues in this shader, but minor, this has helped highlight an issue I have found with MG in that when rendering with a shader to a sprite batch, if I don’t sample the spritebatch texture before doing any other texture samples, the samples are not loaded by the graphics pipeline correctly. It seems to ignore the registers in this case.

So, I am rendering like this in my draw call:-

And the shader now looks like this:

Forgive my images, but I don’t seem to be able to get my head around the formatting on this forum lol

You can get this off my repo if you want to have a look at it here under project CommunityPost11896.

Hope this helps

[AlienScribble]

@nbiefeld

why did you add two textures in the shader file instead of using the auto-texture? Is that something that changed with new HLSL/PixelShader?

Note: for the first texture you should be good. Any textures after the first one you’d need to pass as a parameter (which I’d do during initialization if it doesn’t change). [Also reminder to match up the default VS params to the PS input ones]
ie:

sampler TextureSampler : register(s0);
sampler DisplacementSampler : register(s1)
{
    Texture = (DisplaceTexture); // <--- use SetValue for DisplaceTexture (must pass as parameter for textures when more than one) 
};

in load:
DisplaceFX.Parameters[“DisplaceTexture”].SetValue(DisplaceTex);

Basically just confirming what the others have said – I think you can usually get away with not passing the first texture unless you do not use sprite-batch in which case using device.Textures[0] = tex; should work for first texture but I’ve experienced it messing up before too (older MG version maybe) so I pass as parameter to be safe.

[nbiefeld]

Gotcha, I bet that is what the root issue is. Something MonoGame does differently than XNA. The shader I linked above worked without any modifications in XNA, but it would not work in MonoGame.

[nbiefeld]

Charles_Humphrey, you are the best. Your changes worked just fine for me too. I guess something I need to keep in mind between the differences of XNA and MonoGame is that I need to sample all textures passed to the shader. I wonder if the vsOutput makes a difference too.

[Charles_Humphrey]

vOutput, do you mean the structure passed from the vertex shader to the pixel shader?

I don’t think it will, you can pack that with what ever you like in the vertex shader (if you write your own). But here with a SpriteBatch Draw call, I think it will return the regular stuff as shown in my example. It may include normals in there too, but for what you need, these three should be fine. The GPU pipeline will match the type and channel to the structure we give as best it can (think this is called vertex patching) though thinking about it, I think this describes what’s done when the pipeline is trying to match the vertex stream we pass with the intended vertex structure we describe in our shader. This is digging a little deeper than we need to here though lol

Glad it helped you out

[nbiefeld]

When I was messing around with it, I ended up with this:

float4 PixelShaderFunction(float4 pos : SV_POSITION, float4 color1 : COLOR0, float2 coords: TEXCOORD0) : COLOR0

So I thought maybe the vsOutput struct itself would have any impact. One more quick question. Why does removing this line break the shader?

float4 v = tex2D(TextureSampler, input.texCoord);

I do not see v being used anywhere right?

[Charles_Humphrey]

yea, so the structure is the same as what you have there, just tidier

Right, that is the issue I found, v is not used at all, other than to sample the TextureSampler. If I comment that out, the texture pipeline goes to poop… I think this is a but in MG…

[nbiefeld]

Yeah that is so strange. I know it likes to remove unused variables and stuff, but you are using the TextureSampler in the return line. So I am not sure why that v variable would be needed. Strange!

[Charles_Humphrey]

I think it’s the fact that the texture is being sampled first, before any other in the shader, before the displacement texture was samples first, this just ensured that this texture got sampled first. I imagine if you sample the displacement texture before that it will break it again… it’s v odd, and as I say I think a bug in MG

[nbiefeld]

Awesome. Thank you for the help! I do not know HLSL very well and as you can see, this shader is very simple. I was struggling on this for a few days and was pulling my hair out!

[Charles_Humphrey]

The more you play with them, the better you will get. It’s worth investing the time in them, as ultimately, EVERYTHING you send to the screen will go through a shader, and if you can write your own, you will have total control

[nbiefeld]

Do you recommend any resources (books, videos, or websites)? I tried looking for better resources other than “Here is how to make your sprite look greyscale” type of shaders.

[Charles_Humphrey]

First book I ever got was this, Programming Vertex, Geometry, and Pixel Shaders, I have the first edition, it’s out of print now though as I think this is too

I have a few samples on my repo, you could re purpose the post processing shaders I have in there for 2D sprite rendering, as in that sample I am using them on SpriteBatch Draw calls too Though, in my engine, I use a screen space quad, this stops these sort of issues we have just spoken about here too.

Good luck, it can be frustrating playing with shaders, but at the same time, it can be very rewarding too