[SOLVED] PixelShader for replacing colors not working as expected

Hey,

Iā€™m writing a pixel shader for replacing colors in a texture.
I have a grayscale texture with 4 colors. (source)
Then I have a texture which is 4x1 pixels and it contains the colors of the grayscale texture. (palette_source)
The third texture, which is 4x1 pixels too, has the actual colors to replace the grayscale colors with. (palette)

texture source;
sampler source_sampler = sampler_state
{
    Texture = (source);
};
texture palette_source;
sampler palette_source_sampler = sampler_state
{
    Texture = (palette_source);
};
texture palette;
sampler palette_sampler = sampler_state
{
    Texture = (palette);
};

float4 PixelShaderFunction(float2 coords : TEXCOORD0) : COLOR0
{
    float4 color = tex2D(source_sampler, coords);
    
    if (color.r == tex2D(palette_source_sampler, float2(0, 0)).r)
    {
        return tex2D(palette_sampler, float2(0, 0));
    }
    else if (color.r == tex2D(palette_source_sampler, float2(1, 0)).r)
    {
        return tex2D(palette_sampler, float2(1, 0));
    }
    else if (color.r == tex2D(palette_source_sampler, float2(2, 0)).r)
    {
        return tex2D(palette_sampler, float2(2, 0));
    }
    else if (color.r == tex2D(palette_source_sampler, float2(3, 0)).r)
    {
        return tex2D(palette_sampler, float2(3, 0));
    }

    return color;
}

technique Technique1
{
    pass Pass1
    {
        PixelShader = compile ps_4_0_level_9_1 PixelShaderFunction();
    }
}

Above is my shader script. It takes the three textures and looks up the target color for the pixel and then returns it.
As I am new to shaders, this looks pretty simple and theoretically working to me.
In my ā€œsprite classā€, I execute the following in the Draw method:

			Shader.Parameters["source"].SetValue(texture);
			Shader.Parameters["palette"].SetValue(palette);
			Shader.Parameters["palette_source"].SetValue(paletteSource);
			Shader.CurrentTechnique.Passes[0].Apply();

The texture is drawn afterwards with:

spriteBatch.Draw(texture, new Rectangle(x - GameManager.Instance.CurrentRoom.XView, y - GameManager.Instance.CurrentRoom.YView, (int)Math.Round(keyframeWidth * xScale), (int)Math.Round(keyframeHeight * yScale)), new Rectangle(keyframeX, keyframeY, keyframeWidth, keyframeHeight), blend * alpha, angle, new Vector2(xOffset, yOffset), SpriteEffects.None, 0f);

Now, instead of a train I drew, it looks like this:

Also, for reference, here are my textures I created:

(source) Scaled 2x

(palette_source) Scaled 4x

(palette) Scaled 4x

Can anyone point out what I am doing wrong here?

Thanks in advance!

I canā€™t see your image for some reason. The palettes and the train work, but the larger image above doesnā€™t work.

Iā€™m not super well versed in shaders, but I seem to recall the coordinate system for shaders isnā€™t in pixels. So the part where youā€™re doing your colour lookup, instead of using an offset of 1, 2, or 3, for those pixelsā€¦ you need to calculate what the size of a pixel is.

In this case, since I think your palette image is one pixel each, the size of a pixel would be 1/4 = 0.25.

Index 0 = 0
Index 1 = 0.25
Index 2 = 0.5
Index 3 = 0.75

Maybe try that?

Ah, so 1 equals the full width of the texture.
I did that, but the results are still the same. :frowning:

if (color.r == tex2D(palette_source_sampler, float2(0, 0)).r)
{
    return tex2D(palette_sampler, float2(0, 0));
}
else if (color.r == tex2D(palette_source_sampler, float2(0.25, 0)).r)
{
    return tex2D(palette_sampler, float2(0.25, 0));
}
else if (color.r == tex2D(palette_source_sampler, float2(0.5, 0)).r)
{
    return tex2D(palette_sampler, float2(0.5, 0));
}
else if (color.r == tex2D(palette_source_sampler, float2(0.75, 0)).r)
{
    return tex2D(palette_sampler, float2(0.75, 0));
}

I also noticed the gif not working, but if you click on it, itā€™ll show up in full size, at least for me.

Oh, yea clicking it works :slight_smile:

And yea, in UV, they range from 0 to 1, so you just have to map it.

Iā€™ll try to take a more in depth look at your shader in a bit then, maybe something else is wrong :slight_smile:

1 Like

So, HLSL is a huge pain in the butt. I did some mucking around and managed to get something working, but not in a way that I like.

I couldnā€™t get the sampler2d for the textures being passed into the shader to work correctly. Whenever I tried, everything would just start to mess up. Itā€™s extra difficult because MonoGame seems to do some optimizations when it compiles the shaders and this makes debugging difficult. I might play around with this a bit more later, but hopefully someone with more shader experience will have some input!

So instead, I tried a different approach. Instead of using the textures and associated samplers, I figured putting the palette into float4 arrays would work. This was a huge pain to get working because the compiler again spewed errors. First my loops were too large, and then second, I was using too many instructionsā€¦ even though I canā€™t for the life of me figure out why. Iā€™ve certainly written more complex shaders in XNA before.

Anyway, I pared it down to thisā€¦ which seems to do what you want, but itā€™s pretty specialized. I suspect thereā€™s a better solution here and I think your approach using textures might be more ā€œshader friendlyā€. Iā€™m gonna keep playing around, but hereā€™s something that works at least :wink:

  • Shader Code
#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

sampler2D InputSampler = sampler_state
{
	Texture = <SpriteTexture>;
};

float4 xSourcePal[4];
float4 xTargetPal[4];

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 UV : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 cIn = tex2D(InputSampler, input.UV) * input.Color;
	float4 cOut = cIn;

	for (int i = 0; i < 4; i++)
	{
		if (xSourcePal[i].r == cIn.r && xSourcePal[i].g == cIn.g && xSourcePal[i].b == cIn.b)
		{
			cOut = xTargetPal[i];
			break;
		}
	}

	return cOut;
}

technique Technique1
{
	pass Pass1
	{
		PixelShader = compile PS_SHADERMODEL PixelShaderFunction();
	}
}

Game1.cs

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;

namespace RecolourTest
{
    /// <summary>
    /// This is the main type for your game.
    /// </summary>
    public class Game1 : Game
    {
        GraphicsDeviceManager _graphics;
        SpriteBatch _spriteBatch;

        RenderTarget2D _target;
        Texture2D _train;
        Texture2D _sourcePal;
        Texture2D _targetPal;

        Vector4[] _sourcePalData;
        Vector4[] _targetPalData;

        Effect _effect;

        public Game1()
        {
            _graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";
            this.IsMouseVisible = true;
        }

        protected override void Initialize()
        {
            base.Initialize();
        }

        protected override void LoadContent()
        {
            _spriteBatch = new SpriteBatch(GraphicsDevice);

            _target = new RenderTarget2D(_graphics.GraphicsDevice, _graphics.PreferredBackBufferWidth, _graphics.PreferredBackBufferHeight);
            _train = this.Content.Load<Texture2D>("train");

            _sourcePal = this.Content.Load<Texture2D>("source_pal");
            _targetPal = this.Content.Load<Texture2D>("target_pal");

            _effect = this.Content.Load<Effect>("RecolourEffect");

            Color[] sourceData = new Color[_sourcePal.Width * _sourcePal.Height];
            _sourcePal.GetData<Color>(sourceData);
            _sourcePalData = new Vector4[sourceData.Length];
            for (int i = 0; i < sourceData.Length; i++)
                _sourcePalData[i] = sourceData[i].ToVector4();

            Color[] targetData = new Color[_targetPal.Width * _targetPal.Height];
            _targetPal.GetData<Color>(targetData);
            _targetPalData = new Vector4[targetData.Length];
            for (int i = 0; i < targetData.Length; i++)
                _targetPalData[i] = targetData[i].ToVector4();
        }

        protected override void UnloadContent()
        {
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
                Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            _graphics.GraphicsDevice.SetRenderTarget(_target);
            _graphics.GraphicsDevice.Clear(Color.Transparent);
            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp);
            _spriteBatch.Draw(_train, new Vector2(10, 10), Color.White);
            _spriteBatch.Draw(_sourcePal, new Vector2(10 + _train.Width + 10, 10), Color.White);
            _spriteBatch.Draw(_targetPal, new Vector2(10 + _train.Width + 10, 10 + _sourcePal.Height + 10), Color.White);
            _spriteBatch.End();

            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp, null, null, _effect, null);
            _effect.Parameters["xSourcePal"].SetValue(_sourcePalData);
            _effect.Parameters["xTargetPal"].SetValue(_targetPalData);
            _effect.CurrentTechnique.Passes[0].Apply();
            _spriteBatch.Draw(_train, new Vector2(10, 10 + _train.Height + 10), Color.White);
            _spriteBatch.End();

            float scale = 4f;
            _graphics.GraphicsDevice.SetRenderTarget(null);
            _graphics.GraphicsDevice.Clear(Color.CornflowerBlue);
            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp);
            _spriteBatch.Draw(_target, Vector2.Zero, null, Color.White, 0, Vector2.Zero, scale, SpriteEffects.None, 0f);
            _spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}

(Set up your content project as required)

1 Like

Oh, side note, if your intent here is to do palette swaps, it might not be a bad idea to use a source image where the colour corresponds directly to a palette index, then pass the palette as an array and index it directly (so you donā€™t have to search).

Hey! I figured it out :smiley:

I think the key thing that was needed was to tell the palette samplers that they needed to use the POINT filter. By default I think they were using something else that was causing weird results. Also, not sure if you need to use the VertexShaderOutput struct, or if you can do it your way. Play around and see, I guess.

I still recommend changing your source image to treat each pixel as an index, then using that to look up the correct colour in the resulting texture since itā€™ll be a little more generic and optimized; however, this works. Hereā€™s my code.

  • Shader
#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

sampler2D InputSampler : register(s0);

texture xSourcePal;
sampler2D SourceSampler = sampler_state
{
	Texture = <xSourcePal>;
	Filter = POINT;
};

texture xTargetPal;
sampler2D TargetSampler = sampler_state
{
	Texture = <xTargetPal>;
	Filter = POINT;
};

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 UV : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 cIn = tex2D(InputSampler, input.UV) * input.Color;
	float4 cOut = cIn;

	float4 c0 = tex2D(SourceSampler, float2(0, 0));
	float4 c1 = tex2D(SourceSampler, float2(0.25, 0));
	float4 c2 = tex2D(SourceSampler, float2(0.50, 0));
	float4 c3 = tex2D(SourceSampler, float2(0.75, 0));
	
	if (cIn.r == c0.r && cIn.g == c0.g && cIn.b == c0.b)
	{
		cOut = tex2D(TargetSampler, float2(0, 0));
	}
	else if (cIn.r == c1.r && cIn.g == c1.g && cIn.b == c1.b)
	{
		cOut = tex2D(TargetSampler, float2(0.25, 0));
	}
	else if (cIn.r == c2.r && cIn.g == c2.g && cIn.b == c2.b)
	{
		cOut = tex2D(TargetSampler, float2(0.50, 0));
	}
	else if (cIn.r == c3.r && cIn.g == c3.g && cIn.b == c3.b)
	{
		cOut = tex2D(TargetSampler, float2(0.75, 0));
	}

	return cOut;
}

technique Technique1
{
	pass Pass1
	{
		PixelShader = compile PS_SHADERMODEL PixelShaderFunction();
	}
}
  • Game1.cs
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;

namespace RecolourTest
{
    public class Game1 : Game
    {
        GraphicsDeviceManager _graphics;
        SpriteBatch _spriteBatch;

        RenderTarget2D _target;
        Texture2D _train;
        Texture2D _sourcePal;
        Texture2D _targetPal;

        Effect _effect;

        public Game1()
        {
            _graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";
            this.IsMouseVisible = true;
        }

        protected override void Initialize()
        {
            base.Initialize();
        }

        protected override void LoadContent()
        {
            _spriteBatch = new SpriteBatch(GraphicsDevice);

            _target = new RenderTarget2D(_graphics.GraphicsDevice, _graphics.PreferredBackBufferWidth, _graphics.PreferredBackBufferHeight);
            _train = this.Content.Load<Texture2D>("train");

            _sourcePal = this.Content.Load<Texture2D>("source_pal");
            _targetPal = this.Content.Load<Texture2D>("target_pal");

            _effect = this.Content.Load<Effect>("RecolourEffect2");
        }

        protected override void UnloadContent()
        {
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed || Keyboard.GetState().IsKeyDown(Keys.Escape))
                Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            _graphics.GraphicsDevice.SetRenderTarget(_target);
            _graphics.GraphicsDevice.Clear(Color.Transparent);
            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp);
            _spriteBatch.Draw(_train, new Vector2(10, 10), Color.White);
            _spriteBatch.Draw(_sourcePal, new Vector2(10 + _train.Width + 10, 10), Color.White);
            _spriteBatch.Draw(_targetPal, new Vector2(10 + _train.Width + 10, 10 + _sourcePal.Height + 10), Color.White);
            _spriteBatch.End();

            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp, null, null, _effect, null);
            _effect.Parameters["xSourcePal"].SetValue(_sourcePal);
            _effect.Parameters["xTargetPal"].SetValue(_targetPal);
            _effect.CurrentTechnique.Passes[0].Apply();
            _spriteBatch.Draw(_train, new Vector2(10, 10 + _train.Height + 10), Color.White);
            _spriteBatch.End();

            float scale = 4f;
            _graphics.GraphicsDevice.SetRenderTarget(null);
            _graphics.GraphicsDevice.Clear(Color.CornflowerBlue);
            _spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.PointClamp);
            _spriteBatch.Draw(_target, Vector2.Zero, null, Color.White, 0, Vector2.Zero, scale, SpriteEffects.None, 0f);
            _spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}

  • Result
1 Like

This awesome!
Thank you so much! :smiley:
Iā€™ll try it out as soon as possible and will give a feedback. :slight_smile:

EDIT:

I see that you are using a new SpriteBatch.Begin() for everything.
Is there a way to just apply the shader to a single texture, with only a single SpriteBatch.Begin()?

Nevermind, I modified it so it is working now! :smiley:
Thanks again!

Yea I just did this so I could render to a target and then scale things up :wink: