TiledMap effect (pixel shader) is not working

wakaja · December 10, 2020, 3:00pm

Hi all,

I’m trying to apply an Effect (pixel shader) to a TiledMap of the monogame.extended framework.
This shader is very simple since it only forces the color of the pixels to black and white. It works perfectly on a simple 2D texture displayed with the spriteBatch.Draw method.
My TiledMap is displayed correctly without using any effects:
_mapRenderer.Draw (_camera.GetViewMatrix (), null, null);
on the other hand, when I pass the Effect as a parameter, the map simply no longer appears on the screen:
_mapRenderer.Draw (_camera.GetViewMatrix (), null, effect);

Does anyone use any effects that work on a TiledMap? Or would have an idea on the problem, if it is a bug or a misunderstanding on my part …?

Thanks for your help

Here is the code of the effect used:

#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

sampler s0;

struct VertexShaderOutput
{
float4 Position: SV_POSITION;
float4 Color: COLOR0;
float2 TextureCoordinates: TEXCOORD0;
};

float4 MainPS (VertexShaderOutput input): COLOR0
{
float4 color = tex2D (s0, input.TextureCoordinates);
color.gb = color.r;
return color;
}

SpriteDrawing technique
{
pass P0
{
PixelShader = compile PS_SHADERMODEL MainPS ();
}
};

LithiumToast · December 14, 2020, 2:48am

github.com

craftworkgames/MonoGame.Extended/blob/develop/src/dotnet/MonoGame.Extended.Tiled/Renderers/TiledMapRenderer.cs#L153


				foreach (var subLayer in groupLayer.Layers)
					Draw(subLayer, offset, ref viewMatrix, ref projectionMatrix, effect, depth);
			}
			else
			{
				_worldMatrix.Translation = new Vector3(offset, depth);

				var effect1 = effect ?? _defaultEffect;
				var tiledMapEffect = effect1 as ITiledMapEffect;
				if (tiledMapEffect == null)
					return;

				// model-to-world transform
				tiledMapEffect.World = _worldMatrix;
				tiledMapEffect.View = viewMatrix;
				tiledMapEffect.Projection = projectionMatrix;

				foreach (var layerModel in _mapModel.LayersOfLayerModels[layer])
				{
					// desired alpha
					tiledMapEffect.Alpha = layer.Opacity;

Your Effect needs to implement this interface: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Tiled/Renderers/TiledMapEffect.cs#L6

Also, I don’t see a vertex shader in your shader program. The vertex shader is important as it transforms the positions of the tiles from model space all the way to clip space.

wakaja · December 14, 2020, 10:12pm

Thanks for your reply @LithiumToast !

I tried to apply your advices but I didn’t manage to make my effect working.
I probably have a wrong approach but here is what I did :

Created a TiledMapEffect (in order to implement ITiledMapEffect) using the clone Constructor to use my shader file :

mapEffect = new TiledMapEffect(Content.Load(“MapEffect”));

I added a vertex shader in my shader file, here is the MapEffect.fx content :

#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0_level_9_1
#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

matrix WorldViewProjection;

Texture2D Texture;
sampler2D TextureSampler = sampler_state
{
Texture = (Texture);
};

struct VertexShaderInput
{
float4 Position : POSITION0;
float4 Color : COLOR0;
};

struct VertexShaderOutput
{
float4 Position : SV_POSITION;
float4 Color : COLOR0;
float2 TextureCoordinates : TEXCOORD0;
};

VertexShaderOutput MainVS(in VertexShaderInput input)
{
VertexShaderOutput output = (VertexShaderOutput)0;
output.Position = mul(input.Position, WorldViewProjection);
output.Color = input.Color;

return output;
}

float4 MainPS(VertexShaderOutput input) : COLOR
{
float4 color = tex2D(TextureSampler,input.TextureCoordinates);
if (color.a) {
color.gb = color.r;
}
return color;
}

technique BasicColorDrawing
{
pass P0
{
VertexShader = compile VS_SHADERMODEL MainVS();
PixelShader = compile PS_SHADERMODEL MainPS();
}
};

the call to TiledMapRenderer draw method remains the same :

_mapRenderer.Draw(_camera.GetViewMatrix(), null, effect);

I tried affect or not a WorldViewProjection matrix (
mapEffect.Parameters[“WorldViewProjection”].SetValue(_level.getCameraViewMatrix());) with the same result :
NullReferenceException, in the MonoGame.Extended.Graphics.Effects.DefaultEffect.UpdateMaterialColor() method.

This exception usely deals with trying to set a missing shader variable or setting a shader variable that is not used.
The only one I found in the UpdateMaterialColor() method is : _diffuseColorParameter.SetValue(diffuseColorVector4). If I declare a variable float4 DiffuseColor; then I must use it and I don’t know what to do with it.
(if I try to use it I end up with a IndexOutOfRangeException in MonoGame.Extended.Graphics.Effects.DefaultEffect.OnApply(), so I guess it’s not a good way to go…)

I must misanderstanding something in the TiledMapEffect working and about shaders. Could you put me on the right way ?

LithiumToast · December 14, 2020, 11:46pm

First your matrix is not being updated correctly. The WorldViewProjection matrix is mapped here: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/MatrixChainEffect.cs#L96

This matrix is important part of how 3D rendering works with rasterization. You can see the code in C# for creating the matrix here: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/MatrixChainEffect.cs#L133

If you are unfamiliar with these concepts I recommend you refresh on this image as found here: https://learnopengl.com/Getting-started/Coordinate-Systems

Note that MonoGame has XNA heritage which has roots in DirectX; OpenGL traditionally calls the first matrix “Model” where DirectX calls it “World”. They pragmatically have the same meaning.

Second, another problem which you mentioned in your stack trace is that you are using DefaultEffect which expects a shader parameter of DiffuseColor: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/DefaultEffect.cs#L146

As such, the parameter is null (can’t be found) and you get a null reference exception when calling UpdateMaterialColor https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/DefaultEffect.cs#L188

This method is called from here: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/DefaultEffect.cs#L165

My recommendation is to either not use DefaultEffect and write your own C# code or make sure your shader matches what is expected of TiledMapEffect and it’s inherited classes such as DefaultEffect, and MatrixChainEffect.

LithiumToast · December 14, 2020, 11:49pm

Btw, you can find the .fx for DefaultEffect here: https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/Resources/DefaultEffect.fx

It’s not easy to grasp at first as it uses macros; feel free to ask questions.

wakaja · December 15, 2020, 1:36pm

Thank you very much for your help @LithiumToast !

I choosed the 2nd option, “make sure your shader matches what is expected of TiledMapEffect” as it’s far more simple and it preserves the mechanic of TiledMapEffect, and it’s works

Here is what I did following your advices

C# code remains the same :

Create the TiledMapEffect

mapEffect = new TiledMapEffect(Content.Load(“MapEffect”));

Draw the TiledMap

_mapRenderer.Draw(_camera.GetViewMatrix(), null, mapEffect);

Added the following files to my Content pipeline tool (MGCB editor) :

https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/Resources/DefaultEffect.fx => renamed to MapEffect.fx
https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/Resources/Macros.fxh
https://github.com/craftworkgames/MonoGame.Extended/blob/541e162887745d43669298d687100f564b9644cd/src/dotnet/MonoGame.Extended.Graphics/Effects/Resources/Structures.fxh

for both Macros.fxh and Structures.fxh files in MGCB editor, in the file properties Settings, set Build Action to Copy (instead of Build)

in MapEffect.fx :

change

float4 PixelShaderFunctionPositionTexture(VertexShaderOutputPositionTexture input) : SV_Target0
{
return SAMPLE_TEXTURE(Texture, input.TextureCoordinate) * DiffuseColor;
}

To

float4 PixelShaderFunctionPositionTexture(VertexShaderOutputPositionTexture input) : SV_Target0
{
float4 textureColor = SAMPLE_TEXTURE(Texture, input.TextureCoordinate) * DiffuseColor;
if (textureColor.a) {
textureColor.gb = textureColor.r;
}
return textureColor;
}

Thanks again for your your explanations and links !

LithiumToast · December 15, 2020, 2:56pm

No problem

brian_michelson · January 18, 2021, 5:58pm

Hello there,

I am trying to get the Map effects working too. Man I tried for hours @wakaja could you maybe upload the code snippet of the effect, the effect class in c# and so one? It would really help ^^ I would love to see it as a minimalistic working example and would like to add it to the Monogame-extended wiki too

greetings Brian

wakaja · January 18, 2021, 8:17pm

Hi Brian,

I don’t anderstand what missing you as everything you need is in my previous post. What is not working for you ?

The code of the effect is DefaultEffect.fx that you can find in MonoGame Extended Github repo (see link in previous post) with “PixelShaderFunctionPositionTexture” method changed as explained to turn color into black and white. I renamed the file MapEffect.fx

The effect class is “TiledMapEffect”, also from Monogame Extended framework, instanciated with the clone constructor : new TiledMapEffect(Content.Load(“MapEffect”));

When you draw your TiledMapRenderer in your game, just pass the effect in the draw method.

brian_michelson · January 19, 2021, 3:22pm

Okay it now works but is there a way that the effect only effects specific parts of the map? @LithiumToast