Need help with 3D normal map shader, cannot get it to work right

I’ve been Googling and reading tutorials and checking topics on this site and adapting various example shaders to try to get normal mapping to work in my OpenGL 3D world, but it just does not work. I don’t know what’s wrong, and I don’t know enough about shaders to fix it myself at this point. I don’t know if somehow my light positions are off or something, just have no idea. The textures draw flat, with no depth even though I am providing normal map.

I was hoping someone could at least point me in the right direction so I can stop beating my head against a wall.

This is an example of what I would like to apply the normals to; specifically the brick walls, floor, and ceilling, to add depth to the brick:

Screen111

This is one of the shaders I am trying to use, it is from digitalerr0r.net. I have tried at least 10 different normal map shaders at this point and I have been able to get most of them to compile and run for OpenGL, but the normals don’t seem to work. The texture either draws flat (as above), the same as if I’m using BasicEffect and drawing without a normal map, or not at all. In the case of below shader, it ONLY draws if I provide an ambient color. It does not draw the diffuse or specular at all, which are the portions that have the normal map applied.

float4x4 World;
float4x4 View;
float4x4 Projection;

// Light related
float4 AmbientColor;
float AmbientIntensity;

float3 LightDirection;
float4 DiffuseColor;
float DiffuseIntensity;

float4 SpecularColor;
float3 EyePosition;

texture2D ColorMap;
sampler2D ColorMapSampler = sampler_state
{
    Texture = <ColorMap>;
    MinFilter = linear;
    MagFilter = linear;
    MipFilter = linear;
};

texture2D NormalMap;
sampler2D NormalMapSampler = sampler_state
{
    Texture = <NormalMap>;
    MinFilter = linear;
    MagFilter = linear;
    MipFilter = linear;
};

struct VertexShaderInput
{
    float4 Position : POSITION0;
    float2 TexCoord : TEXCOORD0;
    float3 Normal : NORMAL0;
    float3 Binormal : BINORMAL0;
    float3 Tangent : TANGENT0;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
    float2 TexCoord : TEXCOORD0;
    float3 View : TEXCOORD1;
    float3x3 WorldToTangentSpace : TEXCOORD2;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);
    output.TexCoord = input.TexCoord;

    output.WorldToTangentSpace[0] = mul(normalize(input.Tangent), World);
    output.WorldToTangentSpace[1] = mul(normalize(input.Binormal), World);
    output.WorldToTangentSpace[2] = mul(normalize(input.Normal), World);

    output.View = normalize(float4(EyePosition, 1.0) - worldPosition);

    return output;
}

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(ColorMapSampler, input.TexCoord);

    float3 normalMap = 2.0 * (tex2D(NormalMapSampler, input.TexCoord)) - 1.0;
    normalMap = normalize(mul(normalMap, input.WorldToTangentSpace));
    float4 normal = float4(normalMap,1.0);

    float4 diffuse = saturate(dot(-LightDirection,normal));
    float4 reflect = normalize(2 * diffuse * normal - float4(LightDirection,1.0));
    float4 specular = pow(saturate(dot(reflect,input.View)),8);

    float4 finalColor = color * AmbientColor * AmbientIntensity +
                        color * DiffuseIntensity * DiffuseColor * diffuse +
                        color * SpecularColor * specular;
    finalColor.w = 1.0f; // Make sure it's not transparent?
    return  (finalColor);
}

technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_3_0 VertexShaderFunction();
        PixelShader = compile ps_3_0 PixelShaderFunction();
    }
}
  • World/view/projection seems fine, as the walls do draw correctly - just without normal mapping.
  • Colors are all white.
  • Light direction is diagonal: Vector3(1, -1, 1).
  • eyePosition I am setting to my current camera position in the world (where we’re looking from).

Some part of this must be wrong?

Don’t turn a float3 into a float4 by filling the w-compnent with a 1. You are doing this multiple times. It changes the result of the normalize. If anything, you fill up with a 0. Even better, just stick to float3. Are you turning everything into float4 to get rid of the truncation warning? You can do it like this:
float3 result = someFloat3 - someFloat4.xyz;

To find out where the pixel shader starts failing, you output intermediate results, instead of the final color, and see if the output makes sense. The first step in your calculation is the normalmap. check if you have valid normals:
return normal;

If normals are fine, check the next step:
return diffuse;

Thanks for your help! I was able to get it to work:

Screen116

I took your advice and used the float3’s instead, then I had to make a few more changes for some reason - normalize the normalMap, and I had to remove this line completely:
normalMap = normalize(mul(normalMap, input.WorldToTangentSpace));

I’m not sure why it was doing a matrix multiplication on the world to tangent space.

And then, light direction was reversed for some reason, so I changed -LightDirection to just LightDirection.

Finally, and this is my big question because I’ve seen this in every normal map shader i’ve come across so far, I had to change this:

float3 normalMap = normalize(2.0 * (tex2D(NormalMapSampler, input.TexCoord)) - 1.0);

To this:

float3 normalMap = normalize(2.0 * (tex2D(NormalMapSampler, input.TexCoord)));

Removing that “- 1.0” at the end. Maybe something changed between XNA 4.0 and Monogame? These shaders I’m repurposing were designed for XNA.

You got some bump effect now, but it’s not what you would expect from a normal map shader using a directional light. All sides of the tunnel are equally lit. With a directional light, only half should be lit, the other half should be dark. A better lighting choice for a tunnel like this would be point lights, especially considering those torch holders on the walls, but it’s fine to start with directional lights.

This line is responsible for rotating the normals to match the polygon’s facing direction. If you don’t do that, all normals are facing in the same direction, no matter which way the polygon they are on is facing. That’s why every side looks the same now, which is not right.
Maybe your vertexbuffer doesn’t have tangents and binormals set up properly, so they are all zero. I think you have to check that option in the content builder.

The -1 is important. Without it, there will only be positive numbers, because the text2D is btw 0 and 1. You need negative numbers too. Normals don’t just point to the right, they can also point left.

Ok, I tried reverting those values and I did find a setting on my mesh in content builder called “Generate Tangent Frames” which was set to false, so I set that to true. PremultiplyVertexColors was already set to true, if that matters, and I didn’t change anything else. I don’t see anything else that sounds relevant. Now I’m getting this:

Screen117

Which seems slightly better in terms of the directional lighting but obviously something weird is happening to the texture. This is my current pixel shader:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(ColorMapSampler, input.TexCoord);

    float3 normalMap = normalize(2.0 * (tex2D(NormalMapSampler, input.TexCoord)) - 1.0);
    normalMap = normalize(mul(normalMap, input.WorldToTangentSpace));

    float3 diffuse = saturate(dot(-LightDirection, normalMap));
    float3 reflect = normalize(2 * diffuse * normalMap - LightDirection);
    float3 specular = pow(saturate(dot(reflect, (float3)input.View)),8);

    float3 finalColor = color * AmbientColor * AmbientIntensity +
                        color * DiffuseIntensity * DiffuseColor * diffuse +
                        color * SpecularColor * specular;

    return  (float4(finalColor, 1.0f));
}

Paring that shader down to only return “color * diffuse” results in the same issue.

Is there some other value or checkbox I could be missing in the content builder? On the texture, model or shader or anything I might be missing? I don’t see anything obvious.

LightDirection is not normalized, is it? You should normalize it before setting it as a shader parameter. If that doesn’t help, I’d like to see the normal map texture you are using? And maybe show what this looks like:

normalMap = normalize(mul(normalMap, input.WorldToTangentSpace));
return float4(normalMap, 1);

Normalizing LightDirection actually did help even out the lighting but did not fundamentally resolve the issue:

Screen119

Here is this normal map texture:

And the result of:

normalMap = normalize(mul(normalMap, input.WorldToTangentSpace));
return float4(normalMap, 1);

is this:

Screen118

Just very strange after I checked the GenerateTangentFrames on the mesh (simple cube). Although it is better than before, where I would not see any normal map data at all after doing the matrix mul.

Do you have scale in your world matrix?
You only want to rotate the tangent frame here, not scale it:

The world matrices are scaled before they’re passed into the shader. Should remove one of the muls on the tangent space? I’m not sure what to do with that. I am really new to all of this, i’ve only been seriously working with shaders for the past week or so. I will play around and see if I can get it to work.

EDIT: It seems the WorldToTangentSpace[2] is the scale? If I set that to simply input.Normal, things start to look a lot better. What is WorldToTangentSpace[1] ? It doesn’t seem to matter whether I matrix mul it or just leave it at input.Binormal.

Just make an additional float3x3 shader parameter, call it WorldRotation, and use it instead of World in all three lines. Pass the world matrix without scale into it.

Awesome, I tried that, here’s how it turned out:

Screen120

Seems correct given the directional lighting, right?

Ya it looks right. One thing to note is that some programs reverse the Green this is because bump mapping used to be the other name for normal mapping (some programs will still flip or not flip the green value). Not to be confused with displacement mapping.

That little bit of back lighting is because i think the light is directional or outside of the room.
I add a cut off for flat surfaces to get rid of that in this sort of scenario.

image not found

I added it to the Phong shader below that needs to have vertex data with normals and tangents.

//++++++++++++++++++++++++++++++++++++++++
// D E F I N E S
//++++++++++++++++++++++++++++++++++++++++


#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0 //_level_9_1
#define PS_SHADERMODEL ps_4_0 //_level_9_1
#endif
#define PI 3.14159265359f
#define RECIPROCAL_PI 0.31830988618f
#define EPSILON 1e-6


//++++++++++++++++++++++++++++++++++++++++
// S H A D E R  G L O B A L S
//++++++++++++++++++++++++++++++++++++++++

float3 CameraPosition;
float3 LightPosition;
float3 LightColor;

float AmbientStrength;
float DiffuseStrength;
float SpecularStrength;

matrix World;
matrix View;
matrix Projection;

//++++++++++++++++++++++++++++++++++++++++
// T E X T U R E S  A N D  S A M P L E R S
//++++++++++++++++++++++++++++++++++++++++

Texture2D TextureDiffuse;
sampler2D TextureSamplerDiffuse = sampler_state
{
	texture = (TextureDiffuse);
};

Texture2D TextureNormalMap;
sampler TextureSamplerNormalMap = sampler_state
{
	texture = (TextureNormalMap);
	AddressU = clamp;
	AddressV = clamp;
};


//++++++++++++++++++++++++++++++++++++++++
// S T R U C T S
//++++++++++++++++++++++++++++++++++++++++

struct VertexShaderInput
{
	float4 Position : POSITION0;
	float3 Normal : NORMAL0;
	float3 Tangent : NORMAL1;
	float2 TextureCoordinates : TEXCOORD0;
};

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float3 Normal : NORMAL0;
	float3 Tangent : NORMAL1;
	float2 TextureCoordinates : TEXCOORD0;
	float3 Position3D : TEXCOORD1;
};


//++++++++++++++++++++++++++++++++++++++++
// F U N C T I O N S
//++++++++++++++++++++++++++++++++++++++++

float MaxDot(float3 a, float3 b)
{
	return max(0.0f, dot(a, b));
}

// L , V
float3 HalfNormal(float3 pixelToLight, float3 pixelToCamera)
{
	return normalize(pixelToLight + pixelToCamera);
}

float IsFrontFaceToLight(float3 L, float3 N)
{
	return sign(saturate(dot(L, N)));
}

float3 GammaToLinear(float3 gammaColor)
{
	return pow(gammaColor, 2.2f);
}

// to viewer can be   pos   or  v   aka  cam - pixel
float SpecularPhong(float3 toViewer, float3 toLight, float3 normal, float shininess)
{
	float3 viewDir = normalize(-toViewer);
	float3 reflectDir = reflect(toLight, normal);
	float b = max(dot(reflectDir, viewDir), 0.0f);
	return pow(b, shininess / 4.0f); // note that the exponent is different here
}

float SpecularBlinnPhong(float3 toViewer, float3 toLight, float3 normal, float shininess)
{
	toViewer = normalize(toViewer);
	float3 halfnorm = normalize(toLight + toViewer);
	float cosb = max(dot(normal, halfnorm), 0.0f);
	return pow(cosb, shininess);
}

float SpecularSharpener(float specular, float scalar)
{
	return saturate(specular - scalar) * (1.0f / (1.0f - scalar));
}

float SpecularCurveFit(float3 V, float3 L, float3 N, float sharpness)
{
	float3 h = normalize(L + V);
	float ndoth = max(dot(N, h), 0.0f);

	float a = sharpness / (sharpness + 0.1f); // infinite sliding limit.
	float ndotl = saturate(dot(L, N));
	float r = (dot(V, reflect(-L, N)) + ndoth * 0.07f) / 1.07f;  // *ndotl;
	float result = saturate(r - a) * ( 1.0f / ( 1.0f - a) );

	return result;
}


float Falloff(float distance, float lightRadius)
{
	return pow(saturate(1.0f - pow(distance / lightRadius, 4)), 2) / (distance * distance + 1);
}

// Determines inflection position on the opposite side of a plane defined by (point and a normal) \|/
float3 InflectionPositionFromPlane(float3 anyPositionOnPlaneP, float3 theSurfaceNormalN, float3 theCameraPostionC)
{
	float camToPlaneDist = dot(theSurfaceNormalN, theCameraPostionC - anyPositionOnPlaneP);
	return theCameraPostionC - theSurfaceNormalN * camToPlaneDist * 2;
}

float ReflectionTheta(float3 L, float3 N, float3 V)
{
	return dot(V, reflect(-L, N));
}

float3 FunctionNormalMapGeneratedBiTangent(float3 normal, float3 tangent, float2 texCoords)
{
	// Normal Map
	float3 NormalMap = tex2D(TextureSamplerNormalMap, texCoords).rgb;
	NormalMap.g = 1.0f - NormalMap.g;  // flips the y. the program i used fliped the green,  bump mapping is when you don't do this i guess.
	NormalMap = NormalMap * 2.0f - 1.0f;
	float3 bitangent = normalize(cross(normal, tangent));

	float3x3 mat;
	mat[0] = bitangent; // set right
	mat[1] = tangent; // set up
	mat[2] = normal; // set forward

	return normalize(mul(NormalMap, mat)); // norm to ensure later scaling wont break it.
}




//++++++++++++++++++++++++++++++++++++++++
// V E R T E X  S H A D E R S
//++++++++++++++++++++++++++++++++++++++++

VertexShaderOutput VS(in VertexShaderInput input)
{
	VertexShaderOutput output = (VertexShaderOutput)0;

	matrix vp = mul(View, Projection);
	matrix wvp = mul(World, vp);

	output.TextureCoordinates = input.TextureCoordinates;
	output.Position3D = mul(input.Position, World);
	output.Normal = mul(input.Normal, World);
	output.Tangent = mul(input.Tangent, World);
	output.Position = mul(input.Position, wvp); // we transform the position.

	return output;
}



//++++++++++++++++++++++++++++++++++++++++
// P I X E L  S H A D E R S
//++++++++++++++++++++++++++++++++++++++++

float4 PS_Phong(VertexShaderOutput input) : COLOR
{
	float4 col = tex2D(TextureSamplerDiffuse, input.TextureCoordinates);
	float3 N = FunctionNormalMapGeneratedBiTangent(input.Normal, input.Tangent, input.TextureCoordinates); // this calls to the pixel shader function above.
	float3 P = input.Position3D;
	float3 C = CameraPosition;
	float3 V = normalize(C - P);
	float NdotV = MaxDot(N, V);
	float3 R = 2.0f * NdotV * N - V;
	float3 L = normalize(LightPosition - P);
	float3 H = HalfNormal(L, V);
	float NdotH = MaxDot(N, H);
	float NdotL = MaxDot(N, L);

	float spec= SpecularPhong(V, L, N, 100.0f);
	float3 speccol = col.rgb * LightColor;
	col.rgb =  (speccol.rgb * spec * SpecularStrength) +(col.rgb * NdotL * NdotL * DiffuseStrength) + (col.rgb * AmbientStrength);

	// this is extra it simply removes backface lighting off a polygon surface regardless.
	col.rgb = col.rgb * saturate(dot(input.Normal, L) * 4.0f); 

	return col;
}

float4 PS_BlinnPhong(VertexShaderOutput input) : COLOR
{
	float4 col = tex2D(TextureSamplerDiffuse, input.TextureCoordinates);
	float3 N = FunctionNormalMapGeneratedBiTangent(input.Normal, input.Tangent, input.TextureCoordinates);
	float3 P = input.Position3D;
	float3 C = CameraPosition;
	float3 V = normalize(C - P);
	float NdotV = MaxDot(N, V);
	float3 R = 2.0f * NdotV * N - V;
	float3 L = normalize(LightPosition - P);
	float3 H = HalfNormal(L, V);
	float NdotH = MaxDot(N, H);
	float NdotL = MaxDot(N, L);

	float spec = SpecularBlinnPhong(V, L, N, 100.0f);
	float3 speccol = col.rgb * LightColor;
	col.rgb = (speccol.rgb * spec * SpecularStrength) + (col.rgb * NdotL * NdotL * DiffuseStrength) + (col.rgb * AmbientStrength);
	return col;
}

float4 PS_Wills(VertexShaderOutput input) : COLOR
{
	float4 col = tex2D(TextureSamplerDiffuse, input.TextureCoordinates);
	float3 N = FunctionNormalMapGeneratedBiTangent(input.Normal, input.Tangent, input.TextureCoordinates);
	float3 P = input.Position3D;
	float3 C = CameraPosition;
	float3 V = normalize(C - P);
	float NdotV = MaxDot(N, V);
	float3 R = 2.0f * NdotV * N - V;
	float3 L = normalize(LightPosition - P);
	float3 H = HalfNormal(L, V);
	float NdotH = MaxDot(N, H);
	float NdotL = MaxDot(N, L);

	float spec = SpecularCurveFit(V, L, N, 1.50f);
	float3 speccol = col.rgb * LightColor;
	col.rgb = (speccol.rgb * spec * SpecularStrength) + (col.rgb * NdotL * NdotL * DiffuseStrength) + (col.rgb * AmbientStrength);
	return col;
}


//++++++++++++++++++++++++++++++++++++++++
// T E C H N I Q U E S.
//++++++++++++++++++++++++++++++++++++++++

technique Lighting_Phong
{
	pass P0
	{
		VertexShader = compile VS_SHADERMODEL
			VS();
		PixelShader = compile PS_SHADERMODEL
			PS_Phong();
	}
};

technique Lighting_Blinn
{
	pass P0
	{
		VertexShader = compile VS_SHADERMODEL
			VS();
		PixelShader = compile PS_SHADERMODEL
			PS_BlinnPhong();
	}
};

technique Lighting_Wills
{
	pass P0
	{
		VertexShader = compile VS_SHADERMODEL
			VS();
		PixelShader = compile PS_SHADERMODEL
			PS_Wills();
	}
};

Cool, thanks Will, your shader actually built correctly the first time. Most of the other ones i’ve found have needed a lot of tweaking to get building in latest Monogame for OpenGL.

@markus - Thank you so much for all the help with the normal mapping, I don’t know how I would have figured it out otherwise.

1 Like