[Solved] SSAO Issue

EDIT: This used to be about hemisphere-oriented SSAO, but I was having too many difficulties with that and read that it’s more susceptible to artifacts, so I changed to spherical, but I’m still having problems (although it’s much closer!)

I’ve loosely been following a number of tutorials, but I’m having a few issues with my SSAO.

This screenshot shows world-space normals, encoded depth, and the ambient occlusion map, with the composition in the background. There’s darkness on the floor by the right wall, but not at the base of the wall itself. There’s also occasionally a strange cloud of darkness around the head of the torch, but nothing around the base. Additionally, it seems like maybe one of my transformations is incorrect, because all of the ambient occlusion varies so much when the camera moves around.

Here are the main bits of my shader:

// Vertex Input Structure
struct VSI
{
	float3 Position : POSITION0;
	float2 UV : TEXCOORD0;
};

// Vertex Output Structure
struct VSO
{
	float4 Position : POSITION0;
	float2 UV : TEXCOORD0;
	float3 ViewRay : TEXCOORD1;
};

// Vertex Shader
VSO VS(VSI input)
{
	// Initialize Output
	VSO output;

	// Pass Position
	output.Position = float4(input.Position, 1);

	// Pass Texcoord's
	output.UV = input.UV;

	output.ViewRay = float3(
		-input.Position.x * TanHalfFov * GBufferTextureSize.x / GBufferTextureSize.y,
		input.Position.y * TanHalfFov,
		1);

	// Return
	return output;
}

float3 randomNormal(float2 uv)
{
	float noiseX = (frac(sin(dot(uv, float2(15.8989f, 76.132f) * 1.0f)) * 46336.23745f));
	float noiseY = (frac(sin(dot(uv, float2(11.9899f, 62.223f) * 2.0f)) * 34748.34744f));
	float noiseZ = (frac(sin(dot(uv, float2(13.3238f, 63.122f) * 3.0f)) * 59998.47362f));
	return normalize(float3(noiseX, noiseY, noiseZ));
}

//Pixel Shader
float4 PS(VSO input) : COLOR0
{
	//Sample Vectors
	float3 kernel[8] =
	{
		float3(0.355512, -0.709318, -0.102371),
		float3(0.534186, 0.71511, -0.115167),
		float3(-0.87866, 0.157139, -0.115167),
		float3(0.140679, -0.475516, -0.0639818),
		float3(-0.207641, 0.414286, 0.187755),
		float3(-0.277332, -0.371262, 0.187755),
		float3(0.63864, -0.114214, 0.262857),
		float3(-0.184051, 0.622119, 0.262857)
	};

	// Get Normal from GBuffer
	half3 normal = DecodeNormalRGB(tex2D(GBuffer1, input.UV).rgb);
	normal = normalize(mul(normal, ViewIT));

	// Get Depth from GBuffer
	float2 encodedDepth = tex2D(GBuffer2, input.UV).rg;
	float depth = DecodeFloatRG(encodedDepth) * -FarClip;

	// Scale ViewRay by Depth
	float3 position = normalize(input.ViewRay) * depth;

	// Construct random transformation for the kernel
	float3 rand = randomNormal(input.UV);

	// Sample
	float numSamplesUsed = 0;
	float occlusion = 0.0;
	for (int i = 0; i < NUM_SAMPLES; ++i) {

		// Reflect the sample across a random normal to avoid artifacts
		float3 sampleVec = reflect(kernel[i], rand);

		// Negate the sample if it points into the surface
		sampleVec *= sign(dot(sampleVec, normal));

		// Ignore the sample if it's almost parallel with the surface to avoid depth-imprecision artifacts
		//if (dot(sampleVec, normal) > 0.15) // TODO: is this necessary for our purposes?
		//{
			// Offset the sample by the current pixel's position
			float4 samplePos = float4(position + sampleVec * SampleRadius, 1);

			// Project the sample to screen space
			float2 samplePosUV = float2(dot(samplePos, ProjectionCol1), dot(samplePos, ProjectionCol2));
			samplePosUV /= dot(samplePos, ProjectionCol4);

			// Convert to UV space
			samplePosUV = (samplePosUV + 1) / 2;

			// Get sample depth
			float2 encodedSampleDepth = tex2D(GBuffer2, samplePosUV).rg;
			float sampleDepth = DecodeFloatRG(encodedSampleDepth) * -FarClip;

			// Calculate and accumulate occlusion
			float diff = 1 * max(sampleDepth - depth, 0.0f);
			occlusion += 1.0f / (1.0f + diff * diff * 0.1);
			++numSamplesUsed;
		//}
	}

	occlusion /= numSamplesUsed;
	return float4(occlusion, occlusion, occlusion, 1.0);
}

I’d really appreciate if anyone could give me a hand, because I’ve been stuck on this for a few days scouring the web and these forums. Thanks!

try changing the radius you work with to something smaller, I think the output on the right looks fine for a large radius

In that screenshot I intentionally increased the radius to try to make some of the problems more obvious. Like I said, in the screenshot above, there’s darkness on the floor near the right wall, but no darkness at the bottom of the right wall itself.

When I decrease the radius here, you can also see the dark cloud around the top of the torch but no darkness around the base (which is touching the ground).

To demonstrate the issue with the ambient occlusion varying drastically with camera orientation, compare the first screenshot with this one, rotated towards the wall that was initially on the right. While the first screenshot showed occlusion only on the floor and not the wall, now that I’m facing the wall it’s only on the wall and not on the floor.The corner on the top right also shows a sudden change where one wall only has occlusion on the wall, while an adjacent wall only has occlusion on the floor. Could this be an issue with the projection transformation or with transforming the kernel?

I tried looking at the individual pieces used to compute ambient occlusion. Reconstructed screen space normals and position both looked correct to me. I can post screenshots of those too if it would help.

It’s a screenspace effect, it’s going to vary with the camera orientation drastically.

The dot check isn’t optional, if your samples aren’t in the hemisphere of the origin normal than they’re all fudged and you get bad samples canceling or exacerbating the valid ones. Ideally it’s the bent normal’s hemisphere. I doubt that’s the whole issue though.

You’ve got something going on.

Your wall edges are just because they’re bordering void space - the differences are so extreme that they’re falsely perfectly perpendicular, that’s going to be bad no matter what. If you can, you can stencil that out. Fixing the dot test for the hemisphere should get rid of it though.

Even if it’s expected to vary with camera orientation, there’s still something clearly wrong when the floor straight ahead of the camera never has occlusion, and the walls to the left and right of the camera never have occlusion. And thanks for the tip, but my main question isn’t about the wall edges right now, although that’s still good to know. (Maybe you mentioned it because they’re circled in the picture, but the circles were referring to the other oddities.) Even if this screenspace effect isn’t expected to be perfect, I’m still quite certain something’s wrong with it right now.

To be clear, when I say it varies drastically depending on camera orientation, I don’t mean the fuzziness in particular pixels turning on and off when the camera turns – I mean entire regions that were dark go light and vice versa.

EDIT: Interestingly, it’s starting to look a little better if I subtract my sample vector from my position instead of add it, but I’m not sure why.

I think the negation was due to an issue with the UV coordinates related to the camera’s handedness. I fixed that, and I found a different ambient occlusion calculation that uses a range check, which got rid of the dark cloud around the top of the torch and looks much better in corners. The problem now is that the ambient occlusion seems to scale inversely with distance from the center. I feel like there should be some simple factor I’m missing to fix that.

Here is my updated pixel shader:

float4 PS(VSO input) : COLOR0
{
	//Sample Vectors
	float3 kernel[8] =
	{
		float3(0.03853108, -0.02882797, 0.08766016),
		float3(0.07369451, 0.07844372, 0.03776182),
		float3(-0.03775074, 0.15020743, 0.02065602),
		float3(0.15360640, 0.03035969, 0.16374959),
		float3(0.22924014, 0.08017805, -0.21597555),
		float3(-0.38877658, 0.09169313, 0.21060350),
		float3(0.04701450, -0.56492682, -0.21491018),
		float3(-0.68454995, -0.00186773, -0.39243790),
	};

	// Get Normal from GBuffer
	half3 normal = DecodeNormalRGB(tex2D(GBuffer1, input.UV).rgb);
	normal = normalize(mul(normal, ViewIT));

	// Get Depth from GBuffer
	float2 encodedDepth = tex2D(GBuffer2, input.UV).rg;
	float depth = -DecodeFloatRG(encodedDepth) * FarClip;

	float radius = SampleRadius * FarClip / -depth;

	// Scale ViewRay by Depth
	float3 position = normalize(input.ViewRay) * -depth;

	// Get a random surface normal to randomize the kernel
	float3 rand = randomNormal(input.UV);

	// Sample
	float numSamplesUsed = 0;
	float occlusion = 0.0;
	for (int i = 0; i < NUM_SAMPLES; ++i) {

		// Reflect the sample across a random normal to avoid banding
		float3 sampleVec = reflect(kernel[i], rand);

		// Negate the sample if it points into the surface
		sampleVec *= sign(dot(sampleVec, normal));

		// Ignore the sample if it's almost parallel with the surface to avoid depth-imprecision artifacts
		//if (dot(sampleVec, normal) > 0.15) // TODO: is this necessary for our purposes?
		//{
			// Offset the sample by the current pixel's position
			float4 samplePos = float4(position + sampleVec * radius, 1);

			// Project the sample to screen space
			float2 samplePosUV = float2(dot(samplePos, ProjectionCol1), dot(samplePos, ProjectionCol2));
			samplePosUV /= dot(samplePos, ProjectionCol4);

			// Convert to UV space
			samplePosUV = (samplePosUV + 1) / 2;
			samplePosUV.y = 1 - samplePosUV.y;

			// Get sample depth
			float2 encodedSampleDepth = tex2D(GBuffer2, samplePosUV).rg;
			float sampleDepth = -DecodeFloatRG(encodedSampleDepth) * FarClip;

			// Calculate and accumulate occlusion
			float rangeCheck = abs(depth - sampleDepth) < radius ? 1.0 : 0.0;
			occlusion += (sampleDepth >= samplePos.z ? 1.0 : 0.0) * rangeCheck;
			++numSamplesUsed;
		//}
	}

	occlusion = 1.0 - (occlusion / numSamplesUsed);
	return float4(occlusion, occlusion, occlusion, 1.0);
}

I found the problem! I’m not supposed to be normalizing the ViewRay. That should’ve been obvious, because normalizing it would change z, which means the result after multiplying by depth wouldn’t actually have a z value of depth. This caused the area of AO to be mapped from a frustum to a cone, which explains why I only saw occlusion in a circle around the center of the screen.

I’ll post the final code and screenshot after tinkering with the parameters a bit.

UPDATE: Here’s the final result! I’m quite happy with it, and I’ll probably add the blur at some point.

Here’s the final shader code:

#define NUM_SAMPLES 16

// Vertex Input Structure
struct VSI
{
	float3 Position : POSITION0;
	float2 UV : TEXCOORD0;
};

// Vertex Output Structure
struct VSO
{
	float4 Position : POSITION0;
	float2 UV : TEXCOORD0;
	float3 ViewRay : TEXCOORD1;
};

// Vertex Shader
VSO VS(VSI input)
{
	// Initialize Output
	VSO output;

	// Pass Position
	output.Position = float4(input.Position, 1);

	// Pass Texcoord's
	output.UV = input.UV;

	output.ViewRay = float3(
		input.Position.x * TanHalfFov * GBufferTextureSize.x / GBufferTextureSize.y,
		input.Position.y * TanHalfFov,
		-1);

	// Return
	return output;
}

float3 randomNormal(float2 uv)
{
	float noiseX = (frac(sin(dot(uv, float2(15.8989f, 76.132f) * 1.0f)) * 46336.23745f));
	float noiseY = (frac(sin(dot(uv, float2(11.9899f, 62.223f) * 2.0f)) * 34748.34744f));
	float noiseZ = (frac(sin(dot(uv, float2(13.3238f, 63.122f) * 3.0f)) * 59998.47362f));
	return normalize(float3(noiseX, noiseY, noiseZ));
}

// Pixel Shader
float4 PS(VSO input) : COLOR0
{
	//Sample Vectors
	float3 kernel[16] =
	{
		float3(0.03853108, -0.02882797, 0.08766016),
		float3(0.07369451, 0.07844372, 0.03776182),
		float3(-0.03775074, 0.15020743, 0.02065602),
		float3(0.15360640, 0.03035969, 0.16374959),
		float3(0.22924014, 0.08017805, -0.21597555),
		float3(-0.38877658, 0.09169313, 0.21060350),
		float3(0.04701450, -0.56492682, -0.21491018),
		float3(-0.68454995, -0.00186773, -0.39243790),
		float3(0.03190635, -0.01036435, 0.09420491),
		float3(-0.02874287, 0.08323552, 0.07249792),
		float3(0.03564418, -0.15075309, 0.02042203),
		float3(-0.00353680, 0.12755659, -0.18720944),
		float3(-0.11076603, -0.24070639, 0.18819224),
		float3(-0.24215766, 0.01602741, -0.38080373),
		float3(0.01062293, 0.07784872, 0.60113708),
		float3(0.47454790, -0.60003461, 0.19334525)
	};

	// Get Normal from GBuffer
	half3 normal = DecodeNormalRGB(tex2D(GBuffer1, input.UV).rgb);
	normal = normalize(mul(normal, ViewIT));

	// Get Depth from GBuffer
	float2 encodedDepth = tex2D(GBuffer2, input.UV).rg;
	float depth = DecodeFloatRG(encodedDepth);

	// Scale radius by normalized depth to give the illusion of perspective
	float radius = SampleRadius / depth;

	// Denormalize depth (no need to negate)
	depth *= -FarClip;

	// Scale ViewRay by absolute value of depth
	// (which is always negative from our camera frame)
	float3 position = input.ViewRay * -depth;

	// Get a random surface normal to randomize the kernel
	float3 rand = randomNormal(input.UV);

	// Sample
	float occlusion = 0.0;
	for (int i = 0; i < NUM_SAMPLES; ++i) {

		// Reflect the sample across a random normal to avoid banding
		float3 sampleVec = reflect(kernel[i], rand);

		// Negate the sample if it points into the surface
		sampleVec *= sign(dot(sampleVec, normal));

		// Ignore the sample if it's almost parallel with the surface to avoid depth-imprecision artifacts
		if (dot(sampleVec, normal) > 0.15)
		{
			// Offset the sample by the current pixel's position
			float4 samplePos = float4(position + sampleVec * radius, 1);

			// Project the sample to screen space
			float2 samplePosUV = float2(dot(samplePos, ProjectionCol1), dot(samplePos, ProjectionCol2));
			samplePosUV /= dot(samplePos, ProjectionCol4);

			// Convert to UV space
			samplePosUV = (samplePosUV + 1) / 2;
			samplePosUV.y = 1 - samplePosUV.y;

			// Get sample depth
			float2 encodedSampleDepth = tex2D(GBuffer2, samplePosUV).rg;
			float sampleDepth = -DecodeFloatRG(encodedSampleDepth) * FarClip;

			// Calculate and accumulate occlusion
			float rangeCheck = abs(depth - sampleDepth) < radius ? 1.0 : 0.0;
			occlusion += (sampleDepth >= samplePos.z ? 1.0 : 0.0) * rangeCheck;
		}
	}

	occlusion = 1 - (Intensity * occlusion / NUM_SAMPLES);
	return float4(occlusion, occlusion, occlusion, 1.0);
}
3 Likes

And here’s after blurring, a little more tuning, and changing the composition to use multiplication instead of subtraction. I even got the channel to fit into my normals render target so I don’t need any extra memory for this!

Here’s the same scene without SSAO for comparison.

1 Like

Nice! The SSAO really adds to the scene!

1 Like