Real estate problem in vertex shader output

1

Hi,

I’m having a bit of a problem with the main shader for my game.

Using DirectX, my main shader needs to be able to do the following things :

  • Texture mapping
  • Ambient, diffuse, and specular lighting
  • Normal mapping
  • Shadows using a shadow map
  • Reflections using a cubemap

Now because most of the work will be done in the pixel shader, my vertex shader output is becoming too crowded. I had to move on to shader model 5, because 4.9.3 was giving me an error saying that the max number of output (8) had been reached.

Here are my current VS outputs :

struct InstancingVSoutput { float4 Position : POSITION0; float2 TexCoord : TEXCOORD0; float3 View : TEXCOORD3; float3 Reflection : TEXCOORD4; float4 WorldPos : TEXCOORD5; float4 HighlightColor : TEXCOORD6; float3x3 WorldToTangentSpace : TEXCOORD7; };

Here is what each output is for :

  • View :
    This is a precalculation for specular lighting. The formula is :
    output.View = normalize(float4(EyePosition, 1.0) - worldPosition);

  • Reflection :
    This is the reflection vector. Its formula is :
    reflect(-normalize(worldPosition - EyePosition), normalize(output.WorldToTangentSpace[2]));

  • WorldPos :
    This is the “real world” position of my vertex. I need this for my shadow calculations

  • HighlightColor :
    I am using hardware instancing the draw a bunch of the same model at once, each model is being shaded with a different tint. Since the coloring is happening in the pixel shader, I need to pass the tint color to it.

  • WorldToTangentSpace :
    This is the normal, binormal, and tangent Vectors used by my normal mapping

Now I am aware there are a few things I could cut. For starters, I could cut the tangent vector, since it’s the cross product of my normal and binormal vectors. Also, the reflection and view vectors seem similar enough that only one is needed, and the other could be calculated in the pixel shader.

But I am wondering if there is a better way to approach this? With my main shader getting bigger and bigger, it seems that I am bound to encounter this problem again in the future. I have no experience with multi pass shaders, but it seems like they could help me in this situation?

Thank you for your help

2

I think it’s safe to go with 4, which is basic DirectX10 and it should support everything you need. (In case you have concerns about your players not being on DirectX11 hardware level, which is SM5)

Contrary to popular believe 4.9.3 is much much older than 4.0, which I’m sure is confusing to some.

Apart from that I think it’s fine.

I think it’s okay to have full per-pixel lighting for all objects nowadays (if you want to reduce VSOutput that would help), but the difference when precomputing in VS is relatively small and the performance boost might be good when using heavy instancing.
So if you want to stay with 4_9_3 calculate reflection and specular in pixel shader, these operations are pretty cheap anyways.

I think you are on a good path, if you post the actual shader or more specific questions I would be glad to provide further assistance.

3

Thank you for the advice!

I went with shader model 4.0, and also ended up doing my reflection vector calculation in the pixel shader. The advantage of doing this, appart for less stuff in the vertex shader output, is that I can use my normal map to modulate the normal (instead of just the model normal), and it really looks a lot better.