Issue moving from DX to OpenGL

I’m transitioning my monogame project to OpenGL for compatibility with Android etc down the line. :video_game: :iphone:

Everything is working except one of my custom shaders. I have two custom shaders that I use in the project, one works like normal under OpenGL and the other one does not. The working one is a very basic texture shader, the non working one is more or less the same except it is billboarding and animates the top corners of the mesh.

Spent a day trying to figure this out, temporarily running out of ideas. What could be the issue here?

I get no compiler errors but my billboarding meshes are invisible under OpenGL.

The only difference in the code from a fully working DirectX build and the half working OpenGL is I changed the

compile vs_4_0_level_9_1

to:

compile vs_3_0

Heres my Billboarding shader that seemingly stops working under OpenGL:

float4x4 World;
float4x4 View;
float4x4 Projection;
float Alpha = 1;
float animationTime;
float animationScaleFactor;

texture ColorMap;
sampler colorMapSampler = sampler_state
{
    Texture = <ColorMap>;
    AddressU = clamp;
    AddressV = clamp;
};

struct VertexInput
{
    float4 Position : POSITION;
    float4 TexCoord : TEXCOORD0;
    float2 Offsets : TEXCOORD1;
};

struct VertexOutput
{
    float4 Position : POSITION;
    float2 TexCoord : TEXCOORD0;
};

//-----------------------------------------------------------------------------
// Vertex shaders.
//-----------------------------------------------------------------------------
VertexOutput VS_BillboardingCameraAligned(VertexInput input)
{
    //Input
    float2 offsets = input.Offsets.xy;
    float2 texCoord = input.TexCoord.xy;
    float3 inPos = input.Position.xyz;

    //Billboarding magic
    float4x4 worldViewProjection = mul(mul(World, View), Projection);
    float3 xAxis = float3(View._11, View._21, View._31);
    float3 yAxis = float3(View._12, View._22, View._32);
    float animationDisplacement = input.Position.w * sin(animationTime) * animationScaleFactor;
    float3 pos = inPos + ((offsets.x + animationDisplacement) * xAxis) + ((offsets.y) * yAxis);
    float4 outPos = mul(float4(pos, 1.0f), worldViewProjection)

    //Output
    VertexOutput output;
    output.Position = outPos;
    output.TexCoord.xy = texCoord.xy;
    return output;
}

//-----------------------------------------------------------------------------
// Pixel shaders.
//-----------------------------------------------------------------------------
float4 PS_Standard(VertexOutput input) : COLOR
{
    float alphaTestDirection = 1.0f;
    float alphaTestThreshold = 0.95f;
    float4 outColor = tex2D(colorMapSampler, input.TexCoord);
    outColor.rgba *= Alpha;

    return outColor;
}

//-----------------------------------------------------------------------------
// Techniques.
//-----------------------------------------------------------------------------
technique Standard
{
    pass
    {
        VertexShader = compile vs_3_0 VS_BillboardingCameraAligned();
        PixelShader = compile ps_3_0 PS_Standard();
    }
}

Or am I wrong to assume that it’s shader that’s the issue?

EDIT: Formatting

Jeez, I don’t know how the guts of Monogame convert from HLSL to GLSL, but I do know they use MojoShader to convert it on the fly. It’s possible that you may be touching on some advanced feature of HLSL they don’t know how to translate?

Nothing catches my eye in your code though.

1 Like

When I have situations like this , I play with simple things to see what happens.

One of the less obvious ones is swapping semantics.

Instead of POSITION try SV_POSITION in VertexOutput

Other than that , use renderdoc .

You have to find a draw call that renders one of these billboard quads , but once you do you can look at the mesh viewer to see if the vertex shader output is correct and do a pixel history check to see what the pixel shader outputs

1 Like

Aha I’ll look in to the differences between HLSL and GLSL, maybe there’s a clue to be found. Thanks for taking the time to read my code and answering me, (despite my bad formatting)! :pray:t2:

Thanks so much for your answer! :pray:t2: I’ll try switching semantics and learn about renderdoc, and circle back. What a great community this is!

Thanks to the helpful comments above I now have a theory on what the issue could be. :pray:

I couldn’t get renderdoc to work for some reason but Intel GPA works with both my DirectX build and my OpenGL build on windows.

With Intel GPA I can see that the missing billboards exist as draw calls to the gpu, seemingly with all the proper ‘arguments’, both under DX and OpenGL.

My billboarding shader takes four vertices that are all at the same position and the shader, as part of the billboarding magic, then offsets them to make a rectangle facing the camera. :red_square:

My basic texture shader that is very similar to my billboarding shader but working with OpenGL takes six vertices arranged in two triangles. :trinidad_tobago:

My guess is that OpenGL doesn’t like playing with non triangle meshes or something along those lines.

I skimmed the code and noticed you’re working directly with the matrix in:

float3 xAxis = float3(View._11, View._21, View._31);
float3 yAxis = float3(View._12, View._22, View._32);

Correct me if I’m wrong:
Switching from HLSL to GLSL may mean going from row-major to column-major for direct entries.
Normally the difference isn’t noticed because the math works out the same way but if you manually enter the values I do believe you’d need to transpose the matrix after or just enter them differently.

Yea, I think DirectX (HLSL) is left handed and GLSL is right handed…

1 Like

Thanks for your comments! Row-major vs column-major and left handed vs right handed are new concepts to me. My understanding of matrices and their application in 3D graphics is definitely lacking.

I somehow managed to solve my issue. I noticed that OpenGL doesn’t like when I use TEXCOORD1 for the offset values. When switching to using COLOR0 it works. DX didn’t care. This is all currently outside my understanding, but they’re visible now!

2 Likes

That’s quite unexpected - but at least it’s working XD

2 Likes

Back with DX9 (not sure if it applied to GL too), there was also a variation in GPU’s too, so sampling a texel in a shader on an ATI card was done slightly differently to how it was done in an NVIDIA card :confounded:

Nvidia sampled from the corner, ATI from the center (if I remember correcctly).

1 Like