MonoGame 3.4 2MGFX Parse Error fails to read sampler name

tgraupmann · January 15, 2016, 10:34pm

MonoGame Gurus,

Back doing another MonoGame port, I have an FX shader that parsed correctly in MG 3.2, but MG 3.4 introduced a shader parsing error.

Firstly I discovered because if a missing null check.

github.com

MonoGame/MonoGame/blob/v3.4/Tools/2MGFX/ShaderData.writer.cs#L40


                    writer.Write(sampler.state.BorderColor.A);
					writer.Write((byte)sampler.state.Filter);
					writer.Write(sampler.state.MaxAnisotropy);
					writer.Write(sampler.state.MaxMipLevel);
					writer.Write(sampler.state.MipMapLevelOfDetailBias);
				}
				else
					writer.Write(false);


                if (options.Profile == ShaderProfile.OpenGL)
                    writer.Write(sampler.samplerName);


                writer.Write((byte)sampler.parameter);
            }


            writer.Write((byte)_cbuffers.Length);
            foreach (var cb in _cbuffers)
                writer.Write((byte)cb);


            if (options.Profile != ShaderProfile.OpenGL)
                return;

ShaderData.writer.cs failed to check before writing a null samplerName.

                if (null == sampler.samplerName)
                {
                    System.Console.Error.WriteLine("Error FX parsing, sampler name is null!");
                    writer.Write(string.Empty);
                }
                else
                {
                    writer.Write(sampler.samplerName);
                }

Here’s a script to reproduce a null sampler name.
http://tagenigma.com/monogame/IndexedEffect_3_4_custom.fx

You’ll have to adjust the include paths to the built-in FX header includes.

I think this is telling me the parser couldn’t find.

DECLARE_TEXTURE(Texture, 0);
DECLARE_TEXTURE(PaletteTex, 1);

“Texture”, “PaletteTex”, and “PaletteID” EffectParameters are now turning up null.

I have an easy repro case as I go down the rabbit hole.

The command-line that I’m using is:

2MGFX.exe IndexedEffect_3_4_custom.fx IndexedEffect.xnb

Any help is appreciated!

Thanks,

~Tim Graupmann

tgraupmann · January 15, 2016, 11:01pm

Determined to find the bug. The issue is in the second texture sampler.

Inspecting samplers:

github.com

MonoGame/MonoGame/blob/v3.4/Tools/2MGFX/ShaderData.mojo.cs#L115




					case MojoShader.MOJOSHADER_symbolRegisterSet.MOJOSHADER_SYMREGSET_INT4:
						symbols [i].register_index = int4_index;
						int4_index += symbols [i].register_count;
						break;
					}
				}
			}


			// Get the samplers.
			var samplers = MarshalHelper.UnmarshalArray<MojoShader.MOJOSHADER_sampler> (
					parseData.samplers, parseData.sampler_count);
			dxshader._samplers = new Sampler[samplers.Length];
			for (var i = 0; i < samplers.Length; i++) 
            {
                // We need the original sampler name... look for that in the symbols.
                var originalSamplerName =
                    symbols.First(e => e.register_set == MojoShader.MOJOSHADER_symbolRegisterSet.MOJOSHADER_SYMREGSET_SAMPLER &&
                    e.register_index == samplers[i].index
                ).name;

tgraupmann · January 15, 2016, 11:18pm

Comparing with the working MG 3.2 this screenshot has the expected data.

Inspecting samplers:

github.com

MonoGame/MonoGame/blob/v3.2/Tools/2MGFX/DXShaderData.mojo.cs#L119




					case MojoShader.MOJOSHADER_symbolRegisterSet.MOJOSHADER_SYMREGSET_INT4:
						symbols [i].register_index = int4_index;
						int4_index += symbols [i].register_count;
						break;
					}
				}
			}


			// Get the samplers.
			var samplers = DXHelper.UnmarshalArray<MojoShader.MOJOSHADER_sampler> (
					parseData.samplers, parseData.sampler_count);
			dxshader._samplers = new Sampler[samplers.Length];
			for (var i = 0; i < samplers.Length; i++) 
            {
                // We need the original sampler name... look for that in the symbols.
                var originalSamplerName =
                    symbols.First(e => e.register_set == MojoShader.MOJOSHADER_symbolRegisterSet.MOJOSHADER_SYMREGSET_SAMPLER &&
                    e.register_index == samplers[i].index
                ).name;

tgraupmann · January 16, 2016, 12:05am

The developer had noticed the indexes aren’t set properly and left a comment.

github.com

MonoGame/MonoGame/blob/v3.4/Tools/2MGFX/ShaderData.mojo.cs#L87


				if (b.info.elements == 1)
					vb += 1024;
				return va.CompareTo (vb);
			}
			);//(a, b) => ((int)(a.info.elements > 1))a.register_index.CompareTo(b.register_index));


            // NOTE: It seems the latest versions of MojoShader only 
            // output vec4 register sets.  We leave the code below, but
            // the runtime has been optimized for this case.


			// For whatever reason the register indexing is 
			// incorrect from MojoShader.
			{
				uint bool_index = 0;
				uint float4_index = 0;
				uint int4_index = 0;


				for (var i = 0; i < symbols.Length; i++) {
					switch (symbols [i].register_set) {
					case MojoShader.MOJOSHADER_symbolRegisterSet.MOJOSHADER_SYMREGSET_BOOL:
						symbols [i].register_index = bool_index;

The name is also missing from MojoShader.

MojoShader! Now open source. Ugh 2008.
https://icculus.org/mojoshader/

The complete MojoShader documentation errr header file with MOJOSHADER_sampler definition.
http://hg.icculus.org/icculus/mojoshader/file/tip/mojoshader.h#l201

The source code:
http://hg.icculus.org/icculus/mojoshader

Okay the more I look at this it seems to be a noted MojoShader bug. And I’ll try either updating to the latest, or fix the MojoShader code so it sets the proper index and name.

Tom · January 16, 2016, 2:17am

Well actually that should never be null… the fact it is null is the bug to worry about. Samplers need to have names for the GL path to work.

But this error sounds familiar to me…

I eventually found the bug was caused by someone checking in a bad version of MojoShader. I ended up rebuilding the 64bit version of it and updating it to fix the issue.

Note the version of MojoShader used by MonoGame is modified from the original source. The code for the version used in MonoGame is here:

If you use a different version of MojoShader it can cause weird problems like the issue linked above.

tgraupmann · January 16, 2016, 7:17am

I’ll try one workaround.

I copied the libmojoshader_32.dll from MonoGame 3.2 and used that instead of the 64-bit one. I set the build of 2MGFX to 32-bit. And I set the debug symbol for OLD_CONTENT_PROCESSORS.

That let the FX compile without the null sampler name.

Ah sweet that fixed the character issue! Okay that did it!

Tom · January 18, 2016, 11:52pm

That all sounds like you had that bad libmojoshader_64.dll.