Issues with shader instancing

Hi all,
I’m trying to make instancing work (as in drawing the same mesh multiple times with array of world transformations), and its very broken. Basically looks like this:

I’m not sure what I’m doing wrong here…
This is my effect file:

#if OPENGL
	#define SV_POSITION POSITION
	#define VS_SHADERMODEL vs_3_0
	#define PS_SHADERMODEL ps_3_0
#else
	#define VS_SHADERMODEL vs_4_0_level_9_1
	#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

Texture2D Texture;
sampler2D TextureSampler = sampler_state
{
    Texture = (Texture);
};

matrix ViewProjection;

struct VertexShaderInput
{
	float4 Position : POSITION0;
    float2 Coords : TEXCOORD0;
};

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
    float2 Coords : TEXCOORD0;
};

VertexShaderOutput MainVS(in VertexShaderInput input, float4x4 instanceTransform : BLENDWEIGHT)
{
	VertexShaderOutput output = (VertexShaderOutput)0;
	matrix wvp = mul(instanceTransform, ViewProjection);
	output.Position = mul(input.Position, wvp);
    output.Coords = input.Coords;
	return output;
}

float4 MainPS(VertexShaderOutput input) : COLOR
{
    float4 color = tex2D(TextureSampler, input.Coords);
	return color;
}

technique InstanceDrawing
{
	pass P0
	{
		VertexShader = compile VS_SHADERMODEL MainVS();
		PixelShader = compile PS_SHADERMODEL MainPS();
	}
};

And the code that uses it:

// vertex declaration for instance rendering
        static VertexDeclaration instanceVertexDeclaration = new VertexDeclaration
        (
            new VertexElement(0, VertexElementFormat.Vector4, VertexElementUsage.BlendWeight, 0),
            new VertexElement(sizeof(float) * 4, VertexElementFormat.Vector4, VertexElementUsage.BlendWeight, 1),
            new VertexElement(sizeof(float) * 8, VertexElementFormat.Vector4, VertexElementUsage.BlendWeight, 2),
            new VertexElement(sizeof(float) * 12, VertexElementFormat.Vector4, VertexElementUsage.BlendWeight, 3)
        );

void DrawInstanceQueue(InstancingQueue queue, Camera camera)
        {
            // update effect
            var effect = queue.Material.Effect;
            effect.Texture = queue.Texture;
            effect.ViewProjection = camera.ViewProjection;

            // update rasterizer state
            GlobalResources.GraphicsDevice.RasterizerState = queue.Material.RasterizerState;

            // update draw calls count
            DrawCalls++;

            // create buffer for instance matrices
            var graphics = GlobalResources.GraphicsDevice;
            var instanceVertexBuffer = new DynamicVertexBuffer(graphics, instanceVertexDeclaration, queue.Count, BufferUsage.WriteOnly);
            instanceVertexBuffer.SetData(queue.Transforms, 0, queue.Count, SetDataOptions.Discard);

            // draw with instancing
            for (int i = 0; i < queue.Mesh.MeshParts.Count; i++)
            {
                var part = queue.Mesh.MeshParts[i];
                if (part.PrimitiveCount > 0)
                {
                    graphics.SetVertexBuffers(
                        new VertexBufferBinding(part.VertexBuffer, part.VertexOffset, 0),
                        new VertexBufferBinding(instanceVertexBuffer, 0, 1)
                    );
                    graphics.Indices = part.IndexBuffer;

                    for (int j = 0; j < effect.CurrentTechnique.Passes.Count; j++)
                    {
                        effect.CurrentTechnique.Passes[j].Apply();
                        graphics.DrawInstancedPrimitives(PrimitiveType.TriangleList, part.VertexOffset, part.StartIndex, part.PrimitiveCount, queue.Count);
                    }
                }
            }

            // reset queue count
            queue.Count = 0;
        }

Now its safe to assume the camera ViewProjection and queue.Transforms are correct, because if I use them with non-instancing “regular” effect everything draws nicely. So I guess its something in the shader or the way I pass data to it.

Any hints on what I’m doing wrong?

Thanks :slight_smile:

EDIT: I tried to ignore the instanceTransform and only multiply with the ViewProjection and models render properly, but obviously they are all with identity matrix (ie position 0,0,0, no rotation, scale 1,1,1). This confirms that the issue is the values inside the instanceTransform matrix. However, as mentioned, if I use the same transformations to draw “normally”, it looks alright. So looks like the issue is passing the instanceTransform from C# to shaders?

OpenGL? I use instancing extensively in my MG DX projects, never really had issue (altho, in your vertex declaration be explicit about your struct, it shouldn’t matter in this specific aligned case, but rather a as general tip)

 [StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct VertexMaskedSprite : IVertexType
...

Anyway, recently I was working on one OpenGL project and I couldn’t get it running properly, poked into it a lot, tried to debug with Nvidia Nsight, but for life of me I couldn’t find why per instance data are getting scrambled, it had all symptoms of vertex data bleeding (incorrect semantics / type). Fun thing was, that last 4 bytes of each instance vertex actually been correct. Sadly, this comment isn’t going to have happy ending, just letting you know that if you have issue with instancing in OpenGL project I believe there might be bug in MG somewhere. However it really works perfectly fine in DX (can also do instancing using more than 2 vertex bindings).

However, if this is DX project, I would suggest to make sure that your instance Transform is constructed correctly and you don’t have flipped columns and rows. Alternatively I would try one more thing regarding semantics of that vertex input. Let me know if it is DX / OpenGL and I can take closer look. Good luck!

Hi @Ravendarke, thanks for the tip!
I was working with OpenGL, so to test your theory I switched to DirectX 11 and it looks the same (but it did fix other bug of picking resolution from the wrong monitor which is nice :smiley: ).

The VertexType I use is MonoGame’s so it should be OK.

Any chance you share with me a working instancing code that I can compare to?
For now I will keep working with DX until its resolved, then I’ll try switching to GL again.

PS about this:

Why would it be flipped? If the same transformations work as a matrix for a regular non-instancing shader, does it confirm this, or not necessarily?

Thanks!

Solved it! Needed to do Matrix.Transpose() on the world matrix for some reason… Probably what you said, wrong rows order when passing the data to shader…

Also for future seekers: creating the instanceVertexBuffer every batch is a terrible idea and creates waves of memory consumption and lots of GC work… Better to create it once with max instances count and just fill parts of it every time you draw an instances batch.

1 Like

yeah, if you transposed it, you switched rows / columns. Matrices gets automatically transposed depending on Api (OpenGL/DirectX have different column/row major). You technically don’t send matrices in your vertex buffers but 4 x float4, hence MG can’t transpose data as it would screw any other case (simply it can’t be detected as matrix).

Also yeah, definitely do not recreate vertex buffer every frame.