[SOLVED] Can't get this instanced shader to work right. Transpose

I was working on testing instancing a few days ago and decided id like to just pass in a world orientation matrix per instance. This seemed straightforward at first but no matter what i do, i can’t get proper transformation to occur i honestly can’t even see what is really going wrong the results are weird. Im recreating the matrix on the shader but its as if its screwed up.

So i pulled out my camera class and added that and a second game class.
I changed the Instanced Vertex structure to take a full wvp 4x4 matrix and passed it in for just a single instanced quad. At this point im starting to think that the data is not being passed in properly or it is being optimized out or something weird.

Both tests are here in one project if anyone wants to take a look.

Re: Edit: newer test

It should be working there is nothing left to it i was playing around with the w value thinking that was the problem but now im thinking that the actual values are not being passed in correctly.
Though i should of changed it back to 1 doesnt matter it wont work.

    VSOutputInstanced MainVsInstanced(in VSVertexInputInstanced vertexInput, VSInstanceInputInstanced instanceInput)
    {
        VSOutputInstanced output;
        //float instanceTimeOrId = instanceInput.InstanceTimeOrId;
        float4x4 wvp;
        wvp[0] = instanceInput.InstanceLeft; // its really passing right up back
        wvp[1] = instanceInput.InstanceUp;
        wvp[2] = instanceInput.InstanceForward;
        wvp[3] = instanceInput.InstancePosition;
        float4 posVert = float4(vertexInput.Position, 1.0f); // 0.0f i give up lol.
        float4 pos = mul(posVert, wvp);
        // passthru
        output.Position = pos;
        output.TexCoord = vertexInput.TexCoord;
        output.Color = float4(1.0f, 0.0f, 0.0f, .99f);
        return output;
    }

    float4 MainPsInstanced(VSOutputInstanced input) : COLOR0
    {
        float4 col = tex2D(TexSamplerA, input.TexCoord) *  input.Color;
        // straight clip alpha draws
        clip(col.a - .05f);
        return col;
    }

http://i936.photobucket.com/albums/ad207/xlightwavex/Gl_instanced_matrixbug_zpsaopwd7sh.gif

public struct InstanceDataOrientationB : IVertexType
{
    public Vector4 instancePosition;//here //0
    public Vector4 instanceForward;//1
    public Vector4 instanceUp;//2
    public Vector4 instanceLeft;//3

    /*old
    public Vector4 instanceForward;//0
    public Vector4 instanceUp;//1
    public Vector4 instanceLeft;//2
    public Vector4 instancePosition;//3 <<<
   */
   
    //public float instanceTimeOrId;

    public static readonly VertexDeclaration VertexDeclaration;
    static InstanceDataOrientationB()
    {
        var elements = new VertexElement[]
            {
                new VertexElement(0, VertexElementFormat.Vector4, VertexElementUsage.Position, 1),//0
                new VertexElement(sizeof(float) *4, VertexElementFormat.Vector4, VertexElementUsage.Normal, 1),//1
                new VertexElement(sizeof(float) *8, VertexElementFormat.Vector4, VertexElementUsage.Normal, 2),//2
                new VertexElement(sizeof(float) *12, VertexElementFormat.Vector4, VertexElementUsage.Normal, 3),//3
                //new VertexElement(sizeof(float) *12, VertexElementFormat.Single, VertexElementUsage.BlendWeight, 0)
                //new VertexElement( offset in bytes, VertexElementFormat.Single, VertexElementUsage. option, shader element usage id number )
            };
        VertexDeclaration = new VertexDeclaration(elements);
    }

edit: add

alt

public struct InstanceDataOrientationB : IVertexType
{
    public Matrix Transform;

    public static readonly VertexDeclaration VertexDeclaration;
    static InstanceDataOrientationB()
    {
        var elements = new VertexElement[]
            {
                new VertexElement(sizeof(float) *0, VertexElementFormat.Vector4, VertexElementUsage.Normal, 1),
                new VertexElement(sizeof(float) *4, VertexElementFormat.Vector4, VertexElementUsage.Normal, 2),
                new VertexElement(sizeof(float) *8, VertexElementFormat.Vector4, VertexElementUsage.Normal, 3),
                new VertexElement(sizeof(float) *12, VertexElementFormat.Vector4, VertexElementUsage.Normal, 4),
            };
        VertexDeclaration = new VertexDeclaration(elements);
    }

    VertexDeclaration IVertexType.VertexDeclaration
    {
        get { return VertexDeclaration; }
    }
}

&

struct VSInstanceInputSimple
{
    float4x4 Transform : NORMAL1;    
};

@willmotil Are you having this problem on OpenGL, DirectX, or both?

I am seeing weird issues with instancing on OpenGL, but it works fine / as expected on DirectX.

Edit…

Ok maybe i missed something pumkin did.
I made a new test for comparision. I tried this on dx this time around. I made a regular quad and a instanced one. Other then the way the instanced version is passed in, They are nearly identical, the matrices are proper before hitting the shader ect.

The animated image shows two tests one after the other of the same matrix being passed but in different ways.

http://i936.photobucket.com/albums/ad207/xlightwavex/miscellaneous/Gl_instanced_matrixbug02_zpsfonl0b1v.gif

As per the gif the matrices are proper before they get into the shader. As you can see the same matrix when passed non instanced as a global works ok. The matrix when passed thru the instance isn’t. I mean its sort of working its translating now, but somethings still not right and the weird clipping.

Here is the new test project if anyone wants to take a look.
You need to comment out which Test will run within the Program.cs.

the two shaders are basically identical now other then how the matrix is passed when instanced, as shown below.

// FxInstancedObjMatrices.fx
// April 21 2018
#if OPENGL
#define SV_POSITION POSITION
#define VS_SHADERMODEL vs_3_0
#define PS_SHADERMODEL ps_3_0
#else
#define VS_SHADERMODEL vs_4_0
#define PS_SHADERMODEL ps_4_0
#endif
//_________________________________________________________________

matrix WorldViewProjection;

Texture2D TextureA;
sampler2D TexSamplerA = sampler_state
{
    Texture = <TextureA>;
};

//_______________________________________________________________
// Game_RegularComparison
// techniques QuadDraw  Not instanced
//_______________________________________________________________
struct VsInputQuad
{
    float4 Position : POSITION0;
    float2 TexCoord : TEXCOORD0;
};
struct VsOutputQuad
{
    float4 Position : SV_Position;
    float2 TexCoord : TEXCOORD0;
    float4 Color : COLOR0;
};

VsOutputQuad VsQuadDraw(VsInputQuad input)
{
    VsOutputQuad output;
    output.Position = mul(input.Position, WorldViewProjection); // basically matrix identity
    output.Color = float4(1.0f,0.0f,0.0f,1.0f);
    output.TexCoord = input.TexCoord;
    return output;
}

float4  PsQuadDraw(VsOutputQuad input) : COLOR0
{
    float4 col = tex2D(TexSamplerA, input.TexCoord) * input.Color;
    clip(col.a - .05f); // clip very low alpha pixels
    return col;
}

technique QuadDraw
{
    pass
    {
        VertexShader = compile VS_SHADERMODEL VsQuadDraw();
        PixelShader = compile PS_SHADERMODEL PsQuadDraw();
    }
}

//__________________________________________________________
// Game_HwiMatrixPassing version
// Instanced the instance passes the matrix.
//__________________________________________________________

struct VSVertexInputInstanced
{
    float3 Position : POSITION0;//SV_POSITION;
    float2 TexCoord : TEXCOORD0;
};
struct VSInstanceInputInstanced
{
    float4x4 Transform : NORMAL1;
};
struct VSOutputInstanced
{
    float4 Position : SV_POSITION;
    float2 TexCoord : TEXCOORD0;
    float4 Color : COLOR0;
};

VSOutputInstanced MainVsInstanced(in VSVertexInputInstanced vertexInput, VSInstanceInputInstanced instanceInput)
{
    VSOutputInstanced output;
    output.Position = mul(float4(vertexInput.Position, 1.0f), instanceInput.Transform);
    output.TexCoord = vertexInput.TexCoord;
    output.Color = float4(1.0f, 0.0f, 0.0f, .99f);
    return output;
}

float4 MainPsInstanced(VSOutputInstanced input) : COLOR0
{
    float4 col = tex2D(TexSamplerA, input.TexCoord) *  input.Color;
    clip(col.a - .05f); // clip very low alpha pixels
    return col;
}
technique ParticleDrawingInstanced
{
    pass
    {
        VertexShader = compile VS_SHADERMODEL MainVsInstanced();
        PixelShader = compile PS_SHADERMODEL MainPsInstanced();
    }
};

The vertex structure looks like this now.

    public struct InstanceDataOrientationB : IVertexType
    {
        public Matrix Transform;
        ////public float instanceTimeOrId;

        public static readonly VertexDeclaration VertexDeclaration;
        static InstanceDataOrientationB()
        {
            var elements = new VertexElement[]
            {
                new VertexElement(sizeof(float) *0, VertexElementFormat.Vector4, VertexElementUsage.Normal, 1),
                new VertexElement(sizeof(float) *4, VertexElementFormat.Vector4, VertexElementUsage.Normal, 2),
                new VertexElement(sizeof(float) *8, VertexElementFormat.Vector4, VertexElementUsage.Normal, 3),
                new VertexElement(sizeof(float) *12, VertexElementFormat.Vector4, VertexElementUsage.Normal, 4),
                //new VertexElement(sizeof(float) *12, VertexElementFormat.Single, VertexElementUsage.BlendWeight, 0)
                //new VertexElement( offset in bytes, VertexElementFormat.Single, VertexElementUsage. option, shader element usage id number )
            };
            VertexDeclaration = new VertexDeclaration(elements);
        }
        VertexDeclaration IVertexType.VertexDeclaration
        {
            get { return VertexDeclaration; }
        }
    }

    // the vertexDataType
    //[StructLayout(LayoutKind.Sequential)]
    public struct VertexPositionUvB : IVertexType
    {
        public Vector3 position;
        public Vector2 texCoordinate;

        public static readonly VertexDeclaration VertexDeclaration;
        static VertexPositionUvB()
        {
            var elements = new VertexElement[]
                {
                    new VertexElement(0, VertexElementFormat.Vector3, VertexElementUsage.Position, 0),
                    new VertexElement(sizeof(float) * 3, VertexElementFormat.Vector2, VertexElementUsage.TextureCoordinate, 0),
                };
            VertexDeclaration = new VertexDeclaration(elements);
        }
        VertexDeclaration IVertexType.VertexDeclaration
        {
            get { return VertexDeclaration; }
        }
    }

Edit this is soo weird i changed it again to use texture coordinates no difference. Is it possible that the 4th column is not being feed into the shader fully ? E.G. the w components. Im getting the weirdest cliping artifacts im having trouble even understanding what im seeing but it looks like it is working except for some sort of messed up W values.

      VSOutputInstanced MainVsInstanced(in VSVertexInputInstanced vertexInput, VSInstanceInputInstanced instanceInput)
      {
           VSOutputInstanced output;
           //vertexInput.Position is position of base vertex, multiple with instance transform matrix to get correct position
            // effectparameter setvalue do column->row for matrix but this one not >>> transpose(instanceInput.Transform)
           float4 wpos = mul(vertexInput.Position, transpose(instanceInput.Transform));//instance transform
          

           output.Position = mul(wpos, WorldViewProjection);//wvp     
           output.TexCoord = vertexInput.TexCoord;
           output.Color = float4(1.0f, 0.0f, 0.0f, .99f);
           return output;
      }

example

1 Like

PumpkinPudding, You are a genius…

Both shaders the instanced and non instanced one are working exactly the
same now.



Ahhhh that makes sense… Im guessing monogame does this on it’s own when passing in global matrices it transposes them ?

VSOutputInstanced MainVsInstanced(in VSVertexInputInstanced vertexInput, VSInstanceInputInstanced instanceInput)
{
    VSOutputInstanced output;
    output.Position = mul(float4(vertexInput.Position, 1.0f), transpose(instanceInput.Transform));
    output.TexCoord = vertexInput.TexCoord;
    output.Color = float4(1.0f, 0.0f, 0.0f, .99f);
    return output;
}.

The WorldViewProjection variable is just for the other shader atm to compare.
Later ill probably pass the ViewProjection and the Instances World and let the shader calculate the final matrix to save cpu calculations.which i guess is the whole point of instancing.