@willmotil
Sorry it’s too dificult for me to understand what you have done exactly. But I think I have found the solution. Hope it helps.
The original question was about the difference between the documentation of the D3DXMatrixLookAtRH function and the implementation of this function in MonoGame.
I now think that the DirectX documentation is just wrong! The MonoGame implementation is correct.
There are two operations needed to transform from WorldSpace to ViewSpace:
1.Given the camera rotation, one must apply the inverse of this rotation. In this case the rotation matrix is orthogonal, so the inverse is equivalent to the transpose of the original rotation matrix. In the method below, the rotation matrix is the one containing the inverse rotation, this means the matrix has already been transposed. I hope it is clear what I mean.
2.Given the WorldSpace position of the camera, one must apply the inverse of this translation. The inverse translation matrix is given by using the negative of the position vector.
The Rotation matrix below is Since the XAxis, YAxis and ZAxis vectors are normalized.
I wrote the following method to check this. It gives the same result as the MonoGame Matrix.CreateLookAt method:
private static Matrix CalculateLookAt(Vector3 Position, Vector3 Target, Vector3 UpVector)
{
var ZAxis = Vector3.Normalize(Position - Target);
var XAxis = Vector3.Normalize(Vector3.Cross(UpVector, ZAxis));
var YAxis = Vector3.Cross(ZAxis, XAxis);
//calculate rotation
Matrix Rotation = new Matrix();
Rotation.M11 = XAxis.X;
Rotation.M12 = YAxis.X;
Rotation.M13 = ZAxis.X;
Rotation.M14 = 0f;
Rotation.M21 = XAxis.Y;
Rotation.M22 = YAxis.Y;
Rotation.M23 = ZAxis.Y;
Rotation.M24 = 0f;
Rotation.M31 = XAxis.Z;
Rotation.M32 = YAxis.Z;
Rotation.M33 = ZAxis.Z;
Rotation.M34 = 0f;
Rotation.M41 = 0f;
Rotation.M42 = 0f;
Rotation.M43 = 0f;
Rotation.M44 = 1f;
//calculate translation
Matrix Translation = new Matrix();
Translation.M11 = 1f;
Translation.M12 = 0f;
Translation.M13 = 0f;
Translation.M14 = 0f;
Translation.M21 = 0f;
Translation.M22 = 1f;
Translation.M23 = 0f;
Translation.M24 = 0f;
Translation.M31 = 0f;
Translation.M32 = 0f;
Translation.M33 = 1f;
Translation.M34 = 0f;
Translation.M41 = -Position.X;
Translation.M42 = -Position.Y;
Translation.M43 = -Position.Z;
Translation.M44 = 1f;
return Rotation * Translation;
}