 [SOLVED] How can I get the world coords of the mouse (2D).

I have a 2D game and I’m trying to get the world coordinated of my mouse.

I think I need to use Viewport.Unproject but I’m getting lost as to what the matrices are that I’m supposed to supply. I am translating my spritebatch with a matrix like so:

var Transform =
Matrix.CreateTranslation(pos) *
Matrix.CreateRotationZ(rot) *
Matrix.CreateScale(scale) *
Matrix.CreateTranslation(offset);

Globals.spriteBatch.Begin(
SpriteSortMode.FrontToBack,
null,
SamplerState.PointClamp,
null,
null,
null,
Transform
);

But Viewport.Unproject takes 3 matrices: Projection, View and World. Which matrix is this and where do I get the other two?

I’ve tried combinations like this:

return viewport.Unproject(pos, Matrix.Identity, Transform, Matrix.Identity);
return viewport.Unproject(pos, Transform, Matrix.Identity, Matrix.Identity);
return viewport.Unproject(pos, Matrix.Identity, Matrix.Identity, Transform);

but they always just give me:

{X:NaN Y:NaN Z:NaN}

You’ll need to invert your camera matrix and transform it to get a point in world space:

public Vector2 ScreenToWorldSpace(in Vector2 point)
{
Matrix invertedMatrix = Matrix.Invert(TransformMatrix);
return Vector2.Transform(point, invertedMatrix);
}

You can use the following function directly it returns a ray for your mouse coordinates in world space.

A ray of course has a position which should be usable for your purposes. As well as a direction the direction is a ray into the world from that point in the correspondingly correct direction into the world.

This works for any sort of matrix concation which is why all the matrices are passed as well.

/// <summary>
/// Near plane is typically just 0 in this function or some extremely small value. Far plane cant be more then one and so i just folded it automatically. In truth this isnt the near plane its the min clip but whatever.
/// </summary>
public Ray GetScreenVector2AsRayInto3dWorld(Vector2 screenPosition, Matrix projectionMatrix, Matrix viewMatrix, Matrix cameraWorld, float near, GraphicsDevice device)
{
//if (far > 1.0f) // this is actually a misnomer which caused me a headache this is supposed to be the max clip value not the far plane.
//    throw new ArgumentException("Far Plane can't be more then 1f or this function will fail to work in many cases");
Vector3 nearScreenPoint = new Vector3(screenPosition.X, screenPosition.Y, near); // must be more then zero.
Vector3 nearWorldPoint = Unproject(nearScreenPoint, projectionMatrix, viewMatrix, Matrix.Identity, device);

Vector3 farScreenPoint = new Vector3(screenPosition.X, screenPosition.Y, 1f); // the projection matrice's far plane value.
Vector3 farWorldPoint = Unproject(farScreenPoint, projectionMatrix, viewMatrix, Matrix.Identity, device);

Vector3 worldRaysNormal = Vector3.Normalize((farWorldPoint + nearWorldPoint) - nearWorldPoint);
return new Ray(nearWorldPoint, worldRaysNormal);
}

/// <summary>
/// Note the source position internally expects a Vector3 with a z value.
/// That Z value can Not excced 1.0f or the function will error. I leave it as is for future advanced depth selection functionality which should be apparent.
/// </summary>
public Vector3 Unproject(Vector3 position, Matrix projection, Matrix view, Matrix world, GraphicsDevice gd)
{
if (position.Z > gd.Viewport.MaxDepth)
throw new Exception("Source Z must be less than MaxDepth ");
Matrix wvp = Matrix.Multiply(view, projection);
Matrix inv = Matrix.Invert(wvp);
Vector3 clipSpace = position;
clipSpace.X = (((position.X - gd.Viewport.X) / ((float)gd.Viewport.Width)) * 2f) - 1f;
clipSpace.Y = -((((position.Y - gd.Viewport.Y) / ((float)gd.Viewport.Height)) * 2f) - 1f);
clipSpace.Z = (position.Z - gd.Viewport.MinDepth) / (gd.Viewport.MaxDepth - gd.Viewport.MinDepth); // >> Oo <<
Vector3 invsrc = Vector3.Transform(clipSpace, inv);
float a = (((clipSpace.X * inv.M14) + (clipSpace.Y * inv.M24)) + (clipSpace.Z * inv.M34)) + inv.M44;
return invsrc / a;
}

That looks like what I’m looking for. But the invert method is giving me:

{M11:NaN M12:NaN M13:NaN M14:NaN}
{M21:NaN M22:NaN M23:NaN M24:NaN}
{M31:NaN M32:NaN M33:Infinity M34:NaN}
{M41:NaN M42:NaN M43:NaN M44:NaN}

The transform before it gets inverted is:

{M11:1 M12:0 M13:0 M14:0}
{M21:0 M22:1 M23:0 M24:0}
{M31:0 M32:0 M33:0 M34:0}
{M41:0 M42:-60 M43:0 M44:1}

I cannot speak for the idea of only inverting.

I can only tell you i have tested the method i posted in 3d with a free camera and a billboarded object that changed shape when i put the mouse over it i tested it all directions and at distances as well as when objects were behind the camera.
I debugged out the problems i had with it and the notes to that still remain. The ray is basically both the position and the direction into the world and so it gives more information then just a position anyways.
This is a part of my utility class methods now.

You can look to the unproject function to see how it inverts which is actually right out of the matrix class i may have minorly altered it just to make it clearer when i was debugging a problem with my function.

Both of the methods you supplied take 3 matrices: Matrix projection, Matrix view, Matrix world. Which leads me back to my original question. Where do I get these matrices from?

It is somewhat odd that you would ask what these matrixs are and also need to have a world position for the mouse. Out of curiosity can i ask what this is for ?

Anyways to be as succinct as possible.

If you are only using spritebatch and passing it a BasicEffect basicEffect then these matrices are basicEffect.World
basicEffect.View
basicEffect.Projection;

If you are not passing a basic effect to spritebatch and using it as is then you don’t even need the above functions and can directly caluculate world coordinates by linear math.

pesudo code.

`Vector2 worldpos = ((mousepos / GraphicsDevice.Viewport.Bounds.Size ) * 2)-Vector2.One;`

Edit.
If you want to create your own basic Effect that matches spritebatch and then pass that to spriteBatch.Begin You can see the equivalent matrices and how to make them and pass them to spritebatch begin, in the below post were i put a full example that can be copy pasted.

This also includes a second example of how to make your own custom effect that matches spritebatch matrice setup.

Edit sorry that might not be the best example. If you look at set states it shows how to set up a basic effect though in the actual draw using spritebatch you add the basic effect to begin.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, null, RasterizerState.CullNone, basicEffect, null);

spriteBatch.Draw(generatedTexture, new Rectangle(000, 000, 200, 200), Color.Beige);
spriteBatch.Draw(generatedTexture, new Rectangle(100, 100, 200, 200), Color.Green);

spriteBatch.End();

If you look at my image you will see that there is a character centred on the screen. The character is able to move around freely and the camera is set to follow her.

If you look to the right of her you will see a cross-hair. The player can use the mouse to move the cross-hair in a circle around the character. The red line is the vector.

I’m calculating the angle of the cross-hair as follows:

player.transform.Position - mouse.Position

Then I convert the vector to an angle which I then apply to the transform of the cross-hairs.

The problem is that the mouse position I’m getting is in screen coords and the player is in world coords.

I’d like to do this with a matrix transform because the camera can freely move, scale(zoom) and rotate, so I’d like accurate coords for any situation.

I hope this is enough information.

player.transform.Position - Vector2.Transform(mouse.Position, Matrix.Invert(CameraTransformMatrix));

Where “CameraTransformMatrix” is the “Transform” you pass to spritebatch

See my answer above. This was suggested by @Kimimaru but Inverting my camera transform just give me a NAN matrix. I’m not sure why.

The matrix here looks incorrect, as the Z is translated and scaled; that would explain why M42 has a value like -60. Try this instead to guarantee nothing goes awry if the Z is modified:

var Transform =
Matrix.CreateTranslation(new Vector3(pos.X, pos.Y, 0f)) *
Matrix.CreateRotationZ(rot) *
Matrix.CreateScale(scale.x, scale.y, 1) *
Matrix.CreateTranslation(offset.x, offset.y, 0f);

This worked, thanks.

Although it did mess up my sprite layerDepth. Am I correct in saying that the layerDepth must be a value between 0 and 1? When I set the scale.z to 1 all of my sprites disappeared. I found that I can either set the scale.z to a really low value (0.0000001f) or divide my layerDepth by a large value (2000000). Before implementing your fix my layerDepth was set to the sprites Y position. Is this the correct way to do this?

If memory serves the transform spritebatch takes is set to the world matrix.

Untested but id think this should also work

var transformedMousePosition = (Matrix.CreateTranslation(new Vector3(mouse.Position.X,mouse.Position.Y, 0) ) * player.transform).Translation;

I’m afraid that didn’t work. The solution provided by @Kimimaru worked fine though. Thanks for your help guys.

That’s correct; layerDepth must be between 0 and 1 if you’re using a FrontToBack or BackToFront sorting mode. If the depth is out of this range, a sprite may not render. A sprite’s Y position will be outside of that range fairly quickly, so for the depth, divide the Y by a value you don’t think it’ll ever go over.

1 Like

That solution worked for me! Now how do I turn that vector back to Screen Coordinates when I need to.

The method above gives me inaccurate results (the ray direction seems a bit offset). I came up with the following code using the Unproject method of the GraphicsDevice.Viewport and the result is perfect. I also added the groundplane-ray-intersection stuff.

public Vector3 GetMouseWorldPosition(Point screenPosition)
{
Vector3 nearWorldPoint = GraphicsDevice.Viewport.Unproject(new Vector3(screenPosition.X, screenPosition.Y, 0), Projection, View, World);
Vector3 farWorldPoint = GraphicsDevice.Viewport.Unproject(new Vector3(screenPosition.X, screenPosition.Y, 1), Projection, View, World);
Ray ray = new(nearWorldPoint, farWorldPoint - nearWorldPoint);

Plane groundPlane = new(Vector3.Zero, Vector3.UnitZ);
if (ray.Intersects(groundPlane) is float distance)
{
Vector3 intersection = ray.Position + distance * ray.Direction;
return intersection;
}
else return Vector3.Zero;
}

Credits go to: ViewPort Unproject (...) problem.