2D Cascaded shadow help

[AlexOD]

I am trying to make a “cascaded shadow” effect in my 2D sprite game, the way I currently handle my game map is that I draw my scenes objects in the following order:

Floor tiles → Shadows of “Overlay” tiles → Overlay tiles

The floor tiles is a simple draw call, we use a matrix that simply moves the world position of our camera to follow a game character like so:

      Transform = Matrix.Identity *
                Matrix.CreateTranslation(-Position.X, -Position.Y, 0) *
                Matrix.CreateRotationZ(Rotation) *
                Matrix.CreateTranslation(0, 0, 0) *
                Matrix.CreateScale(new Vector3(Scale, Scale, Scale));

Drawing the floor:
      spriteBatch.Begin(SpriteSortMode.BackToFront,
    BlendState.AlphaBlend,
    SamplerState.PointClamp,
    DepthStencilState.None,
    RasterizerState.CullCounterClockwise,
    null,
    Transform);

// Called for every 32x32 tile in the Floor layer
      spriteBatch.Draw(tileSource, position, sourceRect, Color.White,
        0f, new Vector2(0, 0), 1.0f, SpriteEffects.None, LayerDepth);

  spriteBatch.End();

And then, just before we draw our Overlay tiles, I call another draw with an altered matrix that skews the perspective of the next draw like so:

      Transform = Matrix.Identity *
                  Matrix.CreateRotationX(MathHelper.ToRadians(45)) *
            Matrix.CreateRotationY(MathHelper.ToRadians(45)) *
            Matrix.CreateScale(1, 1, 0) *
            Matrix.CreateTranslation(-Position.X, -Position.Y, 0) *
            Matrix.CreateRotationZ(Rotation) *
            Matrix.CreateTranslation(0, 0, 0) *
            Matrix.CreateScale(new Vector3(Scale, Scale, Scale));

spriteBatch.Begin(...same as before but with updated Transform)

// Called for every 32x32 tiles in the "Overlay" layer, as a Color.Black colour
      spriteBatch.Draw(tileSource, position, sourceRect, Color.Black,
        0f, new Vector2(0, 0), 1.0f, SpriteEffects.None, LayerDepth);

spriteBatch.End()

We then reset the matrix back to what it was before (None-Skewed) and draw our Overlays similar to the floor tiles and get the following:

So it’s obvious as to why this looks a bit jank at the moment, we’re drawing a set of tiles in given Vector2 positions with one matrix, and the exact same Vector2 positions with a different matrix! Of course they’re going to be positioned differently!
So the issue is basically I need to find a mathematical way to get offset my Vector2 positions for my “Shadow tiles” so they can appear in the same position where the Overlay tiles are. I’m not quite sure how to do that, I’m assuming I need to alter the Vector2s of the Shadow Tiles in some way? Would appreciate any suggestions or help!

[markus]

My first attempt would look something like this:

Matrix correctionTransform = sceneTransform * Matrix.Invert(shadowTransform);
Vector3 correctedPos = Vector3.Transform(pos, correctionTransform);

sceneTransform is the transform you are using for the normal non-shadow sprites.
shadowTransform is the transform you are using when drawing shadows.
pos is the position of the sprite.
correctedPos is the sprite position you should use instead of pos when drawing shadows.

[AlexOD]

I appear to be getting NaN values when I Matrix.Invert() my shadow transform:

(the transform I’m inverting, the position in this context is the camera’s position and scale respectively)
Matrix.Identity *
Matrix.CreateRotationX(MathHelper.ToRadians(45)) *
Matrix.CreateRotationY(MathHelper.ToRadians(45)) *
Matrix.CreateScale(1, 1, 0) *
Matrix.CreateTranslation(-Position.X, -Position.Y, 0) *
Matrix.CreateRotationZ(Rotation) *
Matrix.CreateTranslation(0, 0, 0) *
Matrix.CreateScale(new Vector3(Scale, Scale, Scale));Preformatted text

[markus]

That has to be the zero scale in Matrix.CreateScale(1, 1, 0).
Maybe try a small number instead.

[AlexOD]

So I modified the ShadowTransform like so:

Matrix.CreateScale(1, 1, 0.000000000001f) *

and it’s produced this affect, the position of the tiles appear to be aligned with the rotation radians (we’re using 45 x and 45 for y at the moment):

It’s also worth noting that I’m using Vector2s for our draw calls rather than Vector3s:

    public virtual void DrawShadow(SpriteBatch spriteBatch)
{
  Matrix sceneTransform = CameraManager.Instance.GetNormalTransform();
  Matrix shadowTransform = Matrix.Invert(CameraManager.Instance.GetShadowTransform());
  Matrix correctTranform = sceneTransform * shadowTransform;
  Vector2 tempPos = position;
  Vector2 correctedPos = Vector2.Transform(tempPos, correctTranform);

  spriteBatch.Draw(tileSource, correctedPos, sourceRect, Color.Black,
    0f, new Vector2(0, 0), 1.0f, SpriteEffects.None, LayerDepth);
}

[markus]

@AlexOD

It’s also worth noting that I’m using Vector2s for our draw calls rather than Vector3s

It’s not going to work then. I was assuming you could feed a 3D position to SpriteBatch.Draw. As long as the transformations you are making affect the z-dimension, you will also need the z-position for this to work. The x and y rotations are the problem.

Can you not render the shadow objects individually? That would be the simplest solution.
Otherwise you would need to replace the SpriteBatch drawing with some custom solution.

[AlexOD]

@markus

Can you not render the shadow objects individually?

What do you mean by this? like drawing the shadows as actual sprites instead of trying to auto-skew them in code?

Also SpriteBatch doesn’t seem to have any draw solutions for Vector3.

[markus]

Sorry, that was poorly explained. By rendering shadow objects individually I mean giving each object a separate transform. Now they all share the same transform from SpriteBatch.Begin(). You could draw every shadow object in a separate SpriteBatch.Begin/End block for example, or create a custom shader, if you really need to draw them all in one batch for performance reasons.

If every object has it’s own transform, it should be possible to apply your shadow-transformations while still having full control over the objects final position, because you can position the object using the transform, rather than the position parameter from SpriteBatch.Draw. The basic idea would be something like this:

• calculate the final position of the overlay object by transforming the object position with the scene transform. Now that you know where the overlay object ends up, you should be able to place the shadow object in the same spot.

• create a shadow transform that includes only the shadow rotations, maybe scaling, but not the translations used in the scene transform. It should only create the skewing effect you want, without affecting translation.

• to position the shadow object properly, multiply the shadow transform with a translation matrix containing the final object position you calculated in the beginning.

• draw the shadow object using this shadow transform, but pass zero to the position parameter of SpriteBatch.Draw. (Positioning is done in the shadow transform).

Does this make sense? I think it should work.

[Trinith]

I have a couple questions…

1. Are your shadows separate sprites, or are they the actual sprites that you’re then transforming into shadows?
2. Is your game a fixed perspective, or does your camera pseudo-rotate to create a 3D effect?

[AlexOD]

Using a different transform per tile with each tile having their own Spritebatch.Begin() + End() would be very expensive. I’ve started at looking at shader effect files for a possible solution.

Would it be possible to reposition the transform the Shader effect is using before drawing each shadow tile in a single Begin/End call? Or can the uniforms only be set once?

I (think) I understand the points you’re making in your step-by-step. You’re basically saying that if we should position the shadow tiles using the transform we tell the shader and not the position we pass to the SpriteBatch.Draw?

[AlexOD]

1. The shadows are just small portions (32x32) of a tileset contained as a Texture2d Colored Black. Each tile has their own SpriteBatch.Draw call.
2. The game is a fixed perspective yes. It’s a flat 2d perspective with pixel art that has a pseudo 3d angle.

[Trinith]

So each shadow is built out of multiple sprite tiles to make the complete picture? I ask, because if your view is always going to be that fixed perspective and you have no intentions of rotating it, you could always just make the shadows their own sprites, at the correct perspective, and then draw them where they need to be.

You could probably avoid doing any tricksy math to get the projection and positioning just right by doing this.

[AlexOD]

UPDATE: So I’ve been trying using a Shader approach, I’ll show what I have so far:

I have an fx file with a VertexShader that skews our sprites like so:

VS_OUTPUT MainVS(in VS_INPUT input)
{
	VS_OUTPUT output;
	output.texCoord = input.texCoord;
	output.color = input.color;
	float4x4 transformMatrix = WorldViewProjection;
	float v = 0.00075;
	transformMatrix[1][0] = v;
	output.vertex = mul(input.vertex, transformMatrix);
	return output;
}

For the “WorldViewProjection” we do the following:

  Matrix transform =  Matrix.Identity *
          Matrix.CreateTranslation(-cameraMan.Position.X, -cameraMan.Position.Y, 0) *
          Matrix.CreateRotationZ(0) *
          Matrix.CreateTranslation(0, 0, 0) *
          Matrix.CreateScale(new Vector3(1, 1, 1));

  Matrix projection;
  Matrix.CreateOrthographicOffCenter(0, CurrentMapSize.Width,
    CurrentMapSize.Height,
    0, 0, -1, out projection);

SkewEffect.Parameters["WorldViewProjection"].SetValue(transform * projection);

Combined with our previous SpriteBatch.Draw for Shadow tiles, we get this:

So now we’ve managed to recreate the Skew effect but with using an fx shader (and no more black pixels/transparency but we can fix that later), I’ve tried Vertex2.Transform()ing the shadow tile positions with our new combined transform/projection matrix but I couldn’t see any of them.

Any other suggestions for offsetting skewed shadow tiles back to the position of the overlay tile would be appreciated.

For further reference, I’ve been using the following guide. This will explain the visual effect I’m trying to recreate: Skew/Shear Vertex Shader : Sector12 Games

[markus]

Am I understanding this correctly and you are using the same transform for normal tiles and for shadow tiles, with the only difference being the little skewing hack you do in the vertex shader?
transformMatrix[1][0] = v;

It looks like you simplified the shadow transform so much, that you don’t even need a custom vertex shader anymore
This new transform doesn’t affect the z-dimension anymore, which was the original problem, it only shifts in the x-direction. All you need to do is calculated this x-shift, and subtract it from the tile’s x-position, before passing it to SpriteBatch.Draw().

You can do the matrix skewing (transformMatrix[1][0] = v) also in C#. The individual matrix values are accessible. That’s why a custom shader shouldn’t be neccessary. No need to get rid of it though, might come in handy later on.

I think you can do it like this:

// I'll use only the translation here, because the other multipliers don't do anything right now, but you can leave them
Matrix transform =  Matrix.CreateTranslation(-cameraMan.Position.X, -cameraMan.Position.Y, 0);
float v = 0.5f; // needs to be a bigger value than the one you used in the shader, because it's done before the projection matrix enters
trasnform.M21 = v; // or is it M12?

The rest can stay the same, but you have to remove the skewing from the shader, because it already receives a skewed transform.

The x-shift caused by the transform should be a very simple calculation:

float xShift = objectBottom.y * v; // You need the object's bottom here, because that's where you want object and shadow to overlap. 
Vector2 correctedTilePos = new Vector2(tilePos.X - xShift, tilePos.Y); // in case it doesn't work try +xShift instead
SpriteBatch.Draw(..., correctedTilePos ,...);

If objects are built from multiple tiles, you will need the object’s bottom position for this, not the tile’s bottom. I hope that’s not a problem.

[AlexOD]

Hey all, sorry for the late reply, but just to add to markus’ solution I’ll share how else you can modify the pseudo-shadows.

I’ve managed to not only get the “Skew” working, but I can also “Compress” the shadows like so:

In order to do this, you need to apply a value to the “.M22” parameter of the transform Matrix:

  Matrix skewTransform = transform;
  //skewTransform.M11 = 0.5f; X COMPRESSION
  //skewTransform.M12 = 0.5f; Y SKEW
  skewTransform.M21 = shadowParams.SkewX;
  skewTransform.M22 = shadowParams.CompressY;

And then to offset the shadow tiles to the intended position I do the following. (For reference “z” is just a pseudo value I use to give objects perceived height in my scene):

  float yCompressShift = 0.0f;
  float diff = 1.0f - shadowParams.CompressY;
  if (shadowParams.CompressY != 0.0f) 
  {
    yCompressShift = ((position.Y + sourceRect.Height) / shadowParams.CompressY) * diff;
    yCompressShift += (z / shadowParams.CompressY) * diff; // This ensures that overlay tiles towards zero anchor the shadow
  }

  // We need to account for Y compression before X shifting
  float xSkewShift = (position.Y + sourceRect.Height + yCompressShift) * shadowParams.SkewX;
  xSkewShift += (z) * shadowParams.SkewX;

  Vector2 correctedTilePos = new Vector2(position.X - xSkewShift, position.Y + yCompressShift);

Here’s some examples of the shadows with different ShadowParams for compress and skew:

Skew: 0.8, Compress: 0.4

Skew: -0.6, Compress: 0.5

[Kwyrky]

Cool stuff. As in every situation, shadows add a lot to a scene visually.