Working on first shader, but getting weird effects

Taien · August 13, 2019, 10:55pm

I’ve made some progress after reading some more tutorials. The problem definitely seems to be the texture coordinates.

If I run this basic debug shader:

#if OPENGL
#define SV_POSITION POSITION
    #define VS_SHADERMODEL vs_3_0
	#define PS_SHADERMODEL ps_3_0
#else
	#define VS_SHADERMODEL vs_4_0_level_9_1
	#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

Texture2D SpriteTexture;

sampler2D SpriteTextureSampler = sampler_state
{
	Texture = <SpriteTexture>;
};

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 TextureCoordinates : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR
{
	float4 color = tex2D(SpriteTextureSampler, input.TextureCoordinates); // * input.Color;
	color.rg = input.TextureCoordinates;
	color.b = 0;
	return color;
}

technique Technique1
{
	pass Pass1
	{
		PixelShader = compile PS_SHADERMODEL PixelShaderFunction();
	}
}

I get this:

If I delete the lines altering the color channels from the test shader above, it draws the textured tiles as normal. It’s just the alpha calculation that isn’t working.

I’ve tried a bunch of stuff and not sure what I’m doing wrong here.

Here’s my Batch.Begin:

I can’t really understand why the TextureCoordinates are coming in as one number for the whole area being drawn…

I’m so close! Just need to figure out this alpha issue And yes, the texture png in use definitely has an alpha channel (I checked).

Since you guys are nice enough to help me out, as a thank you here’s a link to a song from the game…enjoy
https://soundcloud.com/taien/lepidoptera-1-final

Kimimaru · August 13, 2019, 11:19pm

Try using the COLOR0 semantic instead of COLOR:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0

markus · August 13, 2019, 11:42pm

I don’t think you should apply the effect manually. SpriteBatch handles that internally. You are passing the effect as a parameter to SpriteBatch.Begin after all.

Taien · August 14, 2019, 12:03am

Yeah sorry, that part in the original post is older I removed that line since I changed it to be passed into the begin call now. I will try COLOR0 and see if that helps.

Edit: Tried changing the semantics to match (COLOR0) and there was no change. Same with using COLOR in both places. I’m scratching my head…

Edit 2: Basically, all I’m doing is starting the batch and passing in the test effect from my previous post, drawing tiles, and ending the batch, but the TextureCoordinates seem to be wonky. Does it possibly have something to do with one of the other Batch options I chose?

Taien · August 14, 2019, 8:55pm

Am I just not understanding how TextureCoordinates work? I am under the impression they are supposed to be between 0 and 1 as a float, similar to the colors…

Edit: I’m having a lot of trouble locating good, current resources on how to do this sort of thing. If any of you have some relevant links you can share, I’d really appreciate it. Most of the tutorials I’m finding are either extremely outdated or simply don’t work in practice. Even the default shader created by Monogame’s Content Pipeline Tool doesn’t work. Because of this, I’m having a lot of trouble trying to follow another person’s example…there seems to be a lot of disagreement as to how the signature of the Shader should look, whether it requires VertexShaderOutput or like a color and textcoords as arguments. I’m not sure what variables I actually have the option of accessing through the VertexShaderOutput because I can’t seem to find a list anywhere. So yeah, any relevant links would be great…Google is failing me

kgambill · August 14, 2019, 9:04pm

Out of curiosity, if you hardcode color.a = 0.5f (or whatever number) do you see what you would expect? That should help identify if it’s a problem with the actual alpha calculation in the shader or a more global setting in the graphics device.

It’s always possible it’s an issue with premultiplied alpha. I had quite a few issues with that when I was starting my current project. You can try setting the blend state to BlendState NonPremultiplied instead of AlphaBlend and see if it changes.

kgambill · August 14, 2019, 9:04pm

The song is quite catchy, by the way.

Taien · August 14, 2019, 9:06pm

I tried that as well, and it doesn’t seem to have any appreciable impact on the drawing. I’ll show you the output from color.a = 0.5; So you may be right.

One sec.

Taien · August 14, 2019, 9:11pm

Glad you liked the song There’s a bunch of other ones posted on my soundcloud as well if you get bored.

So if I use color.a = 0.5, I get the following:

If I just return color without any changes, I get this:

They both seem to be drawn with full alpha, except the first one almost seems to be using additive math for the colors that are overlapping, even though I specified AlphaBlend. I’m baffled. I’ll try setting NonPremultiplied and see what happens.

Kimimaru · August 14, 2019, 9:11pm

The input of the pixel shader must be the same as the output of the vertex shader. I’m not 100% certain on this, but I believe MonoGame’s default vertex shader outputs what you currently have, so you should be good with the inputs. I have shaders set up identically that work with no issues.

I recommend testing an empty game state with just your texture on the screen. If that looks good, then apply a very simple shader like @kgambill suggested. There may be something off with your graphics device states.

Taien · August 14, 2019, 9:23pm

Ok, so I tried using color.a = 0.5 with NonPremultiplied, and that seemed to draw it correctly (half alpha):

Then I tried using color.a = input.TextureCoordinates.x and I got this:

The problem has GOT to be the TextureCoordinates. Is it because of the type of Draw call I’m using?

Here’s the method called in that Draw line (it’s inside the TileSheet class I wrote, which stores the texture and info on the tile sizes):

Variations is an int defining how many variations each tile on the tilesheet has. So if it’s 4, that means every tile on the tilesheet gets 4 cells with a slightly different graphic (for non-repetitive-textures-sake). I’m reasonably certain my math there is correct (it’s supposed to be getting the lower-right 16th of a tile, depending on its size, which by default is 64x64, so the 16th in this case is 16x16.

TileWidth is the width of one tile, so in this case, 64. Same with TileHeight.

Width is the width of the entire texture. Same with Height. In this case they are both 1024.

Taien · August 14, 2019, 9:24pm

When you say there may be something off with my GraphicsDevice states, what exactly do you mean? I’m using the GraphicsDevice that’s provided by monogame at program start, but I’m not super familiar with how to mess with its state.

Kimimaru · August 14, 2019, 9:44pm

SpriteBatch internally changes the graphics device’s state when you call Begin. This includes the BlendState and shader used. It needs to do this, so it’s not an issue. What I suspected when I mentioned that is the state is unintentionally changing at some point in between your drawing code.

What happens if you multiply the pixel’s color by 0.5 instead of setting color.a to 0.5 with AlphaBlend? The first output you linked looks like premultiplied alpha at work. If that looks good, then try multiplying the output color by input.TextureCoordinates.x.

Taien · August 14, 2019, 9:50pm

Multiplying the entire color by 0.5 with AlphaBlend produces the same output as setting color.a to 0.5 with NonPremultiplied.

Multiplying the entire color by input.TextureCoordinates.x produces the same output as setting color.a to input.TextureCoordinates.x.

The texturecoordinates still seem to be off I’m so lost, lol.

Kimimaru · August 14, 2019, 10:16pm

What are you expecting the texture coordinates to be? For instance, if all of the tiles are in the same texture and one tile is 1/4 the entire texture vertically and horizontally, the first tile will start at (0,0) and end at (.25,.25). If your tiles are in different textures then you will need to either account for the size of each texture or render tiles in different textures in separate batches. I’d suggest the latter due to improved performance and fewer complications with handling different texture sizes in the shader.

I think the next step would be to omit the tile drawing for now and keep it as simple as possible so it’s easier to diagnose the issue. Simply draw the entire texture at once and see what happens.

Taien · August 14, 2019, 10:42pm

Give me a little time. You’ve revealed something here I wasn’t aware of. I assumed the texture coordinates were related to the section of the texture being drawn, not the entire texture. So it looks like I need to do some more math. Haha.

Edit: In other words, I thought that when I provided a sourceRect to the draw function, the tilecoordinates were how far across that sourceRect we were, not how far across the entire texture

Taien · August 15, 2019, 12:02am

I’m pretty close to solving it now thanks to your help with that texturecoordinate info. Here’s my latest test

Taien · August 15, 2019, 2:01am

I’m really close now. I had to do some additional math, but here’s the current state of things.

As you can see, the corners are still a bit bright, and the borders are a bit dimmer than I want. I found that after figuring out what the alpha value should be, I still had to multiply the color by the alpha to get the desired effect as shown above. But it’s still not perfect… I will keep experimenting

Here’s the current state of the shader:

#if OPENGL
	#define SV_POSITION POSITION
	#define VS_SHADERMODEL vs_3_0
	#define PS_SHADERMODEL ps_3_0
#else
	#define VS_SHADERMODEL vs_4_0_level_9_1
	#define PS_SHADERMODEL ps_4_0_level_9_1
#endif

int style;
float4 coordPoint;
Texture2D SpriteTexture;

sampler2D SpriteTextureSampler = sampler_state
{
	Texture = <SpriteTexture>;
};

struct VertexShaderOutput
{
	float4 Position : SV_POSITION;
	float4 Color : COLOR0;
	float2 TextureCoordinates : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 color = tex2D(SpriteTextureSampler, input.TextureCoordinates) * input.Color;
	float2 coords = (input.TextureCoordinates - coordPoint.xy) / (coordPoint.zw - coordPoint.xy);
	
	if (style == 0)
	{
		if (coords.x < coords.y) color.a = coords.x;
		else color.a = coords.y;
	}
	else if (style == 1) color.a = coords.y;
	else if (style == 2)
	{
		if (coords.y < 1 - coords.x) color.a = coords.y;
		else color.a = 1 - coords.x;
	}
	else if (style == 3) color.a = coords.x;
	else if (style == 5) color.a = 1 - coords.x;
	else if (style == 6)
	{
		if (coords.x < 1 - coords.y) color.a = coords.x;
		else color.a = 1 - coords.y;
	}
	else if (style == 7) color.a = 1 - coords.y;
	else if (style == 8)
	{
		if (1 - coords.x < 1 - coords.y) color.a = 1 - coords.x;
		else color.a = 1 - coords.y;
	}
	
	color *= color.a;
	
	return color;
}

technique Technique1
{
	pass Pass1
	{
		PixelShader = compile PS_SHADERMODEL PixelShaderFunction();
	}
}

Style 0 is the top-left corner segment, and Style 8 is the bottom-right corner. The math seems to still be a bit off but I’m really close now. I greatly appreciate all the help here.

Taien · August 15, 2019, 9:41am

I figured it out I’ll post when I wake up about how I got it working, what I realized I was doing wrong, etc. I’m too tired to explain now…but it’s looking smooth now though I even added a little highlight over the tile the mouse is on.

kgambill · August 15, 2019, 9:18pm

Glad you got it working!