Preamble: I'm very new to shaders.
I'm trying to build an isometric tile-based map with lighting. I'm working in 2d, but the map will somewhat resemble a 3d world in that tiles can be stacked atop each other. My design is thus: each tile is composed of a base image, and four light maps for the direction. I'm using my first shader, which is primarily composed of staples, duct tape, and hot glue, to light each tile using various parameters like color, direction, and the lightmaps. (Yes, I know tools exist to do this already, and do it better than any homegrown solution. I don't care--I won't learn much without writing it myself.)
This technique more-or-less works.
Now, I'd like to 'bake' tiles into columns so that I'm not asking the game to draw several bazillion tiles each frame. For the base tile images, this works fine--I just draw the tiles from bottom to top onto a render target, and because they are opaque, they resemble columns. Each of my lightmaps also has to be rebaked--and because they're not opaque, any lighting on the top of the tile remains in the baked image. The tile N + 1 should block the top of tile N's lighting. I figured I could solve this using a shader to solve it. This shader is not working, and I have no idea why. I could use some help debugging it.
The shader should work like this:
-get the Color from the current lightmap
-get the Color from the texture of the tile above, but offset
-if the Color received from the tile above is opaque, don't draw lighting
-otherwise, draw lighting
Calling the baker:
Note: if this ends up being something resolvable with a quick google search, please tell me what search terms to use, so that I might improve my googling.
for (int i = 0; i < tiles.Count; i++)
setUpMaskEffect(i, transparencyRender, tiles[i].texture.lightNE);//set up shader parameters
batch.Draw(tiles[i].texture.lightNE, calculateCoordsFromY(i), Color.White);//assume calculateCoordsFromY works and returns the appropriate rectangle, because it does.
protected void setUpMaskEffect(int i, Effect transparencyRender, Texture2D current)
transparencyRender.Parameters["currentTex"].SetValue(current);//set the lighting texutre
if (i < tiles.Count - 1 && transparencyRender != null)//if this tile is not the top tile,
transparencyRender.Parameters["aboveTex"].SetValue(tiles[i + 1].texture.texture);//give the shader the texture of the above tile
transparencyRender.Parameters["usingAboveTex"].SetValue(true);//let the shader know this tile will be masked by the above tile
transparencyRender.Parameters["offset"].SetValue((float) height_vertical / (float) height_total);//pass the shader an offset equal to the vertical height of the tile over the total height of the tile image
//the relevant shader code:
float4 MainPS(VertexShaderOutput input) : COLOR
float4 color = tex2D(currenttex_sampler, input.TextureCoordinates);//get the current color
//get the alpha value of the above texture, given an offset
float alphaval = tex2D(abovetex_sampler, input.TextureCoordinates + float2(0, offset)).a;
alphaval = 1 - alphaval;//invert it. If the above texture is opaque, we want the lighting to draw transparent, and if the above texture is transparent, we want the lighting to draw opaque
color.a = alphaval;
The shader simply does not work: no matter what, the above tile's graphic does not block the lighting below from working.
If you know of a better solution to lighting a 2d world, I'd be happy to hear about it so that I can try it after this implementation is fixed.