Something like this…
I’m still playing with ideas but I think my game might want to involve overlapping sprites. Because of the colour scheme I’m wanting to use, I don’t really have enough contrast between sprites so when they overlap each other, such as on the left side, it can be hard to tell them apart. I’m wondering if I can do something like on the right side, where I highlight the parts of the sprite that have something underneath them.
I’ve explored using a shader to create an outline around the sprites, but this has two issues. The first is that it kind of messes with the shadow effect I’m doing and looks a little off. The second is due to the antialiasing Anisotropic filtering gives. I’m using this because I want my sprite rotations to look good, but also to maintain a pixelated appearance. So I take a low resolution sprite, scale it up, then render it rotated with anisotropic instead of point filtering so it can fill in some gaps. When I apply an outline shader to it, it kind of looks off because the outline shader detects the anti alias pixels too and so, it looks bad. I am considering an interior outline, but I worry that might have the same effect as the first issue.
The overall effect that @willmotil posted is what I want to do, but I’m wondering if it’s possible to do without having to pass both textures into the shader. I mean, I could do a detection pass at render time to see if there are any sprites under my current one and, if so, apply the correct shader. I was just wondering if there was a way to do this “automatically”. Someone suggested the stencil buffer to me, and I’ve been looking into that, but I’m might not be getting it. It looks like I have to render all my sprites to the stencil buffer first, then do another render pass after? Not to say it won’t work, I just might not be getting it.
I’m thinking what I want to do might not actually be possible because it’s rendered in a SpriteBatch using deferred mode. My (limited) understanding here is that the contents of the backbuffer might not actually be known until End is called on the SpriteBatch.
Waaaaaaay back when I was younger and programming in QBasic/Assembly, I would have done this by drawing all my graphics to a back buffer one at a time. Then, when I wanted to see if there was anything under my sprite, I would just check the appropriate offsets of the back buffer and make the appropriate adjustment. I realize this isn’t how things are done anymore, just providing it for possible context
This is all just exploratory for me right now. I’m still settling on what my visual style is going to look like. I am kind of drawn to the two colour visuals of the screen shot, I’m going for a more retro look, like gameboy or those LCD handheld games they had when we were kids. But I also want my sprites to be able to rotate freely and not look like garbage at certain angles, as they did when I rendered with Point filtering. Other possibilities though might include rendering at fixed angles (45 degree rotations) so I can use Point filtering, or just abandoning the visual style entirely and going full colour.
So yea, just exploring my options