Question about bitmask render

is anyone can share about how bit masking works for rendering or any source that I can refer with?

Not sure what you mean bitmasking is typically bit math on binary bits that are either xor’ed or anded.

For rendering you mean stenciled ?
like this ?