That's actually cool, as it would also allow clipping against a "foreground" by varying the address of the scanline mask. E.g. imagine foreground trees in a jungle scene.
I did extend it to use a full-screen foreground mask that implemented this sort of clipping. I was able to make the mask scrollable which allowed the compiled sprites to appear "behind" fences and other complex shapes with per-pixel accuracy.
It could even be used to mask out individual pixel bits that allowed for fake "lighting" changes with a carefully chosen palette.
I keep wanting to do a "retro-game" and make use of what I've learned about these types of effects now. Despite how far machines like the C64 for example were pushed, I don't think they were pushed nearly as far in terms of games as with demos and it'd be fascinating to try to push the limits..
