recently i'v added few post effects and i thought i need some AA solution to remove those jaggy edges.
traditional AA solution have its quality, no questions but it have some drawbacks:
1. deferred or semi deferred need special care and will need DX10+.
2. needs more memory
3. eat gpu power
i guess there is more but this is enough for me to consider some other solution, not the same quality but better than none.
so what i did is pretty simple:
1. run edge detection
2. run post AA which samples 4 pixels and use the value from 1 to know if its an edge or not, if its an edge i output the filtered pixel, if not i output none filtered pixel.
the heart of the algorithm is 1, what i did is taking few neighbors pixels and check for a big change in their depths/normals (for normals i check angles, for depth i check the gradients from center pixel)
again, very simple but i think it does the work, very fast and its generic for xbox,ps3,pc...
here is few screenshots that shows you the process