Has anyone here experienced with different post processing effects that could be applied to photos or renders to achieve all kinds of non-photorealistic, traditional media like results? Mainly what I'm looking for is ways to do color segmentation and edge detection (there are lots of software that do this but usually the results aren't good enough).

So far I have had the best results with using Gimp+G'MIC filters and potrace; the idea is to take something like http://www.flickr.com/photos/decade_null/83335687/ and

1) Optional: use a Selective Gaussian Blur to smooth out most of the noise

2) Run it through G'MICs Graphics Boost filter. IMHO this makes the end result look more natural.

3) Use potrace to extract outlines, shadows and highlights.

At this point I have something like this: http://i.imgur.com/6kf3m.png . There is probably a bit too much detail/noise in that image but it can be easily controlled with potrace's options.

4) Fill the outlines and shadows with different hatching patterns: http://i.imgur.com/dcJn8.jpg , maybe use the original image to give some color to the image: http://i.imgur.com/7A2AJ.jpg

Here is one done with slightly different potrace settings and a quickly handpainted color layer (C64 palette ;) ) http://i.imgur.com/PkwMv.png

Currently my biggest problem with this method is that it's hard extract outlines from images that have big dark areas. This can be seen in the examples where the large shadowed areas end up interpreted as outlines. Then there is the problem with color segmentation; to achieve a cartoon like shading the image should have fairly uniform and "flat" colors.

So, has anyone done anything similar? Any good tips to share on the subject or non-photorealistic rendering in general?