my source file has both interlaced and progressive frames in it, it's a documentary so some scenes are shot with different cameras I guess. how should I approach encoding this to xvid? will I loose a lot of quality deinterlacing progressive frames?
If it is a mixture of progressive and pure interlaced video (not 3:2 telecine), then you can use FieldDeinterlace in its conditional postprocessing mode. Only the interlaced frames will be deinterlaced. If you have progressive and telecine, then Decomb with one of its hybrid decimation modes can be used. You could also try some other solutions that have come along since Decomb. scharfis_brain ought to be able to tell you all about them, if you'd like to PM him.
No. It's an Avisynth filter. I do not know of any conditional deinterlacers for VirtualDub. Time for you to learn about a really useful tool called ... AVISYNTH! ;)
Believe me, you won't regret it, and you'll wonder how you ever lived without it! If you want, you can try SmartDeinterlacer in field mode, or even Gunnar Thalin's area-based deinterlacer (both Vdub). Make your thresholds not too agressive and your quality loss on progressive frames might be tolerable.
--------------------- Man, it's a good thing nobody has invented "Scratch 'n Sniff" for the Internet, ain't it?