It depends on whether the output stream would look the same. In other words, something like a more efficient way of estimating motion should not change the output stream, since it's all done by the encoder, the encoder is just making different decisions. Something like a new way of ordering B-frames, though, (yes, I'm pulling things out of my a**) would not work since you need to maintain compliancy with the H.264 standard, you don't want to have to use a hacked decoder to make it work.