If the players in your group are sensitive to what is going on, the flag should not slow down play at all. I prefer the flag out on putts inside of 20 feet. I will always ask someone who prefers the flag in if they want to finish before I pull the pin. If they choose not to proceed I will hand them the pin or hand it to another player who is finished so they can replace it and play moves along pretty quickly. Again it largely depends on the players in your group, if everyone is courteous it should be a non-issue.
Most players who prefer the pin in use the flag to stop the ball on hard hit or downhill putts, once you get inside a certain distance most players do not care if the pin is in or out.