Quickness has no distinction between caster or physical. Everyone gets a huge benefit from quickness, no questions asked.
Also, forget about elite up-time. It's a bad way to compare elites. People keep trying to compare gw2's elites to gw1. They aren't the same thing. Truth be told, a skill being elite in GW1 was a BAD thing because you only had 1 slot for an elite skill while most of the time elites were treated as just 1 more skill in your bar.
By giving elites actual strong effects (like the mass quickness) you also need to give them huge cooldowns, or else you'll be rewarding button mashing instead of actual skill usage.