I don't like chamois either.
It's in the design.
Something perfectly flat has the most surface area. Therefore something flat can absorb faster that something that's textured.
Chamois are relatively flat, so they absorb lots of water fairly quickly.
Something that's flat also doesn't provide anywhere for dirt to be pulled away, so if you rub, you drag any dirt you missed along the surface. Rubbing with a chamois is trouble.
With a chamois, I just lay it on a part, kinda press it, and lift it. No rubbing. It helps a lot to keep the swirls down... eventually I just took Homer2's advice and switched to a bathtowel, which works really well... but a little linty sometimes.