My wife and I call condoms "Novacaine." For me, they deaden the sensation so much, that things just don't work. My wife said that she didn't like the feeling either. Maybe she just didn't like that I wasn't staying as hard or as big. I may be very unusual, but I don't find it terribly surprising that other guys have a hard time keeping it up with a condom on.
We've never tried the female condom route. Since I've had a vascectomy, there is not more need for condoms at our house.