Holy **** swimming in your underwear is nasty. Check the pool rules people, NO STREET CLOTHES ALLOWED. If you're going to bring a pair especially to wear to the pool I guess it's ok, but wearing any street clothes in to the pool just results in getting the pool way dirtier then it already is. If you're really that protective about your privates, buy a speedo and wear it under your shorts, or buy shorts that have support in them.
I'm a lifeguard, and we have to deal with this all them time. It really contributes to pool dirtiness. For the love of God don't do it.