This is a common misconception, perpetuated by ignorant males who really don't have a clue what they are talking about.
In our modern era of rock stars wearing tight leather pants (frankly, that sounds way more gay to me) for some silly reason a bunch of men and women in a classical orchestra playing music that was composed centuries ago is considered "rather fruity".
HOWEVER lets stop and consider the actual music being played... most of it doesn't sound remotely "gay".
Consider the following scores:
Hall of the Mountain King (by Edvard Grieg)
Ride of the Valkyries (by Richard Wagner)
Such music is played by American troops heading into a war zone because they like freaking out the enemy with the sheer massive quality to the music. (See the scene from the Vietnam War film "Apocalypse Now".)
The fact is classical music is used a lot in films, especially in explosion scenes because it is such exciting music. See the explosion scene below from "V for Vendetta".
When I was young I listened to classical music regularly. I found a record in my parents' record collection called "Royal Philharmonic Orchestra : Hooked On Classics: 1982" and I was instantly addicted to listening to it because it was such EXCITING & EXHILARATING MUSIC. It made me hyper just listening to it, faster than 4 cans of Coca-Cola on a seven-year-old.
And its not limited to myself or films... classical music is also rampant in the video game industry because its ideal as background music and there's no copyright on it. These days the video game industry is bigger than the global film industry and as such there are gamers out there who are daily listening to classical music whilst slaying computer generated orcs and dragons.
Small surprise, but classical music is now seeing a revival in sales (both on CD and digital downloads such as iTunes).
Even classical orchestras are seeing new people coming out to see their performances (if you live in Toronto, try Mooredale Concerts). Mooredale has a philosophy that "classical music is for everyone" because it is gender neutral and because there is no lyrics it crosses any language barriers.
Conclusions? Classical music has always been associated with the culturally elite (not necessarily rich, simply for people who are often more educated about the history of music)... but there is no reason regular people can't appreciate it. Try listening to the medley video below and see how many of the different scores you recognize:
Is High Culture making a comeback in 2010?