From my perspective, as newcomer to classical music, I believe that one of the key elements and therefore "drawback" (in relation to the OP) is the width and breadth of the classical music genre, be it in terms of periods, composers, recordings, artists, etc.
Like many said previously, you can be introduced to that world by your relatives or friends who already know about the genre and had LPs/CDs, and that will give you guidelines to carry on with.
But if you don't benefit from the introduction above, you will most likely first meet classical music by :
- a TV ad background theme
- a movie soundtrack
- a phone "on-hold" muzak

- a dance remix (William Orbit and the adagio for strings...good'un, mind you !)
- Katherine Jenkins singing the welsh anthem on telly
or anything else....
Either any of the above will leave you cold and you'll move on... or that will tickle your curiosity and you'll take a step through the classical door ... The "crossover" classical is as big as it is (and will use most of the columns inches) because it is an "ear friendly" first step where most of the "general public" (not to be read in a derogatery way) will stop at, once they have that curiosity tickled. They will not have the urge to go further because that is enough for them to know of/about, or maybe will stop by being put off by the width and breadth mentionned above . They will stick to that "comfort zone" because where do you go from there ??
From there it takes a bigger effort and dedication and patience to dig further across the genre, because it is really daunting initially ... but boy, has it got its rewards !! Despite the undeniable charms and pretty voice of Katherine Jenkins, i'll go now for Mahler 2nd any time (the first exemple that came to mind

) ... That "negative" reaction to the crossover is only existing because people who have made that further effort know or feel that there is something else out there more rewarding, but you have to appreciate that some are content with "crossover" as well.... frustrating it may be but there you go.
As for the snobbish aspect, it is like any other hobbies and its direct "competitors" (headphones Vs Speakers geeks anyone ??

)... Classical Music (maybe Jazz to a lesser extent) suffers probably more of this image again because of the width and breadth of it... The other musical genres do not offer as many approaches and multiples recordings to a piece of music (usually a studio and a live version, maybe an unplugged one, by the same band...very seldomely covers by others artists). Therefore, Classical music offers more grounds for discussions, debates and other heated arguments over such recording, conductor or orchestra. Non classical music listeners will be (rightly-so) oblivious to that side of things, but will maybe wrongly take the reductive step of calling it snobbish gobbledeegook, when they catch an argument over what Beethoven's 9th symphony or Elgar means to you (exemples for Argument sakes

)
Anyway, that was my (long) 2 cents.
As long as everyone enjoy some music of sorts, that's all that matters !

Olivier