Two books I heartily recommend:
!. What to listen for in music
by Aaron Copland. The best music appreciation primer around, and available for about a buck in a well stocked used bookstore.
2. The All Music Guide to Classical Music
by numerous writers. This is a large paperback; my copy of the 2005 edition is 1,607 pages long and was priced at $34.95 USD. It is revised regularly; I recommend getting a new one about every 5 years. Its wonderful. They have brief bios of most important composers, both the who, what, when, where stuff and interpretive information about how they fit into the whole scheme of things. Then essays on each of what they consider said composer's major works, and their judgment in these matters, while not flawless, is pretty good. Then, a list of recommended recordings for each one, also pretty good, though not flawless. At least, I find myself tearing my hair out and yelling, "Why the hell did they recommend that record?" a lot less than with other books which recommend specific recordings.
For the Dvorak 9th, for example, they recommend the Kubelik/BPO set of all nine, and individual recordings by Ashkenazy, Szell, Davis, and Kertesz. Actually, I haven't heard the Ashkenazy or the Davis, but the others are excellent. I do wish they had added three more, though--Ancerl, Giulini/CSO, and my all time favorite, the Zdenek Macal/London Phil recording on the budget Classics for Pleasure label. But, fact is, this AMG list is pretty good, so good that I am putting those Ashkenazy and Davis CDs on my want list today.
In addition to composers, they have articles on specific orchestras and chamber ensembles, conductors, and various soloists.
One thing that people find some difficulty with when starting out is finding out something about the vocabulary of whatever discipline it is. One resource which I find very valuable is a free on-line music dictionary at Virginia Tech. http://www.music.vt.edu/musicdictionary/
The have definitions for thousands of musical terms. One other thing they have that I think is unique (at least, I haven't come across it anywhere else) is that they have an audio pronunciation guide for every word in the dictionary. You click on a button near the word, and you hear a knowledgeable person actually pronounce the word for you. Unfortunately, they don't have the names of any composers or performers or ensemble names, just generic musical terms, but it is a very valuable resource.