Symphonies 2-3
I pulled these out just to see how conservative they are. They are! 
No.2 (Bernstein/DG), in five movements, sounds almost like how I'd want my perfect Victorian symphony to go. Perhaps I think it reminds me of Elgar, but not the that degree. I don't know, maybe your words about this music will make me like it more?
As I'm listening to No.3 (Marriner/Argo), I'm completely underwhelmed by the sheer normalcy of the music. It's nice and semi-pastoral, but that's about it. Had all Ives sounded like these two beasts, he would have been relegated to the Chadwick/Mason/Foote (re: quaint & boring)corner of the room. What do you think?
Both pieces are over. I feel compelled to listen to them further, on academic grounds, but don't feel the need anymore. Perhaps this is where boring Cowell comes from?
Don't get me wrong,... nice music. Nice.
This may sound strange coming from a dyed in the whool Ives fan, but his 2nd symphony has (after many years of studying his music) become my favorite work from Ives. I love his 1st symphony too, even better than his 3rd and 4th.
For my taste, of the most successful recordings of the Ives 2nd is Kenneth Schermerhorn's account on Naxos:

This recording uses the critical edition, prepared by Jonathan Elkus of the Charles Ives Society, which corrects about 1,000 manuscript errors that have been repeated in recordings (and performances) dating back at least as early as Bernstein's late-'50's New York Philharmonic Orchestra recording on Columbia.
Errors or not, that is not my deciding factor in deciding on a performance, it's the end result that counts and I
love what Schermerhorn has done with the 2nd. Absolutely tamer than Bernstein's account, but not as slick either.
My own personal favorite is Harold Farberman's old account, which I have on LP and (out of print) CD. This recording has something special I can't explain.

I like how you described the 2nd as your
perfect Victorian symphony, as that is exactly what I feel about it, and how Schermerhorn and Farberman play this work. I have a suspicion that the "Victorian" nostalgia-feeling is Ives' true self, and that his 1st, 2nd, and (perhaps to a lesser degree) his 3rd are signs of his deepest self. His more experimental work appears to be a reaction to this side of him, brought on one by life's blows and experiences. In the end, we'll never know, but interesting to ponder.