The French seem to like stressing the final syllable of words, so that when I was watching Michael Haneke's engrossing French film The Piano Teacher the other week, I heard Isabelle Huppert saying schön-BERG, schu-BERT, etc. On the other hand, we Americans always refer to Bizet's opera as CAR-men, while the score invariably stresses the final syllable, as in car-MEN. But then again, car-MEN is not a composer, and is therefore irrelevant to this thread.
I will however persist in mispronouncing Dufay with two syllables as du-FAY. I'm sure he will not object, and I prefer it that way.