From a 1977 interview with Richard Wilbur, as found at the Paris Review:
You have said, “I do think that there is nothing more dangerous to the imagination than fantasy.” Many people would equate the two things. Could you elaborate on that?
To me, the imagination is a faculty that fuses things, takes hold of the physical and ideal worlds and makes them one, provisionally. Fantasy, in my mind, is a poetic or artistic activity that leaves something out—it ignores the concrete and the actual in order to create a purely abstract, unreal realm. If we think of fantasy at its least dignified, non-artistic level, this becomes obvious. Sexual reverie very clearly leaves something out, and that something is the physical object of one's desire.