Have just been re-reading some of the other questions raised by Nikam
What is the goal of an improvisation, apart from the obvious use in parts of a church service?
Unsure as to being able to specify a "goal" as such, but I'm aware that sometime in the past, Improvisation Competitions used to be held, but I'm vague about the details. A former teacher of mine once presented a radio programme on that very topic, entitled "Duel for Organs." Two popular organists would meet in the same venue and have a "play off".
Improvisations have featured (and of course do still feature) in concert recitals by organists around the globe (even theatre organists will include an improvisation as part of their concert programmes). This can often be as a further showcase of the organist's talents. Latry, Briggs, and doubtless many other luminaries (Wayne Marshall I think did at some stage?) regularly feature an improvisation at their recitals.
I have a mildly amusing story about improvising at a recital. I did one earlier this year quite by accident. I was doing a recital for the local Organ Music Society and to my shock, despite my usually careful preparation, after arriving at the church I discovered that I could not find one of my scores anywhere in my case, or even the car. The only way to fill in the required amount of time was to improvise (the audience at least got a good laugh during my preamble!). So after requesting a random number from the hymn-book, away I went, and it was thankfully rather well-received.
How should one listen intelligently to what can simply seem a clash of sound?
Again, I'd argue subjectivity on that one. I've heard similar criticisms levelled against improvs by Cochereau, Lefebvre, Latry, et al. An understanding of their own musical language would be fairly important to gain an appreciation of what they do tonally during during their improvisations. Not forgetting the colourful French traditions of tonality, from Tournemire through Dupre, Durufle, Langlais, Messiaen, Alain, etc, etc. If nothing else, to listen "intelligently" as you put it, I guess the simplest thing to do would be to accept the "clash of sound" as an intrinsic part of their musical language at that moment, and put up with it - or, also, perhaps tie it in to whatever conceptual theme the improvisation may be based upon.
Where does the ability to improvise begin, and what are the essential stages of development?
As pcnd has already pointed out, a strong performance technique is basically essential. In terms of developing, (again as pcnd has said) an appreciation of compositional techniques, any and every device you can use, even if it's a way of making the most out of a two or three note motif (transposing, retrograding, inverting, augmenting, , etc, etc) Think on that spectacular improvisation that Latry did after a special service at Notre Dame dedicated to the Emergency service personell of the city - Latry improvised on the two note emergency siren call!! It's on YouTube somewhere.
From my own experience, another important stage of development includes the following: listen, listen, listen, listen, listen, and listen ... and then after that do some more listening - to any and everything, be open to being influenced and inspired by it
To what degree is a musical imagination a contributory factor?
In relation to all of the above, you could be the most knowledgeable player, armed with every clever trick from a thousand books or as many teachers, but if you don't have an imagination to put it together in a creative and interesting way, listeners are possibly likely to get bored!