From the night stand, A Life in Words, Paul Auster conversations with I B Siegumfeldt:

IBS: So what is it that makes some memories stay with us and not others? 

PA: Siri [Auster's wife] has done vast amounts of work on neuroscience and questions of memory. What she tells me, and what I'm convinced is true, is that emotion consolidates memory. If it's just an ordinary occurrence in an everyday moment of your life, you're not going to remember it. You don't recall what you ate for dinner on April ninth thirty-sixth years ago. But on that April ninth thirty-six years ago, your parents had a tremendous argument in front of you, or someone died, or you broke your arm, you might have a clear memory of what you ate that night. It's possible. There must be something sufficiently powerful to make a deep impression on you. Then again, there are things I remember that seem entirely insignificant, and yet, there they are. Little scenes without much emotion to them that come back for no apparent reason... 

IBS: Our senses takes us back to such moments. 

PA: Smell, more than anything else for sure. 

IBS: Sound, music, color... they transport you straight into the past. Of course, it's all connected to emotion, as you say.