Early on, passivity is your enemy. Later, too, but less critical then. You've also got kind of a mixed metaphor going, and you use 'weight' twice in short succession. Dunno if this is a dealbreaker, but a reader is likely to notice.
The past never let go. Outside, rain pulled leaves from...