The part where it falls apart. It's so hilariously tragic when you've spent so much time and effort building it and then you can't get it out the pan or something.
I've only ever worked on one gingerbread house, and that was like 4 years ago with my sister. Decorating is definitely the most fun part, I need more of those happy sister moments in my life.