it doesn't bother me at all but i admit when i had stitches in my last town his house was a bit off-putting. it's probably more of a deal breaker to me than interior but it wouldn't influence who moved in/out of my town!
The exterior is kind of important to me, If I really REALLY love a villager I guess I'd let it slide but if I'm just having a hard time choosing between a few to fit in my town that are not ultimate favorites their house is definetely a deciding factor.
For my town of Moonfall I wanted all my villagers in my town to be animals that typically live in forests and have earth-tones; brown and orange. I also wanted their house exterior to have nice non-bright coloring and have sort of a cottage feel. Marshal does not really fit either, he's white and his house exterior is very ugly to me but he's just adorable and I really don't think any other smug would fit in better since any others I like also don't have very appealing houses or they are not forest animals which was the biggest deciding factor for me.