That's the default for you visiting no matter how long you stay. I usually stay until they ask me if I want to buy something they own. Best way to get rid of weird things they're displaying!
I used to feel more guilty about this, but nowadays the thing that makes me more stressed out is wondering how to rate their house at the end of a scheduled visit. It's so sad when they act like I was just being nice to them when I say their house is great. I really meant it!
Sometimes I still feel a little guilty when leaving a villager's house right away, and it's a little ironic when I consider how eager I am for villagers to leave whenever they pop into my house unscheduled. I love my villagers to bits, but they always seem to choose to visit whenever I'm rearranging stuff and I want to get right back to it.