Well, you are not friend enough with them. If you were close friends, the animal would turn to you and smile.
Are they where villagers tend to hang out? I have 3, and one is by the waterfall -- I've never seen anyone on it, but I rarely see anyone in that part of town. The one by my house (and 4 villagers) gets sitters a couple times a week, I think.
I had one near a cliff in my old town, for some reason I would find villagers sitting there a lot.
I love finding villagers sitting on stumps even more. Especially when they're asleep. Too adorable. <3