You get that dialogue just from people visiting. When people come to my town they just trade at the train station and leave and villagers still talk about them.
I kinda like it ^-^ it's cute that they reminisce about their past. Gives them more personality. But it does get annoying when they repeat it over and over again. Maybe once in a while is fine but constantly on the same day >. < sheesh! I feel like I unintentionally memorized their town tune better than mine! Haha
My villagers always talk about people they never really talked to, even those they saw in dreamtown. Often I have no idea who they are talking about. The only time it annoyed me, was when a visitor talked a while with Deidre and that she moved out without saying a word a few days later. I always wondered if it was linked because that visitor had said they wished to have her.
I get why people don't want it, but personally, I've never minded. I sort of LIKE being reminded of past visitors, even past trades. I think that's what makes visits fun.