I only allow people in after conversing with them about the purpose of their visit, such as a trade. That's why, along with the basic rules, I ask them to follow me at a close distance at all times if they need to walk past the train station area.
To be honest, I'm not strict on how I have visitors. My town is...I guess developed...so common sense tells me don't run on my flowers, just walk. I have this rule. If you break my flowers in an accident, I wouldn't be mad. If you break my flowers but don't say anything...:/ Hm...I wonder how I'll react.
Otherwise, not others. I let people talk to my villagers, as long as they don't write some hate letters to them. :/