I think a huge part of the initial issue is just that: fear from all of the potential damage that visitors could do on past games. Unless you are leaving items on the ground that someone could steal, there's literally nothing they can do to your town that would "ruin" it. The scams people are talking about here are one thing, but someone just taking a walk through your town? There's really no crimes they can commit.
Having played new leaf, I totally get that fear, however. To the point I even feel nervous visiting someone else's town, like I don't even want to step off the path, or leave the sight of my host, because I'm nervous about making them nervous. It all goes back to the days of the older games when an unsupervised visitor could absolutely destroy your town, and people would have rules posted in their threads like "if you leave my sight, I'll end the session immediately and the trade will be voided".
I'm far more laid back as a host than as a visitor. I encourage people to take a stroll through my town, talk to my villagers, visit the shops, see the sights, you know? But what we're seeing here is a clash of new players vs. old. There's a big part of the playerbase that is experiencing animal crossing for the first time and they honestly have no idea of the unspoken etiquette that's already been established by those of us who have played the older games.