I mean, it is -technically- taking an item from them that you didn't ask permission to take, and that's not really cool. Even if it's a renewable resource it's the principal, you know? If someone asked me if they could take some flower tops or fruit, I'd absolutely say yes. But if they just took it I'd be a tad miffed.
Anyway, I don't think it makes a big difference if they DO talk to them. To be honest, it kind of weirds me out how my neighbors keep asking about people who spent a couple of minutes on my island to trade items

I mean, unless the visitor and I arranged a trade, we trade, and then they insist on running around and going to track down and chat with every villager, I wouldn't care.