I feel the same way. I never understood why some people don't allow it. If they don't want the dialog they shouldn't receive visitors in the first placeI think it’s fine if my visitors talk to my villagers
Regardless of if a visitor talks to my villagers or not they will still mention the visitor later that day or the next