I tend to not hold anything when I walk around, I only hold a tool when I need space and even then I just equip my watering can since it uses up my time when I put away my shovel just to talk to someone.
What's even worse is when they ping you and do their little dance for a bit while you're talking to a different villager. You press B as fast as you can but it's too late and you just have to hope that the ping wasn't for a pwp you wanted!