I had a very similar experience once. There were holes blocking off everywhere when I came off the island. I put the dirt back on a few of the holes ( pressing Y) and someone came over shouting " NO!!!!!" I laughed to myself and flipped the switch.
Aw, but I think if you flip the switch it kicks them off of it too! They had to go start all over again on some other island. lol
I don't mind if people come in and chop most of the trees etc down, because it helps everyone with beetle hunting when there are fewer trees. But some people get mad about it, so I never did that myself. Mostly though, it's not anyone's personal island, so no one should yell at you "no" for doing anything there. I mean they can try to do their thing, but you can also do your thing. Hopefully everyone can be civil...but yeah, let's be realistic. XD