I don't have an answer here, but I did notice this happening too. However, it was only in a tour on the island (a non-fishing one). I haven't noticed it elsewhere.
Just tested this in two towns, one updated, one not. No skittish fish encountered. Caught a Pond Smelt in each. What are you wearing? (Wait! That doesn't sound quite right.)