Oh I never posted this when I actually learned it, but something reminded me.
When you're doing cast master, to count as a fail, you either need to leave the fish on the hook too long after it bobs and let it swim away without pressing A, or you need to press A too soon once a fish has started...