Yeah, there's no visual way to tell the difference between a Seed/Normal Yellow Mum and a "Hybrid" Yellow Mum that was bred from a Red and Yellow pair. It's up to you to keep them separate.
Once you've started breeding for a specific flower, you shouldn't replace flowers unless something has gone wrong and you need to start over. But what you should do is always start your breeding from scratch by buying seeds from Nook's Cranny or Leif. For example, when breeding for Green Mums, don't just grab any random Red/Yellow Mums on your island or from someone else's island. It's best to always just buy seeds and use those. For certain easier hybrids, this is less important, but for more difficult ones like Green Mums, it's basically necessary.
The reason is the same as to how you can't tell the difference between Seed and "Hybrid" Yellow Mums. Most flowers of a single color have multiple genotypes, some of which won't work well for breeding. Seed Flowers have the same genotype for any given flower, so using them guarantees that you'll succeed as long as you use the proper layouts and don't accidentally mix them up.