Aww don't feel that way. Most people will be so grateful to come to your town that they won't even care how many PWPs/hybrids/patterns you have. I've had 300+ visitors and they have never said anything bad about my town.
Even when I chopped down all my trees/bushes and my town was completely barren. People even complimented my town and said it was unique, lol.