I like having them, especially because I enjoy using the mushrooms to decorate. Depends on where they are in your town though, if they're really in the way I'd probably dig them up.
I would keep them, especially if they have interesting patterns! I've cut down tons and not gotten any in my town. I don't think they count against perfect town, but I could be wrong.
I have a bunch of stumps with patterns, as i have a foresty town and i think it looks cute to have little nooks with a stump and a book, like this area in my town for example;