Either it's too close to the house, or it could be too close to the paths. I've never tried it before, so I don't know if trying to grow plants near paths will cause the plants to die, or for them to automatically get rid of the path (like a villager house would).