I'm almost in the same situation as you, but I have one staircase leading to the upper cliffs. The thing is, it's located on the leftmost portion of the map, which means I still have to use the ladder when I need access the upper cliffs on the right side of the map.My upper cliffs aren't accessible without a ladder, so I still use that.
But I don't use the vaulting pole very often at all.
Because of this, I still need to build at least another staircase. I also need to build another bridge - to access the same rightmost part of the map on the north section.
But as soon as I finish building everything, I won't need the ladder and the vaulting pole. I think they were designed to be used only in the early game to tackle obstacles before the player gains access to the terraforming tools and bridge/staircase construction. And this is also a good explanation for them to be the only unbreakable tools.