Never fear mods or admins. It's really a demotion, not a promotion.
You don't take the job to have power over users, you do it because you want to help them. In essence, you become a servant of the users. Now if the site owner on the other hand shows up.... there's one to get queazy about, but only if you question your own actions.