You read that correctly! The IRS has the authority to deny or revoke your passport!
When, how, and why do they do this? It all has to do with past debts and you ignoring their attempts to get a hold of you. The good news is, you can avoid this situation. Here is what you need to know:
In 2015, the FAST Act was passed (the Fixing America's Surface Transportation Act) and within it, in order to help enforce collections, the IRS was given permission to cancel, deny or revoke a person's passport if they owed more than $50,000 in past due taxes. This happens when you have an IRS debt that exceeds $50k, and you have been ignoring their attempts to get a hold of you.
When the IRS believes that you owe them money, before their revoke your passport or take other collection actions, they start by sending you a series of letters. The letters start out pretty nice, basically stating "hey, we think you owe us some money. Please either pay us, or send us back a letter telling us why you don't think that you owe us". If they don't hear back from you, the letters get progressively more and more nasty, until you get some final notices. That is when the passport denial or revocation starts to kick in.
Unfortunately, once the IRS decides to revoke or deny your passport, the only way to get it back is to file in tax court. You cannot appeal it at the IRS level (which is the easier way to deal with IRS issues). That is why you definitely want to avoid getting to this point!
The other item to note, is that there is a way that someone can be declared "Currently Not Collectible" (the IRS calls this being in CNC status). This is a method that some people use to get the IRS or other debtors off of their back. That doesn't work in the case of Passport denial/revocation. Even in CNC status, the IRS can still remove your passport.
So how do we avoid this? We open our mail and get ahead of IRS issues! If you are actively working with the IRS, or you are in an agreed upon payment plan (even if that payment agreement is at a reduced rate), then they generally won't be coming after your passport!