Using your IRA effectively to limit taxation on your investments is one sign of a savvy investor, but the IRS says that some people have taken this to an extreme. While most people have less than $5 million in their IRAs (typically much less) just over just over 300 people hold a combined $81 billion in their IRAs.

“IRA balances greater than $5 million could be considered large based on what individuals typically contribute,” the IRS writes in a classic bit of bureaucratic understatement.

IRA accounts by size 1114

Normal investment would need 18% returns to get a $5 million IRA

Since IRAs were intended to encourage people to save for retirement, not to shield large investments from taxes, there is an annual contribution limit (and some other additional eligibility conditions), but there isn’t a limit on the total size of the account. At first glance this policy makes sense: there are penalties for early withdrawal and you don’t want to penalize people because the assets they saved for retirement have done particularly well.

But getting to $5 million should be extremely difficult. As an example, someone who made the maximum contribution every year from 1975 – 2011 would have paid in a total of $99,500 before any gains on the assets, which would amount to $730,000 if it had been invested in the S&P 500. To get to $5 million the assets in the account would need to have a dollar-weighted rate of return of 18%.

ira irs rate of returns 1114

Two scenarios for enormous IRA accounts

But even 18% returns don’t explain what we actually see. For 314 people to have $81 billion in holdings, they would need to have about $250 million apiece, so any kind of normal market returns are out of the question. The IRS relies on account custodians to report fair market value for assets and doesn’t have good data on the assets themselves, but it speculates that two specific kinds of deals are at the heart of these IRA mega-accounts.

Some people have access to non-publicly traded assets, giving them a lot of control over the estimated price of those assets when they are first deposited into the IRA. An entrepreneur who deposits stocks in his or her startup and then later has a big IPO, for instance, could see the IRA account grow exponentially.

founders accumulating large IRA 1114

This example seems like it should be exceedingly rare, and even the IRS should know that going after successful entrepreneurs isn’t going to be popular, so they offer a second possible way to accumulate a large IRA through private equity activity. By rolling over a 401(k) plan into an IRA and using the proceeds to buy a profit interest in a PE firm’s general partner could result in a multi-million dollar IRA as well.

PE IRA use 1114

The IRS admits that it doesn’t really know what’s going on inside these huge IRA accounts, but that’s kind of the point. It wants more authority to see how people are building so much value in retirement accounts that were never meant to be tax havens for the ultra-wealthy.