The housing industry has staunchly defended the mortgage interest tax deduction as an essential tool for promoting homeownership. But attitudes are softening, as data show the low- and moderate-income homebuyers who need the subsidy the most are getting the least benefit.

If the deduction were to be discontinued, it would make housing a little less affordable than it is right now. But with interest rates remaining near all-time lows, it would have less of an effect on borrowers' wallets than it in the 1980s, when rates were above 10%.

"If you were going to do it [take away the deduction] and you were looking for the time to do it when it was least painful now would be the time to do it," said Brent Nyitray, director of capital markets for iServe Residential Lending.

Nearly 33.6 million taxpayers deducted $72.4 billion in mortgage interest from their 2014 taxes, and that figure could rise to as much as $96.4 billion by 2019, according to estimates by the Congressional Joint Committee on Taxation. That makes it one of the largest tax breaks taken by individuals and an easy target for policymakers looking to increase tax revenue without raising tax rates.

Few lower-income borrowers itemize deductions on their tax return, a requirement to receive the benefit. And for those that do, "the benefits they get financially clearly are less than folks that are in the upper end of the income bracket. If you're in a 40% tax bracket, obviously that's going to be a much bigger subsidy to you than somebody that may be in a 15% tax bracket," said Gary Acosta, CEO of the National Association of Hispanic Real Estate Professionals.

Of those that claimed the deduction, 14.3 million returns were from taxpayers with household incomes between $100,000 and $200,000. They claimed a total deduction of $28.8 billion.

An even bigger financial benefit went to taxpayers whose household income was over $200,000. That group represents 17% of returns claiming the benefit, but 42% of the total dollar amount deducted, or more than $30 billion.

Meanwhile, only 2.9 million returns had household incomes of $50,000 or less. The total deduction claimed was $1.6 billion or just 2% of the dollar amount.

In comments made during a television interview and reiterated in an exclusive interview with National Mortgage News, Mortgage Bankers Association CEO David Stevens said lower-income borrowers do not itemize on their taxes and can't take advantage of the deduction.

"If the entire tax code were to be looked at through a formal process and it was thorough and the mortgage interest deduction then became part of that dialogue, we certainly would support that," Stevens said last month.

The deduction is a staple in the real estate industry and to a certain degree, built into home prices. "Eliminating it altogether would be a disruption. But NAHREP would be open to a modification that weighed the benefits more towards first-time home buyers, perhaps even having a first-time homebuyer tax credit versus a tax deduction," said Acosta.

For example, the government could give first-time homebuyers a one-time tax credit, possibly $10,000 and that would be accompanied by lowering the current loan amount cap for the deduction.

If the deduction is changed, the aim shouldn't be "to punish people who aren't taking advantage of it by taking it away; it might be a better idea to come up with something for taxpayers who don't itemize," added Ten-X Chief Marketing Officer Rick Sharga.

"If the belief is still that a high level of homeownership has economic benefits for the country, gives the middle class an opportunity to acquire and maintain some degree of wealth, then anything you can do to stimulate homeownership responsibly would seem to make sense," said Sharga.

But government policies that subsidized home purchases contributed to the most recent housing bubble, said Nyitray. And now the overwhelming majority of mortgages have some sort of government backing or subsidy.

The deduction makes a big difference for the middle to upper middle class clients of Michael Chadwick, a financial planner in Unionville, Conn.

"In Connecticut taxes are so high from a property tax perspective, even if you have a small mortgage of $100,000, people are still itemizing. I have a lot of conversations with people where the buy or not to buy decision is actually revolving around the deduction," Chadwick said.

Then there are senior citizens who are selling their home and electing to rent. Often their income tax bill jumps by $4,000 or $5,000 a year because they lost the deduction, a sign that it "was really a big deal," he added.

"There are higher end of the market people who do get mortgages based upon advice from their accountants. There are certainly situations where, instead of paying all cash, because of the mortgage interest deduction people will get a mortgage," said Total Mortgage Services CEO John Walsh.

"So it would certainly hurt the mortgage industry if the deduction were eliminated or changed" but how much depends on what those changes are, Walsh said, who estimated that 20% of Total Mortgage's closed-loan volume is jumbo mortgages.

Meanwhile, a higher-than-normal percentage of Americans is still opting for renting versus buying. So if the government was trying to stimulate homeownership growth, this might not be "a wise time" to eliminate the deduction, Sharga said.

On the other hand, with the upcoming presidential and congressional elections, now is a good time to have a discussion about the deduction, Acosta said. "Housing policy should be top of mind. We'd certainly like to see that more from the candidates, all the way down the ticket."