The weakening net interest margins reported in July at many banks should remind the market that bank portfolios in the years ahead will continue to walk up to and over an investment cliff. The bonds and loans in these portfolios will mature, and the banks will likely need to reinvest at much lower interest rates. The cumulative difference in interest income and earnings could be breathtaking.

The investment cliff could eventually cut nearly $20 billion of income annually from U.S. bank securities portfolios alone, according to our estimates. And those portfolios only account for a fifth of bank assets. If U.S. banks had to reinvest their $3.4 trillion in securities in similar instruments at today's low rates, their portfolio yield would drop from 2.06% to 1.46%. Although estimates of the investment cliff are imprecise, the prospect is sobering. Every day that a dollar in a bond or loan matures, it rolls off the edge. And this has been happening for years.

The shocking thing is that many banks have done little to prepare. Since early 2011, according to research from Barclays Capital, more than 90% of U.S. banks have set up their balance sheets in anticipation of rising interest rates. To their credit, many bank managers have simply been responding to warnings from regulators that interest rates will rise. The chorus began in early 2010 when the Federal Financial Institutions Examination Council, composed of all bank regulators, urged banks to protect themselves from rising rates. As recently as February, Federal Reserve Bank of Atlanta President Dennis Lockhart warned, "rising rates will create challenges in managing net interest margins and risks."

The problem has turned out to be just the opposite. Rates have stayed persistently low. Persistently low inflation is partly to blame. But more worrisome is persistently low real economic growth, where slow productivity growth has taken a toll. U.S. workers from 1995 to 2004 on average raised output by an extraordinary 3.25% a year, a trend that would double standards of living every 22 years. But average annual productivity growth from 2004 to 2014 slipped to 1.5% and, in the last five years, to 0.5%, a pace that would double living standards every 139 years. Other factors such as excess savings and the falling price of capital good contribute to what Larry Summers has called "secular stagnation." Slow growth should continue to keep rates low.

Even the Fed lately has started sympathizing with the idea that rates may stay low for a long time. The minutes from the June meeting of the Federal Open Market Committee showed that many members of the committee believed interest rates in the long run would fall below historic norms, held down by demographics and, yes, slow productivity growth. The June minutes marked a change from earlier assessments by Fed Chair Yellen that only temporary headwinds stood in the way of higher rates.

Banks and their regulators may now know what many fixed-income investors over the years have learned from bitter experience: predicting the direction of interest rates is hard. Instead of positioning for rates to go in one direction or another, most banks and their regulators at this point should strike a neutral balance and find other acceptable sources of risk and return — ones where the bank has comparative advantage in access or funding or information and can earn good, sustainable returns. If even the Federal Reserve can be wrong about the direction of rates, it is hard to imagine that bank managers can do better.

When it comes to the core business of making loans and investing in securities, circumstances are putting pressure on banks and their regulators to consider new possibilities. The industry has rebuilt capital to the highest levels since the 1930s, and it is time to start using it in new loans and securities and in new business while keeping a sharp eye on costs.

The investment cliff looms. If the banking business continues as usual, it could go over and suffer the rockiest of landings.

Steven Abrahams is a co-founder of Milepost Capital Management and former head of securitization research for Deutsche Bank.