The Department of Housing and Urban Development cut corners when it attempted to upgrade a number of internal computer systems and hastily awarded a $17 million add-on contract to an IT services procurement originally valued at $800 million, according to the findings of a HUD Office of Inspector General audit.
The HUD OIG found that a comprehensive plan was never fully vetted to perform the task and HUD did not follow its own IT policies when it assigned the work. In the end, the HUD OIG said poor planning and miscalculations forced HUD to terminate the failed project, leaving critical systems in HUD’s mortgage operations left using antiquated technology.
A HUD spokesperson declined Mortgage Technology’s request for comment, but in the audit, Jerry Williams, HUD’s chief information officer, and Jermine Bryon, its chief procurement officer, defended their respective offices’ involvement in the initiative, claiming the $17 million project met the requirements to justify the contract modification and that it was not of the size or scope of actions that fall under the purview of HUD technology procurement policies.
The process of a systemwide upgrade of HUD’s aging technology infrastructure has long been hampered by budgeting and planning woes. Some of HUD’s mainframe-based technology has been in place since the 1980s. HUD officials believed upgrading the mainframe systems with modern, server-driven and open-source technology would improve performance, make HUD more agile and avoid increasing costs associated with the maintaining the legacy mainframe infrastructure and its related systems.
It was in this vein that HUD’s Office of the CIO proposed a plan to transfer a number of HUD applications and processes from the legacy mainframe systems—comprised of proprietary Unisys and IBM technology—to a server environment running Sun Microsystems’ Solaris operating system, one of many platforms based on the open-source Unix standard.
The “mission-critical” application systems that were to benefit from the upgrade included the technology that processes Federal Housing Administration single-family mortgage insurance applications, known as Computerized Homes Underwriting Management System, or CHUMS; the system used to track and report on the mortgages HUD insures and the accompanying software that tracks delinquent FHA-insured mortgages; as well as the quality control system lenders use to obtain automated responses to prescreening of federally guaranteed loan borrowers against a database of delinquent borrowers, called the Credit Alert Interactive Voice Response, or CAIVRS.
Other systems that would have been impacted by the rehost initiative included technology related to HUD grant, loan and subsidy programs, as well as two outdated and unused accounting systems.
The upgrade was called a “lift and shift,” because the operation was supposed to retain the same programming language, while migrating the applications onto the new hardware platform, changing as little as possible.
“For instance, screens are not redesigned, and program business logic remains the same,” the OIG report said, citing the chief information officer’s proposal. “Moving the applications from Unisys to Sun Solaris should have very little impact on the rest of the applications.”
Little is known about the remaining contents of HUD’s initial rehost proposal. While cited in multiple OIG reports, the document itself is not publicly available and a HUD spokesperson declined Mortgage Technology’s request for a copy, citing the report as a proprietary document.
According to the audit, the rehost project was planned to begin in June 2008 and be completed in October 2009. But the audit found that during the planning stages of the initiative, the CIO and CPO offices did not ensure HUD’s technology policies and procedures were followed, nor did they assess whether the contractor hired to do the work had previous experience in, or was capable of successfully performing, the rehost operation.
“An OCPO official stated that they relied on the IT support contractor’s history and experience with HUD over the years in deciding to accept their proposal,” the audit said.
HUD’s System Development Methodology is a series of overarching principles that guides the department's IT policies. While HUD did not follow the SDM guidelines, it did consult with one of two third-party contractors it was using for other IT work to gauge the feasibility of the “lift and shift” operation, the audit report said.
The contractor confirmed that the proposal was feasible and estimated it would yield $10 million per year in cost savings associated over the continued use of the legacy system. The contractor estimated the migration would cost $17 million and proposed the work be included as an add-on to its existing contract—one of two deals good for up to 10 years and $400 million awarded to Electronic Data Systems and Lockheed Martin in January 2005. The audit does not name which of the two contractors received the add-on to complete the rehost work, and HUD declined to comment.
At this point in the procurement process, HUD’s SDM guidelines call for a thorough evaluation of the contractor’s technical and price proposals. But, according to the auditor’s findings, “a formal, detailed cost analysis was not conducted before the start of the project.” And the audit takes issue with not only missing information in a two-page cost analysis, but also the perspective in which the two-page document was written.
“The two-page rehost cost analysis that was prepared focused on cost and the amount of funding HUD had available rather than the best possible solution,” the audit said. “Further, while the IT support contractor stated in its technical proposal that it was feasible to rehost HUD’s Unisys application systems, no corresponding formal feasibility study was completed.”
According to the audit, the lack of planning doomed the project before it began. It cites HUD’s own project termination order as evidence of the contractor’s inability to perform the rehost migration.
“HUD pointed out that the contractor was unable to demonstrate the technical viability of its rehosting methodology, and gave no indication that it could successfully complete the rehost project,” the audit said. “Schedule slippage, missed milestones, lack of domain knowledge, and a lack of deliverables substantiated the contractor’s inability to perform.”
“If better planning had been performed prior to the initiation of the Unisys rehost project, the time and resources expended on the project could have been avoided,” it continues.
Both HUD’s CIO and CPO offices were allowed to review and comment on the audit before it was publicly released. Their comments, included in the published report, defend their respective roles in the rehost project.
Williams, HUD’s CIO, claims the project did not fall within the scope of the SDM guidelines.
System development efforts have characteristics based on size, type and environment, Williams wrote. “The Unisys Rehost effort involved moving existing systems or applications from one platform to another to ensure the systems or applications remained on a supported platform…Consequently, OCIO maintains that the SDM did not apply to the project.”
But the OIG took exception to that argument because Williams’ “size, type and environment” position is not defined in any official HUD document.
“However, the SDM Section 1, ‘Introduction’ states that ‘SDM presents a methodology applicable to the development and maintenance of all information technology systems,’” the OIG responded.
When it became apparent that the project had failed, Williams said HUD was proactive in terminating the add-on contract and recovered nearly $3.7 million already spent on the failed endeavor, which he said was “recovery of all costs.”
Bryon, the procurement officer, defended the decision to use the contract add-on to assign the rehost project because it was allowed under the terms of the contract. Bryon added that since HUD performed a due diligence evaluation of the contractor for the original set of work, the department was not required to ensure the company had the ability and experience to undertake the rehost project.
