Vontive, a technology company specializing in business-purpose mortgages, has announced the release of the mortgage industry’s first Large Language Model (LLM) benchmark study. This study, utilizing Vontive’s expert-annotated dataset of 23 critical mortgage underwriting document types, establishes a new industry baseline for measuring AI accuracy and effectiveness in mortgage document analysis, aiming to foster greater trust and informed decision-making in AI initiatives.\n\nWolf Rendall, Director of Data Products at Vontive, emphasized the necessity of precision in mortgage lending AI. “AI produces compelling results that often look ‘pretty good,’ but in mortgage lending, you need to actually be right,” Rendall stated. He added that the benchmark addresses a significant industry gap, as previously there was no reliable method to differentiate between seemingly functional AI and systems that truly perform accurately on mortgage-specific tasks. Rendall noted Vontive’s substantial investment in developing a data pipeline for ingesting structured data from complex mortgage documents, recognizing the need for robust validation in a rapidly evolving AI landscape.\n\nFor the benchmark’s development, Vontive collaborated with Vals AI, providing a dataset with human annotations for various documents, including county tax certificates, insurance binders, rental leases, comprehensive title reports, LLC formation documents, operating agreements, and regulatory HUD forms. Vontive’s proprietary AI systems, designed to process these documents, demonstrated particular strength in mathematical reasoning, crucial for tasks such as calculating annualized tax amounts from diverse assessments.\n\nVals AI’s rigorous testing determined that Anthropic’s Claude 3.7 Sonnet delivered optimal performance for Vontive’s document processing functions, balancing accuracy with cost-effectiveness. The study revealed that more expensive AI models offered minimal performance gains, while smaller models showed notable accuracy decreases. The May 2024 study indicated that Anthropic’s Claude 3.7 Sonnet (Nonthinking) was the top performer at 80.6% accuracy, excelling in both semantic and numerical extraction. In contrast, Claude 4.0 (Thinking) achieved 62.5% accuracy, and Meta’s Llama 3 registered 55.3% accuracy, suggesting that model size is not the sole determinant of performance; prompt engineering and problem-specific customization are also critical. Through iterative re-evaluation with Vals AI, Vontive enhanced its AI’s extraction prompts, achieving 90% accuracy, comparable to human data entry.\n\nBy making the benchmark dataset available through their partnership, Vontive and Vals AI aim to provide a standardized measurement system for other organizations evaluating their AI implementations. This framework is intended to function similarly to established standards in other domains, such as Stanford’s question-and-answer dataset for natural language processing, providing the mortgage industry with its first rigorous framework for AI evaluation. Rayan Krishnan, CEO of Vals AI, affirmed their commitment to the initiative, stating, “We wanted to work with Vontive on this important initiative because of the company’s clear commitment to AI innovation, accuracy, and quality. Now others can benefit as we introduce a new standard for rigor in the industry.” Vontive’s internal AI systems currently achieve 95% accuracy across all supported document types.\n\nExpanding its AI capabilities, Vontive now supports over 40 document types in its AI-powered processing pipeline, encompassing most common documents required in mortgage underwriting. This expansion is designed to enhance accurate underwriting and pricing while potentially reducing loan processing times. Vontive also incorporates automatic data validation across thousands of criteria to ensure consistency throughout the loan process. Vontive’s AI Underwriter entered production workflows on July 23, 2024, and has since parsed 35,000 documents to populate datasets for 1,700 borrowers across 6,845 loans, totaling requested balances of $1.713 billion.\n\nVontive, founded by credit industry and technology veterans, operates as an embedded mortgage platform for investment real estate, providing technology to standardize private credit mortgages. The company enables banks, credit unions, property technology companies, and B2C brands serving real estate investors to establish their own investment-mortgage businesses.