.png)
Why the Heck Do I Need RAG When I’ve Got ChatGPT?
"I don’t need Google, my wife knows everything" – that’s the writing on a T-shirt I bought on vacation 14 years ago.
Recently, I was speaking with a potential partner, explaining RAG (Retrieval-Augmented Generation) and showcasing its capabilities through the FinanceBench benchmark.
(FinanceBench is a dataset and benchmarking suite designed to evaluate the performance of NLP and RAG systems on financial documents. It includes financial reports from the past few years for 40 companies. You can try a demo based on this benchmark here: https://demo.quepasa.ai/fb).
At one point, the guy asked me, "Why would I even need RAG for financial reports? Company financial reports are public information available online, and ChatGPT-4o, praise science, can search for information on the Internet and build its answers based on the search results. It should answer any questions from this benchmark, right?"
"Let’s test that," I suggested. So, we tried it out. ChatGPT answered the first random question from the benchmark correctly, but gave an incorrect answer to the second. This raised the question: can ChatGPT truly be relied upon in the financial domain?
Let's dive in.
This article doesn’t aim to be a formal study – I didn’t set out to collect statistics on correct or incorrect responses. Instead, it’s an analysis of how ChatGPT handles this type of question, along with a breakdown of the kinds of errors you might encounter. It’s important to keep in mind that search results are ever-evolving, so if you replicate my experiment, ChatGPT-4o’s answers may vary.
I chose 32 relatively random questions, and for 9 of them, ChatGPT-4o provided incorrect answers.
Here’s the table with questions and answers (incorrect answers highlighted in red and yellow).
Example 1 - 3M capital expenditure amount
Question:
What is the FY2018 capital expenditure amount (in USD millions) for 3M? Give a response to the question by relying on the details shown in the cash flow statement.
Benchmark answer: $1577.00
ChatGPT-4o + Internet answer:
In fiscal year 2018, 3M's capital expenditures amounted to $1,699 million.
ChatGPT sources: https://stockanalysis.com/stocks/mmm/financials/cash-flow-statement/
What Went Wrong?
We click the link and find that ChatGPT pulled data from the wrong column in the table (selecting 2019 instead of 2018). In its defense, the specific year we needed wasn’t actually present in the table. In other words, the search returned a somewhat irrelevant source, and the LLM used it in its response anyway. On the flip side, let’s be fair — ChatGPT was working with a clean, well-organized HTML format rather than a messy PDF, so it really could have done a better job interpreting the data.
By comparison, QuePasa's answer:
According to the cash flow statement information provided in [3M 3M_2018_10K], 3M's capital expenditure (purchases of property, plant and equipment) for fiscal year 2018 was $1,577 million.
Example 2 - 3M PPNE
Question:
Assume that you are a public equities analyst. Answer the following question by primarily using information that is shown in the balance sheet: what is the year end FY2018 net PPNE for 3M? Answer in USD billions.
Benchmark answer: $8.70
ChatGPT-4o + Internet answer:
At the end of fiscal year 2018, 3M Company's net property, plant, and equipment (PP&E) was approximately $10.2 billion.
ChatGPT sources: https://stockanalysis.com/stocks/mmm/financials/balance-sheet/
What went wrong?
We click the link and see the same error again — the wrong year was selected.
By comparison, QuePasa's answer:
Based on the information provided in the sources, I can answer this question using data from the balance sheet.
According to [3M 3M_2018_10K], which discusses 3M's 2018 financial information: "As of December 31, 2018, net property, plant and equipment totaled $8.7 billion".
Therefore, the year-end FY2018 net PP&E (Property, Plant and Equipment) for 3M was $8.7 billion.
Example 3 - 3M debt securities
Question:
Which debt securities are registered to trade on a national securities exchange under 3M's name as of Q2 of 2023?.
Benchmark answer:
Following debt securities registered under 3M's name are listed to trade on the New York Stock Exchange:
- 1.500% Notes due 2026 (Trading Symbol: MMM26)
- 1.750% Notes due 2030 (Trading Symbol: MMM30)
- 1.500% Notes due 2031 (Trading Symbol: MMM31)
ChatGPT-4o + Internet answer:
As of the second quarter of 2023, 3M Company had no debt securities registered for trading on any national securities exchange. The company's publicly traded securities were limited to its common stock, which is listed on the New York Stock Exchange under the ticker symbol "MMM." 3M's debt instruments, such as bonds and notes, were issued in the over-the-counter (OTC) market and were not listed on national securities exchanges.
ChatGPT sources:
What went wrong?
The search didn’t return the needed results, so the LLM decided that the requested debt securities simply didn’t exist.
By comparison, QuePasa's answer:
According to the information provided in [3M 3M_2023Q2_10Q], as of June 30, 2023, 3M had the following debt securities registered to trade on national securities exchanges:
1.500% Notes due 2026
1.750% Notes due 2030
1.500% Notes due 2031
These notes are registered to trade on the New York Stock Exchange [3M 3M_2023Q2_10Q].
Example 4 - AMCOR's restructuring liability
Question:
What is the nature & purpose of AMCOR's restructuring liability as oF Q2 of FY2023 close?
Benchmark answer: 87% of the total restructuring liability is related Employee liabilities.
ChatGPT-4o + Internet answer:
As of the close of the second quarter of fiscal year 2023, Amcor recognized restructuring liabilities primarily associated with its strategic initiatives aimed at enhancing operational efficiency and reducing costs. These restructuring activities encompassed several key components:
- Employee Severance and Related Benefits*: Costs incurred from workforce reductions, including severance payments and associated benefits.
- Impairment of Property and Equipment*: Write-downs of assets that were either underperforming or rendered redundant due to operational changes.
- Accelerated Depreciation*: Increased depreciation expenses for assets slated for early retirement as part of the restructuring plan.
- Other Associated Costs*: Additional expenses related to the restructuring process, such as facility closures and contract terminations.
These restructuring efforts were part of Amcor's broader strategy to optimize its manufacturing footprint and align its cost structure with evolving market conditions. The company anticipated that these actions would lead to meaningful cost savings in fiscal years 2024 and 2025.
ChatGPT sources: Amcor_FY23_-_Transcript_FINAL.pdf
What went wrong?
What’s interesting here is that we’re looking at a PDF source rather than an HTML page, as in the previous examples. The answer isn’t entirely wrong, but it’s not precise enough — it doesn’t provide the numerical value of the liabilities.
By comparison, QuePasa's answer (we can see that this answer matches the benchmark answer, as $81 million / ($81 million + $2 million + $10 million) = 87%.):
Based on the information provided in [Amcor AMCOR_2023Q2_10Q], Amcor's restructuring liability as of Q2 FY2023 (December 31, 2022) was primarily related to:
- Restructuring activities related to the Russia-Ukraine conflict.
- Employee costs, fixed asset related costs, and other costs associated with restructuring plans.
- The total restructuring liability as of December 31, 2022 was $93 million, broken down as follows:
- Employee costs: $81 million
- Fixed asset related costs: $2 million
- Other costs: $10 million
Example 5 - American Water Works cash dividends
Question:
How much (in USD billions) did American Water Works pay out in cash dividends for FY2020? Compute or extract the answer by primarily using the details outlined in the statement of cash flows.
Benchmark answer: $0.40
ChatGPT-4o + Internet answer:
In fiscal year 2020, American Water Works paid out $467 million in cash dividends. This amount is detailed in the company's cash flow statement under financing activities. To express this in billions, the payout was $0.467 billion.
ChatGPT sources: https://www.marketwatch.com/investing/stock/AWK/financials/cash-flow
What went wrong?
We click the link and see that the LLM made the same mistake again — it selected 2022 instead of 2020. However, unlike previous examples, both years are actually present in the table this time.
By comparison, QuePasa's answer (this answer, when rounded, matches the benchmark answer):
According to the information provided in [American Water Works AMERICANWATERWORKS_2021_10K], American Water Works paid out $389 million in cash dividends for fiscal year 2020.
To convert this to billions of USD, we divide by 1000:
$389 million / 1000 = $0.389 billion
Therefore, American Water Works paid out $0.389 billion in cash dividends for fiscal year 2020 [American Water Works AMERICANWATERWORKS_2021_10K].
Example 6 - Boeing net property, plant, and equipment
Question:
We need to calculate a financial metric by using information only provided within the balance sheet. Please answer the following question: what is Boeing's year end FY2018 net property, plant, and equipment (in USD millions)?
Benchmark answer: $12645.00
ChatGPT-4o + Internet answer:
At the end of fiscal year 2018, Boeing's net property, plant, and equipment (PP&E) was valued at $13,684 million.
ChatGPT sources: https://www.wsj.com/market-data/quotes/BA/financials/annual/balance-sheet
What went wrong? We click the link and see that the LLM made the same mistake again, selecting the wrong column.
By comparison, QuePasa's answer:
Based on the information provided in [Boeing BOEING_2018_10K], Boeing's net property, plant, and equipment at the end of fiscal year 2018 was $12,645 million.
Example 7 - General Mills free cash flow
Question:
According to the information provided in the statement of cash flows, what is the fiscal 2020 free cash flow (FCF) for General Mills? FCF here is defined as: (cash from operations - capex). Answer in USD millions.
Benchmark answer: $3215.00
ChatGPT-4o + Internet answer:
In fiscal year 2020, General Mills reported cash provided by operating activities of $3,680 million and capital expenditures of $461 million. Subtracting capital expenditures from operating cash flow, the free cash flow (FCF) for FY2020 is $3,219 million.
What went wrong? This is truly an interesting case. The search provided the correct resource, and the LLM selected the right data. However, when calculating the formula (cash from operations - capex), GPT decided to round cash from operations to the nearest ten million and capex to the nearest million. As a result, the answer differs from the benchmark by 4 million.
By comparison, QuePasa's answer:
According to the information provided in [General Mills GENERALMILLS_2020_10K], General Mills' free cash flow for fiscal 2020 was $3,215.4 million [General Mills GENERALMILLS_2020_10K].
This is calculated as: Net cash provided by operating activities: $3,676.2 million Minus Capital expenditures: $460.8 million Equals Free cash flow: $3,215.4 million
Example 8 - Lockheed Martin net working capital
Question:
What is Lockheed Martin's FY2021 net working capital? Define net working capital as total current assets less total current liabilities. Answer in USD millions. Respond to the question by assuming the perspective of an investment analyst who can only use the details shown within the balance sheet.
Benchmark answer: $5818.00
ChatGPT-4o + Internet answer:
As an investment analyst reviewing Lockheed Martin's balance sheet for fiscal year 2021, net working capital is calculated as total current assets minus total current liabilities. According to the balance sheet as of December 31, 2021, the figures are as follows:
- Total Current Assets*: $20,991 million
- Total Current Liabilities*: $15,887 million
Therefore, the net working capital is:
$20,991 million (Total Current Assets) - $15,887 million (Total Current Liabilities) = $5,104 million
This indicates that Lockheed Martin had a net working capital of $5,104 million at the end of fiscal year 2021.
ChatGPT sources: https://www.wsj.com/market-data/quotes/LMT/financials/annual/balance-sheet
What went wrong? We click the link and see that the LLM made the same mistake again, selecting the wrong column.
By comparison, QuePasa's answer:
As an investment analyst looking at Lockheed Martin's balance sheet for fiscal year 2021, I can calculate the net working capital as follows:
Total current assets: $19,815 million Total current liabilities: $13,997 million
Net working capital = Total current assets - Total current liabilities = $19,815 million - $13,997 million = $5,818 million
Therefore, Lockheed Martin's net working capital for fiscal year 2021 was $5,818 million [Lockheed Martin LOCKHEEDMARTIN_2021_10K].
Example 9 - Pepsico legal battles
Question:
Has Pepsico reported any materially important ongoing legal battles from FY2022 and FY2021?
Benchmark answer: No, Pepsico is not involved in material legal battles.
ChatGPT-4o + Internet answer:
In its fiscal years 2021 and 2022, PepsiCo disclosed several significant legal proceedings:
- Environmental Litigation: In November 2023, the State of New York initiated legal action against PepsiCo, alleging that the company's packaging contributed to plastic pollution in the Buffalo River. The lawsuit accused PepsiCo of creating a public nuisance and engaging in deceptive practices regarding the recyclability of its plastic packaging.
- Tax Dispute in Australia: In June 2024, the Full Federal Court of Australia ruled in favor of PepsiCo in a case concerning royalty withholding tax and diverted profits tax. The Australian Taxation Office had challenged PepsiCo's tax arrangements, but the court's decision was a significant win for the company.
- Plastic Pollution Lawsuit in Los Angeles: In October 2024, Los Angeles County filed a lawsuit against PepsiCo and Coca-Cola, accusing them of misleading consumers about the recyclability of their plastic bottles and downplaying the environmental harm caused by plastic pollution. The lawsuit seeks to address the companies' alleged deceptive business practices and demands restitution for consumers and civil penalties.
These cases highlight ongoing legal challenges PepsiCo faces related to environmental concerns and tax disputes.
ChatGPT sources:
3.https://apnews.com/article/c326225a08b2a2778afdd27d3db2d628?utm_source=chatgpt.com
What went wrong? The search sources and the LLM reference court cases from different years. Neither the search engine nor the LLM seemed to notice this discrepancy.
By comparison, QuePasa's answer:
Based on the provided sources, there is no specific mention of any materially important ongoing legal battles for PepsiCo in fiscal years 2022 and 2021. However, the company does acknowledge that it faces potential risks from litigation and legal proceedings in general.
[PepsiCo PEPSICO_2015_10K], which is from an older 10-K filing, mentions that PepsiCo and its subsidiaries are party to various legal claims and proceedings in the ordinary course of business. These include litigation related to advertising, marketing, commercial practices, product labels, claims and ingredients, intellectual property rights, environmental, employment, and insurance matters [PepsiCo PEPSICO_2015_10K].
The company evaluates legal claims to assess the likelihood of unfavorable outcomes and estimates potential losses. However, they note that litigation is inherently uncertain, and there's no guarantee that their assessment of materiality, likely outcomes, or potential losses will be consistent with the ultimate outcome of such litigation [PepsiCo PEPSICO_2015_10K].
It's important to note that this information is from an older filing and may not reflect the current situation in fiscal years 2022 and 2021. The more recent sources provided do not contain specific information about ongoing legal battles. For the most up-to-date and accurate information on any materially important ongoing legal battles, it would be best to refer to PepsiCo's latest financial reports and disclosures.
Conclusion
As seen from the examples, there is plenty of evidence that ChatGPT with internet access should not be trusted for analyzing financial documents. This is the domain of RAG, even though the data is publicly available.
