Consuloji: Deep Dives into Overseas AI日本語

What Do DecisionRAG's "+13.8 Points" and PageIndex's "98.7%" Really Measure? Auditing RAG Retrieval Claims Before Enterprise Deployment

Published: November 2, 2026

Listen or watch on YouTube

A forensic analysis of Cornell’s DecisionRAG arXiv preprint and VectifyAI’s PageIndex benchmarks, deconstructing comparative baselines, human scoring labels, and enterprise evaluation rules.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube


Executive Summary & Key Takeaways

  • Core Finding on DecisionRAG: DecisionRAG—a retrieval-augmented generation (RAG) framework introduced in an October 26, 2026 Cornell University preprint (arXiv:2610.35868)—claims in its abstract to outperform an 8B reranker by 13.8 points in Recall@1 on MultiDoc2Dial and 17.4 points in Hit@1 on Natural Questions. However, this comparison is strictly against a single baseline (Qwen3-Reranker-8B). When compared against the strongest baseline in the paper’s own Table 2 (Dense top-20 + GPT-5), DecisionRAG’s advantage shrinks to +2.4 points on MultiDoc2Dial and +4.6 points on Natural Questions, while falling -3.7 points behind GPT-5 on ConditionalQA.
  • Core Finding on PageIndex: PageIndex—an open-source tree-search RAG engine by VectifyAI—claims in its GitHub README to achieve “98.7% accuracy on FinanceBench.” Tracing the underlying evidence reveals that this figure originates from a February 19, 2025 blog post evaluating Mafin 2.5 on a public 150-question sample (out of FinanceBench’s 10,231 questions). A line-item tally of Vectify’s evaluation files reveals that only 136 questions achieved strict alignment with answer keys (90.67%). The 98.7% figure is reached only by including 12 human-adjudicated questions classified under benchmark errors, alternative valid methods, and different interpretations (148/150 = 98.67%).
  • Audience & Purpose: Written for enterprise IT leaders, information retrieval engineers, and business executives evaluating vendor proposals or research papers for enterprise search. This report provides an audit methodology to verify what benchmark numbers measure, who scored them, and how to test candidate RAG architectures on internal corporate repositories.

Slide 1: What Do 13.8 and 98.7% Really Measure?


Methodological Framework: Documentation, Computation, and Interpretation

To ensure objectivity, this audit strictly delineates three categories of statements:

  1. Documented Statements (Primary Sources): Claims, tables, and notes quoted directly from the Cornell preprint, GitHub repositories, evaluation scripts, and benchmark documentation.
  2. Author Calculations: Mathematical values derived directly by the author from published figures (e.g., subtracting baseline scores, calculating label percentages).
  3. Analytical Interpretation: Explicit engineering inferences regarding enterprise operational constraints and reproducibility limits.

All primary artifacts were collected and verified on November 2, 2026. Note that DecisionRAG is an unreviewed arXiv preprint (submission date October 26, 2026; PDF header dated October 27, 2026), while PageIndex claims represent vendor self-reported figures. Neither software package was executed locally; claims are audited purely against published primary documentation.

Slide 2: Two Architectures and Five Evaluation Questions


Core Terminology: Dense Retrieval, Rerankers, R@1, and Hit@1

  • Dense Retrieval: Converting search queries and text passages into dense numerical vectors via embedding models (Qwen3-Embedding-8B in the preprint) and retrieving nearest neighbors via cosine similarity.
  • Reranker: A secondary model that rescores top candidates from dense retrieval. In DecisionRAG’s paper, Qwen3-Reranker-8B rescores top-100 candidates, while GPT-5 rescores top-20.
  • Recall@1 (R@1) & Hit@1: Metrics evaluating whether the ground-truth passage ranks first. In MultiDoc2Dial, where each turn has one ground-truth passage, R@1 measures how often that passage ranks first.
  • Retrieval vs. Generation: Crucially, R@1 and Hit@1 measure retrieval ranking accuracy, not the factual correctness of the final synthesized answer.

Five Auditing Questions Before Evaluating RAG Claims

  1. What was measured? Was it retrieval first-rank accuracy (R@1/Hit@1) or final answer factual accuracy?
  2. Was the comparison against the strongest baseline? Were sample sizes and model capacities identical?
  3. Who conducted and scored the evaluation? Was scoring automated, LLM-judged, or human-adjudicated?
  4. Does the target corpus possess structural hierarchy? Does the enterprise corpus have clear table-of-contents structures, or must they be generated?
  5. What was the generation model and reasoning effort? Can runtime latency and API costs be measured predictably?

Section 1: Deconstructing DecisionRAG’s “+13.8 Points”

Slide 3: DecisionRAG’s +13.8 Points Is Against a Single Reranker

DecisionRAG replaces standard vector similarity with a non-generative decision model (Jev 1.13 from TypeSafe AI, accessed via OpenRouter API). Instead of generating text, the decision model evaluates branches of a document tree (Topic -> Document -> Section -> Paragraph) and prunes unpromising paths at each hierarchical level.

The paper evaluates DecisionRAG across four benchmarks. On MultiDoc2Dial (4,201 validation dialog turns over 413 pages and 4,110 paragraphs) and Natural Questions (500 randomly sampled questions from DAPR over 108,626 Wikipedia pages and 2.68 million paragraphs), the paper reports:

Benchmark & Metric DecisionRAG (Jev 1.13) Dense top-100 + Qwen3-Reranker-8B Author Calculated Difference
MultiDoc2Dial (R@1) 47.4 33.6 +13.8 points
Natural Questions (Hit@1) 65.4 48.0 +17.4 points

The headline claims (+13.8 and +17.4) match the exact difference between DecisionRAG and Qwen3-Reranker-8B.

The architectural core of DecisionRAG is batched branch scoring. On MultiDoc2Dial development data, the paper reports that batched scoring achieved 44.4% R@1 compared to 36.7% for independent per-page scoring (+7.7 points), as the decision model compares relative relevance across competing candidates.

However, three critical constraints must be noted:

  1. Synthetic Hierarchy Requirement: Natural Questions lacked document hierarchies. The authors clustered 108,626 pages into 1,000 k-means clusters and prompted Qwen3-8B to write single-sentence topic descriptions.
  2. Irreversible Pruning Risk: As the limitations section notes, branch pruning is permanent. If a ground-truth paragraph’s parent section is pruned, it cannot be recovered.
  3. External Model Dependency: Jev 1.13 is an external API model. The paper provides no public repository link or weights.

Section 2: Deconstructing PageIndex’s “98.7%” Accuracy Claim

Slide 4: PageIndex’s 98.7% Includes Human Adjudication

PageIndex promotes itself on GitHub as a “vectorless, reasoning-based RAG” engine that builds hierarchical tree indices and navigates them via LLM reasoning. The README highlights a chart showing PageIndex at 98.7% accuracy versus “Vector RAG” at 50% on FinanceBench.

Inspection of the underlying sources reveals critical context:

  1. Historical Source: The 98.7% claim links to a February 19, 2025 blog post describing Mafin 2.5, a financial agent built on PageIndex. It does not represent benchmark results on the current open-source PageIndex codebase.
  2. Sample Size: While FinanceBench contains 10,231 questions, Mafin 2.5 evaluated only the publicly released 150-question sample.
  3. The 50% Vector RAG Baseline: Neither the README chart nor the blog post specifies the parameters used for the 50% Vector RAG baseline.

Line-Item Audit of the 150-Question Evaluation File

An audit of the published evaluation JSON files (result_gpt4o.json and DeepSeek V3 results) across all 150 items yields:

Evaluation Label Official Definition in Mafin 2.5 README Item Count Percentage of 150 Questions
AL (Aligned) Conclusion and reasoning strictly align with benchmark answer key 136 90.67%
BE (Benchmark Error) Benchmark answer key contains errors; model answer is valid 6 4.00%
MVA (Multiple Valid Approaches) Model used an alternative valid analytical methodology 5 3.33%
SEDC (Same Evidence, Different Conclusion) Valid divergence in subjective interpretation 1 0.67%
NAL (Not Aligned) Failed to find evidence or misunderstood financial data 2 1.33%

As shown by the author’s calculation:

  • Strict ground-truth alignment: 136 / 150 = 90.67%.
  • Inclusion of BE, MVA, and SEDC: (136 + 6 + 5 + 1) / 150 = 148 / 150 = 98.67%, rounding to 98.7%.

The automated scoring script (eval.py) utilized gpt-4o-2024-11-20 at temperature 0 with prompts instructing the LLM to accept broader answers and subjective interpretations. Enterprise evaluators must recognize that 98.7% reflects human adjudication and lenient grading, not automated exact-match accuracy.


Section 3: Cross-Table Baseline Comparisons

Slide 5: Comparing Baselines Across Published Tables

Claims must be placed alongside all baselines printed in the primary publications.

DecisionRAG Table 2 Performance Across Baselines

Method (DecisionRAG Table 2) MultiDoc2Dial (R@1) ConditionalQA (Hit@1) Natural Questions (Hit@1)
DecisionRAG (Jev 1.13) 47.4 69.7 65.4
Dense top-20 + GPT-5 (Low Reasoning) 47.5* 73.4 60.8
Dense top-100 + Qwen3-Reranker-8B 33.6 64.6 48.0
Dense top-100 + BGE Reranker 22.8 53.1 54.8
Dense Retrieval Only (Qwen3-Embedding-8B) 27.9 47.2 46.2
SPLADE++ 24.4 28.4 36.0
BM25 20.6 24.7 16.4

*Note: The paper’s appendix specifies that the GPT-5 MultiDoc2Dial baseline was evaluated on a 1,000-turn subset. On that identical subset, DecisionRAG scored 49.9.

Auditing this table reveals critical nuances:

  1. GPT-5 Parity: DecisionRAG is +2.4 points above GPT-5 on MultiDoc2Dial (49.9 vs. 47.5) and +4.6 points on Natural Questions (65.4 vs. 60.8), but falls 3.7 points behind GPT-5 on ConditionalQA (69.7 vs. 73.4).
  2. Reranker Anomalies: On Natural Questions, the smaller BGE reranker scored 54.8, beating the 8B Qwen3 reranker (48.0) by 6.8 points. DecisionRAG’s lead over BGE is 10.6 points, not 17.4 points.
  3. No Direct Comparison with PageIndex: DecisionRAG cites PageIndex only in related work and references; it does not evaluate PageIndex in its benchmark experiments.

PageIndex / Mafin 2.5 FinanceBench Comparison Table

Architecture (Mafin 2.5 README) Stated Accuracy Corpus Scope Evaluated Public Audit Log
Mafin 2.5 98.7% 100% of public sample (150/150) Available on GitHub
Quantly 94% 100% of public sample (150/150) Not disclosed
Fintool 98% Partial sample (66.7%, ~100/150) Not disclosed
ChatGPT 4o + Search 31% Partial sample (66.7%, ~100/150) Not disclosed
Perplexity 45% Partial sample (66.7%, ~100/150) Not disclosed

In this table, non-Mafin rows cite secondary blog posts rather than reproducible benchmark repositories. Rows evaluating only 66.7% of the dataset cannot be compared directly against full-sample evaluations.


Section 4: Enterprise Replication, Latency, and Downstream Generation

Slide 6: Enterprise Constraints for DecisionRAG

DecisionRAG Operational Realities

  • Latency & Cost: DecisionRAG reports query latencies of 0.68 to 1.76 seconds at an API cost of 0.22 to 2.33 cents per query. Speed comparisons were uncontrolled: DecisionRAG ran via cloud API, the Qwen3 reranker ran on a single local GPU, and GPT-5 ran via commercial endpoints.
  • Downstream Answer Convergence: In Table 3, feeding top-5 retrieved passages into a downstream generator (Qwen3.5-27B) on Natural Questions yielded exact-match accuracy of 54.2% for DecisionRAG versus 53.0% for Dense Retrieval Only—a marginal difference of just 1.2 points.
  • Absence of Official Code: As of November 2, 2026, no official repository exists in the paper.

Slide 7: Enterprise Constraints for PageIndex

PageIndex Reasoning Sensitivity

In Vectify’s separate “Financial QA Benchmark” (100 questions over 40 Form 10-K filings), identical PageIndex tree retrieval was tested across different answering LLMs:

Model Configuration Reasoning Effort Setting Benchmark Accuracy
Claude Opus 5.5 Maximum Reasoning 98.7%
Claude Sonnet 5.5 Maximum Reasoning 98.0%
Claude Sonnet 5.5 High Reasoning 81.7%

Switching Sonnet 5.5 from High to Maximum reasoning produced a 16.3-point shift (81.7% to 98.0%). This demonstrates that high benchmark scores in tree-search RAG are heavily governed by the reasoning capacity and token budget of the downstream generator.


Practical Five-Step Evaluation Framework for Corporate Repositories

Slide 8: Conclusion: Five Auditing Questions and Testing Steps

Enterprise teams should execute a local evaluation protocol:

  1. Curate 30 Real Enterprise Queries: Collect approximately 30 realistic queries spanning table lookups, cross-department policy synthesis, and unanswerable questions.
  2. Pre-Label Ground-Truth Passages: Identify exact source documents and paragraphs before running evaluations, preventing post-hoc bias.
  3. Decouple Retrieval from Generation: Score Hit@1 and Recall@5 separately from final generated answer accuracy.
  4. Vary Reasoning Effort & Models: Run identical retrieved passages through multiple generation models and reasoning settings.
  5. Profile Total Lifecycle Economics: Calculate indexing cost per page, storage overhead, query latency, and token costs under concurrency.

Limits of This Audit

  • DecisionRAG findings are based on arXiv preprint v1 (October 26, 2026); subsequent revisions may adjust reported numbers.
  • PageIndex calculations rely on published evaluation logs without executing independent model inferences.
  • Neither framework is endorsed or rejected; this report evaluates documentation rigor and comparative claims.

Final Verification Checklist

  • Confirmed whether reported metrics reflect retrieval ranking (R@1/Hit@1) or final answer generation.
  • Audited whether baseline comparisons represent the strongest competing models in published tables.
  • Verified human adjudication counts and label criteria in evaluation datasets.
  • Evaluated whether corpus hierarchy is organic or synthetically generated.
  • Measured sensitivity to downstream generator reasoning capacity.
  • Verified end-to-end latency, indexing expense, and data governance boundaries.

Back to articles