|
|
--- |
|
|
datasets: |
|
|
- letxbe/BoundingDocs |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: visual-question-answering |
|
|
tags: |
|
|
- Visual-Question-Answering |
|
|
- Question-Answering |
|
|
- Document |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
|
|
|
<div align="center"> |
|
|
|
|
|
<h1>DocExplainer: Document VQA with Bounding Box Localization</h1> |
|
|
|
|
|
</div> |
|
|
|
|
|
DocExplainer is a an approach to Document Visual Question Answering (Document VQA) with bounding box localization. |
|
|
Unlike standard VLMs that only provide text-based answers, DocExplainer adds **visual evidence through bounding boxes**, making model predictions more interpretable. |
|
|
It is designed as a **plug-and-play module** to be combined with existing Vision-Language Models (VLMs), decoupling answer generation from spatial grounding. |
|
|
|
|
|
- **Authors:** Alessio Chen, Simone Giovannini, Andrea Gemelli, Fabio Coppini, Simone Marinai |
|
|
- **Affiliations:** [Letxbe AI](https://letxbe.ai/), [University of Florence](https://www.unifi.it/it) |
|
|
- **License:** apache-2.0 |
|
|
- **Paper:** ["Towards Reliable and Interpretable Document Question Answering via VLMs"](https://arxiv.org/abs/2509.10129) by Alessio Chen et al. |
|
|
|
|
|
<div align="center"> |
|
|
<img src="https://cdn.prod.website-files.com/655f447668b4ad1dd3d4b3d9/664cc272c3e176608bc14a4c_LOGO%20v0%20-%20LetXBebicolore.svg" alt="letxbe ai logo" width="200"> |
|
|
<img src="https://www.dinfo.unifi.it/upload/notizie/Logo_Dinfo_web%20(1).png" alt="Logo Unifi" width="200"> |
|
|
</div> |
|
|
|
|
|
|
|
|
## Model Details |
|
|
|
|
|
DocExplainer is a fine-tuned [SigLIP2 Giant](https://huggingface.co/google/siglip2-giant-opt-patch16-384)-based regressor that predicts bounding box coordinates for answer localization in document images. The system operates in a two-stage process: |
|
|
|
|
|
1. **Question Answering**: Any VLM is used as a black box component to generate a textual answer given in input a document image and question. |
|
|
2. **Bounding Box Explanation**: DocExplainer takes the image, question, and generated answer to predict the coordinates of the supporting evidence. |
|
|
|
|
|
|
|
|
## Model Architecture |
|
|
DocExplainer builds on [SigLIP2 Giant](https://huggingface.co/google/siglip2-giant-opt-patch16-384) visual and text embeddings. |
|
|
|
|
|
 |
|
|
|
|
|
## Training Procedure |
|
|
- Visual and textual embeddings from SigLiP2 are projected into a shared latent space, fused via fully connected layers. |
|
|
- A regression head outputs normalized coordinates `[x1, y1, x2, y2]`. |
|
|
- **Backbone**: SigLiP2 Giant (frozen). |
|
|
- **Loss Function**: Smooth L1 (Huber loss) applied to normalized coordinates in [0,1]. |
|
|
|
|
|
#### Training Setup |
|
|
- **Dataset**: [BoundingDocs v2.0](https://huggingface.co/datasets/letxbe/BoundingDocs) |
|
|
- **Epochs**: 20 |
|
|
- **Optimizer**: AdamW |
|
|
- **Hardware**: 1 × NVIDIA L40S-1-48G GPU |
|
|
- **Model Selection**: Best checkpoint chosen by highest mean IoU on the validation split. |
|
|
|
|
|
|
|
|
|
|
|
## Quick Start |
|
|
|
|
|
Here is a simple example of how to use `DocExplainer` to get an answer and its corresponding bounding box from a document image. |
|
|
|
|
|
```python |
|
|
from PIL import Image |
|
|
import requests |
|
|
import torch |
|
|
from transformers import AutoModel, AutoModelForImageTextToText, AutoProcessor |
|
|
import json |
|
|
|
|
|
url = "https://i.postimg.cc/BvftyvS3/image-1d100e9.jpg" |
|
|
image = Image.open(requests.get(url, stream=True).raw).convert("RGB") |
|
|
question = "What is the invoice number?" |
|
|
|
|
|
# ----------------------- |
|
|
# 1. Load SmolVLM2-2.2B for answer generation |
|
|
# ----------------------- |
|
|
vlm_model = AutoModelForImageTextToText.from_pretrained( |
|
|
"HuggingFaceTB/SmolVLM2-2.2B-Instruct", |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto", |
|
|
attn_implementation="flash_attention_2" |
|
|
) |
|
|
processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM2-2.2B-Instruct") |
|
|
|
|
|
PROMPT = """Based only on the document image, answer the following question: |
|
|
Question: {QUESTION} |
|
|
Provide ONLY a JSON response in the following format (no trailing commas!): |
|
|
{{ |
|
|
"content": "answer" |
|
|
}} |
|
|
""" |
|
|
|
|
|
prompt_text = PROMPT.format(QUESTION=question) |
|
|
|
|
|
messages = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "image", "image": image}, |
|
|
{"type": "text", "text": prompt_text}, |
|
|
] |
|
|
}, |
|
|
] |
|
|
|
|
|
inputs = processor.apply_chat_template( |
|
|
messages, |
|
|
add_generation_prompt=True, |
|
|
tokenize=True, |
|
|
return_dict=True, |
|
|
return_tensors="pt", |
|
|
).to(vlm_model.device, dtype=torch.bfloat16) |
|
|
|
|
|
input_length = inputs['input_ids'].shape[1] |
|
|
generated_ids = vlm_model.generate(**inputs, do_sample=False, max_new_tokens=2056) |
|
|
|
|
|
output_ids = generated_ids[:, input_length:] |
|
|
generated_texts = processor.batch_decode( |
|
|
output_ids, |
|
|
skip_special_tokens=True, |
|
|
) |
|
|
|
|
|
decoded_output = generated_texts[0].replace("Assistant:", "", 1).strip() |
|
|
answer = json.loads(decoded_output)['content'] |
|
|
|
|
|
print(f"Answer: {answer}") |
|
|
|
|
|
# ----------------------- |
|
|
# 2. Load DocExplainer for bounding box prediction |
|
|
# ----------------------- |
|
|
explainer = AutoModel.from_pretrained("letxbe/DocExplainer", trust_remote_code=True) |
|
|
bbox = explainer.predict(image, answer) |
|
|
print(f"Predicted bounding box (normalized): {bbox}") |
|
|
``` |
|
|
|
|
|
|
|
|
<table> |
|
|
<tr> |
|
|
<td width="50%" valign="top"> |
|
|
Example Output: |
|
|
|
|
|
**Question**: What is the invoice number? <br> |
|
|
**Answer**: 3Y8M2d-846<br><br> |
|
|
**Predicted BBox**: [0.6353235244750977, 0.03685223311185837, 0.8617828488349915, 0.058749228715896606] <br> |
|
|
</td> |
|
|
<td width="50%" valign="top"> |
|
|
Visualized Answer Location: |
|
|
<img src="https://i.postimg.cc/0NmBM0b1/invoice-explained.png" alt="Invoice with predicted bounding box" width="100%"> |
|
|
</td> |
|
|
</tr> |
|
|
</table> |
|
|
|
|
|
|
|
|
## Performance |
|
|
|
|
|
| Architecture | Prompting | ANLS | MeanIoU | |
|
|
|--------------------------------|------------|-------|---------| |
|
|
| Smolvlm-2.2B | Zero-shot | 0.527 | 0.011 | |
|
|
| | Anchors | 0.543 | 0.026 | |
|
|
| | CoT | 0.561 | 0.011 | |
|
|
| Qwen2-vl-7B | Zero-shot | 0.691 | 0.048 | |
|
|
| | Anchors | 0.694 | 0.051 | |
|
|
| | CoT | <ins>0.720</ins> | 0.038 | |
|
|
| Claude Sonnet 4 | Zero-shot | **0.737** | 0.031 | |
|
|
| Smolvlm-2.2B + DocExplainer | Zero-shot | 0.572 | 0.175 | |
|
|
| Qwen2-vl-7B + DocExplainer | Zero-shot | 0.689 | 0.188 | |
|
|
| Smol + Naive OCR | Zero-shot | 0.556 | <ins>0.405</ins> | |
|
|
| Qwen + Naive OCR | Zero-shot | 0.690 | **0.494** | |
|
|
|
|
|
|
|
|
Document VQA performance of different models and prompting strategies on the [BoundingDocs v2.0 dataset](https://huggingface.co/datasets/letxbe/BoundingDocs). <br> |
|
|
The best value is shown in **bold**, the second-best value is <ins>underlined</ins>. |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use `DocExplainer`, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{chen2025reliableinterpretabledocumentquestion, |
|
|
title={Towards Reliable and Interpretable Document Question Answering via VLMs}, |
|
|
author={Alessio Chen and Simone Giovannini and Andrea Gemelli and Fabio Coppini and Simone Marinai}, |
|
|
year={2025}, |
|
|
eprint={2509.10129}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL}, |
|
|
url={https://arxiv.org/abs/2509.10129}, |
|
|
} |
|
|
``` |
|
|
|
|
|
## Limitations |
|
|
- **Prototype only**: Intended as a first approach, not a production-ready solution. |
|
|
- **Dataset constraints**: Current evaluation is limited to cases where an answer fits in a single bounding box. Answers requiring reasoning over multiple regions or not fully captured by OCR cannot be properly |
|
|
|
|
|
|
|
|
|
|
|
|
