evaluation_of_RAG_retrieval_chunking_methods.md - how did you guys calculate precision?

[aidando73]

• https://github.com/superlinked/VectorHub/blob/main/docs/articles/evaluation_of_RAG_retrieval_chunking_methods.md

I'm trying to reproduce this calculation (squad dataset)

But I'm getting 0.213 instead of 0.1246

My code is here: https://github.com/aidando73/hello-retrieval-evals/blob/main/squad_1.ipynb

What am I doing differently here? I'm doing:

• Using same model
• Using ScentenceSplitter same chunk size (no chunk overlap)
• Consider the result correct if it contains the answer as a substring
• Using chromadb's default kNN algorithm
• I tried all different splits (train, validation, train + validation)

[svonava-superlinked]

@hkristof03 could you check this out?

[hkristof03]

The code is open sourced, one reason to be able to reproduce it. Sorry but I don't do random requests with different code.

[svonava-superlinked]

Ah that's right, I totally forgot the code was in a separate repo!

@aidando73 could you check https://github.com/superlinked/chunking-research and see where your code might have deviated?

[aidando73]

Sweet - thank you 🙏