ChatQnA Accuracy
ChatQnA is a Retrieval-Augmented Generation (RAG) pipeline, which can enhance generative models through external information retrieval.
For evaluating the accuracy, we use 2 latest published datasets and 10+ metrics which are popular and comprehensive:
- Dataset
- metrics (measure accuracy of both the context retrieval and response generation)
- evaluation for retrieval/reranking
- MRR@10
- MAP@10
- Hits@10
- Hits@4
- LLM-as-a-Judge
- evaluation for the generated response from the end-to-end pipeline
- BLEU
- ROGUE(L)
- LLM-as-a-Judge
- evaluation for retrieval/reranking
Prerequisite
Environment
git clone https://github.com/opea-project/GenAIEval
cd GenAIEval
pip install -r requirements.txt
pip install -e .
MultiHop (English dataset)
MultiHop-RAG: a QA dataset to evaluate retrieval and reasoning across documents with metadata in the RAG pipelines. It contains 2556 queries, with evidence for each query distributed across 2 to 4 documents. The queries also involve document metadata, reflecting complex scenarios commonly found in real-world RAG applications.
Launch Service of RAG System
Please refer to this guide to launch the service of ChatQnA.
Launch Service of LLM-as-a-Judge
To setup a LLM model, we can use tgi-gaudi to launch a service. For example, the follow command is to setup the mistralai/Mixtral-8x7B-Instruct-v0.1 model on 2 Gaudi2 cards:
# please set your llm_port and hf_token
docker run -p {your_llm_port}:80 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e HF_TOKEN={your_hf_token} --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tgi-gaudi:2.3.1 --model-id mistralai/Mixtral-8x7B-Instruct-v0.1 --max-input-tokens 2048 --max-total-tokens 4096 --sharded true --num-shard 2
# for better performance, set `PREFILL_BATCH_BUCKET_SIZE`, `BATCH_BUCKET_SIZE`, `max-batch-total-tokens`, `max-batch-prefill-tokens`
docker run -p {your_llm_port}:80 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e HF_TOKEN={your_hf_token} -e PREFILL_BATCH_BUCKET_SIZE=1 -e BATCH_BUCKET_SIZE=8 --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tgi-gaudi:2.3.1 --model-id mistralai/Mixtral-8x7B-Instruct-v0.1 --max-input-tokens 2048 --max-total-tokens 4096 --sharded true --num-shard 2 --max-batch-total-tokens 65536 --max-batch-prefill-tokens 2048
Prepare Dataset
We use the evaluation dataset from MultiHop-RAG repo, use the below command to prepare the dataset.
git clone https://github.com/yixuantt/MultiHop-RAG.git
Evaluation
Use below command to run the evaluation, please note that for the first run, argument --ingest_docs should be added in the command to ingest the documents into the vector database, while for the subsequent run, this argument should be omitted. Set --retrieval_metrics to get retrieval related metrics (MRR@10/MAP@10/Hits@10/Hits@4). Set --ragas_metrics and --llm_endpoint to get end-to-end rag pipeline metrics (faithfulness/answer_relevancy/...), which are judged by LLMs. We set --limits is 100 as default, which means only 100 examples are evaluated by llm-as-judge as it is very time consuming.
If you are using docker compose to deploy ChatQnA system, you can simply run the evaluation as following:
python eval_multihop.py --docs_path MultiHop-RAG/dataset/corpus.json --dataset_path MultiHop-RAG/dataset/MultiHopRAG.json --ingest_docs --retrieval_metrics --ragas_metrics --llm_endpoint http://{llm_as_judge_ip}:{llm_as_judge_port}/generate
If you are using Kubernetes manifest/helm to deploy ChatQnA system, you must specify more arguments as following:
python eval_multihop.py --docs_path MultiHop-RAG/dataset/corpus.json --dataset_path MultiHop-RAG/dataset/MultiHopRAG.json --ingest_docs --retrieval_metrics --ragas_metrics --llm_endpoint http://{llm_as_judge_ip}:{llm_as_judge_port}/generate --database_endpoint http://{your_dataprep_ip}:{your_dataprep_port}/v1/dataprep/ingest --embedding_endpoint http://{your_embedding_ip}:{your_embedding_port}/v1/embeddings --tei_embedding_endpoint http://{your_tei_embedding_ip}:{your_tei_embedding_port} --retrieval_endpoint http://{your_retrieval_ip}:{your_retrieval_port}/v1/retrieval --service_url http://{your_chatqna_ip}:{your_chatqna_port}/v1/chatqna
The default values for arguments are:
| Argument | Default value |
|---|---|
| service_url | http://localhost:8888/v1/chatqna |
| database_endpoint | http://localhost:6007/v1/dataprep/ingest |
| embedding_endpoint | http://localhost:6000/v1/embeddings |
| tei_embedding_endpoint | http://localhost:8090 |
| retrieval_endpoint | http://localhost:7000/v1/retrieval |
| reranking_endpoint | http://localhost:8000/v1/reranking |
| output_dir | ./output |
| temperature | 0.1 |
| max_new_tokens | 1280 |
| chunk_size | 256 |
| chunk_overlap | 100 |
| search_type | similarity |
| retrival_k | 10 |
| fetch_k | 20 |
| lambda_mult | 0.5 |
| dataset_path | None |
| docs_path | None |
| limits | 100 |
You can check arguments details use below command:
python eval_multihop.py --help
CRUD (Chinese dataset)
CRUD-RAG is a Chinese benchmark for RAG (Retrieval-Augmented Generation) system. This example utilize CRUD-RAG for evaluating the RAG system.
Prepare Dataset
We use the evaluation dataset from CRUD-RAG repo, use the below command to prepare the dataset.
git clone https://github.com/IAAR-Shanghai/CRUD_RAG
mkdir data/
cp CRUD_RAG/data/crud_split/split_merged.json data/
cp -r CRUD_RAG/data/80000_docs/ data/
python process_crud_dataset.py
Launch Service of RAG System
Please refer to this guide to launch the service of ChatQnA system. For Chinese dataset, you should replace the English emebdding and llm model with Chinese, for example, EMBEDDING_MODEL_ID="BAAI/bge-base-zh-v1.5" and LLM_MODEL_ID=Qwen/Qwen2-7B-Instruct.
Evaluation
Use below command to run the evaluation, please note that for the first run, argument --ingest_docs should be added in the command to ingest the documents into the vector database, while for the subsequent run, this argument should be omitted.
If you are using docker compose to deploy ChatQnA system, you can simply run the evaluation as following:
python eval_crud.py --dataset_path ./data/split_merged.json --docs_path ./data/80000_docs --ingest_docs
# if you want to get ragas metrics
python eval_crud.py --dataset_path ./data/split_merged.json --docs_path ./data/80000_docs --contain_original_data --llm_endpoint "http://{llm_as_judge_ip}:{llm_as_judge_port}" --ragas_metrics
If you are using Kubernetes manifest/helm to deploy ChatQnA system, you must specify more arguments as following:
python eval_crud.py --dataset_path ./data/split_merged.json --docs_path ./data/80000_docs --ingest_docs --database_endpoint http://{your_dataprep_ip}:{your_dataprep_port}/v1/dataprep/ingest --embedding_endpoint http://{your_embedding_ip}:{your_embedding_port}/v1/embeddings --retrieval_endpoint http://{your_retrieval_ip}:{your_retrieval_port}/v1/retrieval --service_url http://{your_chatqna_ip}:{your_chatqna_port}/v1/chatqna
The default values for arguments are:
| Argument | Default value |
|---|---|
| service_url | http://localhost:8888/v1/chatqna |
| database_endpoint | http://localhost:6007/v1/dataprep/ingest |
| embedding_endpoint | http://localhost:6000/v1/embeddings |
| retrieval_endpoint | http://localhost:7000/v1/retrieval |
| reranking_endpoint | http://localhost:8000/v1/reranking |
| output_dir | ./output |
| temperature | 0.1 |
| max_new_tokens | 1280 |
| chunk_size | 256 |
| chunk_overlap | 100 |
| dataset_path | ./data/split_merged.json |
| docs_path | ./data/80000_docs |
| tasks | ["question_answering"] |
You can check arguments details use below command:
python eval_crud.py --help
Acknowledgements
This example is mostly adapted from MultiHop-RAG and CRUD-RAG repo, we thank the authors for their great work!