jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Adds audio querying to MultimodalQ&A Example (#1225)

Browse Source 
Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
This commit is contained in:
Melanie Hart Buehler
2024-12-12 00:05:14 -08:00
committed by GitHub
parent a50e4e6f9f
commit c760cac2f4
13 changed files with 389 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
MultimodalQnA/docker_compose/intel/cpu/xeon/README.md
View File

@@ -78,6 +78,9 @@ export https_proxy=${your_http_proxy}
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${host_ip}:$EMBEDDER_PORT/v1/encode"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export REDIS_URL="redis://${host_ip}:6379"
export REDIS_HOST=${host_ip}
export INDEX_NAME="mm-rag-redis"
@@ -144,7 +147,21 @@ docker build --no-cache -t opea/lvm-llava-svc:latest --build-arg https_proxy=$ht
docker build --no-cache -t opea/dataprep-multimodal-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/multimodal/redis/langchain/Dockerfile .
```
### 5. Build MegaService Docker Image
### 5. Build asr images
Build whisper server image
```bash
docker build --no-cache -t opea/whisper:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/asr/whisper/dependency/Dockerfile .
```
Build asr image
```bash
docker build --no-cache -t opea/asr:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/asr/whisper/Dockerfile .
```
### 6. Build MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the [multimodalqna.py](../../../../multimodalqna.py) Python script. Build MegaService Docker image via below command:
@@ -155,7 +172,7 @@ docker build --no-cache -t opea/multimodalqna:latest --build-arg https_proxy=$ht
cd ../..
```
### 6. Build UI Docker Image
### 7. Build UI Docker Image
Build frontend Docker image via below command:
@@ -165,16 +182,19 @@ docker build --no-cache -t opea/multimodalqna-ui:latest --build-arg https_proxy=
cd ../../../
```
Then run the command `docker images`, you will have the following 8 Docker Images:
Then run the command `docker images`, you will have the following 11 Docker Images:
1. `opea/dataprep-multimodal-redis:latest`
2. `opea/lvm-llava-svc:latest`
3. `opea/lvm-llava:latest`
4. `opea/retriever-multimodal-redis:latest`
5. `opea/embedding-multimodal:latest`
6. `opea/embedding-multimodal-bridgetower:latest`
7. `opea/multimodalqna:latest`
8. `opea/multimodalqna-ui:latest`
5. `opea/whisper:latest`
6. `opea/asr:latest`
7. `opea/redis-vector-db`
8. `opea/embedding-multimodal:latest`
9. `opea/embedding-multimodal-bridgetower:latest`
10. `opea/multimodalqna:latest`
11. `opea/multimodalqna-ui:latest`
## 🚀 Start Microservices
@@ -240,7 +260,16 @@ curl http://${host_ip}:7000/v1/multimodal_retrieval \
-d "{\"text\":\"test\",\"embedding\":${your_embedding}}"
```
4. lvm-llava
4. asr
```bash
curl ${ASR_SERVICE_ENDPOINT} \
-X POST \
-H "Content-Type: application/json" \
-d '{"byte_str" : "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}'
```
5. lvm-llava
```bash
curl http://${host_ip}:${LLAVA_SERVER_PORT}/generate \
@@ -249,7 +278,7 @@ curl http://${host_ip}:${LLAVA_SERVER_PORT}/generate \
-d '{"prompt":"Describe the image please.", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
```
5. lvm-llava-svc
6. lvm-llava-svc
```bash
curl http://${host_ip}:9399/v1/lvm \
@@ -274,7 +303,7 @@ curl http://${host_ip}:9399/v1/lvm \
-d '{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
```
6. dataprep-multimodal-redis
7. dataprep-multimodal-redis
Download a sample video, image, and audio file and create a caption
@@ -348,7 +377,7 @@ curl -X POST \
${DATAPREP_DELETE_FILE_ENDPOINT}
```
7. MegaService
8. MegaService
```bash
curl http://${host_ip}:8888/v1/multimodalqna \
@@ -357,6 +386,12 @@ curl http://${host_ip}:8888/v1/multimodalqna \
-d '{"messages": "What is the revenue of Nike in 2023?"}'
```
```bash
curl http://${host_ip}:8888/v1/multimodalqna \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
```
```bash
curl http://${host_ip}:8888/v1/multimodalqna \
-H "Content-Type: application/json" \

24
MultimodalQnA/docker_compose/intel/cpu/xeon/compose.yaml
View File

@@ -2,6 +2,27 @@
# SPDX-License-Identifier: Apache-2.0
services:
whisper-service:
image: ${REGISTRY:-opea}/whisper:${TAG:-latest}
container_name: whisper-service
ports:
- "7066:7066"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
restart: unless-stopped
asr:
image: ${REGISTRY:-opea}/asr:${TAG:-latest}
container_name: asr-service
ports:
- "${ASR_SERVICE_PORT}:9099"
ipc: host
environment:
ASR_ENDPOINT: ${ASR_ENDPOINT}
ASR_SERVICE_PORT: ${ASR_SERVICE_PORT}
ASR_SERVICE_ENDPOINT: ${ASR_SERVICE_ENDPOINT}
redis-vector-db:
image: redis/redis-stack:7.2.0-v9
container_name: redis-vector-db
@@ -102,6 +123,7 @@ services:
- embedding-multimodal
- retriever-multimodal-redis
- lvm-llava-svc
- asr
ports:
- "8888:8888"
environment:
@@ -113,6 +135,8 @@ services:
MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE}
MM_RETRIEVER_SERVICE_HOST_IP: ${MM_RETRIEVER_SERVICE_HOST_IP}
LVM_SERVICE_HOST_IP: ${LVM_SERVICE_HOST_IP}
ASR_SERVICE_PORT: ${ASR_SERVICE_PORT}
ASR_SERVICE_ENDPOINT: ${ASR_SERVICE_ENDPOINT}
ipc: host
restart: always
multimodalqna-ui:

3
MultimodalQnA/docker_compose/intel/cpu/xeon/set_env.sh
View File

@@ -12,6 +12,9 @@ export https_proxy=${your_http_proxy}
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${host_ip}:$EMBEDDER_PORT/v1/encode"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export REDIS_URL="redis://${host_ip}:6379"
export REDIS_HOST=${host_ip}
export INDEX_NAME="mm-rag-redis"

49
MultimodalQnA/docker_compose/intel/hpu/gaudi/README.md
View File

@@ -37,6 +37,9 @@ export LVM_MODEL_ID="llava-hf/llava-v1.6-vicuna-13b-hf"
export WHISPER_MODEL="base"
export MM_EMBEDDING_SERVICE_HOST_IP=${host_ip}
export MM_RETRIEVER_SERVICE_HOST_IP=${host_ip}
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export LVM_SERVICE_HOST_IP=${host_ip}
export MEGA_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/multimodalqna"
@@ -95,7 +98,21 @@ docker build --no-cache -t opea/lvm-tgi:latest --build-arg https_proxy=$https_pr
docker build --no-cache -t opea/dataprep-multimodal-redis:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/multimodal/redis/langchain/Dockerfile .
```
### 5. Build MegaService Docker Image
### 5. Build asr images
Build whisper server image
```bash
docker build --no-cache -t opea/whisper:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/asr/whisper/dependency/Dockerfile .
```
Build asr image
```bash
docker build --no-cache -t opea/asr:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/asr/whisper/Dockerfile .
```
### 6. Build MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the [multimodalqna.py](../../../../multimodalqna.py) Python script. Build MegaService Docker image via below command:
@@ -114,16 +131,19 @@ cd GenAIExamples/MultimodalQnA/ui/
docker build --no-cache -t opea/multimodalqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
Then run the command `docker images`, you will have the following 8 Docker Images:
Then run the command `docker images`, you will have the following 11 Docker Images:
1. `opea/dataprep-multimodal-redis:latest`
2. `opea/lvm-tgi:latest`
3. `ghcr.io/huggingface/tgi-gaudi:2.0.6`
4. `opea/retriever-multimodal-redis:latest`
5. `opea/embedding-multimodal:latest`
6. `opea/embedding-multimodal-bridgetower:latest`
7. `opea/multimodalqna:latest`
8. `opea/multimodalqna-ui:latest`
5. `opea/whisper:latest`
6. `opea/asr:latest`
7. `opea/redis-vector-db`
8. `opea/embedding-multimodal:latest`
9. `opea/embedding-multimodal-bridgetower:latest`
10. `opea/multimodalqna:latest`
11. `opea/multimodalqna-ui:latest`
## 🚀 Start Microservices
@@ -189,7 +209,16 @@ curl http://${host_ip}:7000/v1/multimodal_retrieval \
-d "{\"text\":\"test\",\"embedding\":${your_embedding}}"
```
4. TGI LLaVA Gaudi Server
4. asr
```bash
curl ${ASR_SERVICE_ENDPOINT} \
-X POST \
-H "Content-Type: application/json" \
-d '{"byte_str" : "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}'
```
5. TGI LLaVA Gaudi Server
```bash
curl http://${host_ip}:${LLAVA_SERVER_PORT}/generate \
@@ -198,7 +227,7 @@ curl http://${host_ip}:${LLAVA_SERVER_PORT}/generate \
-H 'Content-Type: application/json'
```
5. lvm-tgi
6. lvm-tgi
```bash
curl http://${host_ip}:9399/v1/lvm \
@@ -223,7 +252,7 @@ curl http://${host_ip}:9399/v1/lvm \
-d '{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
```
6. Multimodal Dataprep Microservice
7. Multimodal Dataprep Microservice
Download a sample video, image, and audio file and create a caption
@@ -297,7 +326,7 @@ curl -X POST \
${DATAPREP_DELETE_FILE_ENDPOINT}
```
7. MegaService
8. MegaService
```bash
curl http://${host_ip}:8888/v1/multimodalqna \

24
MultimodalQnA/docker_compose/intel/hpu/gaudi/compose.yaml
View File

@@ -8,6 +8,27 @@ services:
ports:
- "6379:6379"
- "8001:8001"
whisper-service:
image: ${REGISTRY:-opea}/whisper:${TAG:-latest}
container_name: whisper-service
ports:
- "7066:7066"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
restart: unless-stopped
asr:
image: ${REGISTRY:-opea}/asr:${TAG:-latest}
container_name: asr-service
ports:
- "${ASR_SERVICE_PORT}:9099"
ipc: host
environment:
ASR_ENDPOINT: ${ASR_ENDPOINT}
ASR_SERVICE_PORT: ${ASR_SERVICE_PORT}
ASR_SERVICE_ENDPOINT: ${ASR_SERVICE_ENDPOINT}
dataprep-multimodal-redis:
image: ${REGISTRY:-opea}/dataprep-multimodal-redis:${TAG:-latest}
container_name: dataprep-multimodal-redis
@@ -119,6 +140,7 @@ services:
- embedding-multimodal
- retriever-multimodal-redis
- lvm-tgi
- asr
ports:
- "8888:8888"
environment:
@@ -130,6 +152,8 @@ services:
MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE}
MM_RETRIEVER_SERVICE_HOST_IP: ${MM_RETRIEVER_SERVICE_HOST_IP}
LVM_SERVICE_HOST_IP: ${LVM_SERVICE_HOST_IP}
ASR_SERVICE_PORT: ${ASR_SERVICE_PORT}
ASR_SERVICE_ENDPOINT: ${ASR_SERVICE_ENDPOINT}
ipc: host
restart: always
multimodalqna-ui:

3
MultimodalQnA/docker_compose/intel/hpu/gaudi/set_env.sh
View File

@@ -12,6 +12,9 @@ export https_proxy=${your_http_proxy}
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${host_ip}:$EMBEDDER_PORT/v1/encode"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export REDIS_URL="redis://${host_ip}:6379"
export REDIS_HOST=${host_ip}
export INDEX_NAME="mm-rag-redis"

12
MultimodalQnA/docker_image_build/build.yaml
View File

@@ -59,3 +59,15 @@ services:
dockerfile: comps/dataprep/multimodal/redis/langchain/Dockerfile
extends: multimodalqna
image: ${REGISTRY:-opea}/dataprep-multimodal-redis:${TAG:-latest}
whisper:
build:
context: GenAIComps
dockerfile: comps/asr/whisper/dependency/Dockerfile
extends: multimodalqna
image: ${REGISTRY:-opea}/whisper:${TAG:-latest}
asr:
build:
context: GenAIComps
dockerfile: comps/asr/whisper/Dockerfile
extends: multimodalqna
image: ${REGISTRY:-opea}/asr:${TAG:-latest}

147
MultimodalQnA/multimodalqna.py
View File

@@ -2,6 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
import base64
import json
import os
from io import BytesIO
@@ -16,7 +17,7 @@ from comps.cores.proto.api_protocol import (
)
from comps.cores.proto.docarray import LLMParams
from fastapi import Request
from fastapi.responses import StreamingResponse
from fastapi.responses import JSONResponse, StreamingResponse
from PIL import Image
MEGA_SERVICE_PORT = int(os.getenv("MEGA_SERVICE_PORT", 8888))
@@ -29,6 +30,9 @@ LVM_SERVICE_PORT = int(os.getenv("LVM_SERVICE_PORT", 9399))
class MultimodalQnAService(Gateway):
asr_port = int(os.getenv("ASR_SERVICE_PORT", 3001))
asr_endpoint = os.getenv("ASR_SERVICE_ENDPOINT", "http://0.0.0.0:{}/v1/audio/transcriptions".format(asr_port))
def __init__(self, host="0.0.0.0", port=8000):
self.host = host
self.port = port
@@ -73,7 +77,10 @@ class MultimodalQnAService(Gateway):
# this overrides _handle_message method of Gateway
def _handle_message(self, messages):
images = []
audios = []
b64_types = {}
messages_dicts = []
decoded_audio_input = ""
if isinstance(messages, str):
prompt = messages
else:
@@ -87,16 +94,26 @@ class MultimodalQnAService(Gateway):
system_prompt = message["content"]
elif msg_role == "user":
if type(message["content"]) == list:
# separate each media type and store accordingly
text = ""
text_list = [item["text"] for item in message["content"] if item["type"] == "text"]
text += "\n".join(text_list)
image_list = [
item["image_url"]["url"] for item in message["content"] if item["type"] == "image_url"
]
if image_list:
messages_dict[msg_role] = (text, image_list)
else:
audios = [item["audio"] for item in message["content"] if item["type"] == "audio"]
if audios:
# translate audio to text. From this point forward, audio is treated like text
decoded_audio_input = self.convert_audio_to_text(audios)
b64_types["audio"] = decoded_audio_input
if text and not audios and not image_list:
messages_dict[msg_role] = text
elif audios and not text and not image_list:
messages_dict[msg_role] = decoded_audio_input
else:
messages_dict[msg_role] = (text, decoded_audio_input, image_list)
else:
messages_dict[msg_role] = message["content"]
messages_dicts.append(messages_dict)
@@ -108,55 +125,84 @@ class MultimodalQnAService(Gateway):
if system_prompt:
prompt = system_prompt + "\n"
for messages_dict in messages_dicts:
for i, (role, message) in enumerate(messages_dict.items()):
for i, messages_dict in enumerate(messages_dicts):
for role, message in messages_dict.items():
if isinstance(message, tuple):
text, image_list = message
text, decoded_audio_input, image_list = message
if i == 0:
# do not add role for the very first message.
# this will be added by llava_server
if text:
prompt += text + "\n"
elif decoded_audio_input:
prompt += decoded_audio_input + "\n"
else:
if text:
prompt += role.upper() + ": " + text + "\n"
elif decoded_audio_input:
prompt += role.upper() + ": " + decoded_audio_input + "\n"
else:
prompt += role.upper() + ":"
for img in image_list:
# URL
if img.startswith("http://") or img.startswith("https://"):
response = requests.get(img)
image = Image.open(BytesIO(response.content)).convert("RGBA")
image_bytes = BytesIO()
image.save(image_bytes, format="PNG")
img_b64_str = base64.b64encode(image_bytes.getvalue()).decode()
# Local Path
elif os.path.exists(img):
image = Image.open(img).convert("RGBA")
image_bytes = BytesIO()
image.save(image_bytes, format="PNG")
img_b64_str = base64.b64encode(image_bytes.getvalue()).decode()
# Bytes
else:
img_b64_str = img
images.append(img_b64_str)
else:
if image_list:
for img in image_list:
# URL
if img.startswith("http://") or img.startswith("https://"):
response = requests.get(img)
image = Image.open(BytesIO(response.content)).convert("RGBA")
image_bytes = BytesIO()
image.save(image_bytes, format="PNG")
img_b64_str = base64.b64encode(image_bytes.getvalue()).decode()
# Local Path
elif os.path.exists(img):
image = Image.open(img).convert("RGBA")
image_bytes = BytesIO()
image.save(image_bytes, format="PNG")
img_b64_str = base64.b64encode(image_bytes.getvalue()).decode()
# Bytes
else:
img_b64_str = img
images.append(img_b64_str)
elif isinstance(message, str):
if i == 0:
# do not add role for the very first message.
# this will be added by llava_server
if message:
prompt += role.upper() + ": " + message + "\n"
prompt += message + "\n"
else:
if message:
prompt += role.upper() + ": " + message + "\n"
else:
prompt += role.upper() + ":"
if images:
return prompt, images
b64_types["image"] = images
# If the query has multiple media types, return all types
if prompt and b64_types:
return prompt, b64_types
else:
return prompt
def convert_audio_to_text(self, audio):
# translate audio to text by passing in base64 encoded audio to ASR
if isinstance(audio, dict):
input_dict = {"byte_str": audio["audio"][0]}
else:
input_dict = {"byte_str": audio[0]}
response = requests.post(self.asr_endpoint, data=json.dumps(input_dict))
if response.status_code != 200:
return JSONResponse(
status_code=503, content={"message": "Unable to convert audio to text. {}".format(response.text)}
)
response = response.json()
return response["query"]
async def handle_request(self, request: Request):
data = await request.json()
stream_opt = bool(data.get("stream", False))
@@ -165,16 +211,35 @@ class MultimodalQnAService(Gateway):
stream_opt = False
chat_request = ChatCompletionRequest.model_validate(data)
# Multimodal RAG QnA With Videos has not yet accepts image as input during QnA.
prompt_and_image = self._handle_message(chat_request.messages)
if isinstance(prompt_and_image, tuple):
# print(f"This request include image, thus it is a follow-up query. Using lvm megaservice")
prompt, images = prompt_and_image
num_messages = len(data["messages"]) if isinstance(data["messages"], list) else 1
messages = self._handle_message(chat_request.messages)
decoded_audio_input = ""
if num_messages > 1:
# This is a follow up query, go to LVM
cur_megaservice = self.lvm_megaservice
initial_inputs = {"prompt": prompt, "image": images[0]}
if isinstance(messages, tuple):
prompt, b64_types = messages
if "audio" in b64_types:
# for metadata storage purposes
decoded_audio_input = b64_types["audio"]
if "image" in b64_types:
initial_inputs = {"prompt": prompt, "image": b64_types["image"][0]}
else:
initial_inputs = {"prompt": prompt, "image": ""}
else:
prompt = messages
initial_inputs = {"prompt": prompt, "image": ""}
else:
# print(f"This is the first query, requiring multimodal retrieval. Using multimodal rag megaservice")
prompt = prompt_and_image
# This is the first query. Ignore image input
cur_megaservice = self.megaservice
if isinstance(messages, tuple):
prompt, b64_types = messages
if "audio" in b64_types:
# for metadata storage purposes
decoded_audio_input = b64_types["audio"]
else:
prompt = messages
initial_inputs = {"text": prompt}
parameters = LLMParams(
@@ -207,18 +272,24 @@ class MultimodalQnAService(Gateway):
if "text" in result_dict[last_node].keys():
response = result_dict[last_node]["text"]
else:
# text in not response message
# text is not in response message
# something wrong, for example due to empty retrieval results
if "detail" in result_dict[last_node].keys():
response = result_dict[last_node]["detail"]
else:
response = "The server fail to generate answer to your query!"
response = "The server failed to generate an answer to your query!"
if "metadata" in result_dict[last_node].keys():
# from retrieval results
metadata = result_dict[last_node]["metadata"]
if decoded_audio_input:
metadata["audio"] = decoded_audio_input
else:
# follow-up question, no retrieval
metadata = None
if decoded_audio_input:
metadata = {"audio": decoded_audio_input}
else:
metadata = None
choices = []
usage = UsageInfo()
choices.append(

23
MultimodalQnA/tests/test_compose_on_gaudi.sh
View File

@@ -22,7 +22,7 @@ function build_docker_images() {
cd $WORKPATH/docker_image_build
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
echo "Build all the images with --no-cache, check docker_image_build.log for details..."
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding-multimodal retriever-multimodal-redis lvm-tgi dataprep-multimodal-redis"
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding-multimodal retriever-multimodal-redis lvm-tgi dataprep-multimodal-redis whisper asr"
docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log
docker pull ghcr.io/huggingface/tgi-gaudi:2.0.6
@@ -35,6 +35,9 @@ function setup_env() {
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${host_ip}:$EMBEDDER_PORT/v1/encode"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export REDIS_URL="redis://${host_ip}:6379"
export REDIS_HOST=${host_ip}
export INDEX_NAME="mm-rag-redis"
@@ -239,13 +242,29 @@ function validate_megaservice() {
"multimodalqna-backend-server" \
'{"messages": "What is the revenue of Nike in 2023?"}'
echo "Validate megaservice with first audio query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"time_of_frame_ms":' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
echo "Validate megaservice with follow-up query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": "chao, "}], "max_tokens": 10}'
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
echo "Validate megaservice with multiple text queries"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
}

23
MultimodalQnA/tests/test_compose_on_xeon.sh
View File

@@ -21,9 +21,8 @@ export caption_fn="apple.txt"
function build_docker_images() {
cd $WORKPATH/docker_image_build
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
echo "Build all the images with --no-cache, check docker_image_build.log for details..."
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding-multimodal retriever-multimodal-redis lvm-llava lvm-llava-svc dataprep-multimodal-redis"
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding-multimodal retriever-multimodal-redis lvm-llava lvm-llava-svc dataprep-multimodal-redis whisper asr"
docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log
docker images && sleep 1m
@@ -34,6 +33,9 @@ function setup_env() {
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${host_ip}:$EMBEDDER_PORT/v1/encode"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export ASR_ENDPOINT=http://$host_ip:7066
export ASR_SERVICE_PORT=3001
export ASR_SERVICE_ENDPOINT="http://${host_ip}:${ASR_SERVICE_PORT}/v1/audio/transcriptions"
export REDIS_URL="redis://${host_ip}:6379"
export REDIS_HOST=${host_ip}
export INDEX_NAME="mm-rag-redis"
@@ -238,14 +240,29 @@ function validate_megaservice() {
"multimodalqna-backend-server" \
'{"messages": "What is the revenue of Nike in 2023?"}'
echo "Validate megaservice with first audio query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"time_of_frame_ms":' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
echo "Validate megaservice with follow-up query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": "chao, "}], "max_tokens": 10}'
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
echo "Validate megaservice with multiple text queries"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
}
function validate_delete {

48
MultimodalQnA/ui/gradio/conversation.py
View File

@@ -5,7 +5,7 @@ import dataclasses
from enum import Enum, auto
from typing import List
from utils import get_b64_frame_from_timestamp
from utils import convert_audio_to_base64, get_b64_frame_from_timestamp
class SeparatorStyle(Enum):
@@ -31,6 +31,7 @@ class Conversation:
skip_next: bool = False
split_video: str = None
image: str = None
audio_query_file: str = None
def _template_caption(self):
out = ""
@@ -41,31 +42,32 @@ class Conversation:
def get_prompt(self):
messages = self.messages
if len(messages) > 1 and messages[1][1] is None:
# Need to do RAG. prompt is the query only
ret = messages[0][1]
# Need to do RAG. If the query is text, prompt is the query only
if self.audio_query_file:
ret = [{"role": "user", "content": [{"type": "audio", "audio": self.get_b64_audio_query()}]}]
else:
ret = messages[0][1]
else:
# No need to do RAG. Thus, prompt of chatcompletion format
conv_dict = []
if self.sep_style == SeparatorStyle.SINGLE:
for i, (role, message) in enumerate(messages):
if message:
if i != 0:
dic = {"role": role, "content": message}
dic = {"role": role}
if self.audio_query_file:
content = [{"type": "audio", "audio": self.get_b64_audio_query()}]
else:
dic = {"role": role}
if self.time_of_frame_ms and self.video_file:
content = [{"type": "text", "text": message}]
if self.base64_frame:
base64_frame = self.base64_frame
else:
base64_frame = get_b64_frame_from_timestamp(self.video_file, self.time_of_frame_ms)
self.base64_frame = base64_frame
if base64_frame is None:
base64_frame = ""
content.append({"type": "image_url", "image_url": {"url": base64_frame}})
else:
content = message
dic["content"] = content
content = [{"type": "text", "text": message}]
if i == 0 and self.time_of_frame_ms and self.video_file:
base64_frame = (
self.base64_frame
if self.base64_frame
else get_b64_frame_from_timestamp(self.video_file, self.time_of_frame_ms)
)
if base64_frame is None:
base64_frame = ""
content.append({"type": "image_url", "image_url": {"url": base64_frame}})
dic["content"] = content
conv_dict.append(dic)
else:
raise ValueError(f"Invalid style: {self.sep_style}")
@@ -83,6 +85,12 @@ class Conversation:
b64_img = get_b64_frame_from_timestamp(video_file, time_of_frame_ms)
return b64_img
def get_b64_audio_query(self):
b64_audio = None
if self.audio_query_file:
b64_audio = convert_audio_to_base64(self.audio_query_file)
return b64_audio
def to_gradio_chatbot(self):
ret = []
for i, (role, msg) in enumerate(self.messages[self.offset :]):
@@ -141,6 +149,7 @@ class Conversation:
"base64_frame": self.base64_frame,
"split_video": self.split_video,
"image": self.image,
"audio_query_file": self.audio_query_file,
}
@@ -157,4 +166,5 @@ multimodalqna_conv = Conversation(
base64_frame=None,
split_video=None,
image=None,
audio_query_file=None,
)

80
MultimodalQnA/ui/gradio/multimodalqna_ui_gradio.py
View File

@@ -16,6 +16,7 @@ from fastapi.staticfiles import StaticFiles
from utils import build_logger, make_temp_image, moderation_msg, server_error_msg, split_video
logger = build_logger("gradio_web_server", "gradio_web_server.log")
logflag = os.getenv("LOGFLAG", False)
headers = {"Content-Type": "application/json"}
@@ -50,21 +51,28 @@ def clear_history(state, request: gr.Request):
if state.image and os.path.exists(state.image):
os.remove(state.image)
state = multimodalqna_conv.copy()
return (state, state.to_gradio_chatbot(), None, None, None) + (disable_btn,) * 1
return (state, state.to_gradio_chatbot(), None, None, None, None) + (disable_btn,) * 1
def add_text(state, text, request: gr.Request):
def add_text(state, text, audio, request: gr.Request):
logger.info(f"add_text. ip: {request.client.host}. len: {len(text)}")
if len(text) <= 0:
if audio:
state.audio_query_file = audio
state.append_message(state.roles[0], "--input placeholder--")
state.append_message(state.roles[1], None)
state.skip_next = False
return (state, state.to_gradio_chatbot(), None, None) + (disable_btn,) * 1
elif len(text) <= 0:
state.skip_next = True
return (state, state.to_gradio_chatbot(), None) + (no_change_btn,) * 1
return (state, state.to_gradio_chatbot(), None, None) + (no_change_btn,) * 1
text = text[:2000] # Hard cut-off
state.append_message(state.roles[0], text)
state.append_message(state.roles[1], None)
state.skip_next = False
return (state, state.to_gradio_chatbot(), None) + (disable_btn,) * 1
return (state, state.to_gradio_chatbot(), None, None) + (disable_btn,) * 1
def http_bot(state, request: gr.Request):
@@ -72,6 +80,7 @@ def http_bot(state, request: gr.Request):
logger.info(f"http_bot. ip: {request.client.host}")
url = gateway_addr
is_very_first_query = False
is_audio_query = state.audio_query_file is not None
if state.skip_next:
# This generate call is skipped due to invalid inputs
path_to_sub_videos = state.get_path_to_subvideos()
@@ -84,13 +93,13 @@ def http_bot(state, request: gr.Request):
new_state = multimodalqna_conv.copy()
new_state.append_message(new_state.roles[0], state.messages[-2][1])
new_state.append_message(new_state.roles[1], None)
new_state.audio_query_file = state.audio_query_file
state = new_state
# Construct prompt
prompt = state.get_prompt()
# Make requests
pload = {
"messages": prompt,
}
@@ -99,6 +108,7 @@ def http_bot(state, request: gr.Request):
logger.info(f"==== url request ====\n{gateway_addr}")
state.messages[-1][-1] = "▌"
yield (state, state.to_gradio_chatbot(), state.split_video, state.image) + (disable_btn,) * 1
try:
@@ -108,8 +118,9 @@ def http_bot(state, request: gr.Request):
json=pload,
timeout=100,
)
print(response.status_code)
print(response.json())
logger.info(response.status_code)
if logflag:
logger.info(response.json())
if response.status_code == 200:
response = response.json()
@@ -152,6 +163,11 @@ def http_bot(state, request: gr.Request):
return
state.messages[-1][-1] = message
if is_audio_query:
state.messages[-2][-1] = metadata.get("audio", "--transcribed audio not available--")
state.audio_query_file = None
yield (
state,
state.to_gradio_chatbot(),
@@ -188,10 +204,11 @@ def ingest_gen_transcript(filepath, filetype, request: gr.Request):
"files": open(dest, "rb"),
}
response = requests.post(dataprep_gen_transcript_addr, headers=headers, files=files)
print(response.status_code)
logger.info(response.status_code)
if response.status_code == 200:
response = response.json()
print(response)
if logflag:
logger.info(response)
yield (gr.Textbox(visible=True, value=f"The {filetype} ingestion is done. Saving your uploaded {filetype}..."))
time.sleep(2)
fn_no_ext = Path(dest).stem
@@ -242,10 +259,11 @@ def ingest_gen_caption(filepath, filetype, request: gr.Request):
"files": open(dest, "rb"),
}
response = requests.post(dataprep_gen_caption_addr, headers=headers, files=files)
print(response.status_code)
logger.info(response.status_code)
if response.status_code == 200:
response = response.json()
print(response)
if logflag:
logger.info(response)
yield (gr.Textbox(visible=True, value=f"The {filetype} ingestion is done. Saving your uploaded {filetype}..."))
time.sleep(2)
fn_no_ext = Path(dest).stem
@@ -299,10 +317,11 @@ def ingest_with_text(filepath, text, request: gr.Request):
response = requests.post(dataprep_ingest_addr, headers=headers, files=files)
finally:
os.remove(text_dest)
print(response.status_code)
logger.info(response.status_code)
if response.status_code == 200:
response = response.json()
print(response)
if logflag:
logger.info(response)
yield (gr.Textbox(visible=True, value="Image ingestion is done. Saving your uploaded image..."))
time.sleep(2)
fn_no_ext = Path(dest).stem
@@ -436,21 +455,26 @@ with gr.Blocks() as upload_pdf:
with gr.Blocks() as qna:
state = gr.State(multimodalqna_conv.copy())
with gr.Row():
with gr.Column(scale=4):
with gr.Column(scale=2):
video = gr.Video(height=512, width=512, elem_id="video", visible=True, label="Media")
image = gr.Image(height=512, width=512, elem_id="image", visible=False, label="Media")
with gr.Column(scale=7):
with gr.Column(scale=9):
chatbot = gr.Chatbot(elem_id="chatbot", label="MultimodalQnA Chatbot", height=390)
with gr.Row():
with gr.Column(scale=6):
# textbox.render()
textbox = gr.Textbox(
# show_label=False,
# container=False,
label="Query",
info="Enter a text query below",
# submit_btn=False,
)
with gr.Column(scale=8):
with gr.Tabs():
with gr.TabItem("Text Query"):
textbox = gr.Textbox(
show_label=False,
container=True,
)
with gr.TabItem("Audio Query"):
audio = gr.Audio(
type="filepath",
sources=["microphone", "upload"],
show_label=False,
container=False,
)
with gr.Column(scale=1, min_width=100):
with gr.Row():
submit_btn = gr.Button(value="Send", variant="primary", interactive=True)
@@ -462,13 +486,13 @@ with gr.Blocks() as qna:
[
state,
],
[state, chatbot, textbox, video, image, clear_btn],
[state, chatbot, textbox, audio, video, image, clear_btn],
)
submit_btn.click(
add_text,
[state, textbox],
[state, chatbot, textbox, clear_btn],
[state, textbox, audio],
[state, chatbot, textbox, audio, clear_btn],
).then(
http_bot,
[

8
MultimodalQnA/ui/gradio/utils.py
View File

@@ -163,7 +163,7 @@ def delete_split_video(video_path):
def convert_img_to_base64(image):
"Convert image to base64 string"
"""Convert image to base64 string."""
_, buffer = cv2.imencode(".png", image)
encoded_string = base64.b64encode(buffer)
return encoded_string.decode("utf-8")
@@ -180,3 +180,9 @@ def get_b64_frame_from_timestamp(video_path, timestamp_in_ms, maintain_aspect_ra
b64_img_str = convert_img_to_base64(frame)
return b64_img_str
return None
def convert_audio_to_base64(audio_path):
"""Convert .wav file to base64 string."""
encoded_string = base64.b64encode(open(audio_path, "rb").read())
return encoded_string.decode("utf-8")