jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Adding files to deploy MultimodalQnA application on ROCm vLLM (#1737)

Browse Source 
Signed-off-by: Artem Astafev <a.astafev@datamonsters.com>
This commit is contained in:
Artem Astafev
2025-04-10 08:34:58 +07:00
committed by GitHub
parent 094ca7aefe
commit 795c29fe87
8 changed files with 972 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

622
MultimodalQnA/docker_compose/amd/gpu/rocm/README.md
View File

@@ -1,4 +1,4 @@
# Build Mega Service of MultimodalQnA for AMD ROCm # Build and Deploy MultimodalQnA Application on AMD GPU (ROCm)
This document outlines the deployment process for a MultimodalQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on AMD server with ROCm GPUs. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as `multimodal_embedding` that employs [BridgeTower](https://huggingface.co/BridgeTower/bridgetower-large-itm-mlm-gaudi) model as embedding model, `multimodal_retriever`, `lvm`, and `multimodal-data-prep`. We will publish the Docker images to Docker Hub soon, it will simplify the deployment process for this service. This document outlines the deployment process for a MultimodalQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on AMD server with ROCm GPUs. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as `multimodal_embedding` that employs [BridgeTower](https://huggingface.co/BridgeTower/bridgetower-large-itm-mlm-gaudi) model as embedding model, `multimodal_retriever`, `lvm`, and `multimodal-data-prep`. We will publish the Docker images to Docker Hub soon, it will simplify the deployment process for this service.
@@ -6,95 +6,217 @@ For detailed information about these instance types, you can refer to this [link
After launching your instance, you can connect to it using SSH (for Linux instances) or Remote Desktop Protocol (RDP) (for Windows instances). From there, you'll have full access to your Xeon server, allowing you to install, configure, and manage your applications as needed. After launching your instance, you can connect to it using SSH (for Linux instances) or Remote Desktop Protocol (RDP) (for Windows instances). From there, you'll have full access to your Xeon server, allowing you to install, configure, and manage your applications as needed.
## Setup Environment Variables ## Build Docker Images
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below. ### 1. Build Docker Image
Please use `./set_env.sh` (. set_env.sh) script to set up all needed Environment Variables. - #### Create application install directory and go to it:
**Export the value of the public IP address of your server to the `host_ip` environment variable**
Note: Please replace with `host_ip` with you external IP address, do not use localhost.
## 🚀 Build Docker Images
### 1. Build embedding-multimodal-bridgetower Image
Build embedding-multimodal-bridgetower docker image
```bash ```bash
git clone https://github.com/opea-project/GenAIComps.git mkdir ~/multimodalqna-install && cd multimodalqna-install
cd GenAIComps
docker build --no-cache -t opea/embedding-multimodal-bridgetower:latest --build-arg EMBEDDER_PORT=$EMBEDDER_PORT --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/third_parties/bridgetower/src/Dockerfile .
``` ```
Build embedding microservice image - #### Clone the repository GenAIExamples (the default repository branch "main" is used here):
```bash
docker build --no-cache -t opea/embedding:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/embeddings/src/Dockerfile .
```
### 2. Build LVM Images
Build lvm-llava image
```bash
docker build --no-cache -t opea/lvm-llava:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/third_parties/whisper/src/Dockerfile .
```
### 3. Build retriever-multimodal-redis Image
```bash
docker build --no-cache -t opea/retriever:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/src/Dockerfile .
```
### 4. Build dataprep-multimodal-redis Image
```bash
docker build --no-cache -t opea/dataprep:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/src/Dockerfile .
```
### 5. Build MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the [multimodalqna.py](../../../../multimodalqna.py) Python script. Build MegaService Docker image via below command:
```bash ```bash
git clone https://github.com/opea-project/GenAIExamples.git git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/MultimodalQnA
docker build --no-cache -t opea/multimodalqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
cd ../..
``` ```
### 6. Build UI Docker Image If you need to use a specific branch/tag of the GenAIExamples repository, then (v1.3 replace with its own value):
Build frontend Docker image via below command:
```bash ```bash
cd GenAIExamples/MultimodalQnA/ui/ git clone https://github.com/opea-project/GenAIExamples.git && cd GenAIExamples && git checkout v1.3
docker build --no-cache -t opea/multimodalqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
cd ../../../
``` ```
### 7. Pull TGI AMD ROCm Image We remind you that when using a specific version of the code, you need to use the README from this version:
- #### Go to build directory:
```bash ```bash
docker pull ghcr.io/huggingface/text-generation-inference:2.4.1-rocm cd ~/multimodalqna-install/GenAIExamples/MultimodalQnA/docker_image_build
``` ```
Then run the command `docker images`, you will have the following 8 Docker Images: - Cleaning up the GenAIComps repository if it was previously cloned in this directory.
This is necessary if the build was performed earlier and the GenAIComps folder exists and is not empty:
1. `opea/dataprep:latest` ```bash
2. `ghcr.io/huggingface/text-generation-inference:2.4.1-rocm` echo Y | rm -R GenAIComps
3. `opea/lvm:latest` ```
4. `opea/retriever:latest`
5. `opea/embedding:latest`
6. `opea/embedding-multimodal-bridgetower:latest`
7. `opea/multimodalqna:latest`
8. `opea/multimodalqna-ui:latest`
## 🚀 Start Microservices - #### Clone the repository GenAIComps (the default repository branch "main" is used here):
### Required Models ```bash
git clone https://github.com/opea-project/GenAIComps.git
```
If you use a specific tag of the GenAIExamples repository,
then you should also use the corresponding tag for GenAIComps. (v1.3 replace with its own value):
```bash
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout v1.3
```
We remind you that when using a specific version of the code, you need to use the README from this version.
- #### Setting the list of images for the build (from the build file.yaml)
If you want to deploy a vLLM-based or TGI-based application, then the set of services is installed as follows:
#### vLLM-based application
```bash
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding retriever lvm dataprep whisper vllm-rocm"
```
#### TGI-based application
```bash
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding retriever lvm dataprep whisper"
```
- #### Optional. Pull TGI Docker Image (Do this if you want to use TGI)
```bash
docker pull ghcr.io/huggingface/text-generation-inference:2.3.1-rocm
```
- #### Build Docker Images
```bash
docker compose -f build.yaml build ${service_list} --no-cache
```
After the build, we check the list of images with the command:
```bash
docker image ls
```
The list of images should include:
##### vLLM-based application:
- opea/vllm-rocm:latest
- opea/lvm:latest
- opea/multimodalqna:latest
- opea/multimodalqna-ui:latest
- opea/dataprep:latest
- opea/embedding:latest
- opea/embedding-multimodal-bridgetower:latest
- opea/retriever:latest
- opea/whisper:latest
##### TGI-based application:
- ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
- opea/lvm:latest
- opea/multimodalqna:latest
- opea/multimodalqna-ui:latest
- opea/dataprep:latest
- opea/embedding:latest
- opea/embedding-multimodal-bridgetower:latest
- opea/retriever:latest
- opea/whisper:latest
---
## Deploy the MultimodalQnA Application
### Docker Compose Configuration for AMD GPUs
To enable GPU support for AMD GPUs, the following configuration is added to the Docker Compose file:
- compose_vllm.yaml - for vLLM-based application
- compose.yaml - for TGI-based
```yaml
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/:/dev/dri/
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
```
This configuration forwards all available GPUs to the container. To use a specific GPU, specify its `cardN` and `renderN` device IDs. For example:
```yaml
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/card0:/dev/dri/card0
- /dev/dri/renderD128:/dev/dri/renderD128
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
```
**How to Identify GPU Device IDs:**
Use AMD GPU driver utilities to determine the correct `cardN` and `renderN` IDs for your GPU.
### Set deploy environment variables
#### Setting variables in the operating system environment:
##### Set variable HUGGINGFACEHUB_API_TOKEN:
```bash
### Replace the string 'your_huggingfacehub_token' with your HuggingFacehub repository access token.
export HUGGINGFACEHUB_API_TOKEN='your_huggingfacehub_token'
```
#### Set variables value in set_env\*\*\*\*.sh file:
Go to Docker Compose directory:
```bash
cd ~/multimodalqna-install/GenAIExamples/MultimodalQnA/docker_compose/amd/gpu/rocm
```
The example uses the Nano text editor. You can use any convenient text editor:
#### If you use vLLM
```bash
nano set_env_vllm.sh
```
#### If you use TGI
```bash
nano set_env.sh
```
If you are in a proxy environment, also set the proxy-related environment variables:
```bash
export http_proxy="Your_HTTP_Proxy"
export https_proxy="Your_HTTPs_Proxy"
```
Set the values of the variables:
- **HOST_IP, HOST_IP_EXTERNAL** - These variables are used to configure the name/address of the service in the operating system environment for the application services to interact with each other and with the outside world.
If your server uses only an internal address and is not accessible from the Internet, then the values for these two variables will be the same and the value will be equal to the server's internal name/address.
If your server uses only an external, Internet-accessible address, then the values for these two variables will be the same and the value will be equal to the server's external name/address.
If your server is located on an internal network, has an internal address, but is accessible from the Internet via a proxy/firewall/load balancer, then the HOST_IP variable will have a value equal to the internal name/address of the server, and the EXTERNAL_HOST_IP variable will have a value equal to the external name/address of the proxy/firewall/load balancer behind which the server is located.
We set these values in the file set_env\*\*\*\*.sh
- **Variables with names like "**\*\*\*\*\*\*\_PORT"\*\* - These variables set the IP port numbers for establishing network connections to the application services.
The values shown in the file set_env.sh or set_env_vllm they are the values used for the development and testing of the application, as well as configured for the environment in which the development is performed. These values must be configured in accordance with the rules of network access to your environment's server, and must not overlap with the IP ports of other applications that are already in use.
#### Required Models
By default, the multimodal-embedding and LVM models are set to a default value as listed below: By default, the multimodal-embedding and LVM models are set to a default value as listed below:
@@ -108,57 +230,187 @@ Note:
For AMD ROCm System "Xkev/Llama-3.2V-11B-cot" is recommended to run on ghcr.io/huggingface/text-generation-inference:2.4.1-rocm For AMD ROCm System "Xkev/Llama-3.2V-11B-cot" is recommended to run on ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
### Start all the services Docker Containers #### Set variables with script set_env\*\*\*\*.sh
> Before running the docker compose command, you need to be in the folder that has the docker compose yaml file #### If you use vLLM
```bash
. set_env_vllm.sh
```
#### If you use TGI
```bash ```bash
cd GenAIExamples/MultimodalQnA/docker_compose/amd/gpu/rocm
. set_env.sh . set_env.sh
```
### Start the services:
#### If you use vLLM
```bash
docker compose -f compose_vllm.yaml up -d
```
#### If you use TGI
```bash
docker compose -f compose.yaml up -d docker compose -f compose.yaml up -d
``` ```
Note: Please replace with `host_ip` with your external IP address, do not use localhost. All containers should be running and should not restart:
Note: In order to limit access to a subset of GPUs, please pass each device individually using one or more -device /dev/dri/rendered<node>, where <node> is the card index, starting from 128. (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html#docker-restrict-gpus) ##### If you use vLLM:
Example for set isolation for 1 GPU - multimodalqna-vllm-service
- multimodalqna-lvm
- multimodalqna-backend-server
- multimodalqna-gradio-ui-server
- whisper-service
- embedding-multimodal-bridgetower
- redis-vector-db
- embedding
- retriever-redis
- dataprep-multimodal-redis
``` ##### If you use TGI:
- /dev/dri/card0:/dev/dri/card0
- /dev/dri/renderD128:/dev/dri/renderD128
```
Example for set isolation for 2 GPUs - tgi-llava-rocm-server
- multimodalqna-lvm
- multimodalqna-backend-server
- multimodalqna-gradio-ui-server
- whisper-service
- embedding-multimodal-bridgetower
- redis-vector-db
- embedding
- retriever-redis
- dataprep-multimodal-redis
``` ---
- /dev/dri/card0:/dev/dri/card0
- /dev/dri/renderD128:/dev/dri/renderD128
- /dev/dri/card1:/dev/dri/card1
- /dev/dri/renderD129:/dev/dri/renderD129
```
Please find more information about accessing and restricting AMD GPUs in the link (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html#docker-restrict-gpus) ## Validate the Services
### Validate Microservices ### 1. Validate the vLLM/TGI Service
1. embedding-multimodal-bridgetower #### If you use vLLM:
```bash ```bash
curl http://${host_ip}:${EMBEDDER_PORT}/v1/encode \ DATA='{"model": "Xkev/Llama-3.2V-11B-cot", '\
'"messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens": 256}'
curl http://${HOST_IP}:${MULTIMODALQNA_VLLM_SERVICE_PORT}/v1/chat/completions \
-X POST \
-d "$DATA" \
-H 'Content-Type: application/json'
```
Checking the response from the service. The response should be similar to JSON:
```json
{
"id": "chatcmpl-a3761920c4034131b3cab073b8e8b841",
"object": "chat.completion",
"created": 1742959065,
"model": "Intel/neural-chat-7b-v3-3",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " Deep Learning refers to a modern approach of Artificial Intelligence that aims to replicate the way human brains process information by teaching computers to learn from data without extensive programming",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null
}
],
"usage": { "prompt_tokens": 15, "total_tokens": 47, "completion_tokens": 32, "prompt_tokens_details": null },
"prompt_logprobs": null
}
```
If the service response has a meaningful response in the value of the "choices.message.content" key,
then we consider the vLLM service to be successfully launched
#### If you use TGI:
```bash
DATA='{"inputs":"What is Deep Learning?",'\
'"parameters":{"max_new_tokens":256,"do_sample": true}}'
curl http://${HOST_IP}:${MULTIMODALQNA_TGI_SERVICE_PORT}/generate \
-X POST \
-d "$DATA" \
-H 'Content-Type: application/json'
```
Checking the response from the service. The response should be similar to JSON:
```json
{
"generated_text": "\n\nDeep Learning is a subset of machine learning, which focuses on developing methods inspired by the functioning of the human brain; more specifically, the way it processes and acquires various types of knowledge and information. To enable deep learning, the networks are composed of multiple processing layers that form a hierarchy, with each layer learning more complex and abstraction levels of data representation.\n\nThe principle of Deep Learning is to emulate the structure of neurons in the human brain to construct artificial neural networks capable to accomplish complicated pattern recognition tasks more effectively and accurately. Therefore, these neural networks contain a series of hierarchical components, where units in earlier layers receive simple inputs and are activated by these inputs. The activation of the units in later layers are the results of multiple nonlinear transformations generated from reconstructing and integrating the information in previous layers. In other words, by combining various pieces of information at each layer, a Deep Learning network can extract the input features that best represent the structure of data, providing their outputs at the last layer or final level of abstraction.\n\nThe main idea of using these 'deep' networks in contrast to regular algorithms is that they are capable of representing hierarchical relationships that exist within the data and learn these representations by"
}
```
If the service response has a meaningful response in the value of the "generated_text" key,
then we consider the TGI service to be successfully launched
### 2. Validate the LVM Service
```bash
curl http://${host_ip}:${MULTIMODALQNA_LVM_PORT}/v1/lvm \
-X POST \
-H 'Content-Type: application/json' \
-d '{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
```
Checking the response from the service. The response should be similar to JSON:
```textmate
{"downstream_black_list":[],"id":"1b17e903e8c773be909bde0e7cfdb53f","text":" I will analyze the image and provide a detailed description based on its visual characteristics. I will then compare these characteristics to the standard answer provided to ensure accuracy.\n\n1. **Examine the Image**: The image is a solid color, which appears to be a shade of yellow. There are no additional elements or patterns present in the image.\n\n2. **Compare with Standard Answer**: The standard answer describes the image as a \"yellow image\" without any additional details or context. This matches the observed characteristics of the image being a single, uniform yellow color.\n\n3. **Conclusion**: Based on the visual analysis and comparison with the standard answer, the image can be accurately described as a \"yellow image.\" There are no other features or elements present that would alter this description.\n\nFINAL ANSWER: The image is a yellow image.","metadata":{"video_id":"8c7461df-b373-4a00-8696-9a2234359fe0","source_video":"WeAreGoingOnBullrun_8c7461df-b373-4a00-8696-9a2234359fe0.mp4","time_of_frame_ms":"37000000","transcript_for_inference":"yellow image"}}
```
If the service response has a meaningful response in the value of the "choices.text" key,
then we consider the vLLM service to be successfully launched
### 3. Validate MicroServices
#### embedding-multimodal-bridgetower
Text example:
```bash
curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \ -X POST \
-H "Content-Type:application/json" \ -H "Content-Type:application/json" \
-d '{"text":"This is example"}' -d '{"text":"This is example"}'
``` ```
Checking the response from the service. The response should be similar to text:
```textmate
{"embedding":[0.036936961114406586,-0.0022056063171476126,0.0891181230545044,-0.019263656809926033,-0.049174826592206955,-0.05129311606287956,-0.07172256708145142,0.04365323856472969,0.03275766223669052,0.0059910244308412075,-0.0301326...,-0.0031989417038857937,0.042092420160770416]}
```
Image example:
```bash ```bash
curl http://${host_ip}:${EMBEDDER_PORT}/v1/encode \ curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \ -X POST \
-H "Content-Type:application/json" \ -H "Content-Type:application/json" \
-d '{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}' -d '{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
``` ```
2. embedding Checking the response from the service. The response should be similar to text:
```textmate
{"embedding":[0.024372786283493042,-0.003916610032320023,0.07578050345182419,...,-0.046543147414922714]}
```
#### embedding
Text example:
```bash ```bash
curl http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings \ curl http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings \
@@ -167,143 +419,95 @@ curl http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings \
-d '{"text" : "This is some sample text."}' -d '{"text" : "This is some sample text."}'
``` ```
```bash Checking the response from the service. The response should be similar to text:
curl http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings \
-X POST \ ```textmate
-H "Content-Type: application/json" \ {"id":"4fb722012a2719e38188190e1cb37ed3","text":"This is some sample text.","embedding":[0.043303076177835464,-0.051807764917612076,...,-0.0005179636646062136,-0.0027774290647357702],"search_type":"similarity","k":4,"distance_threshold":null,"fetch_k":20,"lambda_mult":0.5,"score_threshold":0.2,"constraints":null,"url":null,"base64_image":null}
-d '{"text": {"text" : "This is some sample text."}, "image" : {"url": "https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true"}}'
``` ```
3. retriever-multimodal-redis Image example:
```bash
curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \
-H "Content-Type:application/json" \
-d '{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
```
Checking the response from the service. The response should be similar to text:
```textmate
{"id":"cce4eab623255c4c632fb920e277dcf7","text":"This is some sample text.","embedding":[0.02613169699907303,-0.049398183822631836,...,0.03544217720627785],"search_type":"similarity","k":4,"distance_threshold":null,"fetch_k":20,"lambda_mult":0.5,"score_threshold":0.2,"constraints":null,"url":"https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true","base64_image":"iVBORw0KGgoAAAANSUhEUgAAAoEAAAJqCAMAAABjDmrLAAAABGdBTUEAALGPC/.../BCU5wghOc4AQnOMEJTnCCE5zgBCc4wQlOcILzqvO/ARWd2ns+lvHkAAAAAElFTkSuQmCC"}
```
#### retriever-multimodal-redis
set "your_embedding" variable:
```bash ```bash
export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)") export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
curl http://${host_ip}:7000/v1/retrieval \ ```
Test Redis retriever
```bash
curl http://${host_ip}:${REDIS_RETRIEVER_PORT}/v1/retrieval \
-X POST \ -X POST \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d "{\"text\":\"test\",\"embedding\":${your_embedding}}" -d "{\"text\":\"test\",\"embedding\":${your_embedding}}"
``` ```
4. lvm-llava Checking the response from the service. The response should be similar to text:
```textmate
{"id":"80a4f3fc5f5d5cd31ab1e3912f6b6042","retrieved_docs":[],"initial_query":"test","top_n":1,"metadata":[]}
```
#### whisper service
```bash ```bash
curl http://${host_ip}:${LLAVA_SERVER_PORT}/generate \ curl http://${host_ip}:7066/v1/asr \
-X POST \ -X POST \
-d '{"audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}' \
-H 'Content-Type: application/json'
```
Checking the response from the service. The response should be similar to text:
```textmate
{"asr_result":"you"}
```
### 4. Validate the MegaService
```bash
DATA='{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
curl http://${HOST_IP}:${MULTIMODALQNA_BACKEND_SERVICE_PORT}/v1/multimodalqna \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"prompt":"Describe the image please.", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}' -d "$DATA"
``` ```
5. lvm Checking the response from the service. The response should be similar to text:
```textmate
{"id":"chatcmpl-75aK2KWCfxZmVcfh5tiiHj","object":"chat.completion","created":1743568232,"model":"multimodalqna","choices":[{"index":0,"message":{"role":"assistant","content":"There is no video segments retrieved given the query!"},"finish_reason":"stop","metadata":{"audio":"you"}}],"usage":{"prompt_tokens":0,"total_tokens":0,"completion_tokens":0}}
```
If the output lines in the "choices.text" keys contain words (tokens) containing meaning, then the service is considered launched successfully.
### 5. Stop application
#### If you use vLLM
```bash ```bash
curl http://${host_ip}:9399/v1/lvm \ cd ~/multimodalqna-install/GenAIExamples/MultimodalQnA/docker_compose/amd/gpu/rocm
-X POST \ docker compose -f compose_vllm.yaml down
-H 'Content-Type: application/json' \
-d '{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [{"b64_img_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "transcript_for_inference": "yellow image", "video_id": "8c7461df-b373-4a00-8696-9a2234359fe0", "time_of_frame_ms":"37000000", "source_video":"WeAreGoingOnBullrun_8c7461df-b373-4a00-8696-9a2234359fe0.mp4"}], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
``` ```
#### If you use TGI
```bash ```bash
curl http://${host_ip}:9399/v1/lvm \ cd ~/multimodalqna-install/GenAIExamples/MultimodalQnA/docker_compose/amd/gpu/rocm
-X POST \ docker compose -f compose.yaml down
-H 'Content-Type: application/json' \
-d '{"image": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "prompt":"What is this?"}'
```
Also, validate LVM Microservice with empty retrieval results
```bash
curl http://${host_ip}:9399/v1/lvm \
-X POST \
-H 'Content-Type: application/json' \
-d '{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
```
6. dataprep-multimodal-redis
Download a sample video, image, and audio file and create a caption
```bash
export video_fn="WeAreGoingOnBullrun.mp4"
wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/WeAreGoingOnBullrun.mp4 -O ${video_fn}
export image_fn="apple.png"
wget https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true -O ${image_fn}
export caption_fn="apple.txt"
echo "This is an apple." > ${caption_fn}
export audio_fn="AudioSample.wav"
wget https://github.com/intel/intel-extension-for-transformers/raw/main/intel_extension_for_transformers/neural_chat/assets/audio/sample.wav -O ${audio_fn}
```
Test dataprep microservice with generating transcript. This command updates a knowledge base by uploading a local video .mp4 and an audio .wav file.
```bash
curl --silent --write-out "HTTPSTATUS:%{http_code}" \
${DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT} \
-H 'Content-Type: multipart/form-data' \
-X POST \
-F "files=@./${video_fn}" \
-F "files=@./${audio_fn}"
```
Also, test dataprep microservice with generating an image caption using lvm microservice
```bash
curl --silent --write-out "HTTPSTATUS:%{http_code}" \
${DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT} \
-H 'Content-Type: multipart/form-data' \
-X POST -F "files=@./${image_fn}"
```
Now, test the microservice with posting a custom caption along with an image
```bash
curl --silent --write-out "HTTPSTATUS:%{http_code}" \
${DATAPREP_INGEST_SERVICE_ENDPOINT} \
-H 'Content-Type: multipart/form-data' \
-X POST -F "files=@./${image_fn}" -F "files=@./${caption_fn}"
```
Also, you are able to get the list of all files that you uploaded:
```bash
curl -X POST \
-H "Content-Type: application/json" \
${DATAPREP_GET_FILE_ENDPOINT}
```
Then you will get the response python-style LIST like this. Notice the name of each uploaded file e.g., `videoname.mp4` will become `videoname_uuid.mp4` where `uuid` is a unique ID for each uploaded file. The same files that are uploaded twice will have different `uuid`.
```bash
[
"WeAreGoingOnBullrun_7ac553a1-116c-40a2-9fc5-deccbb89b507.mp4",
"WeAreGoingOnBullrun_6d13cf26-8ba2-4026-a3a9-ab2e5eb73a29.mp4",
"apple_fcade6e6-11a5-44a2-833a-3e534cbe4419.png",
"AudioSample_976a85a6-dc3e-43ab-966c-9d81beef780c.wav
]
```
To delete all uploaded files along with data indexed with `$INDEX_NAME` in REDIS.
```bash
curl -X POST \
-H "Content-Type: application/json" \
-d '{"file_path": "all"}' \
${DATAPREP_DELETE_FILE_ENDPOINT}
```
7. MegaService
```bash
curl http://${host_ip}:8888/v1/multimodalqna \
-H "Content-Type: application/json" \
-X POST \
-d '{"messages": "What is the revenue of Nike in 2023?"}'
```
```bash
curl http://${host_ip}:8888/v1/multimodalqna \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": "chao, "}], "max_tokens": 10}'
``` ```

4
MultimodalQnA/docker_compose/amd/gpu/rocm/compose.yaml
View File

@@ -105,7 +105,7 @@ services:
HUGGINGFACEHUB_API_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN} HUGGINGFACEHUB_API_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN}
HUGGING_FACE_HUB_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN} HUGGING_FACE_HUB_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN}
volumes: volumes:
- "/var/opea/multimodalqna-service/data:/data" - "${MODEL_CACHE:-./data}:/data"
shm_size: 64g shm_size: 64g
devices: devices:
- /dev/kfd:/dev/kfd - /dev/kfd:/dev/kfd
@@ -156,7 +156,7 @@ services:
MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE} MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE}
MM_RETRIEVER_SERVICE_HOST_IP: ${MM_RETRIEVER_SERVICE_HOST_IP} MM_RETRIEVER_SERVICE_HOST_IP: ${MM_RETRIEVER_SERVICE_HOST_IP}
LVM_SERVICE_HOST_IP: ${LVM_SERVICE_HOST_IP} LVM_SERVICE_HOST_IP: ${LVM_SERVICE_HOST_IP}
WHISPER_SERVER_PORT: ${WHISPER_SERVER_PORT} WHISPER_SERVER_PORT: ${WHISPER_PORT}
WHISPER_SERVER_ENDPOINT: ${WHISPER_SERVER_ENDPOINT} WHISPER_SERVER_ENDPOINT: ${WHISPER_SERVER_ENDPOINT}
ipc: host ipc: host
restart: always restart: always

187
MultimodalQnA/docker_compose/amd/gpu/rocm/compose_vllm.yaml Normal file
View File

@@ -0,0 +1,187 @@
# Copyright (C) 2024 Advanced Micro Devices, Inc.
# SPDX-License-Identifier: Apache-2.0
services:
whisper-service:
image: ${REGISTRY:-opea}/whisper:${TAG:-latest}
container_name: whisper-service
ports:
- "7066:7066"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
restart: unless-stopped
redis-vector-db:
image: redis/redis-stack:7.2.0-v9
container_name: redis-vector-db
ports:
- "6379:6379"
- "8001:8001"
dataprep-multimodal-redis:
image: ${REGISTRY:-opea}/dataprep:${TAG:-latest}
container_name: dataprep-multimodal-redis
depends_on:
- redis-vector-db
- lvm
ports:
- "6007:5000"
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
REDIS_URL: ${REDIS_URL}
REDIS_HOST: ${REDIS_HOST}
INDEX_NAME: ${INDEX_NAME}
LVM_ENDPOINT: "http://${LVM_SERVICE_HOST_IP}:9399/v1/lvm"
HUGGINGFACEHUB_API_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN}
MULTIMODAL_DATAPREP: true
DATAPREP_COMPONENT_NAME: "OPEA_DATAPREP_MULTIMODALREDIS"
restart: unless-stopped
embedding-multimodal-bridgetower:
image: ${REGISTRY:-opea}/embedding-multimodal-bridgetower:${TAG:-latest}
container_name: embedding-multimodal-bridgetower
ports:
- ${EMBEDDER_PORT}:${EMBEDDER_PORT}
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
PORT: ${EMBEDDER_PORT}
healthcheck:
test: ["CMD-SHELL", "http_proxy='' curl -f http://localhost:${EMBEDDER_PORT}/v1/health_check"]
interval: 10s
timeout: 6s
retries: 18
start_period: 30s
entrypoint: ["python", "bridgetower_server.py", "--device", "cpu", "--model_name_or_path", $EMBEDDING_MODEL_ID]
restart: unless-stopped
embedding:
image: ${REGISTRY:-opea}/embedding:${TAG:-latest}
container_name: embedding
depends_on:
embedding-multimodal-bridgetower:
condition: service_healthy
ports:
- ${MM_EMBEDDING_PORT_MICROSERVICE}:${MM_EMBEDDING_PORT_MICROSERVICE}
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
MMEI_EMBEDDING_ENDPOINT: ${MMEI_EMBEDDING_ENDPOINT}
MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE}
MULTIMODAL_EMBEDDING: true
restart: unless-stopped
retriever-redis:
image: ${REGISTRY:-opea}/retriever:${TAG:-latest}
container_name: retriever-redis
depends_on:
- redis-vector-db
ports:
- "7000:7000"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
REDIS_URL: ${REDIS_URL}
INDEX_NAME: ${INDEX_NAME}
BRIDGE_TOWER_EMBEDDING: ${BRIDGE_TOWER_EMBEDDING}
LOGFLAG: ${LOGFLAG}
RETRIEVER_COMPONENT_NAME: "OPEA_RETRIEVER_REDIS"
restart: unless-stopped
multimodalqna-vllm-service:
image: ${REGISTRY:-opea}/vllm-rocm:${TAG:-latest}
container_name: multimodalqna-vllm-service
ports:
- "${MULTIMODAL_VLLM_SERVICE_PORT:-8081}:8011"
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
HUGGINGFACEHUB_API_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN}
HF_TOKEN: ${MULTIMODAL_HUGGINGFACEHUB_API_TOKEN}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
WILM_USE_TRITON_FLASH_ATTENTION: 0
PYTORCH_JIT: 0
volumes:
- "${MODEL_CACHE:-./data}:/data"
shm_size: 20G
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/:/dev/dri/
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
- apparmor=unconfined
command: "--model ${MULTIMODAL_LLM_MODEL_ID} --swap-space 16 --disable-log-requests --dtype float16 --tensor-parallel-size 1 --host 0.0.0.0 --port 8011 --num-scheduler-steps 1 --distributed-executor-backend \"mp\""
ipc: host
lvm:
image: ${REGISTRY:-opea}/lvm:${TAG:-latest}
container_name: lvm
depends_on:
- multimodalqna-vllm-service
ports:
- "9399:9399"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LVM_COMPONENT_NAME: "OPEA_VLLM_LVM"
LVM_ENDPOINT: ${LVM_ENDPOINT}
LLM_MODEL_ID: ${MULTIMODAL_LLM_MODEL_ID}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
restart: unless-stopped
multimodalqna:
image: ${REGISTRY:-opea}/multimodalqna:${TAG:-latest}
container_name: multimodalqna-backend-server
depends_on:
- redis-vector-db
- dataprep-multimodal-redis
- embedding
- retriever-redis
- lvm
ports:
- "8888:8888"
environment:
no_proxy: ${no_proxy}
https_proxy: ${https_proxy}
http_proxy: ${http_proxy}
MEGA_SERVICE_HOST_IP: ${MEGA_SERVICE_HOST_IP}
MM_EMBEDDING_SERVICE_HOST_IP: ${MM_EMBEDDING_SERVICE_HOST_IP}
MM_EMBEDDING_PORT_MICROSERVICE: ${MM_EMBEDDING_PORT_MICROSERVICE}
MM_RETRIEVER_SERVICE_HOST_IP: ${MM_RETRIEVER_SERVICE_HOST_IP}
LVM_SERVICE_HOST_IP: ${LVM_SERVICE_HOST_IP}
WHISPER_SERVER_PORT: ${WHISPER_PORT}
WHISPER_SERVER_ENDPOINT: ${WHISPER_SERVER_ENDPOINT}
ipc: host
restart: always
multimodalqna-ui:
image: ${REGISTRY:-opea}/multimodalqna-ui:${TAG:-latest}
container_name: multimodalqna-gradio-ui-server
depends_on:
- multimodalqna
ports:
- "5173:5173"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- BACKEND_SERVICE_ENDPOINT=${BACKEND_SERVICE_ENDPOINT}
- DATAPREP_INGEST_SERVICE_ENDPOINT=${DATAPREP_INGEST_SERVICE_ENDPOINT}
- DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT=${DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT}
- DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT=${DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT}
ipc: host
restart: always
networks:
default:
driver: bridge

2
MultimodalQnA/docker_compose/amd/gpu/rocm/set_env.sh
View File

@@ -31,3 +31,5 @@ export DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/datap
export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions" export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions"
export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get" export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get"
export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete" export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete"
export WHISPER_PORT="7066"
export WHISPER_SERVER_ENDPOINT="http://${host_ip}:${WHISPER_PORT}/v1/asr"

35
MultimodalQnA/docker_compose/amd/gpu/rocm/set_env_vllm.sh Normal file
View File

@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Copyright (C) 2024 Advanced Micro Devices, Inc.
# SPDX-License-Identifier: Apache-2.0
export HOST_IP=${your_host_ip_address}
export MULTIMODAL_HUGGINGFACEHUB_API_TOKEN=${your_huggingfacehub_token}
export MULTIMODAL_TGI_SERVICE_PORT="8399"
export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export BRIDGE_TOWER_EMBEDDING=true
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${HOST_IP}:$EMBEDDER_PORT"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export REDIS_URL="redis://${HOST_IP}:6379"
export REDIS_HOST=${HOST_IP}
export INDEX_NAME="mm-rag-redis"
export VLLM_SERVER_PORT=8081
export LVM_ENDPOINT="http://${HOST_IP}:${VLLM_SERVER_PORT}"
export EMBEDDING_MODEL_ID="BridgeTower/bridgetower-large-itm-mlm-itc"
export LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
export WHISPER_MODEL="base"
export MM_EMBEDDING_SERVICE_HOST_IP=${HOST_IP}
export MM_RETRIEVER_SERVICE_HOST_IP=${HOST_IP}
export LVM_SERVICE_HOST_IP=${HOST_IP}
export MEGA_SERVICE_HOST_IP=${HOST_IP}
export BACKEND_SERVICE_ENDPOINT="http://${HOST_IP}:8888/v1/multimodalqna"
export DATAPREP_INGEST_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/ingest"
export DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_transcripts"
export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions"
export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get"
export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete"
export WHISPER_PORT="7066"
export WHISPER_SERVER_ENDPOINT="http://${host_ip}:${WHISPER_PORT}/v1/asr"

5
MultimodalQnA/docker_image_build/build.yaml
View File

@@ -77,3 +77,8 @@ services:
dockerfile: comps/tts/src/Dockerfile dockerfile: comps/tts/src/Dockerfile
extends: multimodalqna extends: multimodalqna
image: ${REGISTRY:-opea}/tts:${TAG:-latest} image: ${REGISTRY:-opea}/tts:${TAG:-latest}
vllm-rocm:
build:
context: GenAIComps
dockerfile: comps/third_parties/vllm/src/Dockerfile.amd_gpu
image: ${REGISTRY:-opea}/vllm-rocm:${TAG:-latest}

11
MultimodalQnA/tests/test_compose_on_rocm.sh
View File

@@ -72,12 +72,21 @@ function setup_env() {
export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions" export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions"
export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get" export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get"
export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete" export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete"
export MODEL_CACHE=${model_cache:-"/var/opea/multimodalqna-service/data"}
} }
function start_services() { function start_services() {
cd $WORKPATH/docker_compose/amd/gpu/rocm cd $WORKPATH/docker_compose/amd/gpu/rocm
docker compose -f compose.yaml up -d > ${LOG_PATH}/start_services_with_compose.log docker compose -f compose.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
sleep 1m n=0
until [[ "$n" -ge 100 ]]; do
docker logs tgi-llava-rocm-server >& $LOG_PATH/tgi-llava-rocm-server_start.log
if grep -q "Connected" $LOG_PATH/tgi-llava-rocm-server_start.log; then
break
fi
sleep 10s
n=$((n+1))
done
} }
function prepare_data() { function prepare_data() {

340
MultimodalQnA/tests/test_compose_vllm_on_rocm.sh Normal file
View File

@@ -0,0 +1,340 @@
#!/bin/bash
# Copyright (C) 2024 Advanced Micro Devices, Inc.
# SPDX-License-Identifier: Apache-2.0
set -ex
IMAGE_REPO=${IMAGE_REPO:-"opea"}
IMAGE_TAG=${IMAGE_TAG:-"latest"}
echo "REGISTRY=IMAGE_REPO=${IMAGE_REPO}"
echo "TAG=IMAGE_TAG=${IMAGE_TAG}"
export REGISTRY=${IMAGE_REPO}
export TAG=${IMAGE_TAG}
WORKPATH=$(dirname "$PWD")
LOG_PATH="$WORKPATH/tests"
ip_address=$(hostname -I | awk '{print $1}')
export image_fn="apple.png"
export video_fn="WeAreGoingOnBullrun.mp4"
export caption_fn="apple.txt"
function build_docker_images() {
opea_branch=${opea_branch:-"main"}
# If the opea_branch isn't main, replace the git clone branch in Dockerfile.
if [[ "${opea_branch}" != "main" ]]; then
cd $WORKPATH
OLD_STRING="RUN git clone --depth 1 https://github.com/opea-project/GenAIComps.git"
NEW_STRING="RUN git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git"
find . -type f -name "Dockerfile*" | while read -r file; do
echo "Processing file: $file"
sed -i "s|$OLD_STRING|$NEW_STRING|g" "$file"
done
fi
cd $WORKPATH/docker_image_build
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
echo "Build all the images with --no-cache, check docker_image_build.log for details..."
service_list="multimodalqna multimodalqna-ui embedding-multimodal-bridgetower embedding retriever lvm dataprep whisper vllm-rocm"
docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log
docker images && sleep 1m
}
function setup_env() {
export HOST_IP=${ip_address}
export host_ip=${ip_address}
export MULTIMODAL_HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
export MULTIMODAL_VLLM_SERVICE_PORT="8399"
export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export BRIDGE_TOWER_EMBEDDING=true
export EMBEDDER_PORT=6006
export MMEI_EMBEDDING_ENDPOINT="http://${HOST_IP}:$EMBEDDER_PORT"
export MM_EMBEDDING_PORT_MICROSERVICE=6000
export WHISPER_SERVER_PORT=7066
export WHISPER_SERVER_ENDPOINT="http://${HOST_IP}:${WHISPER_SERVER_PORT}/v1/asr"
export REDIS_URL="redis://${HOST_IP}:6379"
export REDIS_HOST=${HOST_IP}
export INDEX_NAME="mm-rag-redis"
export LVM_ENDPOINT="http://${HOST_IP}:8399"
export EMBEDDING_MODEL_ID="BridgeTower/bridgetower-large-itm-mlm-itc"
export MULTIMODAL_LLM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
export WHISPER_MODEL="base"
export MM_EMBEDDING_SERVICE_HOST_IP=${HOST_IP}
export MM_RETRIEVER_SERVICE_HOST_IP=${HOST_IP}
export LVM_SERVICE_HOST_IP=${HOST_IP}
export MEGA_SERVICE_HOST_IP=${HOST_IP}
export BACKEND_SERVICE_ENDPOINT="http://${HOST_IP}:8888/v1/multimodalqna"
export DATAPREP_INGEST_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/ingest"
export DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_transcripts"
export DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/generate_captions"
export DATAPREP_GET_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/get"
export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete"
export MODEL_CACHE=${model_cache:-"/var/opea/multimodalqna-service/data"}
}
function start_services() {
cd $WORKPATH/docker_compose/amd/gpu/rocm
docker compose -f compose_vllm.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
n=0
until [[ "$n" -ge 100 ]]; do
docker logs multimodalqna-vllm-service >& $LOG_PATH/search-vllm-service_start.log
if grep -q "Application startup complete" $LOG_PATH/search-vllm-service_start.log; then
break
fi
sleep 10s
n=$((n+1))
done
}
function prepare_data() {
cd $LOG_PATH
echo "Downloading image and video"
wget https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true -O ${image_fn}
wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/WeAreGoingOnBullrun.mp4 -O ${video_fn}
echo "Writing caption file"
echo "This is an apple." > ${caption_fn}
sleep 1m
}
function validate_service() {
local URL="$1"
local EXPECTED_RESULT="$2"
local SERVICE_NAME="$3"
local DOCKER_NAME="$4"
local INPUT_DATA="$5"
if [[ $SERVICE_NAME == *"dataprep-multimodal-redis-transcript"* ]]; then
cd $LOG_PATH
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -F "files=@./${video_fn}" -H 'Content-Type: multipart/form-data' "$URL")
elif [[ $SERVICE_NAME == *"dataprep-multimodal-redis-caption"* ]]; then
cd $LOG_PATH
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -F "files=@./${image_fn}" -H 'Content-Type: multipart/form-data' "$URL")
elif [[ $SERVICE_NAME == *"dataprep-multimodal-redis-ingest"* ]]; then
cd $LOG_PATH
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -F "files=@./${image_fn}" -F "files=@./apple.txt" -H 'Content-Type: multipart/form-data' "$URL")
elif [[ $SERVICE_NAME == *"dataprep_get"* ]]; then
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -H 'Content-Type: application/json' "$URL")
elif [[ $SERVICE_NAME == *"dataprep_del"* ]]; then
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -d '{"file_path": "apple.txt"}' -H 'Content-Type: application/json' "$URL")
else
HTTP_RESPONSE=$(curl --silent --write-out "HTTPSTATUS:%{http_code}" -X POST -d "$INPUT_DATA" -H 'Content-Type: application/json' "$URL")
fi
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
RESPONSE_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g')
docker logs ${DOCKER_NAME} >> ${LOG_PATH}/${SERVICE_NAME}.log
# check response status
if [ "$HTTP_STATUS" -ne "200" ]; then
echo "[ $SERVICE_NAME ] HTTP status is not 200. Received status was $HTTP_STATUS"
exit 1
else
echo "[ $SERVICE_NAME ] HTTP status is 200. Checking content..."
fi
# check response body
if [[ "$RESPONSE_BODY" != *"$EXPECTED_RESULT"* ]]; then
echo "[ $SERVICE_NAME ] Content does not match the expected result: $RESPONSE_BODY"
exit 1
else
echo "[ $SERVICE_NAME ] Content is as expected."
fi
sleep 1s
}
function validate_microservices() {
# Check if the microservices are running correctly.
# Bridgetower Embedding Server
echo "Validating embedding-multimodal-bridgetower"
validate_service \
"http://${host_ip}:${EMBEDDER_PORT}/v1/encode" \
'"embedding":[' \
"embedding-multimodal-bridgetower" \
"embedding-multimodal-bridgetower" \
'{"text":"This is example"}'
validate_service \
"http://${host_ip}:${EMBEDDER_PORT}/v1/encode" \
'"embedding":[' \
"embedding-multimodal-bridgetower" \
"embedding-multimodal-bridgetower" \
'{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
# embedding microservice
echo "Validating embedding"
validate_service \
"http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings" \
'"embedding":[' \
"embedding" \
"embedding" \
'{"text" : "This is some sample text."}'
validate_service \
"http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings" \
'"embedding":[' \
"embedding" \
"embedding" \
'{"text": {"text" : "This is some sample text."}, "image" : {"url": "https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true"}}'
sleep 1m # retrieval can't curl as expected, try to wait for more time
# test data prep
echo "Data Prep with Generating Transcript for Video"
validate_service \
"${DATAPREP_GEN_TRANSCRIPT_SERVICE_ENDPOINT}" \
"Data preparation succeeded" \
"dataprep-multimodal-redis-transcript" \
"dataprep-multimodal-redis"
echo "Data Prep with Image & Caption Ingestion"
validate_service \
"${DATAPREP_INGEST_SERVICE_ENDPOINT}" \
"Data preparation succeeded" \
"dataprep-multimodal-redis-ingest" \
"dataprep-multimodal-redis"
echo "Validating get file returns mp4"
validate_service \
"${DATAPREP_GET_FILE_ENDPOINT}" \
'.mp4' \
"dataprep_get" \
"dataprep-multimodal-redis"
echo "Validating get file returns png"
validate_service \
"${DATAPREP_GET_FILE_ENDPOINT}" \
'.png' \
"dataprep_get" \
"dataprep-multimodal-redis"
sleep 2m
# multimodal retrieval microservice
echo "Validating retriever-redis"
your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
validate_service \
"http://${host_ip}:7000/v1/retrieval" \
"retrieved_docs" \
"retriever-redis" \
"retriever-redis" \
"{\"text\":\"test\",\"embedding\":${your_embedding}}"
sleep 5m
#vLLM Service
echo "Evaluating vllm"
validate_service \
"${host_ip}:${MULTIMODAL_VLLM_SERVICE_PORT}/v1/chat/completions" \
"content" \
"multimodalqna-vllm-service" \
"multimodalqna-vllm-service" \
'{"model": "Xkev/Llama-3.2V-11B-cot", "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens": 17}'
# lvm
echo "Evaluating lvm"
validate_service \
"http://${host_ip}:9399/v1/lvm" \
'"text":"' \
"lvm" \
"lvm" \
'{"retrieved_docs": [], "initial_query": "What is this?", "top_n": 1, "metadata": [{"b64_img_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "transcript_for_inference": "yellow image", "video_id": "8c7461df-b373-4a00-8696-9a2234359fe0", "time_of_frame_ms":"37000000", "source_video":"WeAreGoingOnBullrun_8c7461df-b373-4a00-8696-9a2234359fe0.mp4"}], "chat_template":"The caption of the image is: '\''{context}'\''. {question}"}'
# data prep requiring lvm
echo "Data Prep with Generating Caption for Image"
validate_service \
"${DATAPREP_GEN_CAPTION_SERVICE_ENDPOINT}" \
"Data preparation succeeded" \
"dataprep-multimodal-redis-caption" \
"dataprep-multimodal-redis"
sleep 3m
}
function validate_megaservice() {
# Curl the Mega Service with retrieval
echo "Validate megaservice with first query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"time_of_frame_ms":' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": "What is the revenue of Nike in 2023?"}'
echo "Validate megaservice with first audio query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"time_of_frame_ms":' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
echo "Validate megaservice with follow-up query"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}, {"type": "image_url", "image_url": {"url": "https://www.ilankelman.org/stopsigns/australia.jpg"}}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
echo "Validate megaservice with multiple text queries"
validate_service \
"http://${host_ip}:8888/v1/multimodalqna" \
'"content":"' \
"multimodalqna" \
"multimodalqna-backend-server" \
'{"messages": [{"role": "user", "content": [{"type": "text", "text": "hello, "}]}, {"role": "assistant", "content": "opea project! "}, {"role": "user", "content": [{"type": "text", "text": "goodbye"}]}]}'
}
function validate_delete {
echo "Validate data prep delete files"
export DATAPREP_DELETE_FILE_ENDPOINT="http://${HOST_IP}:6007/v1/dataprep/delete"
validate_service \
"${DATAPREP_DELETE_FILE_ENDPOINT}" \
'{"status":true}' \
"dataprep_del" \
"dataprep-multimodal-redis"
}
function delete_data() {
cd $LOG_PATH
echo "Deleting image, video, and caption"
rm -rf ${image_fn}
rm -rf ${video_fn}
rm -rf ${caption_fn}
}
function stop_docker() {
cd $WORKPATH/docker_compose/amd/gpu/rocm
docker compose -f compose.yaml stop && docker compose -f compose.yaml rm -f
}
function main() {
setup_env
stop_docker
if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi
start_time=$(date +%s)
start_services
end_time=$(date +%s)
duration=$((end_time-start_time))
echo "Mega service start duration is $duration s" && sleep 1s
prepare_data
validate_microservices
echo "==== microservices validated ===="
validate_megaservice
echo "==== megaservice validated ===="
validate_delete
echo "==== delete validated ===="
delete_data
stop_docker
echo y | docker system prune
}
main