jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Adding files to deploy VisualQnA application on ROCm vLLM (#1751)

Browse Source 
Signed-off-by: Artem Astafev <a.astafev@datamonsters.com>
This commit is contained in:
Artem Astafev
2025-04-07 08:27:19 +07:00
committed by GitHub
parent e8cdf7d668
commit 073e5443ec
10 changed files with 715 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
VisualQnA/assets/img/visualqna-ui-result-page.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 395 KiB

BIN
VisualQnA/assets/img/visualqna-ui-starting-page.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 248 KiB

441
VisualQnA/docker_compose/amd/gpu/rocm/README.md
View File

@@ -1,127 +1,358 @@
# Build Mega Service of VisualQnA on AMD ROCm
# Build and Deploy VisualQnA Application on AMD GPU (ROCm)
This document outlines the deployment process for a VisualQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Xeon server. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to Docker Hub soon, it will simplify the deployment process for this service.
## Build Docker Images
## 🚀 Build Docker Images
### 1. Build Docker Image
First of all, you need to build Docker Images locally and install the python package of it.
- #### Create application install directory and go to it:
### 1. Build LVM and NGINX Docker Images
```bash
mkdir ~/visualqna-install && cd visualqna-install
```
- #### Clone the repository GenAIExamples (the default repository branch "main" is used here):
```bash
git clone https://github.com/opea-project/GenAIExamples.git
```
If you need to use a specific branch/tag of the GenAIExamples repository, then (v1.3 replace with its own value):
```bash
git clone https://github.com/opea-project/GenAIExamples.git && cd GenAIExamples && git checkout v1.3
```
We remind you that when using a specific version of the code, you need to use the README from this version:
- #### Go to build directory:
```bash
cd ~/visualqna-install/GenAIExamples/VisualQnA/docker_image_build
```
- Cleaning up the GenAIComps repository if it was previously cloned in this directory.
This is necessary if the build was performed earlier and the GenAIComps folder exists and is not empty:
```bash
echo Y | rm -R GenAIComps
```
- #### Clone the repository GenAIComps (the default repository branch "main" is used here):
```bash
git clone https://github.com/opea-project/GenAIComps.git
```
If you use a specific tag of the GenAIExamples repository,
then you should also use the corresponding tag for GenAIComps. (v1.3 replace with its own value):
```bash
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout v1.3
```
We remind you that when using a specific version of the code, you need to use the README from this version.
- #### Setting the list of images for the build (from the build file.yaml)
If you want to deploy a vLLM-based or TGI-based application, then the set of services is installed as follows:
#### vLLM-based application
```bash
service_list="visualqna visualqna-ui nginx lvm vllm-rocm"
```
#### TGI-based application
```bash
service_list="visualqna visualqna-ui nginx lvm"
```
- #### Optional. Pull TGI Docker Image (Do this if you want to use TGI)
```bash
docker pull ghcr.io/huggingface/text-generation-inference:2.3.1-rocm
```
- #### Build Docker Images
```bash
docker compose -f build.yaml build ${service_list} --no-cache
```
After the build, we check the list of images with the command:
```bash
docker image ls
```
The list of images should include:
##### vLLM-based application:
- opea/vllm-rocm:latest
- opea/lvm:latest
- opea/visualqna:latest
- opea/visualqn-ui:latest
- opea/nginx:latest
##### TGI-based application:
- ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
- opea/lvm:latest
- opea/visualqna:latest
- opea/visualqn-ui:latest
- opea/nginx:latest
---
## Deploy VisualQnA Application
### Docker Compose Configuration for AMD GPUs
To enable GPU support for AMD GPUs, the following configuration is added to the Docker Compose file:
- compose_vllm.yaml - for vLLM-based application
- compose.yaml - for TGI-based
```yaml
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/:/dev/dri/
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
```
This configuration forwards all available GPUs to the container. To use a specific GPU, specify its `cardN` and `renderN` device IDs. For example:
```yaml
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/card0:/dev/dri/card0
- /dev/dri/renderD128:/dev/dri/renderD128
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
```
**How to Identify GPU Device IDs:**
Use AMD GPU driver utilities to determine the correct `cardN` and `renderN` IDs for your GPU.
### Set deploy environment variables
#### Setting variables in the operating system environment:
##### Set variable HUGGINGFACEHUB_API_TOKEN:
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build --no-cache -t opea/lvm:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/lvms/src/Dockerfile .
docker build --no-cache -t opea/nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/third_parties/nginx/src/Dockerfile .
### Replace the string 'your_huggingfacehub_token' with your HuggingFacehub repository access token.
export HUGGINGFACEHUB_API_TOKEN='your_huggingfacehub_token'
```
### 2. Build MegaService Docker Image
#### Set variables value in set_env\*\*\*\*.sh file:
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `visualqna.py` Python script. Build MegaService Docker image via below command:
Go to Docker Compose directory:
```bash
git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/VisualQnA
docker build --no-cache -t opea/visualqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
cd ~/visualqna-install/GenAIExamples/VisualQnA/docker_compose/amd/gpu/rocm
```
### 3. Build UI Docker Image
The example uses the Nano text editor. You can use any convenient text editor:
Build frontend Docker image via below command:
#### If you use vLLM
```bash
cd GenAIExamples/VisualQnA/ui
docker build --no-cache -t opea/visualqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f docker/Dockerfile .
nano set_env_vllm.sh
```
### 4. Pull TGI AMD ROCm Image
#### If you use TGI
```bash
docker pull ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
nano set_env.sh
```
Then run the command `docker images`, you will have the following 5 Docker Images:
1. `ghcr.io/huggingface/text-generation-inference:2.4.1-rocm`
2. `opea/lvm:latest`
3. `opea/visualqna:latest`
4. `opea/visualqna-ui:latest`
5. `opea/nginx`
## 🚀 Start Microservices
### Setup Environment Variables
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
**Export the value of the public IP address of your ROCM server to the `host_ip` environment variable**
> Change the External_Public_IP below with the actual IPV4 value
```
export host_ip="External_Public_IP"
```
**Append the value of the public IP address to the no_proxy list**
```
export your_no_proxy="${your_no_proxy},${host_ip}"
```
If you are in a proxy environment, also set the proxy-related environment variables:
```bash
export HOST_IP=${your_host_ip}
export VISUALQNA_TGI_SERVICE_PORT="8399"
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN={your_hugginface_api_token}
export VISUALQNA_CARD_ID="card1"
export VISUALQNA_RENDER_ID="renderD136"
export LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
export MODEL="llava-hf/llava-v1.6-mistral-7b-hf"
export LVM_ENDPOINT="http://${HOST_IP}:8399"
export LVM_SERVICE_PORT=9399
export MEGA_SERVICE_HOST_IP=${HOST_IP}
export LVM_SERVICE_HOST_IP=${HOST_IP}
export BACKEND_SERVICE_ENDPOINT="http://${HOST_IP}:18003/v1/visualqna"
export FRONTEND_SERVICE_IP=${HOST_IP}
export FRONTEND_SERVICE_PORT=18001
export BACKEND_SERVICE_NAME=visualqna
export BACKEND_SERVICE_IP=${HOST_IP}
export BACKEND_SERVICE_PORT=18002
export NGINX_PORT=18003
export http_proxy="Your_HTTP_Proxy"
export https_proxy="Your_HTTPs_Proxy"
```
Note: Please replace with `host_ip` with you external IP address, do not use localhost.
Set the values of the variables:
Note: You can use set_env.sh file with bash command (. setset_env.sh) to set up needed variables.
- **HOST_IP, HOST_IP_EXTERNAL** - These variables are used to configure the name/address of the service in the operating system environment for the application services to interact with each other and with the outside world.
### Start all the services Docker Containers
If your server uses only an internal address and is not accessible from the Internet, then the values for these two variables will be the same and the value will be equal to the server's internal name/address.
> Before running the docker compose command, you need to be in the folder that has the docker compose yaml file
If your server uses only an external, Internet-accessible address, then the values for these two variables will be the same and the value will be equal to the server's external name/address.
If your server is located on an internal network, has an internal address, but is accessible from the Internet via a proxy/firewall/load balancer, then the HOST_IP variable will have a value equal to the internal name/address of the server, and the EXTERNAL_HOST_IP variable will have a value equal to the external name/address of the proxy/firewall/load balancer behind which the server is located.
We set these values in the file set_env\*\*\*\*.sh
- **Variables with names like "**\*\*\*\*\*\*\_PORT"\*\* - These variables set the IP port numbers for establishing network connections to the application services.
The values shown in the file set_env.sh or set_env_vllm they are the values used for the development and testing of the application, as well as configured for the environment in which the development is performed. These values must be configured in accordance with the rules of network access to your environment's server, and must not overlap with the IP ports of other applications that are already in use.
#### Required Models
By default, LVM models are set to a default value as listed below:
| Service | Model |
| ------- | ------------------------ |
| LVM | llava-hf/llava-1.5-7b-hf |
| LVM | Xkev/Llama-3.2V-11B-cot |
Note:
For AMD ROCm System "Xkev/Llama-3.2V-11B-cot" is recommended to run on ghcr.io/huggingface/text-generation-inference:2.4.1-rocm
#### Set variables with script set_env\*\*\*\*.sh
#### If you use vLLM
```bash
cd GenAIExamples/VisualQnA/docker_compose/amd/gpu/rocm
. set_env_vllm.sh
```
#### If you use TGI
```bash
. set_env.sh
```
### Start the services:
#### If you use vLLM
```bash
docker compose -f compose_vllm.yaml up -d
```
#### If you use TGI
```bash
docker compose -f compose.yaml up -d
```
### Validate Microservices
All containers should be running and should not restart:
Follow the instructions to validate MicroServices.
##### If you use vLLM:
> Note: If you see an "Internal Server Error" from the `curl` command, wait a few minutes for the microserver to be ready and then try again.
- visualqna-vllm-service
- lvm-server
- visualqna-rocm-backend-server
- visualqna-rocm-ui-server
- visualqna-rocm-nginx-server
1. LLM Microservice
##### If you use TGI:
```bash
http_proxy="" curl http://${host_ip}:9399/v1/lvm -XPOST -d '{"image": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "prompt":"What is this?"}' -H 'Content-Type: application/json'
```
- visualqna-tgi-service
- lvm-server
- visualqna-rocm-backend-server
- visualqna-rocm-ui-server
- visualqna-rocm-nginx-server
2. MegaService
---
## Validate the Services
### 1. Validate the vLLM/TGI Service
#### If you use vLLM:
```bash
curl http://${host_ip}:8888/v1/visualqna -H "Content-Type: application/json" -d '{
DATA='{"model": "Xkev/Llama-3.2V-11B-cot", '\
'"messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens": 256}'
curl http://${HOST_IP}:${VISUALQNA_VLLM_SERVICE_PORT}/v1/chat/completions \
-X POST \
-d "$DATA" \
-H 'Content-Type: application/json'
```
Checking the response from the service. The response should be similar to JSON:
```json
{
"id": "chatcmpl-a3761920c4034131b3cab073b8e8b841",
"object": "chat.completion",
"created": 1742959065,
"model": "Intel/neural-chat-7b-v3-3",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " Deep Learning refers to a modern approach of Artificial Intelligence that aims to replicate the way human brains process information by teaching computers to learn from data without extensive programming",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null
}
],
"usage": { "prompt_tokens": 15, "total_tokens": 47, "completion_tokens": 32, "prompt_tokens_details": null },
"prompt_logprobs": null
}
```
If the service response has a meaningful response in the value of the "choices.message.content" key,
then we consider the vLLM service to be successfully launched
#### If you use TGI:
```bash
DATA='{"inputs":"What is Deep Learning?",'\
'"parameters":{"max_new_tokens":256,"do_sample": true}}'
curl http://${HOST_IP}:${VISUALQNA_TGI_SERVICE_PORT}/generate \
-X POST \
-d "$DATA" \
-H 'Content-Type: application/json'
```
Checking the response from the service. The response should be similar to JSON:
```json
{
"generated_text": "\n\nDeep Learning is a subset of machine learning, which focuses on developing methods inspired by the functioning of the human brain; more specifically, the way it processes and acquires various types of knowledge and information. To enable deep learning, the networks are composed of multiple processing layers that form a hierarchy, with each layer learning more complex and abstraction levels of data representation.\n\nThe principle of Deep Learning is to emulate the structure of neurons in the human brain to construct artificial neural networks capable to accomplish complicated pattern recognition tasks more effectively and accurately. Therefore, these neural networks contain a series of hierarchical components, where units in earlier layers receive simple inputs and are activated by these inputs. The activation of the units in later layers are the results of multiple nonlinear transformations generated from reconstructing and integrating the information in previous layers. In other words, by combining various pieces of information at each layer, a Deep Learning network can extract the input features that best represent the structure of data, providing their outputs at the last layer or final level of abstraction.\n\nThe main idea of using these 'deep' networks in contrast to regular algorithms is that they are capable of representing hierarchical relationships that exist within the data and learn these representations by"
}
```
If the service response has a meaningful response in the value of the "generated_text" key,
then we consider the TGI service to be successfully launched
### 2. Validate the LVM Service
```bash
curl http://${host_ip}:${VISUALQNA_LVM_PORT}/v1/lvm \
-X POST \
-H 'Content-Type: application/json' \
-d '{"image": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "prompt":"What is this?"}'
```
Checking the response from the service. The response should be similar to JSON:
```textmate
{"downstream_black_list":[],"id":"53fae0310461ce3e7cde5b0930bd3b92","text":"<SUMMARY> I will analyze the image to determine its color and then provide a structured response using the specified format. </SUMMARY>\n\n<CAPTION> The image is a solid block of color. It appears to be a bright, vibrant hue. </CAPTION>\n\n<REASONING> To determine the color, I will observe the image's hue and saturation. The image is a uniform yellow color, which is a bright and noticeable shade. </REASONING>\n\n<CONCLUSION> The image is a bright yellow color. </CONCLUSION>"}
```
If the service response has a meaningful response in the value of the "choices.text" key,
then we consider the vLLM service to be successfully launched
### 3. Validate the MegaService
```bash
DATA='{
"messages": [
{
"role": "user",
@@ -141,16 +372,48 @@ curl http://${host_ip}:8888/v1/visualqna -H "Content-Type: application/json" -d
],
"max_tokens": 300
}'
curl http://${HOST_IP}:${VISUALQNA_BACKEND_SERVICE_PORT}/v1/visualqna \
-H "Content-Type: application/json" \
-d "$DATA"
```
## 🚀 Launch the UI
Checking the response from the service. The response should be similar to text:
To access the frontend, open the following URL in your browser: http://{host_ip}:5173. By default, the UI runs on port 5173 internally. If you prefer to use a different host port to access the frontend, you can modify the port mapping in the `compose.yaml` file as shown below:
```yaml
visualqna-gaudi-ui-server:
image: opea/visualqna-ui:latest
...
ports:
- "80:5173"
```textmate
{"id":"chatcmpl-ML6oimVzQFmk5dgsAFDSKo","object":"chat.completion","created":1743668853,"model":"visualqna","choices":[{"index":0,"message":{"role":"assistant","content":"<SUMMARY> I will analyze the image to identify key elements and provide a structured response. The focus will be on identifying the main subject of the image. </SUMMARY>\n\n<CAPTION> The image shows a street scene with a prominent red and white \"STOP\" sign in the foreground. In the background, there is a Chinese-style archway with red pillars and decorative elements. The archway is labeled with Chinese characters. </CAPTION>\n\n<REASONING> To determine the main subject of the image, I will focus on the most visually striking and central elements. The \"STOP\" sign is clearly visible and stands out due to its color and position. The archway, while significant, is in the background and does not dominate the image as much as the \"STOP\" sign does. </REASONING>\n\n<CONCLUSION> Stop sign. </CONCLUSION>"},"finish_reason":"stop","metadata":null}],"usage":{"prompt_tokens":0,"total_tokens":0,"completion_tokens":0}}
```
If the output lines in the "choices.text" keys contain words (tokens) containing meaning, then the service is considered launched successfully.
### 4. Validate the Frontend (UI)
To access the UI, use the URL - http://${EXTERNAL_HOST_IP}:${FRONTEND_SERVICE_PORT}
A page should open when you click through to this address:
![UI start page](../../../../assets/img/visualqna-ui-starting-page.png)
If a page of this type has opened, then we believe that the service is running and responding,
and we can proceed to functional UI testing.
To check that service is working push on one of the example pictures and whait for the answer:
![UI result page](../../../../assets/img/visualqna-ui-result-page.png)
If the result shown on the page is correct, then we consider the verification of the UI service to be successful.
### 5. Stop application
#### If you use vLLM
```bash
cd ~/visualqnaa-install/GenAIExamples/VisualQnA/docker_compose/amd/gpu/rocm
docker compose -f compose_vllm.yaml down
```
#### If you use TGI
```bash
cd ~/visualqna-install/GenAIExamples/VisualQnA/docker_compose/amd/gpu/rocm
docker compose -f compose.yaml down
```

2
VisualQnA/docker_compose/amd/gpu/rocm/compose.yaml
View File

@@ -15,7 +15,7 @@ services:
HUGGINGFACEHUB_API_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
HUGGING_FACE_HUB_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
volumes:
- "/var/opea/visualqna-service/data:/data"
- "${MODEL_CACHE:-./data}:/data"
shm_size: 64g
devices:
- /dev/kfd:/dev/kfd

105
VisualQnA/docker_compose/amd/gpu/rocm/compose_vllm.yaml Normal file
View File

@@ -0,0 +1,105 @@
# Copyright (C) 2024 Advanced Micro Devices, Inc.
# SPDX-License-Identifier: Apache-2.0
services:
visualqna-vllm-service:
image: ${REGISTRY:-opea}/vllm-rocm:${TAG:-latest}
container_name: visualqna-vllm-service
ports:
- "${VISUALQNA_VLLM_SERVICE_PORT:-8081}:8011"
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
HUGGINGFACEHUB_API_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
HF_TOKEN: ${VISUALQNA_HUGGINGFACEHUB_API_TOKEN}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
WILM_USE_TRITON_FLASH_ATTENTION: 0
PYTORCH_JIT: 0
volumes:
- "${MODEL_CACHE:-./data}:/data"
shm_size: 20G
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/:/dev/dri/
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
- apparmor=unconfined
command: "--model ${VISUALQNA_LVM_MODEL_ID} --swap-space 16 --disable-log-requests --dtype float16 --tensor-parallel-size 1 --host 0.0.0.0 --port 8011 --num-scheduler-steps 1 --distributed-executor-backend \"mp\""
ipc: host
lvm:
image: ${REGISTRY:-opea}/lvm:${TAG:-latest}
container_name: lvm-server
depends_on:
- visualqna-vllm-service
ports:
- "9399:9399"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LVM_COMPONENT_NAME: "OPEA_VLLM_LVM"
LVM_ENDPOINT: ${LVM_ENDPOINT}
LLM_MODEL_ID: ${VISUALQNA_LVM_MODEL_ID}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
restart: unless-stopped
visualqna-rocm-backend-server:
image: ${REGISTRY:-opea}/visualqna:${TAG:-latest}
container_name: visualqna-rocm-backend-server
depends_on:
- visualqna-vllm-service
- lvm
ports:
- "${BACKEND_SERVICE_PORT:-8888}:8888"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- MEGA_SERVICE_HOST_IP=${MEGA_SERVICE_HOST_IP}
- LVM_SERVICE_HOST_IP=${LVM_SERVICE_HOST_IP}
ipc: host
restart: always
visualqna-rocm-ui-server:
image: ${REGISTRY:-opea}/visualqna-ui:${TAG:-latest}
container_name: visualqna-rocm-ui-server
depends_on:
- visualqna-rocm-backend-server
ports:
- "${FRONTEND_SERVICE_PORT:-5173}:5173"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- BACKEND_BASE_URL=${BACKEND_SERVICE_ENDPOINT}
ipc: host
restart: always
visualqna-nginx-server:
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
container_name: visualqna-rocm-nginx-server
depends_on:
- visualqna-rocm-backend-server
- visualqna-rocm-ui-server
ports:
- "${NGINX_PORT:-80}:80"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- FRONTEND_SERVICE_IP=${HOST_IP}
- FRONTEND_SERVICE_PORT=${FRONTEND_SERVICE_PORT}
- BACKEND_SERVICE_NAME=${BACKEND_SERVICE_NAME}
- BACKEND_SERVICE_IP=${HOST_IP}
- BACKEND_SERVICE_PORT=${BACKEND_SERVICE_PORT}
ipc: host
restart: always
networks:
default:
driver: bridge

3
VisualQnA/docker_compose/amd/gpu/rocm/set_env.sh
View File

@@ -3,7 +3,8 @@
# Copyright (C) 2024 Advanced Micro Devices, Inc
# SPDX-License-Identifier: Apache-2.0
export HOST_IP=${Your_host_ip_address}
export HOST_IP=${host_ip}
export EXTERNAL_HOST_IP=${host_ip}
export VISUALQNA_TGI_SERVICE_PORT="8399"
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN=${Your_HUGGINGFACEHUB_API_TOKEN}
export VISUALQNA_CARD_ID="card1"

23
VisualQnA/docker_compose/amd/gpu/rocm/set_env_vllm.sh Normal file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Copyright (C) 2024 Advanced Micro Devices, Inc
# SPDX-License-Identifier: Apache-2.0
export HOST_IP=${host_ip}
export EXTERNAL_HOST_IP=${host_ip}
export VISUALQNA_VLLM_SERVICE_PORT="8081"
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN=${Your_HUGGINGFACEHUB_API_TOKEN}
export VISUALQNA_CARD_ID="card1"
export VISUALQNA_RENDER_ID="renderD136"
export VISUALQNA_LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
export LVM_ENDPOINT="http://${HOST_IP}:${VISUALQNA_VLLM_SERVICE_PORT}"
export LVM_SERVICE_PORT=9399
export MEGA_SERVICE_HOST_IP=${HOST_IP}
export LVM_SERVICE_HOST_IP=${HOST_IP}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:${BACKEND_SERVICE_PORT}/v1/visualqna"
export FRONTEND_SERVICE_IP=${HOST_IP}
export FRONTEND_SERVICE_PORT=18001
export BACKEND_SERVICE_NAME=visualqna
export BACKEND_SERVICE_IP=${HOST_IP}
export BACKEND_SERVICE_PORT=18002
export NGINX_PORT=18003

5
VisualQnA/docker_image_build/build.yaml
View File

@@ -29,3 +29,8 @@ services:
dockerfile: comps/third_parties/nginx/src/Dockerfile
extends: visualqna
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
vllm-rocm:
build:
context: GenAIComps
dockerfile: comps/third_parties/vllm/src/Dockerfile.amd_gpu
image: ${REGISTRY:-opea}/vllm-rocm:${TAG:-latest}

5
VisualQnA/tests/test_compose_on_rocm.sh
View File

@@ -33,6 +33,7 @@ export BACKEND_SERVICE_IP=${HOST_IP}
export BACKEND_SERVICE_PORT=8888
export NGINX_PORT=18003
export PATH="~/miniconda3/bin:$PATH"
export MODEL_CACHE=${model_cache:-"/var/opea/multimodalqna-service/data"}
function build_docker_images() {
opea_branch=${opea_branch:-"main"}
@@ -63,11 +64,11 @@ function start_services() {
sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env
# Start Docker Containers
docker compose up -d > ${LOG_PATH}/start_services_with_compose.log
docker compose -f compose.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
n=0
until [[ "$n" -ge 100 ]]; do
docker logs visualqna-tgi-service > ${LOG_PATH}/lvm_tgi_service_start.log
docker logs visualqna-tgi-service >& ${LOG_PATH}/lvm_tgi_service_start.log
if grep -q Connected ${LOG_PATH}/lvm_tgi_service_start.log; then
break
fi

224
VisualQnA/tests/test_compose_vllm_on_rocm.sh Normal file
View File

@@ -0,0 +1,224 @@
#!/bin/bash
# Copyright (C) 2024 Advanced Micro Devices, Inc.
# SPDX-License-Identifier: Apache-2.0
set -x
IMAGE_REPO=${IMAGE_REPO:-"opea"}
IMAGE_TAG=${IMAGE_TAG:-"latest"}
echo "REGISTRY=IMAGE_REPO=${IMAGE_REPO}"
echo "TAG=IMAGE_TAG=${IMAGE_TAG}"
WORKPATH=$(dirname "$PWD")
LOG_PATH="$WORKPATH/tests"
ip_address=$(hostname -I | awk '{print $1}')
export REGISTRY=${IMAGE_REPO}
export TAG=${IMAGE_TAG}
export HOST_IP=${ip_address}
export VISUALQNA_VLLM_SERVICE_PORT="8081"
export VISUALQNA_HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
export VISUALQNA_CARD_ID="card1"
export VISUALQNA_RENDER_ID="renderD136"
export VISUALQNA_LVM_MODEL_ID="Xkev/Llama-3.2V-11B-cot"
export MODEL="llava-hf/llava-v1.6-mistral-7b-hf"
export LVM_ENDPOINT="http://${HOST_IP}:${VISUALQNA_VLLM_SERVICE_PORT}"
export LVM_SERVICE_PORT=9399
export MEGA_SERVICE_HOST_IP=${HOST_IP}
export LVM_SERVICE_HOST_IP=${HOST_IP}
export BACKEND_SERVICE_ENDPOINT="http://${HOST_IP}:${BACKEND_SERVICE_PORT}/v1/visualqna"
export FRONTEND_SERVICE_IP=${HOST_IP}
export FRONTEND_SERVICE_PORT=5173
export BACKEND_SERVICE_NAME=visualqna
export BACKEND_SERVICE_IP=${HOST_IP}
export BACKEND_SERVICE_PORT=8888
export NGINX_PORT=18003
export PATH="~/miniconda3/bin:$PATH"
export MODEL_CACHE=${model_cache:-"/var/opea/multimodalqna-service/data"}
function build_docker_images() {
opea_branch=${opea_branch:-"main"}
# If the opea_branch isn't main, replace the git clone branch in Dockerfile.
if [[ "${opea_branch}" != "main" ]]; then
cd $WORKPATH
OLD_STRING="RUN git clone --depth 1 https://github.com/opea-project/GenAIComps.git"
NEW_STRING="RUN git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git"
find . -type f -name "Dockerfile*" | while read -r file; do
echo "Processing file: $file"
sed -i "s|$OLD_STRING|$NEW_STRING|g" "$file"
done
fi
cd $WORKPATH/docker_image_build
git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git
echo "Build all the images with --no-cache, check docker_image_build.log for details..."
docker compose -f build.yaml build --no-cache > ${LOG_PATH}/docker_image_build.log
docker images && sleep 1s
}
function start_services() {
cd $WORKPATH/docker_compose/amd/gpu/rocm
sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env
# Start Docker Containers
docker compose -f compose_vllm.yaml up -d > ${LOG_PATH}/start_services_with_compose.log
n=0
until [[ "$n" -ge 100 ]]; do
docker logs visualqna-vllm-service >& ${LOG_PATH}/visualqna-vllm-service_start.log
if grep -q "Application startup complete" $LOG_PATH/visualqna-vllm-service_start.log; then
break
fi
sleep 10s
n=$((n+1))
done
}
function validate_services() {
local URL="$1"
local EXPECTED_RESULT="$2"
local SERVICE_NAME="$3"
local DOCKER_NAME="$4"
local INPUT_DATA="$5"
local HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST -d "$INPUT_DATA" -H 'Content-Type: application/json' "$URL")
if [ "$HTTP_STATUS" -eq 200 ]; then
echo "[ $SERVICE_NAME ] HTTP status is 200. Checking content..."
local CONTENT=$(curl -s -X POST -d "$INPUT_DATA" -H 'Content-Type: application/json' "$URL" | tee ${LOG_PATH}/${SERVICE_NAME}.log)
if echo "$CONTENT" | grep -q "$EXPECTED_RESULT"; then
echo "[ $SERVICE_NAME ] Content is as expected."
else
echo "[ $SERVICE_NAME ] Content does not match the expected result: $CONTENT"
docker logs ${DOCKER_NAME} >> ${LOG_PATH}/${SERVICE_NAME}.log
exit 1
fi
else
echo "[ $SERVICE_NAME ] HTTP status is not 200. Received status was $HTTP_STATUS"
docker logs ${DOCKER_NAME} >> ${LOG_PATH}/${SERVICE_NAME}.log
exit 1
fi
sleep 1s
}
function validate_microservices() {
# Check if the microservices are running correctly.
# lvm microservice
validate_services \
"${ip_address}:9399/v1/lvm" \
"The image" \
"lvm" \
"visualqna-vllm-service" \
'{"image": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC", "prompt":"What is this?"}'
}
function validate_megaservice() {
# Curl the Mega Service
validate_services \
"${ip_address}:8888/v1/visualqna" \
"The image" \
"visualqna-rocm-backend-server" \
"visualqna-rocm-backend-server" \
'{
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What'\''s in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://www.ilankelman.org/stopsigns/australia.jpg"
}
}
]
}
],
"max_tokens": 300
}'
# test the megeservice via nginx
validate_services \
"${ip_address}:${NGINX_PORT}/v1/visualqna" \
"The image" \
"visualqna-rocm-nginx-server" \
"visualqna-rocm-nginx-server" \
'{
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What'\''s in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://www.ilankelman.org/stopsigns/australia.jpg"
}
}
]
}
],
"max_tokens": 300
}'
}
function validate_frontend() {
cd $WORKPATH/ui/svelte
local conda_env_name="OPEA_e2e"
export PATH=${HOME}/miniforge3/bin/:$PATH
if conda info --envs | grep -q "$conda_env_name"; then
echo "$conda_env_name exist!"
else
conda create -n ${conda_env_name} python=3.12 -y
fi
source activate ${conda_env_name}
sed -i "s/localhost/$ip_address/g" playwright.config.ts
conda install -c conda-forge nodejs -y
npm install && npm ci && npx playwright install --with-deps
node -v && npm -v && pip list
exit_status=0
npx playwright test || exit_status=$?
if [ $exit_status -ne 0 ]; then
echo "[TEST INFO]: ---------frontend test failed---------"
exit $exit_status
else
echo "[TEST INFO]: ---------frontend test passed---------"
fi
}
function stop_docker() {
cd $WORKPATH/docker_compose/amd/gpu/rocm/
docker compose stop && docker compose rm -f
}
function main() {
stop_docker
if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi
start_services
validate_microservices
validate_megaservice
#validate_frontend
stop_docker
echo y | docker system prune
}
main