jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Enable CodeGen vLLM (#1636)

Browse Source 
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
xiguiw
2025-03-13 10:38:47 +08:00
committed by GitHub
parent adcd113f53
commit effa2a28cf
8 changed files with 421 additions and 196 deletions
Download Patch File Download Diff File Expand all files Collapse all files

148
CodeGen/docker_compose/intel/cpu/xeon/README.md
View File

@@ -1,6 +1,7 @@
# Build MegaService of CodeGen on Xeon # Build MegaService of CodeGen on Xeon
This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Xeon server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to Docker Hub soon, further simplifying the deployment process for this service. This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Xeon server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to Docker Hub soon, further simplifying the deployment process for this service.
The default pipeline deploys with vLLM as the LLM serving component. It also provides options of using TGI backend for LLM microservice.
## 🚀 Create an AWS Xeon Instance ## 🚀 Create an AWS Xeon Instance
@@ -10,55 +11,6 @@ For detailed information about these instance types, you can refer to [m7i](http
After launching your instance, you can connect to it using SSH (for Linux instances) or Remote Desktop Protocol (RDP) (for Windows instances). From there, you'll have full access to your Xeon server, allowing you to install, configure, and manage your applications as needed. After launching your instance, you can connect to it using SSH (for Linux instances) or Remote Desktop Protocol (RDP) (for Windows instances). From there, you'll have full access to your Xeon server, allowing you to install, configure, and manage your applications as needed.
## 🚀 Download or Build Docker Images
Should the Docker image you seek not yet be available on Docker Hub, you can build the Docker image locally.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Build the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker Images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest` (optional)
## 🚀 Start Microservices and MegaService ## 🚀 Start Microservices and MegaService
The CodeGen megaservice manages a single microservice called LLM within a Directed Acyclic Graph (DAG). In the diagram above, the LLM microservice is a language model microservice that generates code snippets based on the user's input query. The TGI service serves as a text generation interface, providing a RESTful API for the LLM microservice. The CodeGen Gateway acts as the entry point for the CodeGen application, invoking the Megaservice to generate code snippets in response to the user's input query. The CodeGen megaservice manages a single microservice called LLM within a Directed Acyclic Graph (DAG). In the diagram above, the LLM microservice is a language model microservice that generates code snippets based on the user's input query. The TGI service serves as a text generation interface, providing a RESTful API for the LLM microservice. The CodeGen Gateway acts as the entry point for the CodeGen application, invoking the Megaservice to generate code snippets in response to the user's input query.
@@ -89,41 +41,56 @@ flowchart LR
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below. Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
**Append the value of the public IP address to the no_proxy list** 1. set the host_ip and huggingface token
> Note:
> Please replace the `your_ip_address` with you external IP address, do not use `localhost`.
```bash
export host_ip=${your_ip_address}
export HUGGINGFACEHUB_API_TOKEN=you_huggingface_token
``` ```
export your_no_proxy=${your_no_proxy},"External_Public_IP"
``` 2. Set Netowork Proxy
**If you access public network through proxy, set the network proxy, otherwise, skip this step**
```bash ```bash
export no_proxy=${your_no_proxy} export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy} export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy} export https_proxy=${your_https_proxy}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
export MEGA_SERVICE_HOST_IP=${host_ip}
export LLM_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
``` ```
Note: Please replace the `host_ip` with you external IP address, do not use `localhost`.
### Start the Docker Containers for All Services ### Start the Docker Containers for All Services
CodeGen support TGI service and vLLM service, you can choose start either one of them.
Start CodeGen based on TGI service:
```bash ```bash
cd GenAIExamples/CodeGen/docker_compose/intel/cpu/xeon cd GenAIExamples/CodeGen/docker_compose
docker compose up -d source set_env.sh
cd intel/cpu/xeon
docker compose --profile codegen-xeon-tgi up -d
```
Start CodeGen based on vLLM service:
```bash
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/cpu/xeon
docker compose --profile codegen-xeon-vllm up -d
``` ```
### Validate the MicroServices and MegaService ### Validate the MicroServices and MegaService
1. TGI Service 1. LLM Service (for TGI, vLLM)
```bash ```bash
curl http://${host_ip}:8028/generate \ curl http://${host_ip}:8028/v1/chat/completions \
-X POST \ -X POST \
-d '{"inputs":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","parameters":{"max_new_tokens":256, "do_sample": true}}' \ -d '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}], "max_tokens":32}' \
-H 'Content-Type: application/json' -H 'Content-Type: application/json'
``` ```
@@ -257,3 +224,52 @@ For example:
- Ask question and get answer - Ask question and get answer
![qna](../../../../assets/img/codegen_qna.png) ![qna](../../../../assets/img/codegen_qna.png)
## 🚀 Download or Build Docker Images
Should the Docker image you seek not yet be available on Docker Hub, you can build the Docker image locally.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Build the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker Images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest` (optional)

62
CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
View File

@@ -4,7 +4,9 @@
services: services:
tgi-service: tgi-service:
image: ghcr.io/huggingface/text-generation-inference:2.4.0-intel-cpu image: ghcr.io/huggingface/text-generation-inference:2.4.0-intel-cpu
container_name: tgi-service container_name: tgi-server
profiles:
- codegen-xeon-tgi
ports: ports:
- "8028:80" - "8028:80"
volumes: volumes:
@@ -22,28 +24,66 @@ services:
timeout: 10s timeout: 10s
retries: 100 retries: 100
command: --model-id ${LLM_MODEL_ID} --cuda-graphs 0 command: --model-id ${LLM_MODEL_ID} --cuda-graphs 0
llm: vllm-service:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest} image: ${REGISTRY:-opea}/vllm:${TAG:-latest}
container_name: llm-textgen-server container_name: vllm-server
depends_on: profiles:
tgi-service: - codegen-xeon-vllm
condition: service_healthy
ports: ports:
- "9000:9000" - "8028:80"
ipc: host volumes:
- "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub"
shm_size: 1g
environment: environment:
no_proxy: ${no_proxy} no_proxy: ${no_proxy}
http_proxy: ${http_proxy} http_proxy: ${http_proxy}
https_proxy: ${https_proxy} https_proxy: ${https_proxy}
LLM_ENDPOINT: ${TGI_LLM_ENDPOINT} HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
host_ip: ${host_ip}
healthcheck:
test: ["CMD-SHELL", "curl -f http://$host_ip:8028/health || exit 1"]
interval: 10s
timeout: 10s
retries: 100
command: --model ${LLM_MODEL_ID} --host 0.0.0.0 --port 80
llm-base:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-server
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${LLM_ENDPOINT}
LLM_MODEL_ID: ${LLM_MODEL_ID} LLM_MODEL_ID: ${LLM_MODEL_ID}
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN} HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
restart: unless-stopped restart: unless-stopped
llm-tgi-service:
extends: llm-base
container_name: llm-codegen-tgi-server
profiles:
- codegen-xeon-tgi
ports:
- "9000:9000"
ipc: host
depends_on:
tgi-service:
condition: service_healthy
llm-vllm-service:
extends: llm-base
container_name: llm-codegen-vllm-server
profiles:
- codegen-xeon-vllm
ports:
- "9000:9000"
ipc: host
depends_on:
vllm-service:
condition: service_healthy
codegen-xeon-backend-server: codegen-xeon-backend-server:
image: ${REGISTRY:-opea}/codegen:${TAG:-latest} image: ${REGISTRY:-opea}/codegen:${TAG:-latest}
container_name: codegen-xeon-backend-server container_name: codegen-xeon-backend-server
depends_on: depends_on:
- llm - llm-base
ports: ports:
- "7778:7778" - "7778:7778"
environment: environment:

148
CodeGen/docker_compose/intel/hpu/gaudi/README.md
View File

@@ -2,54 +2,7 @@
This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Gaudi2 server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to the Docker Hub soon, further simplifying the deployment process for this service. This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Gaudi2 server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to the Docker Hub soon, further simplifying the deployment process for this service.
## 🚀 Build Docker Images The default pipeline deploys with vLLM as the LLM serving component. It also provides options of using TGI backend for LLM microservice.
First of all, you need to build the Docker images locally. This step can be ignored after the Docker images published to the Docker Hub.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build the MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Construct the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest`
## 🚀 Start MicroServices and MegaService ## 🚀 Start MicroServices and MegaService
@@ -81,36 +34,56 @@ flowchart LR
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below. Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
1. set the host_ip and huggingface token
> [!NOTE]
> Please replace the `your_ip_address` with you external IP address, do not use `localhost`.
```bash
export host_ip=${your_ip_address}
export HUGGINGFACEHUB_API_TOKEN=you_huggingface_token
```
2. Set Netowork Proxy
**If you access public network through proxy, set the network proxy, otherwise, skip this step**
```bash ```bash
export no_proxy=${your_no_proxy} export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy} export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy} export https_proxy=${your_https_proxy}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
export MEGA_SERVICE_HOST_IP=${host_ip}
export LLM_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
``` ```
> [!NOTE]
> Please replace the `host_ip` with you external IP address, do not use `localhost`.
### Start the Docker Containers for All Services ### Start the Docker Containers for All Services
CodeGen support TGI service and vLLM service, you can choose start either one of them.
Start CodeGen based on TGI service:
```bash ```bash
cd GenAIExamples/CodeGen/docker_compose/intel/hpu/gaudi cd GenAIExamples/CodeGen/docker_compose
docker compose up -d source set_env.sh
cd intel/hpu/gaudi
docker compose --profile codegen-gaudi-tgi up -d
```
Start CodeGen based on vLLM service:
```bash
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/hpu/gaudi
docker compose --profile codegen-gaudi-vllm up -d
``` ```
### Validate the MicroServices and MegaService ### Validate the MicroServices and MegaService
1. TGI Service 1. LLM Service (for TGI, vLLM)
```bash ```bash
curl http://${host_ip}:8028/generate \ curl http://${host_ip}:8028/v1/chat/completions \
-X POST \ -X POST \
-d '{"inputs":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","parameters":{"max_new_tokens":256, "do_sample": true}}' \ -d '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}], "max_tokens":32}' \
-H 'Content-Type: application/json' -H 'Content-Type: application/json'
``` ```
@@ -240,3 +213,52 @@ For example:
- Ask question and get answer - Ask question and get answer
![qna](../../../../assets/img/codegen_qna.png) ![qna](../../../../assets/img/codegen_qna.png)
## 🚀 Build Docker Images
First of all, you need to build the Docker images locally. This step can be ignored after the Docker images published to the Docker Hub.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build the MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Construct the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest`

68
CodeGen/docker_compose/intel/hpu/gaudi/compose.yaml
View File

@@ -5,6 +5,8 @@ services:
tgi-service: tgi-service:
image: ghcr.io/huggingface/tgi-gaudi:2.3.1 image: ghcr.io/huggingface/tgi-gaudi:2.3.1
container_name: tgi-gaudi-server container_name: tgi-gaudi-server
profiles:
- codegen-gaudi-tgi
ports: ports:
- "8028:80" - "8028:80"
volumes: volumes:
@@ -30,28 +32,74 @@ services:
- SYS_NICE - SYS_NICE
ipc: host ipc: host
command: --model-id ${LLM_MODEL_ID} --max-input-length 1024 --max-total-tokens 2048 command: --model-id ${LLM_MODEL_ID} --max-input-length 1024 --max-total-tokens 2048
llm: vllm-service:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest} image: ${REGISTRY:-opea}/vllm-gaudi:${TAG:-latest}
container_name: llm-textgen-gaudi-server container_name: vllm-gaudi-server
depends_on: profiles:
tgi-service: - codegen-gaudi-vllm
condition: service_healthy
ports: ports:
- "9000:9000" - "8028:80"
ipc: host volumes:
- "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub"
shm_size: 1g
environment: environment:
no_proxy: ${no_proxy} no_proxy: ${no_proxy}
http_proxy: ${http_proxy} http_proxy: ${http_proxy}
https_proxy: ${https_proxy} https_proxy: ${https_proxy}
LLM_ENDPOINT: ${TGI_LLM_ENDPOINT} HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
HABANA_VISIBLE_DEVICES: all
OMPI_MCA_btl_vader_single_copy_mechanism: none
VLLM_SKIP_WARMUP: ${VLLM_SKIP_WARMUP:-false}
NUM_CARDS: ${NUM_CARDS:-1}
VLLM_TORCH_PROFILER_DIR: "/mnt"
healthcheck:
test: ["CMD-SHELL", "curl -f http://$host_ip:8028/health || exit 1"]
interval: 10s
timeout: 10s
retries: 100
runtime: habana
cap_add:
- SYS_NICE
ipc: host
command: --model ${LLM_MODEL_ID} --tensor-parallel-size ${NUM_CARDS} --host 0.0.0.0 --port 80 --block-size 128 --max-num-seqs 256
llm-base:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-gaudi-server
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${LLM_ENDPOINT}
LLM_MODEL_ID: ${LLM_MODEL_ID} LLM_MODEL_ID: ${LLM_MODEL_ID}
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN} HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
restart: unless-stopped restart: unless-stopped
llm-tgi-service:
extends: llm-base
container_name: llm-codegen-tgi-gaudi-server
profiles:
- codegen-gaudi-tgi
ports:
- "9000:9000"
ipc: host
depends_on:
tgi-service:
condition: service_healthy
llm-vllm-service:
extends: llm-base
container_name: llm-codegen-gaudi-vllm-server
profiles:
- codegen-gaudi-vllm
ports:
- "9000:9000"
ipc: host
depends_on:
vllm-service:
condition: service_healthy
codegen-gaudi-backend-server: codegen-gaudi-backend-server:
image: ${REGISTRY:-opea}/codegen:${TAG:-latest} image: ${REGISTRY:-opea}/codegen:${TAG:-latest}
container_name: codegen-gaudi-backend-server container_name: codegen-gaudi-backend-server
depends_on: depends_on:
- llm - llm-base
ports: ports:
- "7778:7778" - "7778:7778"
environment: environment:

14
CodeGen/docker_compose/set_env.sh
View File

@@ -6,9 +6,21 @@ pushd "../../" > /dev/null
source .set_env.sh source .set_env.sh
popd > /dev/null popd > /dev/null
export host_ip=$(hostname -I | awk '{print $1}')
if [ -z "${HUGGINGFACEHUB_API_TOKEN}" ]; then
echo "Error: HUGGINGFACEHUB_API_TOKEN is not set. Please set HUGGINGFACEHUB_API_TOKEN"
fi
if [ -z "${host_ip}" ]; then
echo "Error: host_ip is not set. Please set host_ip first."
fi
export no_proxy=${no_proxy},${host_ip}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct" export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028" export LLM_ENDPOINT="http://${host_ip}:8028"
export MEGA_SERVICE_HOST_IP=${host_ip} export MEGA_SERVICE_HOST_IP=${host_ip}
export LLM_SERVICE_HOST_IP=${host_ip} export LLM_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen" export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
export MODEL_CACHE="./data"

12
CodeGen/docker_image_build/build.yaml
View File

@@ -29,3 +29,15 @@ services:
dockerfile: comps/llms/src/text-generation/Dockerfile dockerfile: comps/llms/src/text-generation/Dockerfile
extends: codegen extends: codegen
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest} image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
vllm:
build:
context: vllm
dockerfile: Dockerfile.cpu
extends: codegen
image: ${REGISTRY:-opea}/vllm:${TAG:-latest}
vllm-gaudi:
build:
context: vllm-fork
dockerfile: Dockerfile.hpu
extends: codegen
image: ${REGISTRY:-opea}/vllm-gaudi:${TAG:-latest}

74
CodeGen/tests/test_compose_on_gaudi.sh
View File

@@ -30,34 +30,44 @@ function build_docker_images() {
cd $WORKPATH/docker_image_build cd $WORKPATH/docker_image_build
git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git
# Download Gaudi vllm of latest tag
git clone https://github.com/HabanaAI/vllm-fork.git && cd vllm-fork
VLLM_VER=$(git describe --tags "$(git rev-list --tags --max-count=1)")
echo "Check out vLLM tag ${VLLM_VER}"
git checkout ${VLLM_VER} &> /dev/null && cd ../
echo "Build all the images with --no-cache, check docker_image_build.log for details..." echo "Build all the images with --no-cache, check docker_image_build.log for details..."
service_list="codegen codegen-ui llm-textgen" service_list="codegen codegen-ui llm-textgen vllm-gaudi"
docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log
docker pull ghcr.io/huggingface/tgi-gaudi:2.0.6
docker images && sleep 1s docker images && sleep 1s
} }
function start_services() { function start_services() {
local compose_profile="$1"
local llm_container_name="$2"
cd $WORKPATH/docker_compose/intel/hpu/gaudi cd $WORKPATH/docker_compose/intel/hpu/gaudi
export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3" export http_proxy=${http_proxy}
export TGI_LLM_ENDPOINT="http://${ip_address}:8028" export https_proxy=${https_proxy}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export LLM_ENDPOINT="http://${ip_address}:8028"
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
export MEGA_SERVICE_HOST_IP=${ip_address} export MEGA_SERVICE_HOST_IP=${ip_address}
export LLM_SERVICE_HOST_IP=${ip_address} export LLM_SERVICE_HOST_IP=${ip_address}
export BACKEND_SERVICE_ENDPOINT="http://${ip_address}:7778/v1/codegen" export BACKEND_SERVICE_ENDPOINT="http://${ip_address}:7778/v1/codegen"
export NUM_CARDS=1
export host_ip=${ip_address} export host_ip=${ip_address}
sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env
# Start Docker Containers # Start Docker Containers
docker compose up -d > ${LOG_PATH}/start_services_with_compose.log docker compose --profile ${compose_profile} up -d | tee ${LOG_PATH}/start_services_with_compose.log
n=0 n=0
until [[ "$n" -ge 100 ]]; do until [[ "$n" -ge 100 ]]; do
docker logs tgi-gaudi-server > ${LOG_PATH}/tgi_service_start.log docker logs ${llm_container_name} > ${LOG_PATH}/llm_service_start.log 2>&1
if grep -q Connected ${LOG_PATH}/tgi_service_start.log; then if grep -E "Connected|complete" ${LOG_PATH}/llm_service_start.log; then
break break
fi fi
sleep 5s sleep 5s
@@ -94,13 +104,15 @@ function validate_services() {
} }
function validate_microservices() { function validate_microservices() {
local llm_container_name="$1"
# tgi for llm service # tgi for llm service
validate_services \ validate_services \
"${ip_address}:8028/generate" \ "${ip_address}:8028/v1/chat/completions" \
"generated_text" \ "completion_tokens" \
"tgi-llm" \ "llm-service" \
"tgi-gaudi-server" \ "${llm_container_name}" \
'{"inputs":"def print_hello_world():","parameters":{"max_new_tokens":256, "do_sample": true}}' '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "def print_hello_world():"}], "max_tokens": 256}'
# llm microservice # llm microservice
validate_services \ validate_services \
@@ -152,24 +164,50 @@ function validate_frontend() {
} }
function stop_docker() { function stop_docker() {
local docker_profile="$1"
cd $WORKPATH/docker_compose/intel/hpu/gaudi cd $WORKPATH/docker_compose/intel/hpu/gaudi
docker compose stop && docker compose rm -f docker compose --profile ${docker_profile} down
} }
function main() { function main() {
# all docker docker compose profiles for XEON Platform
docker_compose_profiles=("codegen-gaudi-vllm" "codegen-gaudi-tgi")
docker_llm_container_names=("vllm-gaudi-server" "tgi-gaudi-server")
stop_docker # get number of profiels and container
len_profiles=${#docker_compose_profiles[@]}
len_containers=${#docker_llm_container_names[@]}
# number of profiels and docker container names must be matched
if [ ${len_profiles} -ne ${len_containers} ]; then
echo "Error: number of profiles ${len_profiles} and container names ${len_containers} mismatched"
exit 1
fi
# stop_docker, stop all profiles
for ((i = 0; i < len_profiles; i++)); do
stop_docker "${docker_compose_profiles[${i}]}"
done
# build docker images
if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi
start_services
validate_microservices # loop all profiles
for ((i = 0; i < len_profiles; i++)); do
echo "Process [${i}]: ${docker_compose_profiles[$i]}, ${docker_llm_container_names[${i}]}"
start_services "${docker_compose_profiles[${i}]}" "${docker_llm_container_names[${i}]}"
docker ps -a
validate_microservices "${docker_llm_container_names[${i}]}"
validate_megaservice validate_megaservice
validate_frontend validate_frontend
stop_docker stop_docker "${docker_compose_profiles[${i}]}"
echo y | docker system prune sleep 5s
done
echo y | docker system prune
} }
main main

75
CodeGen/tests/test_compose_on_xeon.sh
View File

@@ -31,8 +31,14 @@ function build_docker_images() {
cd $WORKPATH/docker_image_build cd $WORKPATH/docker_image_build
git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git git clone --depth 1 --branch ${opea_branch} https://github.com/opea-project/GenAIComps.git
git clone https://github.com/vllm-project/vllm.git && cd vllm
VLLM_VER="$(git describe --tags "$(git rev-list --tags --max-count=1)" )"
echo "Check out vLLM tag ${VLLM_VER}"
git checkout ${VLLM_VER} &> /dev/null
cd ../
echo "Build all the images with --no-cache, check docker_image_build.log for details..." echo "Build all the images with --no-cache, check docker_image_build.log for details..."
service_list="codegen codegen-ui llm-textgen" service_list="codegen codegen-ui llm-textgen vllm"
docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log docker compose -f build.yaml build ${service_list} --no-cache > ${LOG_PATH}/docker_image_build.log
docker pull ghcr.io/huggingface/text-generation-inference:2.4.0-intel-cpu docker pull ghcr.io/huggingface/text-generation-inference:2.4.0-intel-cpu
@@ -40,10 +46,13 @@ function build_docker_images() {
} }
function start_services() { function start_services() {
local compose_profile="$1"
local llm_container_name="$2"
cd $WORKPATH/docker_compose/intel/cpu/xeon/ cd $WORKPATH/docker_compose/intel/cpu/xeon/
export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3" export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${ip_address}:8028" export LLM_ENDPOINT="http://${ip_address}:8028"
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
export MEGA_SERVICE_HOST_IP=${ip_address} export MEGA_SERVICE_HOST_IP=${ip_address}
export LLM_SERVICE_HOST_IP=${ip_address} export LLM_SERVICE_HOST_IP=${ip_address}
@@ -53,12 +62,12 @@ function start_services() {
sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env sed -i "s/backend_address/$ip_address/g" $WORKPATH/ui/svelte/.env
# Start Docker Containers # Start Docker Containers
docker compose up -d > ${LOG_PATH}/start_services_with_compose.log docker compose --profile ${compose_profile} up -d > ${LOG_PATH}/start_services_with_compose.log
n=0 n=0
until [[ "$n" -ge 100 ]]; do until [[ "$n" -ge 100 ]]; do
docker logs tgi-service > ${LOG_PATH}/tgi_service_start.log docker logs ${llm_container_name} > ${LOG_PATH}/llm_service_start.log 2>&1
if grep -q Connected ${LOG_PATH}/tgi_service_start.log; then if grep -E "Connected|complete" ${LOG_PATH}/llm_service_start.log; then
break break
fi fi
sleep 5s sleep 5s
@@ -95,13 +104,15 @@ function validate_services() {
} }
function validate_microservices() { function validate_microservices() {
local llm_container_name="$1"
# tgi for llm service # tgi for llm service
validate_services \ validate_services \
"${ip_address}:8028/generate" \ "${ip_address}:8028/v1/chat/completions" \
"generated_text" \ "completion_tokens" \
"tgi-llm" \ "llm-service" \
"tgi-service" \ "${llm_container_name}" \
'{"inputs":"def print_hello_world():","parameters":{"max_new_tokens":256, "do_sample": true}}' '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens": 256}'
# llm microservice # llm microservice
validate_services \ validate_services \
@@ -109,7 +120,7 @@ function validate_microservices() {
"data: " \ "data: " \
"llm" \ "llm" \
"llm-textgen-server" \ "llm-textgen-server" \
'{"query":"def print_hello_world():"}' '{"query":"def print_hello_world():", "max_tokens": 256}'
} }
@@ -120,7 +131,7 @@ function validate_megaservice() {
"print" \ "print" \
"mega-codegen" \ "mega-codegen" \
"codegen-xeon-backend-server" \ "codegen-xeon-backend-server" \
'{"messages": "def print_hello_world():"}' '{"messages": "def print_hello_world():", "max_tokens": 256}'
} }
@@ -154,24 +165,50 @@ function validate_frontend() {
function stop_docker() { function stop_docker() {
local docker_profile="$1"
cd $WORKPATH/docker_compose/intel/cpu/xeon/ cd $WORKPATH/docker_compose/intel/cpu/xeon/
docker compose stop && docker compose rm -f docker compose --profile ${docker_profile} down
} }
function main() { function main() {
# all docker docker compose profiles for Xeon Platform
docker_compose_profiles=("codegen-xeon-tgi" "codegen-xeon-vllm")
docker_llm_container_names=("tgi-server" "vllm-server")
stop_docker # get number of profiels and LLM docker container names
len_profiles=${#docker_compose_profiles[@]}
len_containers=${#docker_llm_container_names[@]}
# number of profiels and docker container names must be matched
if [ ${len_profiles} -ne ${len_containers} ]; then
echo "Error: number of profiles ${len_profiles} and container names ${len_containers} mismatched"
exit 1
fi
# stop_docker, stop all profiles
for ((i = 0; i < len_profiles; i++)); do
stop_docker "${docker_compose_profiles[${i}]}"
done
# build docker images
if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi if [[ "$IMAGE_REPO" == "opea" ]]; then build_docker_images; fi
start_services
validate_microservices # loop all profiles
for ((i = 0; i < len_profiles; i++)); do
echo "Process [${i}]: ${docker_compose_profiles[$i]}, ${docker_llm_container_names[${i}]}"
docker ps -a
start_services "${docker_compose_profiles[${i}]}" "${docker_llm_container_names[${i}]}"
validate_microservices "${docker_llm_container_names[${i}]}"
validate_megaservice validate_megaservice
validate_frontend validate_frontend
stop_docker stop_docker "${docker_compose_profiles[${i}]}"
echo y | docker system prune sleep 5s
done
echo y | docker system prune
} }
main main