jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Enable CodeGen vLLM (#1636)

Browse Source 
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
xiguiw
2025-03-13 10:38:47 +08:00
committed by GitHub
parent adcd113f53
commit effa2a28cf
8 changed files with 421 additions and 196 deletions
Download Patch File Download Diff File Expand all files Collapse all files

152
CodeGen/docker_compose/intel/cpu/xeon/README.md
View File

@@ -1,6 +1,7 @@
# Build MegaService of CodeGen on Xeon
This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Xeon server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to Docker Hub soon, further simplifying the deployment process for this service.
The default pipeline deploys with vLLM as the LLM serving component. It also provides options of using TGI backend for LLM microservice.
## 🚀 Create an AWS Xeon Instance
@@ -10,55 +11,6 @@ For detailed information about these instance types, you can refer to [m7i](http
After launching your instance, you can connect to it using SSH (for Linux instances) or Remote Desktop Protocol (RDP) (for Windows instances). From there, you'll have full access to your Xeon server, allowing you to install, configure, and manage your applications as needed.
## 🚀 Download or Build Docker Images
Should the Docker image you seek not yet be available on Docker Hub, you can build the Docker image locally.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Build the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker Images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest` (optional)
## 🚀 Start Microservices and MegaService
The CodeGen megaservice manages a single microservice called LLM within a Directed Acyclic Graph (DAG). In the diagram above, the LLM microservice is a language model microservice that generates code snippets based on the user's input query. The TGI service serves as a text generation interface, providing a RESTful API for the LLM microservice. The CodeGen Gateway acts as the entry point for the CodeGen application, invoking the Megaservice to generate code snippets in response to the user's input query.
@@ -89,42 +41,57 @@ flowchart LR
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
**Append the value of the public IP address to the no_proxy list**
1. set the host_ip and huggingface token
> Note:
> Please replace the `your_ip_address` with you external IP address, do not use `localhost`.
```bash
export host_ip=${your_ip_address}
export HUGGINGFACEHUB_API_TOKEN=you_huggingface_token
```
export your_no_proxy=${your_no_proxy},"External_Public_IP"
```
2. Set Netowork Proxy
**If you access public network through proxy, set the network proxy, otherwise, skip this step**
```bash
export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
export MEGA_SERVICE_HOST_IP=${host_ip}
export LLM_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
export https_proxy=${your_https_proxy}
```
Note: Please replace the `host_ip` with you external IP address, do not use `localhost`.
### Start the Docker Containers for All Services
CodeGen support TGI service and vLLM service, you can choose start either one of them.
Start CodeGen based on TGI service:
```bash
cd GenAIExamples/CodeGen/docker_compose/intel/cpu/xeon
docker compose up -d
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/cpu/xeon
docker compose --profile codegen-xeon-tgi up -d
```
Start CodeGen based on vLLM service:
```bash
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/cpu/xeon
docker compose --profile codegen-xeon-vllm up -d
```
### Validate the MicroServices and MegaService
1. TGI Service
1. LLM Service (for TGI, vLLM)
```bash
curl http://${host_ip}:8028/generate \
-X POST \
-d '{"inputs":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","parameters":{"max_new_tokens":256, "do_sample": true}}' \
-H 'Content-Type: application/json'
curl http://${host_ip}:8028/v1/chat/completions \
-X POST \
-d '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}], "max_tokens":32}' \
-H 'Content-Type: application/json'
```
2. LLM Microservices
@@ -257,3 +224,52 @@ For example:
- Ask question and get answer
![qna](../../../../assets/img/codegen_qna.png)
## 🚀 Download or Build Docker Images
Should the Docker image you seek not yet be available on Docker Hub, you can build the Docker image locally.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Build the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker Images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest` (optional)

62
CodeGen/docker_compose/intel/cpu/xeon/compose.yaml
View File

@@ -4,7 +4,9 @@
services:
tgi-service:
image: ghcr.io/huggingface/text-generation-inference:2.4.0-intel-cpu
container_name: tgi-service
container_name: tgi-server
profiles:
- codegen-xeon-tgi
ports:
- "8028:80"
volumes:
@@ -22,28 +24,66 @@ services:
timeout: 10s
retries: 100
command: --model-id ${LLM_MODEL_ID} --cuda-graphs 0
llm:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-server
depends_on:
tgi-service:
condition: service_healthy
vllm-service:
image: ${REGISTRY:-opea}/vllm:${TAG:-latest}
container_name: vllm-server
profiles:
- codegen-xeon-vllm
ports:
- "9000:9000"
ipc: host
- "8028:80"
volumes:
- "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub"
shm_size: 1g
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
host_ip: ${host_ip}
healthcheck:
test: ["CMD-SHELL", "curl -f http://$host_ip:8028/health || exit 1"]
interval: 10s
timeout: 10s
retries: 100
command: --model ${LLM_MODEL_ID} --host 0.0.0.0 --port 80
llm-base:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-server
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${LLM_ENDPOINT}
LLM_MODEL_ID: ${LLM_MODEL_ID}
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
restart: unless-stopped
llm-tgi-service:
extends: llm-base
container_name: llm-codegen-tgi-server
profiles:
- codegen-xeon-tgi
ports:
- "9000:9000"
ipc: host
depends_on:
tgi-service:
condition: service_healthy
llm-vllm-service:
extends: llm-base
container_name: llm-codegen-vllm-server
profiles:
- codegen-xeon-vllm
ports:
- "9000:9000"
ipc: host
depends_on:
vllm-service:
condition: service_healthy
codegen-xeon-backend-server:
image: ${REGISTRY:-opea}/codegen:${TAG:-latest}
container_name: codegen-xeon-backend-server
depends_on:
- llm
- llm-base
ports:
- "7778:7778"
environment:

152
CodeGen/docker_compose/intel/hpu/gaudi/README.md
View File

@@ -2,54 +2,7 @@
This document outlines the deployment process for a CodeGen application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Gaudi2 server. The steps include Docker images creation, container deployment via Docker Compose, and service execution to integrate microservices such as `llm`. We will publish the Docker images to the Docker Hub soon, further simplifying the deployment process for this service.
## 🚀 Build Docker Images
First of all, you need to build the Docker images locally. This step can be ignored after the Docker images published to the Docker Hub.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build the MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Construct the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest`
The default pipeline deploys with vLLM as the LLM serving component. It also provides options of using TGI backend for LLM microservice.
## 🚀 Start MicroServices and MegaService
@@ -81,37 +34,57 @@ flowchart LR
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
1. set the host_ip and huggingface token
> [!NOTE]
> Please replace the `your_ip_address` with you external IP address, do not use `localhost`.
```bash
export host_ip=${your_ip_address}
export HUGGINGFACEHUB_API_TOKEN=you_huggingface_token
```
2. Set Netowork Proxy
**If you access public network through proxy, set the network proxy, otherwise, skip this step**
```bash
export no_proxy=${your_no_proxy}
export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
export MEGA_SERVICE_HOST_IP=${host_ip}
export LLM_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7778/v1/codegen"
export https_proxy=${your_https_proxy}
```
> [!NOTE]
> Please replace the `host_ip` with you external IP address, do not use `localhost`.
### Start the Docker Containers for All Services
CodeGen support TGI service and vLLM service, you can choose start either one of them.
Start CodeGen based on TGI service:
```bash
cd GenAIExamples/CodeGen/docker_compose/intel/hpu/gaudi
docker compose up -d
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/hpu/gaudi
docker compose --profile codegen-gaudi-tgi up -d
```
Start CodeGen based on vLLM service:
```bash
cd GenAIExamples/CodeGen/docker_compose
source set_env.sh
cd intel/hpu/gaudi
docker compose --profile codegen-gaudi-vllm up -d
```
### Validate the MicroServices and MegaService
1. TGI Service
1. LLM Service (for TGI, vLLM)
```bash
curl http://${host_ip}:8028/generate \
-X POST \
-d '{"inputs":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","parameters":{"max_new_tokens":256, "do_sample": true}}' \
-H 'Content-Type: application/json'
curl http://${host_ip}:8028/v1/chat/completions \
-X POST \
-d '{"model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [{"role": "user", "content": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."}], "max_tokens":32}' \
-H 'Content-Type: application/json'
```
2. LLM Microservices
@@ -240,3 +213,52 @@ For example:
- Ask question and get answer
![qna](../../../../assets/img/codegen_qna.png)
## 🚀 Build Docker Images
First of all, you need to build the Docker images locally. This step can be ignored after the Docker images published to the Docker Hub.
### 1. Build the LLM Docker Image
```bash
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
docker build -t opea/llm-textgen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/src/text-generation/Dockerfile .
```
### 2. Build the MegaService Docker Image
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `codegen.py` Python script. Build the MegaService Docker image via the command below:
```bash
git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen
docker build -t opea/codegen:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```
### 3. Build the UI Docker Image
Construct the frontend Docker image via the command below:
```bash
cd GenAIExamples/CodeGen/ui
docker build -t opea/codegen-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
```
### 4. Build CodeGen React UI Docker Image (Optional)
Build react frontend Docker image via below command:
**Export the value of the public IP address of your Xeon server to the `host_ip` environment variable**
```bash
cd GenAIExamples/CodeGen/ui
docker build --no-cache -t opea/codegen-react-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .
```
Then run the command `docker images`, you will have the following Docker images:
- `opea/llm-textgen:latest`
- `opea/codegen:latest`
- `opea/codegen-ui:latest`
- `opea/codegen-react-ui:latest`

68
CodeGen/docker_compose/intel/hpu/gaudi/compose.yaml
View File

@@ -5,6 +5,8 @@ services:
tgi-service:
image: ghcr.io/huggingface/tgi-gaudi:2.3.1
container_name: tgi-gaudi-server
profiles:
- codegen-gaudi-tgi
ports:
- "8028:80"
volumes:
@@ -30,28 +32,74 @@ services:
- SYS_NICE
ipc: host
command: --model-id ${LLM_MODEL_ID} --max-input-length 1024 --max-total-tokens 2048
llm:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-gaudi-server
depends_on:
tgi-service:
condition: service_healthy
vllm-service:
image: ${REGISTRY:-opea}/vllm-gaudi:${TAG:-latest}
container_name: vllm-gaudi-server
profiles:
- codegen-gaudi-vllm
ports:
- "9000:9000"
ipc: host
- "8028:80"
volumes:
- "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub"
shm_size: 1g
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
HABANA_VISIBLE_DEVICES: all
OMPI_MCA_btl_vader_single_copy_mechanism: none
VLLM_SKIP_WARMUP: ${VLLM_SKIP_WARMUP:-false}
NUM_CARDS: ${NUM_CARDS:-1}
VLLM_TORCH_PROFILER_DIR: "/mnt"
healthcheck:
test: ["CMD-SHELL", "curl -f http://$host_ip:8028/health || exit 1"]
interval: 10s
timeout: 10s
retries: 100
runtime: habana
cap_add:
- SYS_NICE
ipc: host
command: --model ${LLM_MODEL_ID} --tensor-parallel-size ${NUM_CARDS} --host 0.0.0.0 --port 80 --block-size 128 --max-num-seqs 256
llm-base:
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
container_name: llm-textgen-gaudi-server
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
LLM_ENDPOINT: ${LLM_ENDPOINT}
LLM_MODEL_ID: ${LLM_MODEL_ID}
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
restart: unless-stopped
llm-tgi-service:
extends: llm-base
container_name: llm-codegen-tgi-gaudi-server
profiles:
- codegen-gaudi-tgi
ports:
- "9000:9000"
ipc: host
depends_on:
tgi-service:
condition: service_healthy
llm-vllm-service:
extends: llm-base
container_name: llm-codegen-gaudi-vllm-server
profiles:
- codegen-gaudi-vllm
ports:
- "9000:9000"
ipc: host
depends_on:
vllm-service:
condition: service_healthy
codegen-gaudi-backend-server:
image: ${REGISTRY:-opea}/codegen:${TAG:-latest}
container_name: codegen-gaudi-backend-server
depends_on:
- llm
- llm-base
ports:
- "7778:7778"
environment: