113 lines
4.2 KiB
Markdown
Executable File
113 lines
4.2 KiB
Markdown
Executable File
# Build Mega Service of CodeTrans on Gaudi
|
|
|
|
This document outlines the deployment process for a CodeTrans application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on Intel Gaudi server. The steps include Docker image creation, container deployment via Docker Compose, and service execution using microservices `llm`. We will publish the Docker images to Docker Hub soon, it will simplify the deployment process for this service.
|
|
|
|
## 🚀 Build Docker Images
|
|
|
|
First of all, you need to build Docker Images locally and install the python package of it. This step can be ignored after the Docker images published to Docker hub.
|
|
|
|
### 1. Source Code install GenAIComps
|
|
|
|
```bash
|
|
git clone https://github.com/opea-project/GenAIComps.git
|
|
cd GenAIComps
|
|
```
|
|
|
|
### 2. Build the LLM Docker Image with the following command
|
|
|
|
```bash
|
|
docker build -t opea/llm-tgi:latest --no-cache --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/text-generation/tgi/Dockerfile .
|
|
```
|
|
|
|
### 3. Build MegaService Docker Image
|
|
|
|
```bash
|
|
git clone https://github.com/opea-project/GenAIExamples.git
|
|
cd GenAIExamples/CodeTrans/docker
|
|
docker build -t opea/codetrans:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
|
|
```
|
|
|
|
### 4. Build UI Docker Image
|
|
|
|
```bash
|
|
cd GenAIExamples/CodeTrans/ui
|
|
docker build -t opea/codetrans-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
|
|
```
|
|
|
|
Then run the command `docker images`, you will have the following Docker Images:
|
|
|
|
- `opea/llm-tgi:latest`
|
|
- `opea/codetrans:latest`
|
|
- `opea/codetrans-ui:latest`
|
|
|
|
## 🚀 Start Microservices
|
|
|
|
### Setup Environment Variables
|
|
|
|
Since the `docker_compose.yaml` will consume some environment variables, you need to setup them in advance as below. Notice that the `LLM_MODEL_ID` indicates the LLM model used for TGI service.
|
|
|
|
```bash
|
|
export http_proxy=${your_http_proxy}
|
|
export https_proxy=${your_http_proxy}
|
|
export LLM_MODEL_ID="HuggingFaceH4/mistral-7b-grok"
|
|
export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
|
|
export HF_TOKEN=${your_hf_api_token}
|
|
export MEGA_SERVICE_HOST_IP=${host_ip}
|
|
export LLM_SERVICE_HOST_IP=${host_ip}
|
|
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:7777/v1/codetrans"
|
|
```
|
|
|
|
### Start Microservice Docker Containers
|
|
|
|
```bash
|
|
cd GenAIExamples/CodeTrans/docker-composer/gaudi
|
|
docker compose -f docker_compose.yaml up -d
|
|
```
|
|
|
|
### Validate Microservices
|
|
|
|
1. TGI Service
|
|
|
|
```bash
|
|
curl http://${host_ip}:8008/generate \
|
|
-X POST \
|
|
-d '{"inputs":" ### System: Please translate the following Golang codes into Python codes. ### Original codes: '\'''\'''\''Golang \npackage main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n '\'''\'''\'' ### Translated codes:","parameters":{"max_new_tokens":17, "do_sample": true}}' \
|
|
-H 'Content-Type: application/json'
|
|
```
|
|
|
|
2. LLM Microservice
|
|
|
|
```bash
|
|
curl http://${host_ip}:9000/v1/chat/completions\
|
|
-X POST \
|
|
-d '{"text":" ### System: Please translate the following Golang codes into Python codes. ### Original codes: '\'''\'''\''Golang \npackage main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n '\'''\'''\'' ### Translated codes:"}' \
|
|
-H 'Content-Type: application/json'
|
|
```
|
|
|
|
3. MegaService
|
|
|
|
```bash
|
|
curl http://${host_ip}:7777/v1/codetrans \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"language_from": "Golang","language_to": "Python","source_code": "package main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n}"}'
|
|
```
|
|
|
|
## Enable LangSmith to Monitor an Application (Optional)
|
|
|
|
LangSmith offers tools to debug, evaluate, and monitor language models and intelligent agents. It can be used to assess benchmark data for each microservice. Before launching your services with `docker compose -f docker_compose.yaml up -d`, you need to enable LangSmith tracing by setting the `LANGCHAIN_TRACING_V2` environment variable to true and configuring your LangChain API key.
|
|
|
|
Here's how you can do it:
|
|
|
|
1. Install the latest version of LangSmith:
|
|
|
|
```bash
|
|
pip install -U langsmith
|
|
```
|
|
|
|
2. Set the necessary environment variables:
|
|
|
|
```bash
|
|
export LANGCHAIN_TRACING_V2=true
|
|
export LANGCHAIN_API_KEY=ls_...
|
|
```
|