GenAIExamples/CodeGen/microservice/gaudi/README.md

Build MegaService of CodeGen on Gaudi

This document outlines the deployment process for a CodeGen application utilizing the GenAIComps microservice pipeline on Intel Gaudi server. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as llm. We will publish the Docker images to Docker Hub, it will simplify the deployment process for this service.

🚀 Build Docker Images

First of all, you need to build Docker Images locally. This step can be ignored after the Docker images published to Docker hub.

1. Git clone GenAIComps

git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps

2. Build LLM Image

docker build -t opea/gen-ai-comps:llm-tgi-gaudi-server --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/langchain/docker/Dockerfile .

3. Build MegaService Docker Image

To construct the Mega Service, we utilize the GenAIComps microservice pipeline within the codegen.py Python script. Build the MegaService Docker image using the command below:

git clone https://github.com/opea-project/GenAIExamples
cd GenAIExamples/CodeGen/microservice/gaudi/
docker build -t opea/gen-ai-comps:codegen-megaservice-server --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f docker/Dockerfile .

4. Build UI Docker Image

Construct the frontend Docker image using the command below:

cd GenAIExamples/CodeGen/ui/
docker build -t opea/gen-ai-comps:codegen-ui-server --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .

Then run the command docker images, you will have the following 3 Docker Images:

1. opea/gen-ai-comps:llm-tgi-server
2. opea/gen-ai-comps:codegen-megaservice-server
3. opea/gen-ai-comps:codegen-ui-server

🚀 Start MicroServices and MegaService

Setup Environment Variables

Since the docker_compose.yaml will consume some environment variables, you need to setup them in advance as below.

export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export LLM_MODEL_ID="ise-uiuc/Magicoder-S-DS-6.7B"
export TGI_LLM_ENDPOINT="http://${host_ip}:8028"
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
export MEGA_SERVICE_HOST_IP=${host_ip}
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:6666/v1/codegen"

Note: Please replace with host_ip with you external IP address, do not use localhost.

Start all the services Docker Containers

docker compose -f docker_compose.yaml up -d

Validate MicroServices and MegaService

1. TGI Service

curl http://${host_ip}:8028/generate \
  -X POST \
  -d '{"inputs":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","parameters":{"max_new_tokens":256, "do_sample": true}}' \
  -H 'Content-Type: application/json'

2. LLM Microservice

curl http://${host_ip}:9000/v1/chat/completions\
  -X POST \
  -d '{"query":"Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception.","max_new_tokens":256,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"streaming":true}' \
  -H 'Content-Type: application/json'

3. MegaService

curl http://${host_ip}:6666/v1/codegen -H "Content-Type: application/json" -d '{
     "model": "ise-uiuc/Magicoder-S-DS-6.7B",
     "messages": "Implement a high-level API for a TODO list application. The API takes as input an operation request and updates the TODO list in place. If the request is invalid, raise an exception."
     }'

🚀 Launch the UI

To access the frontend, open the following URL in your browser: http://{host_ip}:5173. By default, the UI runs on port 5173 internally. If you prefer to use a different host port to access the frontend, you can modify the port mapping in the docker_compose.yaml file as shown below:

  chaqna-gaudi-ui-server:
    image: opea/gen-ai-comps:codegen-ui-server
    ...
    ports:
      - "80:5173"

Install Copilot VSCode extension from Plugin Marketplace as the frontend

In addition to the Svelte UI, users can also install the Copilot VSCode extension from the Plugin Marketplace as the frontend.

Install Neural Copilot in VSCode as below.

How to use

Service URL setting

Please adjust the service URL in the extension settings based on the endpoint of the code generation backend service.

Customize

The Copilot enables users to input their corresponding sensitive information and tokens in the user settings according to their own needs. This customization enhances the accuracy and output content to better meet individual requirements.

Code Suggestion

To trigger inline completion, you'll need to type # {your keyword} (start with your programming language's comment keyword, like // in C++ and # in python). Make sure Inline Suggest is enabled from the VS Code Settings. For example:

To provide programmers with a smooth experience, the Copilot supports multiple ways to trigger inline code suggestions. If you are interested in the details, they are summarized as follows:

• Generate code from single-line comments: The simplest way introduced before.
• Generate code from consecutive single-line comments:

• Generate code from multi-line comments, which will not be triggered until there is at least one space outside the multi-line comment):

• Automatically complete multi-line comments:

Chat with AI assistant

You can start a conversation with the AI programming assistant by clicking on the robot icon in the plugin bar on the left:

Then you can see the conversation window on the left, where you can chat with AI assistant:

There are 4 areas worth noting:

• Enter and submit your question
• Your previous questions
• Answers from AI assistant (Code will be highlighted properly according to the programming language it is written in, also support streaming output)
• Copy or replace code with one click (Note that you need to select the code in the editor first and then click "replace", otherwise the code will be inserted)

You can also select the code in the editor and ask AI assistant question about it. For example:

• Select code

• Ask question and get answer