Add code translation example (#81)

* add example of Code Translation

Signed-off-by: LetongHan <letong.han@intel.com>

CodeTrans/README.md

@@ -0,0 +1,42 @@
+# Code Translation
+
+Code translation is the process of converting code written in one programming language to another programming language while maintaining the same functionality. This process is also known as code conversion, source-to-source translation, or transpilation. Code translation is often performed when developers want to take advantage of new programming languages, improve code performance, or maintain legacy systems. Some common examples include translating code from Python to Java, or from JavaScript to TypeScript.
+
+The workflow falls into the following architecture:
+
+![architecture](https://i.imgur.com/ums0brC.png)
+
+# Start Backend Service
+
+1. Start the TGI service to deploy your LLM
+
+```sh
+cd serving/tgi_gaudi
+bash build_docker.sh
+bash launch_tgi_service.sh
+```
+
+`launch_tgi_service.sh` by default uses `8080` as the TGI service's port. Please replace it if there are any port conflicts.
+
+2. Start the CodeTranslation service
+
+```sh
+cd langchain/docker
+bash build_docker.sh
+docker run -it --name code_trans_server --net=host --ipc=host -e TGI_ENDPOINT=${TGI ENDPOINT} -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACE_API_TOKEN} -e SERVER_PORT=8000 -e http_proxy=${http_proxy} -e https_proxy=${https_proxy} code_tranlation:latest bash
+```
+
+Here is the explanation of some of the above parameters:
+
+- `TGI_ENDPOINT`: The endpoint of your TGI service, usually equal to `<ip of your machine>:<port of your TGI service>`.
+- `HUGGINGFACEHUB_API_TOKEN`: Your HuggingFace hub API token, usually generated [here](https://huggingface.co/settings/tokens).
+- `SERVER_PORT`: The port of the CodeTranslation service on the host.
+
+3. Quick test
+
+```sh
+curl http://localhost:8000/v1/code_translation \
+    -X POST \
+    -d '{"language_from": "Python","language_to": "Java","source_code": "\ndef hello(name):\n    print(\"Hello, \" + name)\n"}' \
+    -H 'Content-Type: application/json'
+```

CodeTrans/langchain/docker/Dockerfile

@@ -0,0 +1,50 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# SCRIPT USAGE NOTICE:  By downloading and using any script file included
+# with the associated software package (such as files with .bat, .cmd, or 
+# .JS extensions, Docker files, or any other type of file that, when executed,
+# automatically downloads and/or installs files onto your system) (the “Script File”),
+# it is your obligation to review the Script File to understand what files (e.g.,
+# other software, AI models, AI Datasets) the Script File will download to your system
+# (“Downloaded Files”). Furthermore, by downloading and using the Downloaded Files,
+# even if they are installed through a silent install, you agree to any and all
+# terms and conditions associated with such files, including but not limited to,
+# license terms, notices, or disclaimers.
+
+FROM intel/intel-optimized-pytorch:2.2.0-pip-jupyter
+
+RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
+    libgl1-mesa-glx \
+    libjemalloc-dev \
+    vim
+
+RUN useradd -m -s /bin/bash user && \
+    mkdir -p /home/user && \
+    chown -R user /home/user/
+
+USER user
+
+COPY requirements.txt /tmp/requirements.txt
+
+RUN pip install --no-cache-dir --upgrade pip && \
+    pip install --no-cache-dir -r /tmp/requirements.txt
+
+ENV PYTHONPATH=/home/user:/home/user/codetrans-app/app
+
+WORKDIR /home/user/codetrans-app
+COPY --chown=user:user codetrans-app /home/user/codetrans-app
+
+ENTRYPOINT ["python", "server.py"]

CodeTrans/langchain/docker/build_docker.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+docker build . -t code_tranlation:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy

CodeTrans/langchain/docker/codetrans-app/prompts.py

@@ -0,0 +1,11 @@
+from langchain.prompts import PromptTemplate
+
+prompt_template = """
+    ### System: Please translate the following {language_from} codes into {language_to} codes.
+
+    ### Original codes:
+    {source_code}
+
+    ### Translated codes:
+"""
+codetrans_prompt_template = PromptTemplate.from_template(prompt_template)

CodeTrans/langchain/docker/codetrans-app/server.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from fastapi import APIRouter, FastAPI, HTTPException, Request
+from fastapi.responses import StreamingResponse
+from langchain_community.llms import HuggingFaceEndpoint
+from prompts import codetrans_prompt_template
+from starlette.middleware.cors import CORSMiddleware
+
+app = FastAPI()
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+TGI_ENDPOINT = os.getenv("TGI_ENDPOINT", "http://localhost:8080")
+SERVICE_PORT = os.getenv("SERVER_PORT", 8000)
+
+
+class CodeTranslationAPIRouter(APIRouter):
+    """The router for CodeTranslation example."""
+
+    def __init__(self, entrypoint: str, prompt_template: str) -> None:
+        super().__init__()
+        self.entrypoint = entrypoint
+
+        # setup TGI endpoint
+        self.llm = HuggingFaceEndpoint(
+            endpoint_url=entrypoint,
+            max_new_tokens=1024,
+            top_k=10,
+            top_p=0.95,
+            typical_p=0.95,
+            temperature=0.01,
+            repetition_penalty=1.03,
+            streaming=True,
+        )
+
+        self.prompt_template = prompt_template
+
+    def handle_code_translation(self, language_from: str, language_to: str, source_code: str):
+        prompt = self.prompt_template.format(
+            language_from=language_from, language_to=language_to, source_code=source_code
+        )
+        print(f"[codetrans - nonstream] prompt:{prompt}")
+        try:
+            response = self.llm(prompt)
+        except Exception as e:
+            print(f"[codetrans - nonstream] Error occurred: {e}")
+            raise Exception(f"[codetrans - nonstream] {e}")
+        print(f"[codetrans - nonstream] response:\n{response}")
+        return response
+
+    async def handle_code_translation_stream(self, language_from: str, language_to: str, source_code: str):
+        prompt = self.prompt_template.format(
+            language_from=language_from, language_to=language_to, source_code=source_code
+        )
+        print(f"[codetrans - stream] prompt:{prompt}")
+
+        async def stream_generator():
+            async for chunk in self.llm.astream_log(prompt):
+                print(f"[codetrans - stream] data: {chunk}")
+                yield f"data: {chunk}\n\n"
+            yield "data: [DONE]\n\n"
+
+        return StreamingResponse(stream_generator(), media_type="text/event-stream")
+
+
+router = CodeTranslationAPIRouter(entrypoint=TGI_ENDPOINT, prompt_template=codetrans_prompt_template)
+
+
+@router.post("/v1/code_translation")
+async def code_translation(request: Request):
+    params = await request.json()
+    print(f"[codetrans - nonstream] POST request: /v1/code_translation, params:{params}")
+    language_from = params["language_from"]
+    language_to = params["language_to"]
+    source_code = params["source_code"]
+    try:
+        return router.handle_code_translation(
+            language_from=language_from, language_to=language_to, source_code=source_code
+        )
+    except Exception as e:
+        print(f"[codetrans - nonstream] Error occurred: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+@router.post("/v1/code_translation_stream")
+async def code_translation_stream(request: Request):
+    params = await request.json()
+    print(f"[codetrans - stream] POST request: /v1/code_translation_stream, params:{params}")
+    language_from = params["language_from"]
+    language_to = params["language_to"]
+    source_code = params["source_code"]
+    try:
+        return await router.handle_code_translation_stream(
+            language_from=language_from, language_to=language_to, source_code=source_code
+        )
+    except Exception as e:
+        print(f"[codetrans - stream] Error occurred: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+app.include_router(router)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=int(SERVICE_PORT))

CodeTrans/langchain/docker/requirements.txt

@@ -0,0 +1,4 @@
+fastapi
+huggingface_hub
+langchain
+uvicorn

CodeTrans/serving/tgi_gaudi/build_docker.sh

@@ -0,0 +1,19 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/bin/bash
+
+git clone https://github.com/huggingface/tgi-gaudi.git
+cd ./tgi-gaudi/
+docker build -t ghcr.io/huggingface/tgi-gaudi:1.2.1 . --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy

CodeTrans/serving/tgi_gaudi/launch_tgi_service.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Set default values
+default_port=8080
+default_model="HuggingFaceH4/mistral-7b-grok"
+default_num_cards=1
+
+# Check if all required arguments are provided
+if [ "$#" -lt 0 ] || [ "$#" -gt 3 ]; then
+    echo "Usage: $0 [num_cards] [port_number] [model_name]"
+    exit 1
+fi
+
+# Assign arguments to variables
+num_cards=${1:-$default_num_cards}
+port_number=${2:-$default_port}
+model_name=${3:-$default_model}
+
+# Check if num_cards is within the valid range (1-8)
+if [ "$num_cards" -lt 1 ] || [ "$num_cards" -gt 8 ]; then
+    echo "Error: num_cards must be between 1 and 8."
+    exit 1
+fi
+
+# Set the volume variable
+volume=$PWD/data
+
+# Build the Docker run command based on the number of cards
+if [ "$num_cards" -eq 1 ]; then
+    docker_cmd="docker run -d --name tgi-gaudi-server -p $port_number:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/tgi-gaudi:1.2.1 --model-id $model_name"
+else
+    docker_cmd="docker run -d --name tgi-gaudi-server -p $port_number:80 -v $volume:/data --runtime=habana -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/tgi-gaudi:1.2.1 --model-id $model_name --sharded true --num-shard $num_cards"
+fi
+
+# Execute the Docker run command
+eval $docker_cmd