jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Update path for finetuning (#1306)

Browse Source 
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
This commit is contained in:
XinyuYe-Intel
2025-01-09 12:01:59 +08:00
committed by GitHub
parent 370d6928c1
commit 113281d073
10 changed files with 11 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
InstructionTuning/README.md
View File

@@ -38,7 +38,7 @@ curl http://${your_ip}:8015/v1/fine_tuning/jobs \
}' }'
``` ```
The outputs of the finetune job (adapter_model.safetensors, adapter_config,json... ) are stored in `/home/user/comps/finetuning/output` and other execution logs are stored in `/home/user/ray_results` The outputs of the finetune job (adapter_model.safetensors, adapter_config,json... ) are stored in `/home/user/comps/finetuning/src/output` and other execution logs are stored in `/home/user/ray_results`
### 3. Manage fine-tuning job ### 3. Manage fine-tuning job

2
InstructionTuning/docker_compose/intel/cpu/xeon/README.md
View File

@@ -14,7 +14,7 @@ Build docker image with below command:
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps cd GenAIComps
export HF_TOKEN=${your_huggingface_token} export HF_TOKEN=${your_huggingface_token}
docker build -t opea/finetuning:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy --build-arg HF_TOKEN=$HF_TOKEN -f comps/finetuning/Dockerfile . docker build -t opea/finetuning:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy --build-arg HF_TOKEN=$HF_TOKEN -f comps/finetuning/src/Dockerfile .
``` ```
### 2. Run Docker with CLI ### 2. Run Docker with CLI

2
InstructionTuning/docker_compose/intel/hpu/gaudi/README.md
View File

@@ -13,7 +13,7 @@ Build docker image with below command:
```bash ```bash
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps cd GenAIComps
docker build -t opea/finetuning-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/finetuning/Dockerfile.intel_hpu . docker build -t opea/finetuning-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/finetuning/src/Dockerfile.intel_hpu .
``` ```
### 2. Run Docker with CLI ### 2. Run Docker with CLI

2
InstructionTuning/docker_image_build/build.yaml
View File

@@ -9,5 +9,5 @@ services:
https_proxy: ${https_proxy} https_proxy: ${https_proxy}
no_proxy: ${no_proxy} no_proxy: ${no_proxy}
context: GenAIComps context: GenAIComps
dockerfile: comps/finetuning/Dockerfile dockerfile: comps/finetuning/src/Dockerfile
image: ${REGISTRY:-opea}/finetuning:${TAG:-latest} image: ${REGISTRY:-opea}/finetuning:${TAG:-latest}

2
InstructionTuning/tests/test_compose_on_xeon.sh
View File

@@ -19,7 +19,7 @@ ray_port=8265
function build_docker_images() { function build_docker_images() {
cd $WORKPATH/docker_image_build cd $WORKPATH/docker_image_build
if [ ! -d "GenAIComps" ] ; then if [ ! -d "GenAIComps" ] ; then
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
fi fi
docker compose -f build.yaml build --no-cache > ${LOG_PATH}/docker_image_build.log docker compose -f build.yaml build --no-cache > ${LOG_PATH}/docker_image_build.log
} }

2
RerankFinetuning/docker_compose/intel/cpu/xeon/README.md
View File

@@ -14,7 +14,7 @@ Build docker image with below command:
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps cd GenAIComps
export HF_TOKEN=${your_huggingface_token} export HF_TOKEN=${your_huggingface_token}
docker build -t opea/finetuning:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy --build-arg HF_TOKEN=$HF_TOKEN -f comps/finetuning/Dockerfile . docker build -t opea/finetuning:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy --build-arg HF_TOKEN=$HF_TOKEN -f comps/finetuning/src/Dockerfile .
``` ```
### 2. Run Docker with CLI ### 2. Run Docker with CLI

2
RerankFinetuning/docker_compose/intel/hpu/gaudi/README.md
View File

@@ -13,7 +13,7 @@ Build docker image with below command:
```bash ```bash
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps cd GenAIComps
docker build -t opea/finetuning-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/finetuning/Dockerfile.intel_hpu . docker build -t opea/finetuning-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/finetuning/src/Dockerfile.intel_hpu .
``` ```
### 2. Run Docker with CLI ### 2. Run Docker with CLI

2
RerankFinetuning/docker_image_build/build.yaml
View File

@@ -9,5 +9,5 @@ services:
https_proxy: ${https_proxy} https_proxy: ${https_proxy}
no_proxy: ${no_proxy} no_proxy: ${no_proxy}
context: GenAIComps context: GenAIComps
dockerfile: comps/finetuning/Dockerfile dockerfile: comps/finetuning/src/Dockerfile
image: ${REGISTRY:-opea}/finetuning:${TAG:-latest} image: ${REGISTRY:-opea}/finetuning:${TAG:-latest}

2
RerankFinetuning/tests/test_compose_on_xeon.sh
View File

@@ -18,7 +18,7 @@ ray_port=8265
function build_docker_images() { function build_docker_images() {
cd $WORKPATH/docker_image_build cd $WORKPATH/docker_image_build
if [ ! -d "GenAIComps" ] ; then if [ ! -d "GenAIComps" ] ; then
git clone https://github.com/opea-project/GenAIComps.git git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
fi fi
docker compose -f build.yaml build --no-cache > ${LOG_PATH}/docker_image_build.log docker compose -f build.yaml build --no-cache > ${LOG_PATH}/docker_image_build.log
} }

4
docker_images_list.md
View File

@@ -61,8 +61,8 @@ Take ChatQnA for example. ChatQnA is a chatbot application service based on the
| [opea/embedding-multimodal-bridgetower](https://hub.docker.com/r/opea/embedding-multimodal-bridgetower) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/embeddings/src/integrations/dependency/bridgetower/Dockerfile) | The docker image exposes OPEA multimodal embedded microservices based on bridgetower for use by GenAI applications | | [opea/embedding-multimodal-bridgetower](https://hub.docker.com/r/opea/embedding-multimodal-bridgetower) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/embeddings/src/integrations/dependency/bridgetower/Dockerfile) | The docker image exposes OPEA multimodal embedded microservices based on bridgetower for use by GenAI applications |
| [opea/embedding-multimodal-bridgetower-gaudi](https://hub.docker.com/r/opea/embedding-multimodal-bridgetower-gaudi) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/embeddings/src/integrations/dependency/bridgetower/Dockerfile.intel_hpu) | The docker image exposes OPEA multimodal embedded microservices based on bridgetower for use by GenAI applications on the Gaudi | | [opea/embedding-multimodal-bridgetower-gaudi](https://hub.docker.com/r/opea/embedding-multimodal-bridgetower-gaudi) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/embeddings/src/integrations/dependency/bridgetower/Dockerfile.intel_hpu) | The docker image exposes OPEA multimodal embedded microservices based on bridgetower for use by GenAI applications on the Gaudi |
| [opea/feedbackmanagement](https://hub.docker.com/r/opea/feedbackmanagement) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/feedback_management/src/Dockerfile) | The docker image exposes that the OPEA feedback management microservice uses a MongoDB database for GenAI applications. | | [opea/feedbackmanagement](https://hub.docker.com/r/opea/feedbackmanagement) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/feedback_management/src/Dockerfile) | The docker image exposes that the OPEA feedback management microservice uses a MongoDB database for GenAI applications. |
| [opea/finetuning](https://hub.docker.com/r/opea/finetuning) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/finetuning/Dockerfile) | The docker image exposed the OPEA Fine-tuning microservice for GenAI application use | | [opea/finetuning](https://hub.docker.com/r/opea/finetuning) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/finetuning/src/Dockerfile) | The docker image exposed the OPEA Fine-tuning microservice for GenAI application use |
| [opea/finetuning-gaudi](https://hub.docker.com/r/opea/finetuning-gaudi) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/finetuning/Dockerfile.intel_hpu) | The docker image exposed the OPEA Fine-tuning microservice for GenAI application use on the Gaudi | | [opea/finetuning-gaudi](https://hub.docker.com/r/opea/finetuning-gaudi) | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/finetuning/src/Dockerfile.intel_hpu) | The docker image exposed the OPEA Fine-tuning microservice for GenAI application use on the Gaudi |
| [opea/gmcrouter](https://hub.docker.com/r/opea/gmcrouter) | [Link](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/Dockerfile.manager) | The docker image served as one of key parts of the OPEA GenAI Microservice Connector(GMC) to route the traffic among the microservices defined in GMC | | [opea/gmcrouter](https://hub.docker.com/r/opea/gmcrouter) | [Link](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/Dockerfile.manager) | The docker image served as one of key parts of the OPEA GenAI Microservice Connector(GMC) to route the traffic among the microservices defined in GMC |
| [opea/gmcmanager](https://hub.docker.com/r/opea/gmcmanager) | [Link](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/Dockerfile.router) | The docker image served as one of key parts of the OPEA GenAI Microservice Connector(GMC) to be controller manager to handle GMC CRD | | [opea/gmcmanager](https://hub.docker.com/r/opea/gmcmanager) | [Link](https://github.com/opea-project/GenAIInfra/blob/main/microservices-connector/Dockerfile.router) | The docker image served as one of key parts of the OPEA GenAI Microservice Connector(GMC) to be controller manager to handle GMC CRD |
| [opea/guardrails]() | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/guardrails/src/guardrails/Dockerfile) | The docker image exposed the OPEA guardrail microservice to provide content review for GenAI application use | | [opea/guardrails]() | [Link](https://github.com/opea-project/GenAIComps/blob/main/comps/guardrails/src/guardrails/Dockerfile) | The docker image exposed the OPEA guardrail microservice to provide content review for GenAI application use |