jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Image2video code refactor (#1075)

Browse Source 
* image2video code refactor.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix spell error.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* Update opea_image2video_microservice.py

* changed naming

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

---------

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
XinyuYe-Intel
2024-12-31 13:18:15 +08:00
committed by GitHub
parent a19c222636
commit 90a86345c5
17 changed files with 276 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
.github/workflows/docker/compose/image2video-compose.yaml vendored
View File

@@ -5,13 +5,9 @@
services:
image2video:
build:
dockerfile: comps/image2video/Dockerfile
dockerfile: comps/image2video/src/Dockerfile
image: ${REGISTRY:-opea}/image2video:${TAG:-latest}
svd:
image2video-gaudi:
build:
dockerfile: comps/image2video/dependency/Dockerfile
image: ${REGISTRY:-opea}/svd:${TAG:-latest}
svd-gaudi:
build:
dockerfile: comps/image2video/dependency/Dockerfile.intel_hpu
image: ${REGISTRY:-opea}/svd-gaudi:${TAG:-latest}
dockerfile: comps/image2video/src/Dockerfile.intel_hpu
image: ${REGISTRY:-opea}/image2video-gaudi:${TAG:-latest}

18
comps/image2video/Dockerfile
View File

@@ -1,18 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
FROM python:3.11-slim
# Set environment variables
ENV LANG=en_US.UTF-8
COPY comps /home/comps
RUN pip install --no-cache-dir --upgrade pip setuptools && \
pip install --no-cache-dir -r /home/comps/image2video/requirements.txt
ENV PYTHONPATH=$PYTHONPATH:/home
WORKDIR /home/comps/image2video
ENTRYPOINT ["python", "image2video.py"]

83
comps/image2video/README.md
View File

@@ -1,83 +0,0 @@
# Image-to-Video Microservice
Image-to-Video is a task that generate video conditioning on the provided image(s). This microservice supports image-to-video task by using Stable Video Diffusion (SVD) model.
# 🚀1. Start Microservice with Python (Option 1)
## 1.1 Install Requirements
```bash
pip install -r requirements.txt
pip install -r dependency/requirements.txt
```
## 1.2 Start SVD Service
```bash
# Start SVD service
cd dependency/
python svd_server.py
```
## 1.3 Start Image-to-Video Microservice
```bash
cd ..
# Start the OPEA Microservice
python image2video.py
```
# 🚀2. Start Microservice with Docker (Option 2)
## 2.1 Build Images
### 2.1.1 SVD Server Image
Build SVD server image on Xeon with below command:
```bash
cd ../..
docker build -t opea/svd:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/image2video/dependency/Dockerfile .
```
Build SVD server image on Gaudi with below command:
```bash
cd ../..
docker build -t opea/svd-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/image2video/dependency/Dockerfile.intel_hpu .
```
### 2.1.2 Image-to-Video Service Image
```bash
docker build -t opea/image2video:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/image2video/Dockerfile .
```
## 2.2 Start SVD and Image-to-Video Service
### 2.2.1 Start SVD server
Start SVD server on Xeon with below command:
```bash
docker run --ipc=host -p 9368:9368 -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/svd:latest
```
Start SVD server on Gaudi with below command:
```bash
docker run -p 9368:9368 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/svd-gaudi:latest
```
### 2.2.2 Start Image-to-Video service
```bash
ip_address=$(hostname -I | awk '{print $1}')
docker run -p 9369:9369 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e SVD_ENDPOINT=http://$ip_address:9368 opea/image2video:latest
```
### 2.2.3 Test
```bash
http_proxy="" curl http://localhost:9369/v1/image2video -XPOST -d '{"images_path":[{"image_path":"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png"}]}' -H 'Content-Type: application/json'
```

7
comps/image2video/dependency/requirements.txt
View File

@@ -1,7 +0,0 @@
accelerate
diffusers
fastapi
opencv-python
torch
transformers
uvicorn

76
comps/image2video/dependency/svd_server.py
View File

@@ -1,76 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""Stand-alone Stable Video Diffusion FastAPI Server."""
import argparse
import os
import time
import torch
import uvicorn
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import export_to_video, load_image
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
app = FastAPI()
@app.post("/generate")
async def generate(request: Request) -> Response:
print("SVD generation begin.")
request_dict = await request.json()
images_path = request_dict.pop("images_path")
start = time.time()
images = [load_image(img) for img in images_path]
images = [image.resize((1024, 576)) for image in images]
generator = torch.manual_seed(args.seed) if args.device == "cpu" else None
frames = pipe(images, decode_chunk_size=8, generator=generator).frames[0]
video_path = os.path.join(os.getcwd(), args.video_path)
export_to_video(frames, video_path, fps=7)
end = time.time()
print(f"SVD video output in {video_path}, time = {end-start}s")
return JSONResponse({"video_path": video_path})
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--host", type=str, default="0.0.0.0")
parser.add_argument("--port", type=int, default=9368)
parser.add_argument("--model_name_or_path", type=str, default="stabilityai/stable-video-diffusion-img2vid-xt")
parser.add_argument("--video_path", type=str, default="generated.mp4")
parser.add_argument("--use_hpu_graphs", default=False, action="store_true")
parser.add_argument("--device", type=str, default="cpu")
parser.add_argument("--seed", type=int, default=42)
args = parser.parse_args()
if args.device == "hpu":
from optimum.habana.diffusers import GaudiEulerDiscreteScheduler, GaudiStableVideoDiffusionPipeline
from optimum.habana.utils import set_seed
set_seed(args.seed)
scheduler = GaudiEulerDiscreteScheduler.from_pretrained(args.model_name_or_path, subfolder="scheduler")
kwargs = {
"scheduler": scheduler,
"use_habana": True,
"use_hpu_graphs": args.use_hpu_graphs,
"gaudi_config": "Habana/stable-diffusion",
}
pipe = GaudiStableVideoDiffusionPipeline.from_pretrained(
args.model_name_or_path,
**kwargs,
)
elif args.device == "cpu":
pipe = StableVideoDiffusionPipeline.from_pretrained(args.model_name_or_path)
else:
raise NotImplementedError(f"Only support cpu and hpu device now, device {args.device} not supported.")
print("Stable Video Diffusion model initialized.")
uvicorn.run(
app,
host=args.host,
port=args.port,
log_level="debug",
)

0
comps/image2video/deployment/docker_compose/README.md Normal file
View File

0
comps/image2video/deployment/kubernetes/README.md Normal file
View File

47
comps/image2video/image2video.py
View File

@@ -1,47 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import json
import os
import time
import requests
from comps import (
ImagesPath,
ServiceType,
VideoPath,
opea_microservices,
register_microservice,
register_statistics,
statistics_dict,
)
@register_microservice(
name="opea_service@image2video",
service_type=ServiceType.IMAGE2VIDEO,
endpoint="/v1/image2video",
host="0.0.0.0",
port=9369,
input_datatype=ImagesPath,
output_datatype=VideoPath,
)
@register_statistics(names=["opea_service@image2video"])
async def image2video(input: ImagesPath):
start = time.time()
images_path = [img.image_path for img in input.images_path]
inputs = {"images_path": images_path}
video_path = requests.post(url=f"{svd_endpoint}/generate", data=json.dumps(inputs), proxies={"http": None}).json()[
"video_path"
]
statistics_dict["opea_service@image2video"].append_latency(time.time() - start, None)
return VideoPath(video_path=video_path)
if __name__ == "__main__":
svd_endpoint = os.getenv("SVD_ENDPOINT", "http://localhost:9368")
print("Image2video server started.")
opea_microservices["opea_service@image2video"].start()

6
comps/image2video/dependency/Dockerfile → comps/image2video/src/Dockerfile
View File

@@ -13,10 +13,10 @@ COPY comps /home/comps
RUN apt-get update && apt-get install python3-opencv -y && \
pip install --no-cache-dir --upgrade pip setuptools && \
if [ ${ARCH} = "cpu" ]; then pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu; fi && \
pip install --no-cache-dir -r /home/comps/image2video/dependency/requirements.txt
pip install --no-cache-dir -r /home/comps/image2video/src/requirements.txt
ENV PYTHONPATH=$PYTHONPATH:/home
WORKDIR /home/comps/image2video/dependency
WORKDIR /home/comps/image2video/src
ENTRYPOINT ["python", "svd_server.py"]
ENTRYPOINT ["python", "opea_image2video_microservice.py"]

6
comps/image2video/dependency/Dockerfile.intel_hpu → comps/image2video/src/Dockerfile.intel_hpu
View File

@@ -19,11 +19,11 @@ ENV PYTHONPATH=/home/user:/usr/lib/habanalabs/:/optimum-habana
# Install requirements and optimum habana
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r /home/user/comps/image2video/dependency/requirements.txt && \
pip install --no-cache-dir -r /home/user/comps/image2video/src/requirements.txt && \
pip install --no-cache-dir optimum[habana]
ENV PYTHONPATH=$PYTHONPATH:/home/user
WORKDIR /home/user/comps/image2video/dependency
WORKDIR /home/user/comps/image2video/src
ENTRYPOINT ["python", "svd_server.py", "--device", "hpu"]
ENTRYPOINT ["python", "opea_image2video_microservice.py", "--device", "hpu"]

57
comps/image2video/src/README.md Normal file
View File

@@ -0,0 +1,57 @@
# Image-to-Video Microservice
Image-to-Video is a task that generate video conditioning on the provided image(s). This microservice supports image-to-video task by using Stable Video Diffusion (SVD) model.
# 🚀1. Start Microservice with Python (Option 1)
## 1.1 Install Requirements
```bash
pip install -r src/requirements.txt
```
## 1.2 Start Image-to-Video Microservice
```bash
cd ..
# Start the OPEA Microservice
python opea_image2video_microservice.py
```
# 🚀2. Start Microservice with Docker (Option 2)
## 2.1 Build Images
Build Image-to-Video Service image on Xeon with below command:
```bash
cd ../..
docker build -t opea/image2video:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/image2video/src/Dockerfile .
```
Build Image-to-Video Service image on Gaudi with below command:
```bash
cd ../..
docker build -t opea/image2video-gaudi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/image2video/src/Dockerfile.intel_hpu .
```
## 2.2 Start Image-to-Video Service
Start SVD server on Xeon with below command:
```bash
docker run --ipc=host -p 9369:9369 -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/image2video:latest
```
Start SVD server on Gaudi with below command:
```bash
docker run -p 9369:9369 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/image2video-gaudi:latest
```
## 2.3 Test
```bash
http_proxy="" curl http://localhost:9369/v1/image2video -XPOST -d '{"images_path":[{"image_path":"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png"}]}' -H 'Content-Type: application/json'
```

0
comps/image2video/__init__.py → comps/image2video/src/integrations/__init__.py
View File

67
comps/image2video/src/integrations/opea.py Normal file
View File

@@ -0,0 +1,67 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import os
import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import export_to_video, load_image
from comps import CustomLogger, ImagesPath, OpeaComponent, ServiceType, VideoPath
logger = CustomLogger("opea")
class OpeaImage2video(OpeaComponent):
"""A specialized image2video component derived from OpeaComponent for image2video services."""
def __init__(self, name: str, description: str, config: dict = None):
super().__init__(name, ServiceType.IMAGE2VIDEO.name.lower(), description, config)
# initialize model
self.device = config["device"]
self.seed = config["seed"]
if self.device == "hpu":
from optimum.habana.diffusers import GaudiEulerDiscreteScheduler, GaudiStableVideoDiffusionPipeline
from optimum.habana.utils import set_seed
set_seed(self.seed)
scheduler = GaudiEulerDiscreteScheduler.from_pretrained(config["model_name_or_path"], subfolder="scheduler")
kwargs = {
"scheduler": scheduler,
"use_habana": True,
"use_hpu_graphs": config["use_hpu_graphs"],
"gaudi_config": "Habana/stable-diffusion",
}
self.pipe = GaudiStableVideoDiffusionPipeline.from_pretrained(
config["model_name_or_path"],
**kwargs,
)
elif self.device == "cpu":
self.pipe = StableVideoDiffusionPipeline.from_pretrained(config["model_name_or_path"])
else:
raise NotImplementedError(f"Only support cpu and hpu device now, device {self.device} not supported.")
logger.info("Stable Video Diffusion model initialized.")
async def invoke(self, input: ImagesPath) -> VideoPath:
"""Invokes the image2video service to generate video(s) for the provided input.
Args:
input (ImagesPath): The input for image2video service, including image paths.
Returns:
VideoPath: The response is path to the generated video.
"""
logger.info("SVD generation begin.")
images_path = [img_path.image_path for img_path in input.images_path]
images = [load_image(img) for img in images_path]
images = [image.resize((1024, 576)) for image in images]
generator = torch.manual_seed(self.seed) if self.device == "cpu" else None
frames = self.pipe(images, decode_chunk_size=8, generator=generator).frames[0]
video_path = os.path.join(os.getcwd(), "generated.mp4")
export_to_video(frames, video_path, fps=7)
return VideoPath(video_path=video_path)
def check_health(self) -> bool:
return True

76
comps/image2video/src/opea_image2video_microservice.py Normal file
View File

@@ -0,0 +1,76 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import time
from integrations.opea import OpeaImage2video
from comps import (
CustomLogger,
ImagesPath,
OpeaComponentController,
ServiceType,
VideoPath,
opea_microservices,
register_microservice,
register_statistics,
statistics_dict,
)
logger = CustomLogger("opea_image2video_microservice")
# Initialize OpeaComponentController
controller = OpeaComponentController()
@register_microservice(
name="opea_service@image2video",
service_type=ServiceType.IMAGE2VIDEO,
endpoint="/v1/image2video",
host="0.0.0.0",
port=9369,
input_datatype=ImagesPath,
output_datatype=VideoPath,
)
@register_statistics(names=["opea_service@image2video"])
async def image2video(input: ImagesPath):
start = time.time()
try:
# Use the controller to invoke the active component
results = await controller.invoke(input)
statistics_dict["opea_service@image2video"].append_latency(time.time() - start, None)
return results
except Exception as e:
logger.error(f"Error during image2video invocation: {e}")
raise
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--host", type=str, default="0.0.0.0")
parser.add_argument("--port", type=int, default=9368)
parser.add_argument("--model_name_or_path", type=str, default="stabilityai/stable-video-diffusion-img2vid-xt")
parser.add_argument("--use_hpu_graphs", default=False, action="store_true")
parser.add_argument("--device", type=str, default="cpu")
parser.add_argument("--seed", type=int, default=42)
args = parser.parse_args()
# Register components
try:
# Instantiate Image2video components
opea_image2video = OpeaImage2video(
name="OpeaImage2video", description="OPEA Image2video Service", config=args.__dict__
)
# Register components with the controller
controller.register(opea_image2video)
# Discover and activate a healthy component
controller.discover_and_activate()
except Exception as e:
logger.error(f"Failed to initialize components: {e}")
logger.info("Image2video server started.")
opea_microservices["opea_service@image2video"].start()

5
comps/image2video/requirements.txt → comps/image2video/src/requirements.txt
View File

@@ -1,6 +1,9 @@
accelerate
datasets
diffusers
docarray[full]
fastapi
opencv-python
opentelemetry-api
opentelemetry-exporter-otlp
opentelemetry-sdk
@@ -8,4 +11,6 @@ prometheus-fastapi-instrumentator
pydantic==2.7.2
pydub
shortuuid
torch
transformers
uvicorn

13
tests/image2video/test_image2video.sh → tests/image2video/test_image2video_opea.sh
View File

@@ -10,14 +10,7 @@ ip_address=$(hostname -I | awk '{print $1}')
function build_docker_images() {
cd $WORKPATH
echo $(pwd)
docker build --no-cache -t opea/svd:latest -f comps/image2video/dependency/Dockerfile .
if [ $? -ne 0 ]; then
echo "opea/svd built fail"
exit 1
else
echo "opea/svd built successful"
fi
docker build --no-cache -t opea/image2video:latest -f comps/image2video/Dockerfile .
docker build --no-cache -t opea/image2video:latest -f comps/image2video/src/Dockerfile .
if [ $? -ne 0 ]; then
echo "opea/image2video built fail"
exit 1
@@ -28,8 +21,7 @@ function build_docker_images() {
function start_service() {
unset http_proxy
docker run -d --name="test-comps-image2video-svd" -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9368:9368 --ipc=host opea/svd:latest
docker run -d --name="test-comps-image2video" -e SVD_ENDPOINT=http://$ip_address:9368 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9369:9369 --ipc=host opea/image2video:latest
docker run -d --name="test-comps-image2video" -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9369:9369 --ipc=host opea/image2video:latest
sleep 3m
}
@@ -39,7 +31,6 @@ function validate_microservice() {
echo "Result correct."
else
echo "Result wrong."
docker logs test-comps-image2video-svd
docker logs test-comps-image2video
exit 1
fi

59
tests/image2video/test_image2video_opea_on_intel_hpu.sh Normal file
View File

@@ -0,0 +1,59 @@
#!/bin/bash
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
set -x
WORKPATH=$(dirname "$PWD")
ip_address=$(hostname -I | awk '{print $1}')
function build_docker_images() {
cd $WORKPATH
echo $(pwd)
docker build --no-cache -t opea/image2video-gaudi:latest -f comps/image2video/src/Dockerfile.intel_hpu .
if [ $? -ne 0 ]; then
echo "opea/image2video-gaudi built fail"
exit 1
else
echo "opea/image2video-gaudi built successful"
fi
}
function start_service() {
unset http_proxy
docker run -d --name="test-comps-image2video-gaudi" -p 9369:9369 --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/image2video-gaudi:latest
sleep 3m
}
function validate_microservice() {
result=$(http_proxy="" curl http://localhost:9369/v1/image2video -XPOST -d '{"images_path":[{"image_path":"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png"}]}' -H 'Content-Type: application/json')
if [[ $result == *"generated.mp4"* ]]; then
echo "Result correct."
else
echo "Result wrong."
docker logs test-comps-image2video-gaudi
exit 1
fi
}
function stop_docker() {
cid=$(docker ps -aq --filter "name=test-comps-image2video*")
if [[ ! -z "$cid" ]]; then docker stop $cid && docker rm $cid && sleep 1s; fi
}
function main() {
stop_docker
build_docker_images
start_service
validate_microservice
stop_docker
echo y | docker system prune
}
main