Files

Letong Han 7e40475b6d Refine READMEs after reorg (#666 )

* update dataprep readme

Signed-off-by: letonghan <letong.han@intel.com>

* update vectorstore readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* update retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate embedding readme

Signed-off-by: letonghan <letong.han@intel.com>

* update guardrails readme

Signed-off-by: letonghan <letong.han@intel.com>

* update other readmes

Signed-off-by: letonghan <letong.han@intel.com>

* update reranks readme

Signed-off-by: letonghan <letong.han@intel.com>

* update llm&lvms readme

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

2024-09-11 17:23:40 +08:00

1.5 KiB

Raw Blame History

Dataprep Microservice

The Dataprep Microservice aims to preprocess the data from various sources (either structured or unstructured data) to text data, and convert the text data to embedding vectors then store them in the database.

Install Requirements

apt-get update
apt-get install libreoffice

Use LVM (Large Vision Model) for Summarizing Image Data

Occasionally unstructured data will contain image data, to convert the image data to the text data, LVM can be used to summarize the image. To leverage LVM, please refer to this readme to start the LVM microservice first and then set the below environment variable, before starting any dataprep microservice.

export SUMMARIZE_IMAGE_VIA_LVM=1