jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
698 Commits 43 Branches 9 Tags
enable_persistent_db
Commit Graph

91 Commits

Author SHA1 Message Date
Louie Tsai
ca89cf4c9f add opentelemetry tracing into OPEA DAG and couple microservices code path related to ChatQnA (#1122) 
* add opentelemetry tracing into ChatQnA workflow

Signed-off-by: Louie, Tsai <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>

* handle stream/non-stream tracing

Signed-off-by: louie-tsai <louie.tsai@intel.com>

* pre-commit fix

Signed-off-by: louie-tsai <louie.tsai@intel.com>

* add a tag for async llm microservice execute

Signed-off-by: louie-tsai <louie.tsai@intel.com>

* add oltp tracing for retriever

Signed-off-by: louie-tsai <louie.tsai@intel.com>

* fix CI issue

Signed-off-by: louie-tsai <louie.tsai@intel.com>

---------

Signed-off-by: Louie, Tsai <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Co-authored-by: Spycsh <sihan.chen@intel.com>
 2025-01-16 21:42:40 -08:00
Letong Han
c49db2b037 Refactor Retrievers (#1138) 
Refactor retrievers into E-RAG style.
The folder structure after refactored is as below:

- retrievers
    - deployment
        - docker_compose
        - kubernetes
    - src
        - integrations
            - redis.py
            - milvus.py
            - ...
        - Dockerfile
        - opea_retrievers_microservice.py
        - README.md
        - requirements.txt
        - set_env.sh

#1004

Signed-off-by: letonghan <letong.han@intel.com>
 2025-01-16 14:12:30 +08:00
dolpher
1cc4d2119d Add kubernetes deployment for GenAIComps (#1104) 
* Add kubernetes deployment for GenAIComps

---------

Signed-off-by: Dolpher Du <dolpher.du@intel.com>
 2025-01-13 15:42:33 +08:00
Liang Lv
b91911a543 Refine embedding naming and move dependency to 3rd_party (#1125) 
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
 2025-01-10 14:44:25 +08:00
Liang Lv
bf09739585 Refine Component Interface (#1106) 
* Refine component interface

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update env

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* add health check

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update mulimodal embedding

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update import

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* refine other components

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix dataprepissue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix tts issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix ci issues

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix tts response issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix comments

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
 2025-01-07 09:24:47 +08:00
chen, suyue
f57e30dde6 GenAIComps microservices refactor (#1072) 
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: WenjiaoYue <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>
 2025-01-02 16:31:01 +08:00
Cameron Morin
8d6b4b0ac7 Add opensearch integration for OPEA (#1024) 
* Add opensearch integration for OPEA

Signed-off-by: Cameron Morin <cammorin@amazon.com>

* Update docker compose yaml workflows files

Signed-off-by: Cameron Morin <cammorin@amazon.com>

* Fix empty files

Signed-off-by: Cameron Morin <cammorin@amazon.com>

* Address PR comments

Signed-off-by: Cameron Morin <cammorin@amazon.com>

---------

Signed-off-by: Cameron Morin <cammorin@amazon.com>
 2024-12-26 11:09:59 +08:00
Letong Han
a6cdd17242 fix multimodalqna issue (#1042) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-12-17 16:57:45 +08:00
Letong Han
f3aaaebf5a [Reorg] Remove redundant file in retrievers/redis (#1016) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-12-17 12:01:13 +08:00
kkrishTa
5ed041bded Feature/elasticsearch vector store integration - Infosys (#972) 
* Feature/elastic

Elasticsearch vectorstore, dataprep and retriever

---------

Co-authored-by: Adarsh <reachaadi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
 2024-12-10 09:40:44 +08:00
Chaunte W. Lacewell
750e501bee [Bug] Fix VDMS retriever and apply fix to VDMS dataprep (#928) 
* Update requirements to pin protobuf version and fix grpc conflict, and limit vdms version

Signed-off-by: Lacewell, Chaunte W <chaunte.w.lacewell@intel.com>

* Update fix by removing grpcio pin and pinning opentelemetry-proto to 1.23.0

Signed-off-by: Lacewell, Chaunte W <chaunte.w.lacewell@intel.com>

---------

Signed-off-by: Lacewell, Chaunte W <chaunte.w.lacewell@intel.com>
 2024-11-21 11:05:05 +08:00
minmin-intel
1cf27817aa fix retriever and reranker to process chat completion request (#915) 
* fix retriever and reranker to process chat completion request

Signed-off-by: minmin-intel <minmin.hou@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: minmin-intel <minmin.hou@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-11-19 13:54:59 +08:00
rbrugaro
0163ea6f4e trim input to TGI, moved clustering and summarization to dataprep and store in DB (#893) 
* trim input to TGI, moved clustering and summarization to dataprep and DB store

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed inspect_db causing error in precommit

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add HF token to dataprep container because tokenizer is used now

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* updated READMEs to reflect latest changes

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix all files are ingested and graph extracted first followed by 1 cluster call for full graph in database

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update README based on fix for multifile

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* Changes to make graphrag ui work

Signed-off-by: theresa <theresa.shan@intel.com>

* fix bug build communities done once at end of ingestion

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* minor fixes

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* README fixes

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

---------

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: theresa <theresa.shan@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: theresa <theresa.shan@intel.com>
 2024-11-14 15:29:23 -08:00
Dan
d2e9c0a9dd Pinecone retriever index fix (#816) 
* Update huggingface endpoint api

Replace deprecated HuggingFace Embeddings class.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Change timeout strategy launching retriever

Use wait-for-it script rather than an arbitrarily determined sleep.
This should be more resilient to timing variances both saving time
vs unnecessarily long sleeps and avoiding spurious failures if
CI is slow causing a container to take longer than normal to start.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Supress irrelevant warning during docker build

This path warning during docker build is not relevant.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Fix Pinecone service using ServerlessSpec index

Pinecone requires a hosted index.
That index can be a ServerlessSpec instance at a CSP or a more expensive
PodSpec based instance also hosted at a CSP.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Recreate index each run

For CI we need the index to be removed so the tests can be rerun
from a clean state.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Dan Middleton <dan.middleton@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-30 21:35:16 +08:00
rbrugaro
19330ea23f GraphRAG with llama-index (#793) 
* graphRAG dataprep llama-index validated w openai endpoints

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* llama-index graphRAG retrieval validated with openai models

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* validated code usng TGI/TEI instead of openai

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* compose.yaml for dataprep validated with neo4j, TGI/TEI, openai

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* graphRAG retriever validated and full compose.yaml

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* minor fix

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add graphRAGGateway WIP

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* graphragGateway working in E2E Example

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* fix schedule in orchestrator to support ChatCompletionRequest input

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* change default to TGI instead of openAI and add test code for neo4jretriever

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* test code for dataprep-neo4j microservice

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* improved READMES

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update docker build path for tests

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor fix

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* fix typo in container name

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* resolve image name conflict for hub publishing

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* add tgi validation to miicroservice tests

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* rename test script to trigger cicd correctly w hpu

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* rename test containers for cicd

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* update HF_TOKEN in TGI/TEI test scripts

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* swaped larger model so the graph isnt empty in ci test

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* set 4 hpu for 70B model in ci test

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* add extra time for large model loading cicd

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* fix tgi gaudi shard args

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* switch to chat cause chat template is needed

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* enable logs in test

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* use locally downloaded model in CI machine

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* use local model path and reduce wait time

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* clear ports before ci run

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* fix cache model access

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* fix cache model access

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* incrased wait time for tgi shards ready

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* wait until tgi connected

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* switch back to small model for testing

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* minor readability fixes

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

* README fixes

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>

---------

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-29 22:45:44 -07:00
chen, suyue
6518c0f046 upgrade setuptools version to fix CVE-2024-6345 (#806) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-10-22 15:02:48 +08:00
bjzhjing
7e085a4bc1 retriever/milvus: Update README.md (#789) 
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
 2024-10-21 17:32:26 +08:00
Dan
367b3aa5db Remove unused imports across all comps (#762) 
* Remove unused imports across all comps

Linting was configured to ignore unused imports.
Remove this accumulation of lint.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Explain why we ignore unused imports with ruff

In general unused imports should be removed, but ruff can't and probably
shouldn't automatically fix conditional imports. Consequently,
conditional imports will always be left over causing false
positive errors in CI.

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* Revert pinecone import changes

Signed-off-by: Dan Middleton <dan.middleton@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Dan Middleton <dan.middleton@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-11 22:11:59 +08:00
bjzhjing
84374a5710 Refactor milvus dataprep and retriever (#728) 
* milvus: Refactor embedding settings for mivlus dataprep and retriever

Milvus dataprep and retriever leverage the same embedding enpoints, but
the embedding-related code is somewhat messed up, unify the namings and
logic to improve code readability and facilitate user-friendly
configuration.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>

* MOSEC: Rename EMB_MODEL env as MOSEC_EMBEDDING_MODEL

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>

* milvus/dataprep: Update README for milvus dataprep

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>

* Add OCR package for Milvus dataprep

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>

* Update Milvus dataprep test script

This is to fix the CI issue for MILVUS environment variable name is
update.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-09 23:57:19 -07:00
Abolfazl Shahbazi
9e85b35b0b Remove proxy definition from within Dockerfiles (#778) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-10-10 12:00:32 +08:00
Abolfazl Shahbazi
251907d494 Remove 'vim' from Dockerfiles (#776) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-10-10 11:37:35 +08:00
Abolfazl Shahbazi
4f5010cfbb add 'no-cache-dir' to all pip installs (#777) 
* add 'no-cache-dir' to all pip installs

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Fix a malformed line break

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

---------

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-10-10 11:09:47 +08:00
lvliang-intel
f3746dc85c Async support for some microservices (#763) 
* Async support for some microservices

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix issues

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix issues

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix import issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add dependency library

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* roll back pinecone change

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-10 09:51:27 +08:00
Abolfazl Shahbazi
dfd89affe6 Misc docker updates (#761) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-10-08 08:49:47 +08:00
lvliang-intel
f416f8415c Remove fixed version in requirements.txt (#712) 
* Remove fixed version

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
 2024-09-19 20:13:15 +08:00
XuhuiRen
29fe569d34 Enable GraphRAG with Neo4J (#682) 
* add graphrag for neo4j

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* fix ut

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* fix

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update retriever_neo4j.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* Update test_retrievers_neo4j_langchain.sh

* add

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* Update test_retrievers_neo4j_langchain.sh

* Update test_retrievers_neo4j_langchain.sh

* Update test_retrievers_neo4j_langchain.sh

* add docker

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* Update retrievers-compose-cd.yaml

* Update test_retrievers_neo4j_langchain.sh

* Update config.py

* Update test_retrievers_neo4j_langchain.sh

* Update test_retrievers_neo4j_langchain.sh

* Update config.py

* Update test_retrievers_neo4j_langchain.sh

* Update requirements.txt

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update requirements.txt

* Update requirements.txt

* Update requirements.txt

---------

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
 2024-09-15 18:12:29 +08:00
Tiep Le
2ca56f3ed9 Update example name into MultimodalQnA and update image names (#686) 
* update image names

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update image names for dataprep service

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update image name for retriever

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update image name for lvm

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update lvm include caption in output

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update gateway name into MultimodalQnAGateway

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test gateway for rename

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-13 17:07:38 +08:00
chen, suyue
0458443baf update torch cpu installation (#681) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-12 22:01:38 +08:00
chen, suyue
6ce65514e1 replace langchain/langchain:latest with python:3.11-slim (#669) 
* replace langchain/langchain:latest with python:3.11-slim

Signed-off-by: chensuyue <suyue.chen@intel.com>

* fix test

Signed-off-by: chensuyue <suyue.chen@intel.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-12 13:24:50 +08:00
XuhuiRen
a672569551 Fix security issue (#673) 
Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
 2024-09-12 00:01:58 +08:00
Letong Han
7e40475b6d Refine READMEs after reorg (#666) 
* update dataprep readme

Signed-off-by: letonghan <letong.han@intel.com>

* update vectorstore readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* update retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate embedding readme

Signed-off-by: letonghan <letong.han@intel.com>

* update guardrails readme

Signed-off-by: letonghan <letong.han@intel.com>

* update other readmes

Signed-off-by: letonghan <letong.han@intel.com>

* update reranks readme

Signed-off-by: letonghan <letong.han@intel.com>

* update llm&lvms readme

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 17:23:40 +08:00
Steve Zhang
1d761fa0a6 Fix the Retriever README error (#655) 
* change based on comments for missing places.
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-11 14:36:21 +08:00
Letong Han
bea9bb0242 [Reorg] Reorg Folder to Support Different Vendors (#625) 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 23:24:09 +08:00
Huiling Bao
2c48bc80c7 Add Megaservice support for MMRAG VideoRAGQnA usecase (#603) 
* add videoragqna gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test script for gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm ip

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix exist bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
 2024-09-09 23:37:17 +09:00
Tiep Le
23cc3ea5ee Add schema to Redis initialization & Improve LVM-TGI For Multimodal Retriever Microservice (#638) 
* add schema to Redis initialization

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update lvm_tgi

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-09 11:59:05 +08:00
Steve Zhang
824a7e2c0d clear some unnecessary scripts and Dockerfile commands. (#635) 
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-06 18:54:57 +08:00
Tiep Le
1513998950 Retriever and lvm update for multimodal rag on videos (#606) 
* updates

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update redis schema

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update multimodal config and docker compose retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update requirements

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update retriever redis

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* multimodal retriever implementation

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* test for multimodal retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* include prompt preparation for multimodal rag on videos application

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix template

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add test for llava for mm_rag_on_videos

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* update test

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix index not found

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add LVMSearchedMultimodalDoc

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove INDEX_SCHEMA

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* revise folder structure to comps/retrievers/langchain/redis_multimodal

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* change port of redis to resolve CI test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update lvms test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com>
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-06 13:46:10 +08:00
David Kinder
a8a46bc102 doc: fix heading levels in markdown content (#627) 
* only one H1 (#) heading for the title is allowed, so fix the extra H1
  headings (and the subheadings under those) to appropriate levels
* fix some inline code blocks containing leading/trailing spaces
* fix some indenting issues under an ordered list item

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
 2024-09-06 11:09:27 +08:00
Abolfazl Shahbazi
f1f866f7c1 Using '--no-cache-dir' within all Dockerfiles (#607) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-05 12:01:46 -07:00
Huiling Bao
54aa943d82 Bugfix for PR 496 to add format_video_name function (#602) 
* add format_video_name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test for negative case

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update file path

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 23:26:15 +08:00
Sameh Gobriel
445c9b19dd add VDMS retriever microservice for v0.9 Milestone (#539) 
* add VDMS retriever microservice

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* add retrieval gateway and logger back to init

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use 5009 in CI

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* change index_name to collection_name

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* fix var name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use index name all

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add deps

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* changes to address code reviews

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolve docarray

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add optional docarray embeddoc constraints

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug in comment

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* import DEBUG

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
 2024-09-04 14:33:36 +08:00
ZePan110
b541fd8bdb update image build yaml (#529) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
 2024-08-30 17:44:08 +08:00
berkecanrizai
2c2322e7be add: Pathway vector store and retriever as LangChain component (#342) 
* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* example data

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update, add commands

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused parts

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: image name, compose env vars

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: logging name

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: env var

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update input sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: mv vector store, naming, clarify instructions, improve ingestion components

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests: add pw retriever test
fix: update docker to include libmagic

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement suggestions from review, entrypoint, reqs, comments, https_proxy.

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: update docker tags in test and readme

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* tests: add separate pathway vectorstore test

Signed-off-by: Berke <berkecanrizai1@gmail.com>

---------

Signed-off-by: Berke <berkecanrizai1@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
 2024-08-29 21:04:41 +08:00
Abolfazl Shahbazi
25174c0eb5 Remove 'vim' from all Dockerfiles (#557) 
* Remove 'vim' from all Dockerfiles

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-08-28 07:30:42 +08:00
XinyaoWa
fab1fbdfe9 Add logging for unified debug (#521) 
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com>
 2024-08-20 22:09:04 +08:00
lvliang-intel
cc8cd702ae Remove unnecessary langsmith dependency (#369) 
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
 2024-08-18 17:11:13 +08:00
David Kinder
77e0e7be1c doc: fix multiple H1 headings (#481) 
Only one H1 heading (first for the title) allowed.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
 2024-08-17 23:43:12 +08:00
ZePan110
4a67d427bd Change /root/ to /home/user/. (#475) 
* Change /root/ to /hmoe/user/.

Signed-off-by: zepan <ze.pan@intel.com>

* Fix issue.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
 2024-08-14 15:46:28 +08:00
Letong Han
7aee7e4689 fix empty content issue (#476) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-08-14 10:14:41 +08:00
Wang, Kai Lawrence
eb51018f65 Refine the instructions to run the retriever example with qdrant (#442) 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
 2024-08-13 15:59:01 +08:00