jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
414 Commits 43 Branches 9 Tags
1785149ecfe8c29c49f14c3b5627b63000396ab0
Commit Graph

414 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chen, suyue
1785149ecf minor fix for CI detect (#658) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-11 00:45:34 +08:00
Letong Han
a3da7c1e3f Fix Reorg Issues (#657) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-09-11 00:45:12 +08:00
Letong Han
bea9bb0242 [Reorg] Reorg Folder to Support Different Vendors (#625) 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 23:24:09 +08:00
ZePan110
2a91903627 Fix Dockerfile detect issue (#652) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-10 17:22:35 +08:00
lkk
c718602abf add resume finetuning checkpoint ut. (#646) 
* add resume finetuning checkpoint ut.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add final tuned model.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 17:35:23 +09:00
chen, suyue
94eb60f168 add PREDICTIONGUARD_API_KEY for CI (#650) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 15:56:47 +08:00
XinyuYe-Intel
29311479bb added finetuned model deployment tutorial in readme. (#648) 
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
 2024-09-10 15:52:15 +08:00
XuhuiRen
2fbce3e53d Fix the loading error of jsonl file (#644) 
* fixjsonl

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 15:50:40 +08:00
chen, suyue
48d4e53396 unify default reranking model with BAAI/bge-reranker-base (#623) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-10 15:23:10 +08:00
ZePan110
8a1141314d Optimize the content of the alerts. (#640) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 13:43:06 +08:00
XinyuYe-Intel
58e9972d82 Add LLM pretraining support (#622) 
* support rerank model finetuning.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* adapt rerank model to transformers' scheme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* refined readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add LLM pretraining support.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* added rerank finetuning test.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added LLM pretraining test.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

---------

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 13:18:35 +08:00
Dong, Bo
d76751ad81 optimize rerank with backend ref (#579) 
* add rerank with neural speed

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix mismatched response format w/wo streaming guardrails (#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Fix guardrails out handle logics for space linebreak and quote (#571)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* debug

* debug

* debug

* fix pre-space and linebreak

* fix pre-space and linebreak

* fix single/double quote

* fix single/double quote

* remove debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* BUG FIX: LVM security fix (#572)

* add url validator

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Modify output messages. (#569)

* Reduced output.

Signed-off-by: zepan <ze.pan@intel.com>

* Output the location where the modified Dockerfile file is referenced.

Signed-off-by: zepan <ze.pan@intel.com>

* for test

Signed-off-by: zepan <ze.pan@intel.com>

* Restore test file.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* refine logging code. (#559)

* add ut and refine logging code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update microservice port.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* adding lancedb to langchain vectorstores (#291)

* adding lancedb to langchain vectorstores

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Refine Dataprep Milvus MS (#570)

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* final version

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the readme

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the sign

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix error for pre ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update docker file

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update CI test log achieve (#577)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Multimodal dataprep (#575)

* multimodal embedding for MM RAG for videos

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* develop data prep first commit

Signed-off-by: Tiep Le <tiep.le@intel.com>

* develop dataprep microservice for multimodal data

Signed-off-by: Tiep Le <tiep.le@intel.com>

* multimodal langchain for dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: Tiep Le <tiep.le@intel.com>

* test for multimodal dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic update

Signed-off-by: Tiep Le <tiep.le@intel.com>

* remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update API to remove /dataprep from API names and remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update the error message per PR reviewer

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add: Pathway vector store and retriever as LangChain component (#342)

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* example data

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update, add commands

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused parts

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: image name, compose env vars

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: logging name

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: env var

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update input sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: mv vector store, naming, clarify instructions, improve ingestion components

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests: add pw retriever test
fix: update docker to include libmagic

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement suggestions from review, entrypoint, reqs, comments, https_proxy.

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: update docker tags in test and readme

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* tests: add separate pathway vectorstore test

Signed-off-by: Berke <berkecanrizai1@gmail.com>

---------

Signed-off-by: Berke <berkecanrizai1@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add local Rerank microservice for VideoRAGQnA (#496)

* initial commit

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* save

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add readme, test script, fix bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update video URL

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use default

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update core dependency

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use p 5000

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use 5037

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update ctnr name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* remove langsmith

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add rerank algo desc in readme

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add Scan Container. (#560)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix SearchedMultimodalDoc in docarray (#583)

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update image build yaml (#529)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add microservice for intent detection (#131)

* add microservice for intent detection

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update license copyright

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* add ut

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* refine

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update folder

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

---------

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Make the scanning method optional. (#580)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add code owners (#586)

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* remove revision for tei (#584)

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Bug fix (#591)

* Check if the document exists.

Signed-off-by: zepan <ze.pan@intel.com>

* Add flag output.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify nginx readme.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify document detection logic

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix ut issue

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* merge the main

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* align with new pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* align with newest pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* upload code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

---------

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: Berke <berkecanrizai1@gmail.com>
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Huiling Bao <huiling.bao@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: Sharan Shirodkar <91109427+sharanshirodkar7@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Tiep Le <97980157+tileintel@users.noreply.github.com>
Co-authored-by: berkecanrizai <63911408+berkecanrizai@users.noreply.github.com>
Co-authored-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
 2024-09-10 12:08:42 +08:00
Huiling Bao
2c48bc80c7 Add Megaservice support for MMRAG VideoRAGQnA usecase (#603) 
* add videoragqna gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test script for gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm ip

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix exist bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
 2024-09-09 23:37:17 +09:00
Tiep Le
23cc3ea5ee Add schema to Redis initialization & Improve LVM-TGI For Multimodal Retriever Microservice (#638) 
* add schema to Redis initialization

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update lvm_tgi

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-09 11:59:05 +08:00
lvliang-intel
fb4b8d27e3 Fix LVM streaming issue (#637) 
* Fix LVM streaming issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-09 08:34:44 +09:00
lkk
7e1a2e57fd enable embedding finetuning (#639) 
* add embedding finetuning.
* enhance training on hpu.
* add embedding training ut.
 2024-09-09 07:29:19 +08:00
Tiep Le
99be1bdba7 Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase (#626) 
* updates

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update redis schema

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update multimodal config and docker compose retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update requirements

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update retriever redis

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* multimodal retriever implementation

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* test for multimodal retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* include prompt preparation for multimodal rag on videos application

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix template

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add test for llava for mm_rag_on_videos

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* update test

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* first update on gateaway

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix index not found

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add LVMSearchedMultimodalDoc

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement gateway for MultimodalRagQnAWithVideos

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* remove INDEX_SCHEMA

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update MultimodalRAGQnAWithVideosGateway with 2 megaservices

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* revise folder structure to comps/retrievers/langchain/redis_multimodal

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* add unittest for multimodalrag_qna_with_videos_gateway

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test mmrag qna with videos

Signed-off-by: Tiep Le <tiep.le@intel.com>

* change port of redis to resolve CI test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update lvms test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test for multimodal rag qna with videos gateway

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add more test to increase coverage

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic

Signed-off-by: Tiep Le <tiep.le@intel.com>

* add more test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update name of gateway

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com>
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
 2024-09-06 20:19:00 +08:00
ZePan110
2705e93d9f Dockerflie check (#609) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-06 18:58:17 +08:00
Steve Zhang
824a7e2c0d clear some unnecessary scripts and Dockerfile commands. (#635) 
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-06 18:54:57 +08:00
kevinintel
5537b7f6e0 Update CODEOWNERS (#632) 2024-09-06 18:54:42 +08:00
XinyuYe-Intel
7d9265f651 Support rerank model finetuning (#578) 
* support rerank model finetuning.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* adapt rerank model to transformers' scheme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* refined readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* modify command due to api change.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
 2024-09-06 14:14:27 +08:00
ZePan110
89197e54f3 To avoid port conflicts change port to other. (#628) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-06 13:56:54 +08:00
Tiep Le
1513998950 Retriever and lvm update for multimodal rag on videos (#606) 
* updates

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update redis schema

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update multimodal config and docker compose retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update requirements

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update retriever redis

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* multimodal retriever implementation

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* test for multimodal retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* include prompt preparation for multimodal rag on videos application

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix template

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add test for llava for mm_rag_on_videos

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* update test

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix index not found

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add LVMSearchedMultimodalDoc

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove INDEX_SCHEMA

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* revise folder structure to comps/retrievers/langchain/redis_multimodal

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* change port of redis to resolve CI test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update lvms test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com>
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-06 13:46:10 +08:00
Abolfazl Shahbazi
90cc44fd35 switch to using upstream 'tgi-gaudi' on HuggingFace (#616) 
* switch to using upstream 'tei-gaudi' on HuggingFace

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-06 12:51:29 +08:00
David Kinder
a8a46bc102 doc: fix heading levels in markdown content (#627) 
* only one H1 (#) heading for the title is allowed, so fix the extra H1
  headings (and the subheadings under those) to appropriate levels
* fix some inline code blocks containing leading/trailing spaces
* fix some indenting issues under an ordered list item

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
 2024-09-06 11:09:27 +08:00
Hoong Tee, Yeoh
b2e64d2844 feedback_management: Remove 'vim' from Dockerfile (#620) 
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-05 13:46:08 -07:00
Abolfazl Shahbazi
f1f866f7c1 Using '--no-cache-dir' within all Dockerfiles (#607) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-05 12:01:46 -07:00
WenjiaoYue
8369fbf537 Update checkpoint format (#619) 
* update upload_training_files format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update test api

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkpoint

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkpoint response

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* add defi

* add def

* update output_dir

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkoutPoint definition

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* map CHECKPOINT_ID and Model_PATH

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* delete test code

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* Update api_protocol.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* delete runtime_env

* update FineTuningJobCheckpoint format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update step_number to optional

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* Update the checkpoint return format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-05 22:48:59 +08:00
lkk
7d2cd6b4ff update finetuning doc (#615) 
* revert code because of overwriting by other pr.

* update finetuning doc.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-05 08:20:40 +08:00
Abolfazl Shahbazi
f4d123c442 updates to containers for finetuning composite (#556) 
* updates to containers for finetuning composite

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* updates to headers and pip installs

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Adding 'launch.sh' back

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

---------

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-04 14:28:03 -07:00
Huiling Bao
54aa943d82 Bugfix for PR 496 to add format_video_name function (#602) 
* add format_video_name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test for negative case

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update file path

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 23:26:15 +08:00
Lianhao Lu
b873cf833a dataprep: Fix issue in uploading docx with embedding image (#561) 
Fix issue #407

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
 2024-09-04 20:57:48 +08:00
Hoong Tee, Yeoh
72123b2321 Feedback management microservice component (#581) 
* comps: Implement feedback_management component

Implement feedback_management component with following endpoint:
- /v1/feedback/create: Create and save user feedback data from mongo
  database.
- /v1/feedback/get: Retrieve user feedback data from mongo database.
- /v1/feedback/delete: Delete user feedback data from mongo database
  entry.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* feedback_management: Include README

Include steps to build feedback_management microservice.
Include example to invoke API endpoint for the microservice.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* Include test for feedback_management microservice

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* feedback_management: Update schema

Update endpoint schema for data to be collected and stored in database.
Updated README and test script for the same.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typo in test_feedback_management_mongo

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

---------

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 20:49:04 +08:00
Sihan Chen
0bb69ac176 Optimize mega flow by removing microservice wrapper (#582) 
* refactor orchestrator

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove no_wrapper

* fix

* fix

* add align_gen

* add retriever and rerank params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add fake test for customize params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix dep

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 19:49:49 +08:00
WenjiaoYue
3367b76276 update upload_training_files format (#613) 
* update upload_training_files format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>
 2024-09-04 18:48:48 +08:00
chen, suyue
9007212af6 enhance image publish job (#614) 
* remove default services

Signed-off-by: chensuyue <suyue.chen@intel.com>

* update service name

Signed-off-by: chensuyue <suyue.chen@intel.com>

* fix image name

Signed-off-by: chensuyue <suyue.chen@intel.com>

* Add compose file. (#612)

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
 2024-09-04 17:33:43 +08:00
lkk
a924579a41 remove finetuning models limitation. (#573) 
* remove finetuning models limitation.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ut.

* update ut and add dashboard.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ut port.

* update finetuning params for customization.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change name.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 15:16:19 +08:00
Sameh Gobriel
445c9b19dd add VDMS retriever microservice for v0.9 Milestone (#539) 
* add VDMS retriever microservice

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* add retrieval gateway and logger back to init

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use 5009 in CI

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* change index_name to collection_name

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* fix var name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use index name all

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add deps

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* changes to address code reviews

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolve docarray

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add optional docarray embeddoc constraints

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug in comment

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* import DEBUG

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: s-gobriel <sameh.gobriel@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
 2024-09-04 14:33:36 +08:00
bjzhjing
01886fee6a Dataprep fetch page fix (#588) 
* dataprep: Fix to decode url start with "http";Get the encoding attr of web page

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 13:36:26 +08:00
ZePan110
3ce387ad71 Fix image build fail issue. (#611) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-04 13:21:59 +08:00
chen, suyue
341f97af96 image scan and publish cd enhance (#604) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-04 10:01:17 +08:00
ZePan110
2093558e8e Change image tag. (#601) 
* Change image tag.

Signed-off-by: zepan <ze.pan@intel.com>

* Change tag in workflow.

Signed-off-by: zepan <ze.pan@intel.com>

* Remove the CD portion of the retriever.

Signed-off-by: zepan <ze.pan@intel.com>

* Update lvms-compose-cd.yaml

Change '_' to '-'.

* Update lvms-compose-cd.yaml

* Update llms-compose-cd.yaml

* Add images.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
 2024-09-04 09:58:41 +08:00
feng-intel
f510b69eac ollama: Update curl proxy. (#508) 
Signed-off-by: fengding <feng1.ding@intel.com>
 2024-09-04 09:36:54 +08:00
sri-intel
2a53e2547d adding embedding support for CLIP based models for VideoRAGQnA example for v0.9 (#538) 
* clip embedding support

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* test script for embedding

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* fix freeze workflow (#522)

Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix Dataprep Potential Error in get_file (#540)

* fix get file error & refine logs

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Support SearchedDoc input type in LLM for No Rerank Pipeline (#541)

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add dependency for pdf2image and OCR processing (#421)

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add local_embedding return 768 length to align with chatqna example (#313)

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add telemetry doc (#536)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add video-llama LVM microservice under lvms  (#495)

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix the data load issue for structured files (#505)

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add finetuning component (#502)

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: test <test@intel.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add torchvision into requirements (#546)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Use Gaudi base images from Dockerhub (#526)

* Use Gaudi base images from Dockerhub

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Fixing the malformed tag

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* fix another malformed tag

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

---------

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add toxicity detection microservice (#338)

* Add toxicity detection microservice

Signed-off-by: Qun Gao <qun.gao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Modification to toxicity plugin PR  (#432)

* changed microservice to use Service.GUARDRAILS and input/output to TextDoc

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* simplify dockerfile to use langchain

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* sort requirements

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Minor SPDX header update (#434)

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Remove 'langsmith' per code review (#534)

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Add toxicity detection microservices with E2E testing

Signed-off-by: Qun Gao <qun.gao@intel.com>

---------

Signed-off-by: Qun Gao <qun.gao@intel.com>
Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: Tyler W <tyler.wilbers@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* rename script and use 5xxx

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add proxy for build

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* fixed commit issues

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix docarray constraint

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* updated docarray

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm telemetry which cause error in mega

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* renamed dirs

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* renamed test

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

---------

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Signed-off-by: Qun Gao <qun.gao@intel.com>
Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sun, Xuehao <xuehao.sun@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: Zaili Wang <109502517+ZailiWang@users.noreply.github.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Huiling Bao <huiling.bao@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: test <test@intel.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: qgao007 <108324932+qgao007@users.noreply.github.com>
Co-authored-by: Tyler W <tyler.wilbers@intel.com>
 2024-09-04 08:55:33 +08:00
ZePan110
8325d5d9ba Remove duplicate code (#596) 
Signed-off-by: zepan <ze.pan@intel.com>
 2024-09-03 14:17:26 +08:00
ZePan110
12fd97aec1 Bug fix (#591) 
* Check if the document exists.

Signed-off-by: zepan <ze.pan@intel.com>

* Add flag output.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify nginx readme.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify document detection logic

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
 2024-09-03 12:00:54 +08:00
Letong Han
d60907163e remove revision for tei (#584) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-09-03 10:27:40 +08:00
kevinintel
0379aebd28 add code owners (#586) 2024-09-03 06:53:55 +08:00
ZePan110
ae71eeee27 Make the scanning method optional. (#580) 
Signed-off-by: zepan <ze.pan@intel.com>
 2024-09-02 14:48:48 +08:00
Liangyx2
84a7e573b8 add microservice for intent detection (#131) 
* add microservice for intent detection

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update license copyright

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* add ut

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* refine

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update folder

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

---------

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-01 22:19:25 +08:00