jandres/GenAIExamples
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
431 Commits 43 Branches 9 Tags
1b6989797fa00b21ccc58f1a1593f7f5bd81917a
Commit Graph

431 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lvliang-intel
1b6989797f Add OpenAI client access OPEA microservice UT cases (#653) 
* Add OpeanAI access microservice UT cases

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add the absolute path

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* refactor code

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* add openai denpendency

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* install openai in scripts

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix ci issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-12 00:08:14 +08:00
XuhuiRen
a672569551 Fix security issue (#673) 
Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
 2024-09-12 00:01:58 +08:00
lvliang-intel
cff0a4dfbb Support export megaservice yaml to docker compose file (#642) 
* Support export megaservice yaml tp docker compose file

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ut case

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add code spell ignore

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix ut issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* disable cli case

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 22:26:43 +08:00
Zhenzhong1
3ac391a1d2 Fixed CI yaml (#672) 
* fixed CI yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 21:40:00 +08:00
Zhenzhong1
af4e0f8775 [ChatQnA] Support K8S Python Client to export ChatQnA E2E manifests (#634) 
* initialize the exporter

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update manifests_exporter

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update manifests_exporter

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* baseline

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove HF_TOKEN

* E2E pass

* gloden manifests & remove duplicated configs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* gloden manifests & remove duplicated configs

* golden scripts

* rename

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added replicas and removed useless6008 6009

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add the yaml load func

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added UT

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cleancode

* added kubernetes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added UT

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
 2024-09-11 21:05:11 +08:00
Steve Zhang
161c338782 fix vllm components images building. (#667) 
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-11 20:50:45 +08:00
chen, suyue
2d5130fc15 temp remove dockerfile check (#670) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-11 18:32:02 +08:00
ZePan110
61b8fa9b51 Opt filecheck (#663) 
* update description

Signed-off-by: chensuyue <suyue.chen@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Optimize path-scan.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* For test.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test1

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test2.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test3

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Restore test file.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test4

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix syntax error

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test

Signed-off-by: ZePan110 <ze.pan@intel.com>

* test

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix issue.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Remove debug output

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Restore test file and remove debug output.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Remove deprecated code.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Modify output.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* modify output

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix file path error.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix issue

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix error

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix error.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* Fix error.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error

Signed-off-by: ZePan110 <ze.pan@intel.com>

* resolve confilct.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error.

Signed-off-by: ZePan110 <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 18:05:35 +08:00
Letong Han
7e40475b6d Refine READMEs after reorg (#666) 
* update dataprep readme

Signed-off-by: letonghan <letong.han@intel.com>

* update vectorstore readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* update retriever readme

Signed-off-by: letonghan <letong.han@intel.com>

* udpate embedding readme

Signed-off-by: letonghan <letong.han@intel.com>

* update guardrails readme

Signed-off-by: letonghan <letong.han@intel.com>

* update other readmes

Signed-off-by: letonghan <letong.han@intel.com>

* update reranks readme

Signed-off-by: letonghan <letong.han@intel.com>

* update llm&lvms readme

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 17:23:40 +08:00
Tiep Le
7664578c97 Support for UI of MultimodalRAGWithVideos in GenAIExamples (#656) 
* using png instead of jpg due to tgi-gaudi. return message include video_id maps

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update lvm and lvm_tgi to handle empty retrieval results

Signed-off-by: Tiep Le <tiep.le@intel.com>

* handle no retrieval results

Signed-off-by: Tiep Le <tiep.le@intel.com>

* add metadata to textdoc returned by LVM

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add metadatatextdoc

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* add metadata to Chatcompletionresponsechoice

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug

Signed-off-by: Tiep Le <tiep.le@intel.com>

* fix bug

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor fix for more consice

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update port for test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
 2024-09-11 17:22:59 +08:00
XinyaoWa
8e3f553a55 Support Llama index for vLLM (#665) 
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 16:59:47 +08:00
Letong Han
267fb02c76 Move fintuning test script path (#664) 
Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-11 16:46:03 +08:00
lvliang-intel
98c62a016d Move neuralspeed embedding rerank and vllm-xft to catalog (#647) 
* Move neuralspeed embedding rerank and vllm-xft to catalog

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* delete xft code

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update d yaml

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
 2024-09-11 14:51:38 +08:00
Steve Zhang
1d761fa0a6 Fix the Retriever README error (#655) 
* change based on comments for missing places.
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-11 14:36:21 +08:00
sri-intel
f84d91a3b0 adding dataprep support for CLIP based models for VideoRAGQnA example for v1.0 (#621) 
* dataprep service

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* dataprep updates

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* rearranged dirs

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* added readme

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* removed checks

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* added features

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* added get method

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add dim at init, rm unused

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add wait after connect DB

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* Update comps/dataprep/vdms/README.md

Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add test script for mm case

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add return value and update readme

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* fix mm-script

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add into dataprep workflow

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* rm whitespace

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* updated readme and added test script

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* removed unused file

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move test script

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* restructured repo

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates path in test script

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add name for build

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com>
 2024-09-11 13:19:04 +08:00
chen, suyue
4165c7d8c1 optimize ci test scope (#662) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-11 12:04:01 +08:00
minmin-intel
42cde68dd6 fix ragagent text generator bug (#660) 
* fix ragagent text generator bug

Signed-off-by: minmin-intel <minmin.hou@intel.com>
 2024-09-11 08:16:25 +08:00
chen, suyue
1785149ecf minor fix for CI detect (#658) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-11 00:45:34 +08:00
Letong Han
a3da7c1e3f Fix Reorg Issues (#657) 
Signed-off-by: letonghan <letong.han@intel.com>
 2024-09-11 00:45:12 +08:00
Letong Han
bea9bb0242 [Reorg] Reorg Folder to Support Different Vendors (#625) 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 23:24:09 +08:00
ZePan110
2a91903627 Fix Dockerfile detect issue (#652) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-10 17:22:35 +08:00
lkk
c718602abf add resume finetuning checkpoint ut. (#646) 
* add resume finetuning checkpoint ut.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add final tuned model.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 17:35:23 +09:00
chen, suyue
94eb60f168 add PREDICTIONGUARD_API_KEY for CI (#650) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 15:56:47 +08:00
XinyuYe-Intel
29311479bb added finetuned model deployment tutorial in readme. (#648) 
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
 2024-09-10 15:52:15 +08:00
XuhuiRen
2fbce3e53d Fix the loading error of jsonl file (#644) 
* fixjsonl

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 15:50:40 +08:00
chen, suyue
48d4e53396 unify default reranking model with BAAI/bge-reranker-base (#623) 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-10 15:23:10 +08:00
ZePan110
8a1141314d Optimize the content of the alerts. (#640) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
 2024-09-10 13:43:06 +08:00
XinyuYe-Intel
58e9972d82 Add LLM pretraining support (#622) 
* support rerank model finetuning.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* adapt rerank model to transformers' scheme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* refined readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add LLM pretraining support.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* added rerank finetuning test.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added LLM pretraining test.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

---------

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-10 13:18:35 +08:00
Dong, Bo
d76751ad81 optimize rerank with backend ref (#579) 
* add rerank with neural speed

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix mismatched response format w/wo streaming guardrails (#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Fix guardrails out handle logics for space linebreak and quote (#571)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* debug

* debug

* debug

* fix pre-space and linebreak

* fix pre-space and linebreak

* fix single/double quote

* fix single/double quote

* remove debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* BUG FIX: LVM security fix (#572)

* add url validator

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Modify output messages. (#569)

* Reduced output.

Signed-off-by: zepan <ze.pan@intel.com>

* Output the location where the modified Dockerfile file is referenced.

Signed-off-by: zepan <ze.pan@intel.com>

* for test

Signed-off-by: zepan <ze.pan@intel.com>

* Restore test file.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* refine logging code. (#559)

* add ut and refine logging code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update microservice port.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* adding lancedb to langchain vectorstores (#291)

* adding lancedb to langchain vectorstores

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Refine Dataprep Milvus MS (#570)

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* final version

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the readme

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the sign

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix error for pre ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update docker file

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update CI test log achieve (#577)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Multimodal dataprep (#575)

* multimodal embedding for MM RAG for videos

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* develop data prep first commit

Signed-off-by: Tiep Le <tiep.le@intel.com>

* develop dataprep microservice for multimodal data

Signed-off-by: Tiep Le <tiep.le@intel.com>

* multimodal langchain for dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: Tiep Le <tiep.le@intel.com>

* test for multimodal dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic update

Signed-off-by: Tiep Le <tiep.le@intel.com>

* remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update API to remove /dataprep from API names and remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update the error message per PR reviewer

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add: Pathway vector store and retriever as LangChain component (#342)

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* example data

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update, add commands

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused parts

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: image name, compose env vars

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: logging name

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: env var

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update input sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: mv vector store, naming, clarify instructions, improve ingestion components

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests: add pw retriever test
fix: update docker to include libmagic

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement suggestions from review, entrypoint, reqs, comments, https_proxy.

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: update docker tags in test and readme

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* tests: add separate pathway vectorstore test

Signed-off-by: Berke <berkecanrizai1@gmail.com>

---------

Signed-off-by: Berke <berkecanrizai1@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add local Rerank microservice for VideoRAGQnA (#496)

* initial commit

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* save

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add readme, test script, fix bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update video URL

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use default

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update core dependency

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use p 5000

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use 5037

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update ctnr name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* remove langsmith

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add rerank algo desc in readme

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add Scan Container. (#560)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix SearchedMultimodalDoc in docarray (#583)

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update image build yaml (#529)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add microservice for intent detection (#131)

* add microservice for intent detection

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update license copyright

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* add ut

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* refine

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update folder

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

---------

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Make the scanning method optional. (#580)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add code owners (#586)

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* remove revision for tei (#584)

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Bug fix (#591)

* Check if the document exists.

Signed-off-by: zepan <ze.pan@intel.com>

* Add flag output.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify nginx readme.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify document detection logic

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix ut issue

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* merge the main

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* align with new pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* align with newest pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* upload code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

---------

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: Berke <berkecanrizai1@gmail.com>
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Huiling Bao <huiling.bao@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: Sharan Shirodkar <91109427+sharanshirodkar7@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Tiep Le <97980157+tileintel@users.noreply.github.com>
Co-authored-by: berkecanrizai <63911408+berkecanrizai@users.noreply.github.com>
Co-authored-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
 2024-09-10 12:08:42 +08:00
Huiling Bao
2c48bc80c7 Add Megaservice support for MMRAG VideoRAGQnA usecase (#603) 
* add videoragqna gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test script for gateway

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm ip

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix exist bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
 2024-09-09 23:37:17 +09:00
Tiep Le
23cc3ea5ee Add schema to Redis initialization & Improve LVM-TGI For Multimodal Retriever Microservice (#638) 
* add schema to Redis initialization

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update lvm_tgi

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-09 11:59:05 +08:00
lvliang-intel
fb4b8d27e3 Fix LVM streaming issue (#637) 
* Fix LVM streaming issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-09 08:34:44 +09:00
lkk
7e1a2e57fd enable embedding finetuning (#639) 
* add embedding finetuning.
* enhance training on hpu.
* add embedding training ut.
 2024-09-09 07:29:19 +08:00
Tiep Le
99be1bdba7 Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase (#626) 
* updates

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update redis schema

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update multimodal config and docker compose retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update requirements

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update retriever redis

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* multimodal retriever implementation

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* test for multimodal retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* include prompt preparation for multimodal rag on videos application

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix template

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add test for llava for mm_rag_on_videos

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* update test

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* first update on gateaway

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix index not found

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add LVMSearchedMultimodalDoc

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement gateway for MultimodalRagQnAWithVideos

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* remove INDEX_SCHEMA

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update MultimodalRAGQnAWithVideosGateway with 2 megaservices

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* revise folder structure to comps/retrievers/langchain/redis_multimodal

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* add unittest for multimodalrag_qna_with_videos_gateway

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test mmrag qna with videos

Signed-off-by: Tiep Le <tiep.le@intel.com>

* change port of redis to resolve CI test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update lvms test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test for multimodal rag qna with videos gateway

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add more test to increase coverage

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic

Signed-off-by: Tiep Le <tiep.le@intel.com>

* add more test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update name of gateway

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com>
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
 2024-09-06 20:19:00 +08:00
ZePan110
2705e93d9f Dockerflie check (#609) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-06 18:58:17 +08:00
Steve Zhang
824a7e2c0d clear some unnecessary scripts and Dockerfile commands. (#635) 
Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
 2024-09-06 18:54:57 +08:00
kevinintel
5537b7f6e0 Update CODEOWNERS (#632) 2024-09-06 18:54:42 +08:00
XinyuYe-Intel
7d9265f651 Support rerank model finetuning (#578) 
* support rerank model finetuning.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* adapt rerank model to transformers' scheme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* refined readme.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* modify command due to api change.

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
 2024-09-06 14:14:27 +08:00
ZePan110
89197e54f3 To avoid port conflicts change port to other. (#628) 
Signed-off-by: ZePan110 <ze.pan@intel.com>
 2024-09-06 13:56:54 +08:00
Tiep Le
1513998950 Retriever and lvm update for multimodal rag on videos (#606) 
* updates

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update redis schema

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update multimodal config and docker compose retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update requirements

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update retriever redis

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* multimodal retriever implementation

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* test for multimodal retriever

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* include prompt preparation for multimodal rag on videos application

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* fix template

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add test for llava for mm_rag_on_videos

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* update test

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix index not found

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* add LVMSearchedMultimodalDoc

Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove INDEX_SCHEMA

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* revise folder structure to comps/retrievers/langchain/redis_multimodal

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* change port of redis to resolve CI test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

* update lvms test

Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com>
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-06 13:46:10 +08:00
Abolfazl Shahbazi
90cc44fd35 switch to using upstream 'tgi-gaudi' on HuggingFace (#616) 
* switch to using upstream 'tei-gaudi' on HuggingFace

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-06 12:51:29 +08:00
David Kinder
a8a46bc102 doc: fix heading levels in markdown content (#627) 
* only one H1 (#) heading for the title is allowed, so fix the extra H1
  headings (and the subheadings under those) to appropriate levels
* fix some inline code blocks containing leading/trailing spaces
* fix some indenting issues under an ordered list item

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
 2024-09-06 11:09:27 +08:00
Hoong Tee, Yeoh
b2e64d2844 feedback_management: Remove 'vim' from Dockerfile (#620) 
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-05 13:46:08 -07:00
Abolfazl Shahbazi
f1f866f7c1 Using '--no-cache-dir' within all Dockerfiles (#607) 
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-05 12:01:46 -07:00
WenjiaoYue
8369fbf537 Update checkpoint format (#619) 
* update upload_training_files format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update test api

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkpoint

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkpoint response

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* add defi

* add def

* update output_dir

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* update checkoutPoint definition

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* map CHECKPOINT_ID and Model_PATH

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* delete test code

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* Update api_protocol.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* delete runtime_env

* update FineTuningJobCheckpoint format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update step_number to optional

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* Update the checkpoint return format

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-05 22:48:59 +08:00
lkk
7d2cd6b4ff update finetuning doc (#615) 
* revert code because of overwriting by other pr.

* update finetuning doc.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-05 08:20:40 +08:00
Abolfazl Shahbazi
f4d123c442 updates to containers for finetuning composite (#556) 
* updates to containers for finetuning composite

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* updates to headers and pip installs

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Adding 'launch.sh' back

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

---------

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
 2024-09-04 14:28:03 -07:00
Huiling Bao
54aa943d82 Bugfix for PR 496 to add format_video_name function (#602) 
* add format_video_name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test for negative case

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update file path

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 23:26:15 +08:00
Lianhao Lu
b873cf833a dataprep: Fix issue in uploading docx with embedding image (#561) 
Fix issue #407

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
 2024-09-04 20:57:48 +08:00
Hoong Tee, Yeoh
72123b2321 Feedback management microservice component (#581) 
* comps: Implement feedback_management component

Implement feedback_management component with following endpoint:
- /v1/feedback/create: Create and save user feedback data from mongo
  database.
- /v1/feedback/get: Retrieve user feedback data from mongo database.
- /v1/feedback/delete: Delete user feedback data from mongo database
  entry.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* feedback_management: Include README

Include steps to build feedback_management microservice.
Include example to invoke API endpoint for the microservice.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* Include test for feedback_management microservice

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* feedback_management: Update schema

Update endpoint schema for data to be collected and stored in database.
Updated README and test script for the same.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typo in test_feedback_management_mongo

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

---------

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-04 20:49:04 +08:00