XinyaoWa
3a31295ef8
Align parameters for "max_token, repetition_penalty,presence_penalty,frequency_penalty" ( #608 )
...
* align max_tokens
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* aligin repetition_penalty
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* align penalty parameters
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* align max_tokens
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* debug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* debug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix langchain version bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* fix langchain version bug
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
Co-authored-by: kevinintel <hanwen.chang@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com >
2024-09-18 20:39:23 +08:00
lvliang-intel
04385c97e7
Fix microservices which affected by langchain version upgrade ( #704 )
...
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
2024-09-18 17:26:55 +08:00
chen, suyue
3d00a33e09
update image build compose ( #698 )
...
Signed-off-by: chensuyue <suyue.chen@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-18 11:11:59 +08:00
Sharan Shirodkar
1249c4fe72
Prediction Guard LVM component ( #676 )
...
* added files for PG lvm component
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* Fix pre-commit issues: end-of-file, requirements.txt, trailing whitespace, imports, and formatting
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* added package
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
---------
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
2024-09-18 10:32:16 +08:00
lvliang-intel
8adbcceacd
Fix Ollama langchain upgrade issue ( #697 )
...
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
2024-09-18 09:52:35 +08:00
liuzhenwei
2e41dcf960
Support Llama index for vLLM native ( #692 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2024-09-18 07:12:44 +08:00
Sharan Shirodkar
391c4a5801
Prediction Guard LLM component ( #674 )
...
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
2024-09-17 21:57:06 +08:00
Sharan Shirodkar
191061b642
Prediction Guard embeddings component ( #675 )
...
* added files for PG embeddingso component
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* added package
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* fixed dockerfile link
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* Fix pre-commit issues: end-of-file, requirements.txt, trailing whitespace, imports, and formatting
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* added package
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* added package
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* fixed embedoc call
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* file structure updated to latest
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* Fix pre-commit issues: end-of-file, requirements.txt, trailing whitespace, imports, and formatting
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* added package
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
---------
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
2024-09-17 21:33:19 +08:00
qgao007
812c85cefb
Add Bias Detection Microservice ( #659 )
...
* Add Bias Detection Microservice
Signed-off-by: Qun Gao <qun.gao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adding Bias Detection Container to CI (#695 )
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com >
---------
Signed-off-by: Qun Gao <qun.gao@intel.com >
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com >
2024-09-17 10:01:22 +08:00
XuhuiRen
29fe569d34
Enable GraphRAG with Neo4J ( #682 )
...
* add graphrag for neo4j
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* fix ut
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* fix
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update retriever_neo4j.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* Update test_retrievers_neo4j_langchain.sh
* add
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* Update test_retrievers_neo4j_langchain.sh
* Update test_retrievers_neo4j_langchain.sh
* Update test_retrievers_neo4j_langchain.sh
* add docker
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* Update retrievers-compose-cd.yaml
* Update test_retrievers_neo4j_langchain.sh
* Update config.py
* Update test_retrievers_neo4j_langchain.sh
* Update test_retrievers_neo4j_langchain.sh
* Update config.py
* Update test_retrievers_neo4j_langchain.sh
* Update requirements.txt
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update requirements.txt
* Update requirements.txt
* Update requirements.txt
---------
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com >
2024-09-15 18:12:29 +08:00
Harsha Ramayanam
18092f3dec
Changes to comps/llms/text-generation/README ( #678 )
...
* Added changes per dbkinder's review
Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-14 13:52:56 +08:00
David Kinder
f8f8854f9c
doc: fix heading levels ( #690 )
...
Only one H1 for the title is allowed
Signed-off-by: David B. Kinder <david.b.kinder@intel.com >
2024-09-14 09:36:41 +08:00
Hoong Tee, Yeoh
0f2c2b1ed7
vllm langchain: Add Document Retriever Support ( #687 )
...
* vllm langchain: Add Document Retriever Support
Include SearchedDoc in /v1/chat/completions endpoint to accept document
data retreived from retriever service to parse into LLM for answer
generation.
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* vllm: Update README documentation
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-13 20:08:56 +08:00
Tiep Le
2ca56f3ed9
Update example name into MultimodalQnA and update image names ( #686 )
...
* update image names
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update image names for dataprep service
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update image name for retriever
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update image name for lvm
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update lvm include caption in output
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update gateway name into MultimodalQnAGateway
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test gateway for rename
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-13 17:07:38 +08:00
Huiling Bao
2b68323c2c
Rename videoragqna to videoqna to align with other examples ( #680 )
...
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
2024-09-12 22:12:39 +08:00
chen, suyue
0458443baf
update torch cpu installation ( #681 )
...
Signed-off-by: chensuyue <suyue.chen@intel.com >
2024-09-12 22:01:38 +08:00
XinyaoWa
ca94c60fb7
fix vllm llamaindex stream bug ( #679 )
...
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
2024-09-12 14:05:14 +08:00
chen, suyue
6ce65514e1
replace langchain/langchain:latest with python:3.11-slim ( #669 )
...
* replace langchain/langchain:latest with python:3.11-slim
Signed-off-by: chensuyue <suyue.chen@intel.com >
* fix test
Signed-off-by: chensuyue <suyue.chen@intel.com >
---------
Signed-off-by: chensuyue <suyue.chen@intel.com >
2024-09-12 13:24:50 +08:00
XuhuiRen
a672569551
Fix security issue ( #673 )
...
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
2024-09-12 00:01:58 +08:00
lvliang-intel
cff0a4dfbb
Support export megaservice yaml to docker compose file ( #642 )
...
* Support export megaservice yaml tp docker compose file
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add ut case
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix issue
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add code spell ignore
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* fix ut issue
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* disable cli case
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
---------
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-11 22:26:43 +08:00
Zhenzhong1
af4e0f8775
[ChatQnA] Support K8S Python Client to export ChatQnA E2E manifests ( #634 )
...
* initialize the exporter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update manifests_exporter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update manifests_exporter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* baseline
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove HF_TOKEN
* E2E pass
* gloden manifests & remove duplicated configs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gloden manifests & remove duplicated configs
* golden scripts
* rename
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* added replicas and removed useless6008 6009
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add the yaml load func
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* added UT
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cleancode
* added kubernetes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* added UT
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com >
2024-09-11 21:05:11 +08:00
Steve Zhang
161c338782
fix vllm components images building. ( #667 )
...
Signed-off-by: zhlsunshine <huailong.zhang@intel.com >
2024-09-11 20:50:45 +08:00
Letong Han
7e40475b6d
Refine READMEs after reorg ( #666 )
...
* update dataprep readme
Signed-off-by: letonghan <letong.han@intel.com >
* update vectorstore readme
Signed-off-by: letonghan <letong.han@intel.com >
* udpate retriever readme
Signed-off-by: letonghan <letong.han@intel.com >
* update retriever readme
Signed-off-by: letonghan <letong.han@intel.com >
* udpate embedding readme
Signed-off-by: letonghan <letong.han@intel.com >
* update guardrails readme
Signed-off-by: letonghan <letong.han@intel.com >
* update other readmes
Signed-off-by: letonghan <letong.han@intel.com >
* update reranks readme
Signed-off-by: letonghan <letong.han@intel.com >
* update llm&lvms readme
Signed-off-by: letonghan <letong.han@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: letonghan <letong.han@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-11 17:23:40 +08:00
Tiep Le
7664578c97
Support for UI of MultimodalRAGWithVideos in GenAIExamples ( #656 )
...
* using png instead of jpg due to tgi-gaudi. return message include video_id maps
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update lvm and lvm_tgi to handle empty retrieval results
Signed-off-by: Tiep Le <tiep.le@intel.com >
* handle no retrieval results
Signed-off-by: Tiep Le <tiep.le@intel.com >
* add metadata to textdoc returned by LVM
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add metadatatextdoc
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* add metadata to Chatcompletionresponsechoice
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix bug
Signed-off-by: Tiep Le <tiep.le@intel.com >
* fix bug
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix bug
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* minor fix for more consice
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update port for test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test
Signed-off-by: Tiep Le <tiep.le@intel.com >
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com >
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com >
Co-authored-by: chen, suyue <suyue.chen@intel.com >
2024-09-11 17:22:59 +08:00
XinyaoWa
8e3f553a55
Support Llama index for vLLM ( #665 )
...
Signed-off-by: Xinyao Wang <xinyao.wang@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-11 16:59:47 +08:00
lvliang-intel
98c62a016d
Move neuralspeed embedding rerank and vllm-xft to catalog ( #647 )
...
* Move neuralspeed embedding rerank and vllm-xft to catalog
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* delete xft code
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* update d yaml
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
---------
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
Co-authored-by: chen, suyue <suyue.chen@intel.com >
2024-09-11 14:51:38 +08:00
Steve Zhang
1d761fa0a6
Fix the Retriever README error ( #655 )
...
* change based on comments for missing places.
Signed-off-by: zhlsunshine <huailong.zhang@intel.com >
2024-09-11 14:36:21 +08:00
sri-intel
f84d91a3b0
adding dataprep support for CLIP based models for VideoRAGQnA example for v1.0 ( #621 )
...
* dataprep service
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* dataprep updates
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* rearranged dirs
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* added readme
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* removed checks
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* added features
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* added get method
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add dim at init, rm unused
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add wait after connect DB
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* Update comps/dataprep/vdms/README.md
Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com >
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* add test script for mm case
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* add return value and update readme
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* check bug
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* fix mm-script
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* add into dataprep workflow
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* rm whitespace
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* updated readme and added test script
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* removed unused file
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* move test script
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* restructured repo
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* updates path in test script
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
* add name for build
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
---------
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com >
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: BaoHuiling <huiling.bao@intel.com >
Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com >
2024-09-11 13:19:04 +08:00
minmin-intel
42cde68dd6
fix ragagent text generator bug ( #660 )
...
* fix ragagent text generator bug
Signed-off-by: minmin-intel <minmin.hou@intel.com >
2024-09-11 08:16:25 +08:00
Letong Han
a3da7c1e3f
Fix Reorg Issues ( #657 )
...
Signed-off-by: letonghan <letong.han@intel.com >
2024-09-11 00:45:12 +08:00
Letong Han
bea9bb0242
[Reorg] Reorg Folder to Support Different Vendors ( #625 )
...
Signed-off-by: letonghan <letong.han@intel.com >
Signed-off-by: chensuyue <suyue.chen@intel.com >
2024-09-10 23:24:09 +08:00
lkk
c718602abf
add resume finetuning checkpoint ut. ( #646 )
...
* add resume finetuning checkpoint ut.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add final tuned model.
---------
Co-authored-by: root <root@idc708073.jf.intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-10 17:35:23 +09:00
XinyuYe-Intel
29311479bb
added finetuned model deployment tutorial in readme. ( #648 )
...
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
2024-09-10 15:52:15 +08:00
XuhuiRen
2fbce3e53d
Fix the loading error of jsonl file ( #644 )
...
* fixjsonl
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: XuhuiRen <xuhui.ren@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-10 15:50:40 +08:00
chen, suyue
48d4e53396
unify default reranking model with BAAI/bge-reranker-base ( #623 )
...
Signed-off-by: chensuyue <suyue.chen@intel.com >
Signed-off-by: ZePan110 <ze.pan@intel.com >
2024-09-10 15:23:10 +08:00
XinyuYe-Intel
58e9972d82
Add LLM pretraining support ( #622 )
...
* support rerank model finetuning.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* adapt rerank model to transformers' scheme.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* refined readme.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add LLM pretraining support.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* added readme.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* added rerank finetuning test.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* added LLM pretraining test.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
---------
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-10 13:18:35 +08:00
Dong, Bo
d76751ad81
optimize rerank with backend ref ( #579 )
...
* add rerank with neural speed
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add the code
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add the code
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* fix mismatched response format w/wo streaming guardrails (#568 )
* fix mismatched response format w/wo streaming guardrails
* fix & debug
* fix & rm debug
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Fix guardrails out handle logics for space linebreak and quote (#571 )
* fix mismatched response format w/wo streaming guardrails
* fix & debug
* fix & rm debug
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* debug
* debug
* debug
* fix pre-space and linebreak
* fix pre-space and linebreak
* fix single/double quote
* fix single/double quote
* remove debug
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* BUG FIX: LVM security fix (#572 )
* add url validator
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add validation for video_url
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
---------
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Modify output messages. (#569 )
* Reduced output.
Signed-off-by: zepan <ze.pan@intel.com >
* Output the location where the modified Dockerfile file is referenced.
Signed-off-by: zepan <ze.pan@intel.com >
* for test
Signed-off-by: zepan <ze.pan@intel.com >
* Restore test file.
Signed-off-by: zepan <ze.pan@intel.com >
---------
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* refine logging code. (#559 )
* add ut and refine logging code.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update microservice port.
---------
Co-authored-by: root <root@idc708073.jf.intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* adding lancedb to langchain vectorstores (#291 )
* adding lancedb to langchain vectorstores
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Refine Dataprep Milvus MS (#570 )
Signed-off-by: letonghan <letong.han@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* final version
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* update the readme
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add the sign
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* fix error for pre ci
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add the ut
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* update docker file
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* update CI test log achieve (#577 )
Signed-off-by: chensuyue <suyue.chen@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Multimodal dataprep (#575 )
* multimodal embedding for MM RAG for videos
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* develop data prep first commit
Signed-off-by: Tiep Le <tiep.le@intel.com >
* develop dataprep microservice for multimodal data
Signed-off-by: Tiep Le <tiep.le@intel.com >
* multimodal langchain for dataprep
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update README
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update README
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update README
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update README
Signed-off-by: Tiep Le <tiep.le@intel.com >
* cosmetic
Signed-off-by: Tiep Le <tiep.le@intel.com >
* test for multimodal dataprep
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cosmetic update
Signed-off-by: Tiep Le <tiep.le@intel.com >
* remove langsmith
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update API to remove /dataprep from API names and remove langsmith
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* update the error message per PR reviewer
Signed-off-by: Tiep Le <tiep.le@intel.com >
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add: Pathway vector store and retriever as LangChain component (#342 )
* nb
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* init changes
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* docker
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* example data
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* docs(readme): update, add commands
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: formatting, data sources
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* docs(readme): update instructions, add comments
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: rm unused parts
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: image name, compose env vars
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: rm unused part
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: logging name
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: env var
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: rename pw docker
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* docs(readme): update input sources
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* nb
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* init changes
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: formatting, data sources
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* docs(readme): update instructions, add comments
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: rm unused part
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* fix: rename pw docker
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* feat: mv vector store, naming, clarify instructions, improve ingestion components
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: add pw retriever test
fix: update docker to include libmagic
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* implement suggestions from review, entrypoint, reqs, comments, https_proxy.
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: update docker tags in test and readme
Signed-off-by: Berke <berkecanrizai1@gmail.com >
* tests: add separate pathway vectorstore test
Signed-off-by: Berke <berkecanrizai1@gmail.com >
---------
Signed-off-by: Berke <berkecanrizai1@gmail.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Add local Rerank microservice for VideoRAGQnA (#496 )
* initial commit
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* save
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* add readme, test script, fix bug
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* update video URL
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* use default
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update core dependency
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use p 5000
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* use 5037
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* update ctnr name
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* remove langsmith
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add rerank algo desc in readme
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Add Scan Container. (#560 )
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* fix SearchedMultimodalDoc in docarray (#583 )
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* update image build yaml (#529 )
Signed-off-by: chensuyue <suyue.chen@intel.com >
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add microservice for intent detection (#131 )
* add microservice for intent detection
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update license copyright
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
* add ut
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
* refine
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update folder
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix test
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
---------
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Make the scanning method optional. (#580 )
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add code owners (#586 )
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* remove revision for tei (#584 )
Signed-off-by: letonghan <letong.han@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* Bug fix (#591 )
* Check if the document exists.
Signed-off-by: zepan <ze.pan@intel.com >
* Add flag output.
Signed-off-by: zepan <ze.pan@intel.com >
* Modify nginx readme.
Signed-off-by: zepan <ze.pan@intel.com >
* Modify document detection logic
Signed-off-by: zepan <ze.pan@intel.com >
---------
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* fix ut issue
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* merge the main
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* align with new pipeline
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* align with newest pipeline
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* upload code
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* update the ut
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add docker path
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
* add the docker path
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
---------
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com >
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Signed-off-by: zepan <ze.pan@intel.com >
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com >
Signed-off-by: letonghan <letong.han@intel.com >
Signed-off-by: chensuyue <suyue.chen@intel.com >
Signed-off-by: Tiep Le <tiep.le@intel.com >
Signed-off-by: Berke <berkecanrizai1@gmail.com >
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com >
Co-authored-by: Huiling Bao <huiling.bao@intel.com >
Co-authored-by: ZePan110 <ze.pan@intel.com >
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com >
Co-authored-by: root <root@idc708073.jf.intel.com >
Co-authored-by: Sharan Shirodkar <91109427+sharanshirodkar7@users.noreply.github.com >
Co-authored-by: lvliang-intel <liang1.lv@intel.com >
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com >
Co-authored-by: chen, suyue <suyue.chen@intel.com >
Co-authored-by: Tiep Le <97980157+tileintel@users.noreply.github.com >
Co-authored-by: berkecanrizai <63911408+berkecanrizai@users.noreply.github.com >
Co-authored-by: Liangyx2 <yuxiang.liang@intel.com >
Co-authored-by: kevinintel <hanwen.chang@intel.com >
2024-09-10 12:08:42 +08:00
Huiling Bao
2c48bc80c7
Add Megaservice support for MMRAG VideoRAGQnA usecase ( #603 )
...
* add videoragqna gateway
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add test script for gateway
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* rm ip
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix exist bug
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: BaoHuiling <huiling.bao@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com >
2024-09-09 23:37:17 +09:00
Tiep Le
23cc3ea5ee
Add schema to Redis initialization & Improve LVM-TGI For Multimodal Retriever Microservice ( #638 )
...
* add schema to Redis initialization
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update lvm_tgi
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-09 11:59:05 +08:00
lvliang-intel
fb4b8d27e3
Fix LVM streaming issue ( #637 )
...
* Fix LVM streaming issue
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: lvliang-intel <liang1.lv@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-09 08:34:44 +09:00
lkk
7e1a2e57fd
enable embedding finetuning ( #639 )
...
* add embedding finetuning.
* enhance training on hpu.
* add embedding training ut.
2024-09-09 07:29:19 +08:00
Tiep Le
99be1bdba7
Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase ( #626 )
...
* updates
Signed-off-by: Tiep Le <tiep.le@intel.com >
* cosmetic
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update redis schema
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update multimodal config and docker compose retriever
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update requirements
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update retriever redis
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* multimodal retriever implementation
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* test for multimodal retriever
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* include prompt preparation for multimodal rag on videos application
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* fix template
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* add test for llava for mm_rag_on_videos
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* update test
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* first update on gateaway
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* fix index not found
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* add LVMSearchedMultimodalDoc
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* implement gateway for MultimodalRagQnAWithVideos
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* remove INDEX_SCHEMA
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update MultimodalRAGQnAWithVideosGateway with 2 megaservices
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* revise folder structure to comps/retrievers/langchain/redis_multimodal
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* add unittest for multimodalrag_qna_with_videos_gateway
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test mmrag qna with videos
Signed-off-by: Tiep Le <tiep.le@intel.com >
* change port of redis to resolve CI test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update lvms test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test for multimodal rag qna with videos gateway
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add more test to increase coverage
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cosmetic
Signed-off-by: Tiep Le <tiep.le@intel.com >
* add more test
Signed-off-by: Tiep Le <tiep.le@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update name of gateway
Signed-off-by: Tiep Le <tiep.le@intel.com >
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com >
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com >
2024-09-06 20:19:00 +08:00
Steve Zhang
824a7e2c0d
clear some unnecessary scripts and Dockerfile commands. ( #635 )
...
Signed-off-by: zhlsunshine <huailong.zhang@intel.com >
2024-09-06 18:54:57 +08:00
XinyuYe-Intel
7d9265f651
Support rerank model finetuning ( #578 )
...
* support rerank model finetuning.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* adapt rerank model to transformers' scheme.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* refined readme.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* modify command due to api change.
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com >
2024-09-06 14:14:27 +08:00
ZePan110
89197e54f3
To avoid port conflicts change port to other. ( #628 )
...
Signed-off-by: ZePan110 <ze.pan@intel.com >
2024-09-06 13:56:54 +08:00
Tiep Le
1513998950
Retriever and lvm update for multimodal rag on videos ( #606 )
...
* updates
Signed-off-by: Tiep Le <tiep.le@intel.com >
* cosmetic
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update redis schema
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update multimodal config and docker compose retriever
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update requirements
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update retriever redis
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* multimodal retriever implementation
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* test for multimodal retriever
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* include prompt preparation for multimodal rag on videos application
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* fix template
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* add test for llava for mm_rag_on_videos
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* update test
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix index not found
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* add LVMSearchedMultimodalDoc
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove INDEX_SCHEMA
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* revise folder structure to comps/retrievers/langchain/redis_multimodal
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* change port of redis to resolve CI test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
* update lvms test
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
---------
Signed-off-by: Tiep Le <tiep.le@intel.com >
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com >
Signed-off-by: sjagtap1803 <siddhant.jagtap@intel.com >
Co-authored-by: siddhivelankar23 <siddhi.velankar@intel.com >
Co-authored-by: sjagtap1803 <siddhant.jagtap@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-06 13:46:10 +08:00
David Kinder
a8a46bc102
doc: fix heading levels in markdown content ( #627 )
...
* only one H1 (#) heading for the title is allowed, so fix the extra H1
headings (and the subheadings under those) to appropriate levels
* fix some inline code blocks containing leading/trailing spaces
* fix some indenting issues under an ordered list item
Signed-off-by: David B. Kinder <david.b.kinder@intel.com >
2024-09-06 11:09:27 +08:00
Hoong Tee, Yeoh
b2e64d2844
feedback_management: Remove 'vim' from Dockerfile ( #620 )
...
Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com >
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com >
2024-09-05 13:46:08 -07:00
Abolfazl Shahbazi
f1f866f7c1
Using '--no-cache-dir' within all Dockerfiles ( #607 )
...
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com >
2024-09-05 12:01:46 -07:00
WenjiaoYue
8369fbf537
Update checkpoint format ( #619 )
...
* update upload_training_files format
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update test api
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* update checkpoint
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* update checkpoint response
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* add defi
* add def
* update output_dir
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* update checkoutPoint definition
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* map CHECKPOINT_ID and Model_PATH
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* delete test code
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* Update api_protocol.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* delete runtime_env
* update FineTuningJobCheckpoint format
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update step_number to optional
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* Update the checkpoint return format
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com >
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-05 22:48:59 +08:00
