dearwolf/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2025-05-21 11:04:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

feat: Openvino runtime for transformer backend and streaming support for Openvino and CUDA (#1892)

Browse source 
* fixes #1775 and #1774

Add BitsAndBytes Quantization and fixes embedding on CUDA devices

* Manage 4bit and 8 bit quantization

Manage different BitsAndBytes options with the quantization: parameter in yaml

* fix compilation errors on non CUDA environment

* OpenVINO draft

First draft of OpenVINO integration in transformer backend

* first working implementation

* Streaming working

* Small fix for regression on CUDA and XPU

* use pip version of optimum[openvino]

* Update backend/python/transformers/transformers_server.py

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
This commit is contained in:
fakezeta 2024-03-27 00:31:43 +01:00 • committed by GitHub
parent b500ceaf73
commit e7cbe32601
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 90 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

7
backend/python/common-env/transformers/transformers.yml
View file

@ -34,6 +34,7 @@ dependencies:
- boto3==1.28.61
- botocore==1.31.61
- certifi==2023.7.22
- coloredlogs==15.0.1
- TTS==0.22.0
- charset-normalizer==3.3.0
- datasets==2.14.5
@ -48,6 +49,7 @@ dependencies:
- funcy==2.0
- grpcio==1.59.0
- huggingface-hub
- humanfriendly==10.0
- idna==3.4
- jinja2==3.1.2
- jmespath==1.0.1
@ -57,7 +59,10 @@ dependencies:
- multiprocess==0.70.15
- networkx
- numpy==1.26.0
- optimum==1.17.1
- onnx==1.15.0
- openvino==2024.0.0
- openvino-telemetry==2023.2.1
- optimum[openvino]==1.17.1
- packaging==23.2
- pandas
- peft==0.5.0