dearwolf/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2025-05-20 10:35:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 21

fix(autogptq): do not use_triton with qwen-vl (#1985)

Browse source 
* Enhance autogptq backend to support VL models

* update dependencies for autogptq

* remove redundant auto-gptq dependency

* Convert base64 to image_url for Qwen-VL model

* implemented model inference for qwen-vl

* remove user prompt from generated answer

* fixed write image error

* fixed use_triton issue when loading Qwen-VL model

---------

Co-authored-by: Binghua Wu <bingwu@estee.com>
This commit is contained in:
Sebastian.W 2024-04-10 18:36:10 +08:00 • committed by GitHub
parent d692b2c32a
commit d23e73b118
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 0 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
backend/python/autogptq/autogptq.py
View file

@ -39,7 +39,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
self.model_name = "Qwen-VL-Chat" self.model_name = "Qwen-VL-Chat"
model = AutoModelForCausalLM.from_pretrained(model_path, model = AutoModelForCausalLM.from_pretrained(model_path,
trust_remote_code=request.TrustRemoteCode, trust_remote_code=request.TrustRemoteCode,
use_triton=request.UseTriton,
device_map="auto").eval() device_map="auto").eval()
else: else:
model = AutoGPTQForCausalLM.from_quantized(model_path, model = AutoGPTQForCausalLM.from_quantized(model_path,