diff --git a/README.md b/README.md index a65903d5..c5846094 100644 --- a/README.md +++ b/README.md @@ -45,26 +45,45 @@ Tested with: - [GPT4ALL-J](https://gpt4all.io/models/ggml-gpt4all-j.bin) - Koala - [cerebras-GPT with ggml](https://huggingface.co/lxe/Cerebras-GPT-2.7B-Alpaca-SP-ggml) +- WizardLM - [RWKV](https://github.com/BlinkDL/RWKV-LM) models with [rwkv.cpp](https://github.com/saharNooby/rwkv.cpp) -It should also be compatible with StableLM and GPTNeoX ggml models (untested) +### Vicuna, Alpaca, LLaMa... + +[llama.cpp](https://github.com/ggerganov/llama.cpp) based models are compatible + +### GPT4ALL Note: You might need to convert older models to the new format, see [here](https://github.com/ggerganov/llama.cpp#using-gpt4all) for instance to run `gpt4all`. +### GPT4ALL-J + +No changes required to the model. + ### RWKV
-For `rwkv` models, you need to put also the associated tokenizer along with the ggml model: +A full example on how to run a rwkv model is in the [examples](https://github.com/go-skynet/LocalAI/tree/master/examples/rwkv). + +Note: rwkv models have an associated tokenizer along that needs to be provided with it: ``` -ls models 36464540 -rw-r--r-- 1 mudler mudler 1.2G May 3 10:51 rwkv_small 36464543 -rw-r--r-- 1 mudler mudler 2.4M May 3 10:51 rwkv_small.tokenizer.json ```
+### Others + +It should also be compatible with StableLM and GPTNeoX ggml models (untested). + +### Hardware requirements + +Depending on the model you are attempting to run might need more RAM or CPU resources. Check out also [here](https://github.com/ggerganov/llama.cpp#memorydisk-requirements) for `ggml` based backends. `rwkv` is less expensive on resources. + + ## Usage > `LocalAI` comes by default as a container image. You can check out all the available images with corresponding tags [here](https://quay.io/repository/go-skynet/local-ai?tab=tags&tag=latest). diff --git a/main.go b/main.go index 98b7d71b..a57ac727 100644 --- a/main.go +++ b/main.go @@ -5,11 +5,8 @@ import ( "os" "path/filepath" - - api "github.com/go-skynet/LocalAI/api" model "github.com/go-skynet/LocalAI/pkg/model" - "github.com/jaypipes/ghw" "github.com/rs/zerolog" "github.com/rs/zerolog/log" "github.com/urfave/cli/v2" @@ -24,12 +21,6 @@ func main() { os.Exit(1) } - threads := 4 - cpu, err := ghw.CPU() - if err == nil { - threads = int(cpu.TotalCores) - } - app := &cli.App{ Name: "LocalAI", Usage: "OpenAI compatible API for running LLaMA/GPT models locally on CPU with consumer grade hardware.", @@ -46,7 +37,7 @@ func main() { Name: "threads", DefaultText: "Number of threads used for parallel computation. Usage of the number of physical cores in the system is suggested.", EnvVars: []string{"THREADS"}, - Value: threads, + Value: 4, }, &cli.StringFlag{ Name: "models-path",