feat: tokenization endpoint (#3710)

endpoint to access the tokenizer

Signed-off-by: shraddhazpy <shraddha@shraddhafive.in>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Co-authored-by: Dave <dave@gray101.com>
This commit is contained in:
Shraddha 2024-10-02 12:26:18 +05:30 committed by GitHub
parent 0965c6cd68
commit 5488fc3bc1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 121 additions and 0 deletions

View file

@ -63,4 +63,7 @@ func RegisterLocalAIRoutes(app *fiber.App,
app.Get("/system", localai.SystemInformations(ml, appConfig))
// misc
app.Post("/v1/tokenize", localai.TokenizeEndpoint(cl, ml, appConfig))
}