mirror of
https://github.com/mudler/LocalAI.git
synced 2025-06-02 08:55:00 +00:00
feat(p2p): Federation and AI swarms (#2723)
* Wip p2p enhancements * get online state * Pass-by token to show in the dashboard Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Style * Minor fixups * parametrize SearchID * Refactoring * Allow to expose/bind more services Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add federation * Display federated mode in the WebUI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * make federated nodes visible from the WebUI * Fix version display * improve web page * live page update * visual enhancements * enhancements * visual enhancements --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
GitHub
parent
dd95ae130f
commit
cca881ec49
23 changed files with 815 additions and 82 deletions
|
|
@ -37,7 +37,7 @@ SOFTWARE.
|
|||
<body class="bg-gray-900 text-gray-200" x-data="{ key: $store.chat.key }">
|
||||
<div class="flex flex-col min-h-screen">
|
||||
|
||||
{{template "views/partials/navbar"}}
|
||||
{{template "views/partials/navbar" .}}
|
||||
<div class="chat-container mt-2 mr-2 ml-2 mb-2 bg-gray-800 shadow-lg rounded-lg" >
|
||||
<!-- Chat Header -->
|
||||
<div class="border-b border-gray-700 p-4" x-data="{ component: 'menu' }">
|
||||
|
|
|
|||
150
core/http/views/p2p.html
Normal file
150
core/http/views/p2p.html
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
{{template "views/partials/head" .}}
|
||||
|
||||
<body class="bg-gray-900 text-gray-200">
|
||||
<div class="flex flex-col min-h-screen">
|
||||
|
||||
{{template "views/partials/navbar" .}}
|
||||
<div class="container mx-auto px-4 flex-grow">
|
||||
<div class="workers mt-12 text-center">
|
||||
|
||||
<h2 class="text-3xl font-semibold text-gray-100 mb-8">
|
||||
<i class="fa-solid fa-circle-nodes"></i> Distributed inference with P2P
|
||||
<a href="https://localai.io/features/distribute/" target="_blank">
|
||||
<i class="fas fa-circle-info pr-2"></i>
|
||||
</a>
|
||||
</h2>
|
||||
<h5 class="mb-4 text-justify">LocalAI uses P2P technologies to enable distribution of work between peers. It is possible to share an instance with Federation and/or split the weights of a model across peers (only available with llama.cpp models). You can now share computational resources between your devices or your friends!</h5>
|
||||
|
||||
<!-- Federation Box -->
|
||||
<div class="bg-gray-800 p-6 rounded-lg shadow-lg mb-12 text-left">
|
||||
|
||||
<p class="text-xl font-semibold text-gray-200"> <i class="text-gray-200 fa-solid fa-circle-nodes"></i> Federated Nodes: <span hx-get="/p2p/ui/workers-federation-stats" hx-trigger="every 1s"></span> </p>
|
||||
<p class="mb-4">You can start LocalAI in federated mode to share your instance, or start the federated server to balance requests between nodes of the federation.</p>
|
||||
|
||||
<div class="grid grid-cols-1 sm:grid-cols-2 md:grid-cols-3 gap-4 mb-12">
|
||||
<div hx-get="/p2p/ui/workers-federation" hx-trigger="every 1s"></div>
|
||||
</div>
|
||||
|
||||
<hr class="border-gray-700 mb-12">
|
||||
|
||||
<h3 class="text-2xl font-semibold text-gray-100 mb-6"><i class="fa-solid fa-book"></i> Start a federated instance</h3>
|
||||
|
||||
|
||||
<!-- Tabs navigation -->
|
||||
<ul class="mb-5 flex list-none flex-row flex-wrap ps-0" role="tablist" data-twe-nav-ref>
|
||||
<li role="presentation" class="flex-auto text-center">
|
||||
<a href="#tabs-federated-cli" class="tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700 active" data-twe-toggle="pill" data-twe-target="#tabs-federated-cli" data-twe-nav-active role="tab" aria-controls="tabs-federated-cli" aria-selected="true"><i class="fa-solid fa-terminal"></i> CLI</a>
|
||||
</li>
|
||||
<li role="presentation" class="flex-auto text-center">
|
||||
<a href="#tabs-federated-docker" class="tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700" data-twe-toggle="pill" data-twe-target="#tabs-federated-docker" role="tab" aria-controls="tabs-federated-docker" aria-selected="false"><i class="fa-solid fa-box-open"></i> Container images</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<!-- Tabs content -->
|
||||
<div class="mb-6">
|
||||
|
||||
<div class="tabcontent hidden opacity-100 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id="tabs-federated-cli" role="tabpanel" aria-labelledby="tabs-federated-cli" data-twe-tab-active>
|
||||
|
||||
|
||||
<p class="mb-2">To start a new instance to share:</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
# Start a new instance to share with --federated and a TOKEN<br>
|
||||
export TOKEN="<span class="token">{{.P2PToken}}</span>"<br>
|
||||
local-ai run --federated --p2p
|
||||
</code>
|
||||
|
||||
<p class="mt-2">Note: If you don't have a token do not specify it and use the generated one that you can find in this page.</p>
|
||||
|
||||
<p class="mb-2">To start a new federated load balancer:</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
export TOKEN="<span class="token">{{.P2PToken}}</span>"<br>
|
||||
local-ai federated
|
||||
</code>
|
||||
|
||||
<p class="mt-2">Note: Token is needed when starting the federated server.</p>
|
||||
|
||||
<p class="mt-2">For all the options available, please refer to the <a href="https://localai.io/features/distribute/#starting-workers" target="_blank" class="text-yellow-300 hover:text-yellow-400">documentation</a>.</p>
|
||||
</div>
|
||||
<div class="tabcontent hidden opacity-0 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id="tabs-federated-docker" role="tabpanel" aria-labelledby="tabs-federated-docker">
|
||||
<p class="mb-2">To start a new federated instance:</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
docker run -ti --net host -e TOKEN="<span class="token">{{.P2PToken}}</span>" --name local-ai -p 8080:8080 localai/localai:latest-cpu run --federated --p2p
|
||||
</code>
|
||||
|
||||
<p class="mb-2">To start a new federated server (port to 9090):</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
docker run -ti --net host -e TOKEN="<span class="token">{{.P2PToken}}</span>" --name local-ai -p 9090:8080 localai/localai:latest-cpu federated
|
||||
</code>
|
||||
|
||||
<p class="mt-2">For all the options available and see what image to use, please refer to the <a href="https://localai.io/basics/container/" target="_blank" class="text-yellow-300 hover:text-yellow-400">Container images documentation</a> and <a href="https://localai.io/advanced/#cli-parameters" target="_blank" class="text-yellow-300 hover:text-yellow-400">CLI parameters documentation</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Llama.cpp Box -->
|
||||
|
||||
<div class="bg-gray-800 p-6 rounded-lg shadow-lg mb-12 text-left">
|
||||
|
||||
<p class="text-xl font-semibold text-gray-200"> <i class="text-gray-200 fa-solid fa-circle-nodes"></i> Workers (llama.cpp): <span hx-get="/p2p/ui/workers-stats" hx-trigger="every 1s"></span> </p>
|
||||
<p class="mb-4">You can start llama.cpp workers to distribute weights between the workers and offload part of the computation. To start a new worker, you can use the CLI or Docker.</p>
|
||||
|
||||
<div class="grid grid-cols-1 sm:grid-cols-2 md:grid-cols-3 gap-4 mb-12">
|
||||
<div hx-get="/p2p/ui/workers" hx-trigger="every 1s"></div>
|
||||
</div>
|
||||
<hr class="border-gray-700 mb-12">
|
||||
|
||||
<h3 class="text-2xl font-semibold text-gray-100 mb-6"><i class="fa-solid fa-book"></i> Start a new llama.cpp P2P worker</h3>
|
||||
|
||||
<!-- Tabs navigation -->
|
||||
<ul class="mb-5 flex list-none flex-row flex-wrap ps-0" role="tablist" data-twe-nav-ref>
|
||||
<li role="presentation" class="flex-auto text-center">
|
||||
<a href="#tabs-cli" class="tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700 active" data-twe-toggle="pill" data-twe-target="#tabs-cli" data-twe-nav-active role="tab" aria-controls="tabs-cli" aria-selected="true"><i class="fa-solid fa-terminal"></i> CLI</a>
|
||||
</li>
|
||||
<li role="presentation" class="flex-auto text-center">
|
||||
<a href="#tabs-docker" class="tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700" data-twe-toggle="pill" data-twe-target="#tabs-docker" role="tab" aria-controls="tabs-docker" aria-selected="false"><i class="fa-solid fa-box-open"></i> Container images</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<!-- Tabs content -->
|
||||
<div class="mb-6">
|
||||
<div class="tabcontent hidden opacity-100 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id="tabs-cli" role="tabpanel" aria-labelledby="tabs-cli" data-twe-tab-active>
|
||||
<p class="mb-2">To start a new worker, run the following command:</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
export TOKEN="<span class="token">{{.P2PToken}}</span>"<br>
|
||||
local-ai worker p2p-llama-cpp-rpc
|
||||
</code>
|
||||
|
||||
<p class="mt-2">For all the options available, please refer to the <a href="https://localai.io/features/distribute/#starting-workers" target="_blank" class="text-yellow-300 hover:text-yellow-400">documentation</a>.</p>
|
||||
</div>
|
||||
<div class="tabcontent hidden opacity-0 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id="tabs-docker" role="tabpanel" aria-labelledby="tabs-docker">
|
||||
<p class="mb-2">To start a new worker with docker, run the following command:</p>
|
||||
<code class="block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words">
|
||||
docker run -ti --net host -e TOKEN="<span class="token">{{.P2PToken}}</span>" --name local-ai -p 8080:8080 localai/localai:latest-cpu worker p2p-llama-cpp-rpc
|
||||
</code>
|
||||
|
||||
<p class="mt-2">For all the options available and see what image to use, please refer to the <a href="https://localai.io/basics/container/" target="_blank" class="text-yellow-300 hover:text-yellow-400">Container images documentation</a> and <a href="https://localai.io/advanced/#cli-parameters" target="_blank" class="text-yellow-300 hover:text-yellow-400">CLI parameters documentation</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Llama.cpp Box END -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{template "views/partials/footer" .}}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.token {
|
||||
word-break: break-all;
|
||||
}
|
||||
.workers .grid div {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
justify-content: space-between;
|
||||
}
|
||||
</style>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -1,4 +1,5 @@
|
|||
<footer class="text-center py-8">
|
||||
LocalAI Version {{.Version}}<br>
|
||||
<a href='https://localai.io' class="text-blue-400 hover:text-blue-600" target="_blank">LocalAI</a> © 2023-2024 <a href='https://mudler.pm' class="text-blue-400 hover:text-blue-600" target="_blank">Ettore Di Giacinto</a>
|
||||
</footer>
|
||||
</footer>
|
||||
<script src="/static/assets/tw-elements.js"></script>
|
||||
|
|
|
|||
|
|
@ -21,6 +21,9 @@
|
|||
<a href="/text2image/" class="text-gray-400 hover:text-white px-3 py-2 rounded"><i class="fas fa-image pr-2"></i> Generate images</a>
|
||||
<a href="/tts/" class="text-gray-400 hover:text-white px-3 py-2 rounded"><i class="fa-solid fa-music pr-2"></i> TTS </a>
|
||||
<a href="/talk/" class="text-gray-400 hover:text-white px-3 py-2 rounded"><i class="fa-solid fa-phone pr-2"></i> Talk </a>
|
||||
{{ if .IsP2PEnabled }}
|
||||
<a href="/p2p/" class="text-gray-400 hover:text-white px-3 py-2 rounded"><i class="fa-solid fa-circle-nodes"></i> Swarm </a>
|
||||
{{ end }}
|
||||
<a href="/swagger/" class="text-gray-400 hover:text-white px-3 py-2 rounded"><i class="fas fa-code pr-2"></i> API</a>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -34,6 +37,9 @@
|
|||
<a href="/text2image/" class="block text-gray-400 hover:text-white px-3 py-2 rounded mt-1"><i class="fas fa-image pr-2"></i> Generate images</a>
|
||||
<a href="/tts/" class="block text-gray-400 hover:text-white px-3 py-2 rounded mt-1"><i class="fa-solid fa-music pr-2"></i> TTS </a>
|
||||
<a href="/talk/" class="block text-gray-400 hover:text-white px-3 py-2 rounded mt-1"><i class="fa-solid fa-phone pr-2"></i> Talk </a>
|
||||
{{ if .IsP2PEnabled }}
|
||||
<a href="/p2p/" class="block text-gray-400 hover:text-white px-3 py-2 rounded mt-1"><i class="fa-solid fa-circle-nodes"></i> Swarm </a>
|
||||
{{ end }}
|
||||
<a href="/swagger/" class="block text-gray-400 hover:text-white px-3 py-2 rounded mt-1"><i class="fas fa-code pr-2"></i> API</a>
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
<body class="bg-gray-900 text-gray-200" x-data="{ key: $store.chat.key }">
|
||||
<div class="flex flex-col min-h-screen">
|
||||
|
||||
{{template "views/partials/navbar"}}
|
||||
{{template "views/partials/navbar" .}}
|
||||
<div class="chat-container mt-2 mr-2 ml-2 mb-2 bg-gray-800 shadow-lg rounded-lg " >
|
||||
<!-- Chat Header -->
|
||||
<div class="border-b border-gray-700 p-4" x-data="{ component: 'menu' }">
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue