diff --git a/graphs/.$openwebuilocalllms.drawio.bkp b/graphs/.$openwebuilocalllms.drawio.bkp index f0ca867..f781c06 100644 --- a/graphs/.$openwebuilocalllms.drawio.bkp +++ b/graphs/.$openwebuilocalllms.drawio.bkp @@ -1,9 +1,12 @@ - + + + + @@ -227,73 +230,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -304,73 +307,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -381,73 +384,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -458,88 +461,88 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -550,73 +553,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -627,76 +630,76 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -707,6 +710,120 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/graphs/openwebuilocalllms.drawio b/graphs/openwebuilocalllms.drawio index 0f0f894..96ab4f4 100644 --- a/graphs/openwebuilocalllms.drawio +++ b/graphs/openwebuilocalllms.drawio @@ -1,9 +1,12 @@ - + + + + @@ -227,73 +230,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -304,73 +307,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -381,73 +384,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -458,88 +461,88 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -550,73 +553,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -627,76 +630,76 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -707,15 +710,121 @@ - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/opsec/openwebuilocalllms/9.png b/opsec/openwebuilocalllms/9.png new file mode 100644 index 0000000..8f088e6 Binary files /dev/null and b/opsec/openwebuilocalllms/9.png differ diff --git a/opsec/openwebuilocalllms/index.html b/opsec/openwebuilocalllms/index.html index 91e4d1c..1eb2fa5 100644 --- a/opsec/openwebuilocalllms/index.html +++ b/opsec/openwebuilocalllms/index.html @@ -104,7 +104,7 @@
- Previous Page

oxeo0 - 2025 / 04 / 18

+ Previous Page

oxeo0 - 2025 / 04 / 20

Anonymity - Self-Hosted LLM Hidden Service

@@ -308,9 +308,19 @@ Personally, I was interested in Open LLMs since their inception - when ollama pr

To follow this tutorial, you'll need an AMD64 system running Debian 12. Although ollama can work on CPU only, the performance will be much worse than having model that fits in GPU's VRAM.
To comfortably use an 8B model, it's strongly advised to have a dedicated GPU with at least 6GB of VRAM. You can check the supported GPU models here.

This tutorial showcases ollama setup with Nvidia drivers, but AMD GPUs are also supported.

-

If you want to expose Open WebUI via Tor to access it remotely, you should have an onion v3 vanity address and Tor installed.

-

It's also possible to set this up inside a Proxmox VE or any KVM based VM. You just need to PCI passthrough appropriate GPU inside the Hardware tab:

- + +

+Here is how your setup may look like in the end of this tutorial: +

+ + +


+In case you decide to go with Proxmox VE hypervisor, you just need to PCI passthrough appropriate GPU inside the Hardware > Add > PCI device:

+ + +


+If you want to expose Open WebUI via Tor to access it remotely, you should have an onion v3 vanity address and Tor installed.

+
@@ -321,8 +331,9 @@ To comfortably use an 8B model, it's strongly advised to have a dedicated GPU wi

Docker Setup

-

To install Docker, follow the official guide: Install Docker Engine on Debian. After installation, add your user to the docker group:

-
oxeo@andromeda:~$ sudo /sbin/usermod -aG docker oxeo
+
Install Docker from Debian repositories using apt. After installation, add your user to the docker group:

+
oxeo@andromeda:~$ sudo apt install docker.io
+oxeo@andromeda:~$ sudo /sbin/usermod -aG docker oxeo
 oxeo@andromeda:~$ sudo systemctl enable docker
 

 
This ensures you can manage Docker without needing sudo privileges. Finally, reboot your system.