diff --git a/graphs/.$monitoring.drawio.bkp b/graphs/.$monitoring.drawio.bkp
new file mode 100644
index 0000000..b69124f
--- /dev/null
+++ b/graphs/.$monitoring.drawio.bkp
@@ -0,0 +1,77 @@
+
+ +
+
+apt update
+apt install prometheus-node-exporter tor
+systemctl stop tor #stop the tor service
+
+mkdir -p /var/lib/tor/onion/prometheus/authorized_clients #create the client auth keys folder to store our second layer of authentication
+chmod 400 -R /var/lib/tor/prometheus #set restrictive file permissions
+
+vi /etc/tor/torrc #edit the torrc file to add content
+
+cat /etc/tor/torrc
+AutomapHostsSuffixes .onion,.exit
+DataDirectory /var/lib/tor
+SOCKSPort 127.0.0.1:9050 IsolateDestAddr
+HiddenServiceDir /var/lib/tor/onion/prometheus
+HiddenServicePort 9100 127.0.0.1:9100
+
+tor-client-auth-gen
+private_key=descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ #paste this key to your local machine as your prometheus node will need it
+echo "descriptor:x25519:6HDNHLLKIFNU5Q6T75B6Q3GBYDO5ZF4SQUX7EYDEKWNLPQUWUBTA" > /var/lib/tor/onion/prometheus/0.auth
+
+chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder
+
+systemctl start tor #restart tor
+systemctl status tor #check that everything works
+
+cat /var/lib/tor/onion/prometheus/hostname
+[clientaddr].onion
+
+
+What's that tor-client-auth-gen you ask? In order to protect this critical service from attacks that could be done against the grafana servers or from stolen credentials we need more than just security by obscurity
+(relying on the attacker not knowing our hidden service address).
+
+
+sudo systemctl stop tor #stop the tor service
+
+mkdir -p /var/lib/tor/auth_keys #create the client auth keys folder to store our second layer of authentication
+mkdir -p /var/lib/tor/onion/grafana #create the client auth keys folder to store our second layer of authentication
+chmod 400 -R /var/lib/tor/auth_keys #set restrictive file permissions
+
+#line below will allow your aggregator to connect to your monitored server. Without it no requests can even reach it
+echo "[prometheusclientaddr].onion:descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ" > /var/lib/tor/auth_keys/prometheus_server.auth_private
+
+chmod 400 -R /var/lib/tor/onion #set restrictive file permissions
+
+vi /etc/tor/torrc #edit the torrc file to add content
+
+cat /etc/tor/torrc
+AutomapHostsSuffixes .onion,.exit
+DataDirectory /var/lib/tor
+SOCKSPort 127.0.0.1:9050 IsolateDestAddr
+HiddenServiceDir /var/lib/tor/onion/grafana
+HiddenServicePort 80 127.0.0.1:3000
+ClientOnionAuthDir /var/lib/tor/auth_keys
+
+tor-client-auth-gen
+private_key=descriptor:x25519:YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ
+public_key=descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ
+
+echo "YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ" > ~/mygrafana_auth_key
+echo "descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ" > /var/lib/tor/onion/grafana/0.auth
+
+chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder
+
+systemctl start tor #restart tor
+systemctl status tor #check that everything works
+
+
+And that's all you'll need! one hidden service for grafana.
+
+vi /etc/prometheus/prometheus.yml
+
+cat /etc/prometheus/prometheus.yml
+
+alerting:
+ alertmanagers: []
+global:
+ scrape_interval: 10s
+remote_read: []
+remote_write: []
+scrape_configs:
+- job_name: remote-nodes
+ proxy_url: socks5h://localhost:9050
+ static_configs:
+ - labels: {}
+ targets:
+ - [clientaddr].onion:9100
+- job_name: local-node
+ static_configs:
+ - labels: {}
+ targets:
+ - localhost:9100
+
+
+
+
+ docker run -d -p 127.0.0.1:3000:3000 --name=grafana grafana/grafana
+
+
+
+
+
+
+ + +
+
+ RSS Feed
SimpleX Chatrooms
+
+
In this tutorial we're going to cover how you can monitor the resource consumption of your remote servers while maintaining your anonymity, using Grafana, Prometheus, and node exporter.
+ + + Sidenote: Help us improve this tutorial by letting us know if there's anything missing or incorrect on this git issue directly!
In this setup, we have 3 servers. Server A is going to monitor Server B and C.
+Server A is going to have the following 3 services:
+
+-Grafana: to display the data retrieved by prometheus
+-Prometheus: to retrieve the data served by the node-exporters
+-node-exporter: to serve the server resource usage data (ex: CPU consumption, RAM consumption, etc)
+
+
+
+Server B and C are going to have the node-exporter service on them, and thanks to it Server A's prometheus service is going to be able to monitor their resource consumption.
+Now the added complexity that we have to navigate in this tutorial is that each server can only be reached by their own .onion hostnames. we cannot query them via their IPs directly because otherwise we'd reveal the origin and destination of the servers. Hence in this tutorial we're going to maintain the Serverside anonymity.
+First let's setup Server A:
+
+[ Wonderland ] [ /dev/pts/22 ] [/srv/]
+→ sudo apt install docker.io docker-compose -y
+
+[ Wonderland ] [ /dev/pts/22 ] [/srv/]
+→ mkdir /srv/grafana ; cd /srv/grafana
+
+[ Wonderland ] [ /dev/pts/22 ] [/srv/]
+→ vim docker-compose.yml
+
+
+Then, we have the following docker-compose.yml:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ cat docker-compose.yml
+
+networks:
+ tor-monitoring:
+ driver: bridge
+ ipam:
+ config:
+ - subnet: 10.16.0.0/24
+ gateway: 10.16.0.1
+
+services:
+ grafana:
+ image: grafana/grafana-enterprise:latest
+ container_name: grafana
+ restart: unless-stopped
+ ports:
+ - '127.0.0.1:3222:3000'
+ volumes:
+ - grafana-data:/var/lib/grafana
+ environment:
+ GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,grafana-worldmap-panel,grafana-piechart-panel"
+ networks:
+ tor-monitoring:
+ ipv4_address: 10.16.0.5
+
+ prometheus:
+ image: prom/prometheus:latest
+ volumes:
+ - ./prometheus/:/etc/prometheus/
+ - prometheus_data:/prometheus
+ command:
+ - '--config.file=/etc/prometheus/prometheus.yml'
+ - '--storage.tsdb.path=/prometheus'
+ - '--web.console.libraries=/usr/share/prometheus/console_libraries'
+ - '--web.console.templates=/usr/share/prometheus/consoles'
+ networks:
+ tor-monitoring:
+ ipv4_address: 10.16.0.4
+ restart: always
+
+ tor:
+ image: osminogin/tor-simple
+ container_name: tormonitoring
+ volumes:
+ - ./tor-data:/var/lib/tor
+ - ./tor-data/torrc:/etc/tor
+ networks:
+ tor-monitoring:
+ ipv4_address: 10.16.0.3
+
+volumes:
+ prometheus_data: {}
+ grafana-data: {}
+
+
+
+From there, you can already pull the containers and activate them:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ docker-compose pull ; docker-compose up -d
+
+
+It's going to first pull the containers and then activate them, but we need to do some changes first to ensure that the tor container works as intended:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ docker-compose down
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ vim tor-data/torrc/torrc
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ cat tor-data/torrc/torrc
+SOCKSPort 0.0.0.0:9050
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ chown -R 100:65533 tor-data/
+
+
+then we also configure prometheus to scrape the destination servers:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ vim prometheus/prometheus.yml
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ cat prometheus/prometheus.yml
+global:
+ scrape_interval: 15s # By default, scrape targets every 15 seconds.
+ external_labels:
+ monitor: 'datura-monitor'
+
+scrape_configs:
+ - job_name: 'wonderland'
+ scrape_interval: 5s
+ proxy_url: socks5h://10.16.0.3:9050
+ static_configs:
+ - labels: {}
+ targets:
+ - serverAjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
+ - serverBjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
+ - serverCjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
+ basic_auth:
+ username: 'admin'
+ password: 'P@SSW0RD'
+
+
+And then we can re-activate the containers:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ docker-compose up -d
+
+
+once done, we can configure the nginx reverse proxy to make sure that we can access our grafana instance:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ vim /etc/nginx/sites-available/monitoring
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ cat /etc/nginx/sites-available/monitoring
+upstream monitoringend {
+ server 127.0.0.1:3222;
+ #server 10.8.0.2:3009;
+}
+
+
+server {
+ listen 4443;
+ listen [::]:4443;
+ server_name monitoring.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion;
+
+ location / {
+ proxy_set_header Host $http_host;
+ proxy_pass http://monitoringend;
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection "Upgrade";
+ #client_max_body_size 1G;
+ }
+}
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ ln -s /etc/nginx/sites-available/monitoring /etc/nginx/sites-enabled/
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ nginx -t
+nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
+nginx: configuration file /etc/nginx/nginx.conf test is successful
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
+→ systemctl restart nginx
+
+
+
+
+
- -
Next, we can access our grafana instance to create the admin account and login:
+Once logged in we make sure that our grafana instance uses our prometheus instance as a datasource:
+here we mention the local IP of the prometheus container, being 10.16.0.4, with the service accessible on port 9090:
+Then, we import a dashboard to make sure that we can visualize the data we are monitoring, out of which i recommend the excellent "Node Exporter Full" dashboard (whose ID is 1860)
+There, the dashboard is imported, but there's no data to be seen yet because we didn't configure the node-exporter daemons on the servers that we want to monitor.
--
Next, we're going to configure node-exporter on the servers that we want to monitor:
-apt update
-apt install prometheus-node-exporter tor
-systemctl stop tor #stop the tor service
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ cat docker-compose.yml
+version: '3.7'
-mkdir -p /var/lib/tor/onion/prometheus/authorized_clients #create the client auth keys folder to store our second layer of authentication
-chmod 400 -R /var/lib/tor/prometheus #set restrictive file permissions
+services:
-vi /etc/tor/torrc #edit the torrc file to add content
+ node-exporter:
+ container_name: node-exporter
+ image: quay.io/prometheus/node-exporter:latest
+ #command:
+ #- '--path.rootfs=/host'
+ network_mode: host
+ pid: host
+ volumes:
+ - /proc:/host/proc:ro
+ - /sys:/host/sys:ro
+ - /:/rootfs:ro
+ - /:/host:ro,rslave
+ - ./web.yml:/etc/prometheus/web.yml
+ command:
+ - '--path.rootfs=/host'
+ - '--path.procfs=/host/proc'
+ - '--path.sysfs=/host/sys'
+ - --collector.filesystem.ignored-mount-points
+ - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
+ - "--web.config.file=/etc/prometheus/web.yml"
+ restart: always
+ deploy:
+ mode: global
-cat /etc/tor/torrc
-AutomapHostsSuffixes .onion,.exit
-DataDirectory /var/lib/tor
-SOCKSPort 127.0.0.1:9050 IsolateDestAddr
-HiddenServiceDir /var/lib/tor/onion/prometheus
+
+Now that docker-compose.yml is written, we need to write a small python script to hash the basicauth password:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ vim gen-pass.py
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ cat gen-pass.py
+import getpass
+import bcrypt
+
+#sudo apt install python3-bcrypt
+
+password = getpass.getpass("password: ")
+hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
+print(hashed_password.decode())
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ sudo apt install python3-bcrypt -y
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ python3 gen-pass.py
+password: P@SSW0RD
+$2b$12$AZg14Yp.hvDLk/iaYk9.ReqXyfonW94cwqzzxewZDWzTdAQZFo3zy
+
+
+now with the hashed password, we can write the web.yml config file that the node exporter will use:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ vim web.yml
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ cat web.yml
+basic_auth_users:
+ admin: $2b$12$AZg14Yp.hvDLk/iaYk9.ReqXyfonW94cwqzzxewZDWzTdAQZFo3zy
+
+
+And now finally we can pull the container image and activate it:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ docker-compose pull ; docker-compose up -d
+
+
+Then we also make sure that the node-exporter port 9100 is accessible via the onion domain, as otherwise we can't access it while maintaining the serverside anonymity:
+
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ cat /etc/tor/torrc | grep 9100
HiddenServicePort 9100 127.0.0.1:9100
-tor-client-auth-gen
-private_key=descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ #paste this key to your local machine as your prometheus node will need it
-echo "descriptor:x25519:6HDNHLLKIFNU5Q6T75B6Q3GBYDO5ZF4SQUX7EYDEKWNLPQUWUBTA" > /var/lib/tor/onion/prometheus/0.auth
+[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
+→ systemctl restart tor@default
-chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder
-
-systemctl start tor #restart tor
-systemctl status tor #check that everything works
-
-cat /var/lib/tor/onion/prometheus/hostname
-[clientaddr].onion
-
-
-What's that tor-client-auth-gen you ask? In order to protect this critical service from attacks that could be done against the grafana servers or from stolen credentials we need more than just security by obscurity
-(relying on the attacker not knowing our hidden service address).
-
-
-sudo systemctl stop tor #stop the tor service
-
-mkdir -p /var/lib/tor/auth_keys #create the client auth keys folder to store our second layer of authentication
-mkdir -p /var/lib/tor/onion/grafana #create the client auth keys folder to store our second layer of authentication
-chmod 400 -R /var/lib/tor/auth_keys #set restrictive file permissions
-
-#line below will allow your aggregator to connect to your monitored server. Without it no requests can even reach it
-echo "[prometheusclientaddr].onion:descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ" > /var/lib/tor/auth_keys/prometheus_server.auth_private
-
-chmod 400 -R /var/lib/tor/onion #set restrictive file permissions
-
-vi /etc/tor/torrc #edit the torrc file to add content
-
-cat /etc/tor/torrc
-AutomapHostsSuffixes .onion,.exit
-DataDirectory /var/lib/tor
-SOCKSPort 127.0.0.1:9050 IsolateDestAddr
-HiddenServiceDir /var/lib/tor/onion/grafana
-HiddenServicePort 80 127.0.0.1:3000
-ClientOnionAuthDir /var/lib/tor/auth_keys
-
-tor-client-auth-gen
-private_key=descriptor:x25519:YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ
-public_key=descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ
-
-echo "YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ" > ~/mygrafana_auth_key
-echo "descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ" > /var/lib/tor/onion/grafana/0.auth
-
-chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder
-
-systemctl start tor #restart tor
-systemctl status tor #check that everything works
-
-
-And that's all you'll need! one hidden service for grafana.
-
-vi /etc/prometheus/prometheus.yml
-
-cat /etc/prometheus/prometheus.yml
-
-alerting:
- alertmanagers: []
-global:
- scrape_interval: 10s
-remote_read: []
-remote_write: []
-scrape_configs:
-- job_name: remote-nodes
- proxy_url: socks5h://localhost:9050
- static_configs:
- - labels: {}
- targets:
- - [clientaddr].onion:9100
-- job_name: local-node
- static_configs:
- - labels: {}
- targets:
- - localhost:9100
-
-
-
-
- docker run -d -p 127.0.0.1:3000:3000 --name=grafana grafana/grafana
-
-
-
-
-
-
- And now from there the server should be monitored as intended.
++
Now that the remote server's node-exporter is reachable via it's onion domain, we can see that the resource usage started to appear in the dashboard that we setup earlier:
-In this case, in the event of the remote server being seized, the adversary would only be able to see that it has been queried to from a tor exit node IP. This is to make sure that your anonymous infrastructure is able to sustain having each of it's individual servers being taken down, as long as they are not all taken down at once.
+And that's it ! We can now monitor our own remote servers, while maintaining serverside anonymity at the same time.
+Donate XMR:
86NCojqYmjwim4NGZzaoLS2ozbLkMaQTnd3VVa9MdW1jVpQbseigSfiCqYGrM1c5rmZ173mrp8RmvPsvspG8jGr99yK3PSs
Contact: mulligansecurity@riseup.net
website
SimpleX
Donate XMR: 8AUYjhQeG3D5aodJDtqG499N5jXXM71gYKD8LgSsFB9BUV1o7muLv3DXHoydRTK4SZaaUBq4EAUqpZHLrX2VZLH71Jrd9k8