oxeo0/blog-contributions
1
0
Fork 0
mirror of http://git.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/nihilist/blog-contributions.git synced 2025-07-02 11:56:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
blog-contributions/opsec/anonymous_server_monitoring/index.html
nihilist 68196c4d47 simplified the monitoring tutorial
2025-04-27 18:36:18 +02:00

459 lines
16 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<link rel="shortcut icon" href="../../../../../../assets/img/favicon.png">
<title>Anonymous Monitoring (Grafana, Prometheus, Node-exporter)</title>
<!-- Bootstrap core CSS -->
<link href="../../assets/css/bootstrap.css" rel="stylesheet">
<link href="../../assets/css/xt256.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="../../assets/css/main.css" rel="stylesheet">
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
</head>
<body>
<!-- Static navbar -->
<div class="navbar navbar-inverse-anon navbar-static-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand-anon" href="\index.html">The Nihilism Opsec Blog</a>
</div>
<div class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="/about.html">About</a></li>
<li><a href="/blog.html">Categories</a></li>
<li><a href="/contact.html">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
</div>
<!-- +++++ Posts Lists +++++ -->
<!-- +++++ First Post +++++ -->
<div id="anon2">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<a href="../index.html">Previous Page</a></br></br><p><img src="../../assets/img/user.png" width="50px" height="50px"> <ba>nihilist - 27 / 04 / 2025</ba></p>
<h1>Anonymous Monitoring (Grafana, Prometheus, Node-exporter) </h1>
<p>In this tutorial we're going to cover how you can monitor the resource consumption of your remote servers while maintaining your anonymity, using Grafana, Prometheus, and node exporter.</p>
<p><img src="../logos/daturagit.png" style="width:100px"> <u>Sidenote:</u> Help us improve this tutorial by letting us know if there's anything missing or incorrect on this <a href="http://git.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/nihilist/blog-contributions/issues/221">git issue</a> directly!</p>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /grey -->
<!-- +++++ Second Post +++++ -->
<div id="anon3">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<h2><b>What is the targeted setup? </b></h2>
<p>In this setup, we have 3 servers. Server A is going to monitor Server B and C.</p>
<img src="0.png" class="imgRz">
<p>Server A is going to have the following 3 services:</p>
<pre><code class="nim">
-Grafana: to display the data retrieved by prometheus
-Prometheus: to retrieve the data served by the node-exporters
-node-exporter: to serve the server resource usage data (ex: CPU consumption, RAM consumption, etc)
</code></pre>
<p>Server B and C are going to have the node-exporter service on them, and thanks to it Server A's prometheus service is going to be able to monitor their resource consumption.</p>
<p>Now the added complexity that we have to navigate in this tutorial is that <b>each server can only be reached by their own .onion hostnames.</b> we cannot query them via their IPs directly because otherwise we'd reveal the origin and destination of the servers. Hence in this tutorial we're going to maintain the Serverside anonymity.</p>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /white -->
<div id="anon2">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<h2><b>Server A setup (Grafana, Prometheus, Node-exporter)</b></h2> </br> </br>
<p>First let's setup Server A:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/22 ] [/srv/]
→ sudo apt install docker.io docker-compose -y
[ Wonderland ] [ /dev/pts/22 ] [/srv/]
→ mkdir /srv/grafana ; cd /srv/grafana
[ Wonderland ] [ /dev/pts/22 ] [/srv/]
→ vim docker-compose.yml
</code></pre>
<p>Then, we have the following docker-compose.yml:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ cat docker-compose.yml
networks:
tor-monitoring:
driver: bridge
ipam:
config:
- subnet: 10.16.0.0/24
gateway: 10.16.0.1
services:
grafana:
image: grafana/grafana-enterprise:latest
container_name: grafana
restart: unless-stopped
ports:
- '127.0.0.1:3222:3000'
volumes:
- grafana-data:/var/lib/grafana
environment:
GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,grafana-worldmap-panel,grafana-piechart-panel"
networks:
tor-monitoring:
ipv4_address: 10.16.0.5
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
networks:
tor-monitoring:
ipv4_address: 10.16.0.4
restart: always
tor:
image: osminogin/tor-simple
container_name: tormonitoring
volumes:
- ./tor-data:/var/lib/tor
- ./tor-data/torrc:/etc/tor
networks:
tor-monitoring:
ipv4_address: 10.16.0.3
volumes:
prometheus_data: {}
grafana-data: {}
</code></pre>
<p>From there, you can already pull the containers and activate them:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ docker-compose pull ; docker-compose up -d
</code></pre>
<p>It's going to first pull the containers and then activate them, but we need to do some changes first to ensure that the tor container works as intended:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ docker-compose down
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ vim tor-data/torrc/torrc
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ cat tor-data/torrc/torrc
SOCKSPort 0.0.0.0:9050
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ chown -R 100:65533 tor-data/
</code></pre>
<p>then we also configure prometheus to scrape the destination servers:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ vim prometheus/prometheus.yml
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ cat prometheus/prometheus.yml
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
external_labels:
monitor: 'datura-monitor'
scrape_configs:
- job_name: 'wonderland'
scrape_interval: 5s
proxy_url: socks5h://10.16.0.3:9050
static_configs:
- labels: {}
targets:
- serverAjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
- serverBjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
- serverCjezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion:9100
basic_auth:
username: 'admin'
password: 'P@SSW0RD'
</code></pre>
<p>And then we can re-activate the containers:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ docker-compose up -d
</code></pre>
<p>once done, we can configure the nginx reverse proxy to make sure that we can access our grafana instance:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ vim /etc/nginx/sites-available/monitoring
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ cat /etc/nginx/sites-available/monitoring
upstream monitoringend {
server 127.0.0.1:3222;
#server 10.8.0.2:3009;
}
server {
listen 4443;
listen [::]:4443;
server_name monitoring.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion;
location / {
proxy_set_header Host $http_host;
proxy_pass http://monitoringend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
#client_max_body_size 1G;
}
}
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ ln -s /etc/nginx/sites-available/monitoring /etc/nginx/sites-enabled/
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[ Wonderland ] [ /dev/pts/19 ] [/srv/grafana]
→ systemctl restart nginx
</code></pre>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /white -->
<!-- +++++ Second Post +++++ -->
<div id="anon1">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<h2><b>Configuring Grafana</b></h2> </br> </br>
<p>Next, we can access our grafana instance to create the admin account and login:</p>
<img src="1.png" class="imgRz">
<p>Once logged in we make sure that our grafana instance uses our prometheus instance as a datasource:</p>
<img src="2.png" class="imgRz">
<img src="3.png" class="imgRz">
<img src="4.png" class="imgRz">
<p>here we mention the local IP of the prometheus container, being 10.16.0.4, with the service accessible on port 9090:</p>
<img src="5.png" class="imgRz">
<img src="6.png" class="imgRz">
<p>Then, we import a dashboard to make sure that we can visualize the data we are monitoring, out of which i recommend the excellent <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/">"Node Exporter Full"</a> dashboard (whose ID is 1860) </p>
<img src="7.png" class="imgRz">
<img src="8.png" class="imgRz">
<img src="9.png" class="imgRz">
<img src="10.png" class="imgRz">
<img src="11.png" class="imgRz">
<p>There, the dashboard is imported, but there's no data to be seen yet because we didn't configure the node-exporter daemons on the servers that we want to monitor.</p>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /white -->
<!-- +++++ Second Post +++++ -->
<div id="anon2">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<h2><b>Configuring node-exporter on the remote servers</b></h2> </br> </br>
<p>Next, we're going to configure node-exporter on the servers that we want to monitor:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ cat docker-compose.yml
version: '3.7'
services:
node-exporter:
container_name: node-exporter
image: quay.io/prometheus/node-exporter:latest
#command:
#- '--path.rootfs=/host'
network_mode: host
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
- /:/host:ro,rslave
- ./web.yml:/etc/prometheus/web.yml
command:
- '--path.rootfs=/host'
- '--path.procfs=/host/proc'
- '--path.sysfs=/host/sys'
- --collector.filesystem.ignored-mount-points
- "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
- "--web.config.file=/etc/prometheus/web.yml"
restart: always
deploy:
mode: global
</pre></code>
<p>Now that docker-compose.yml is written, we need to write a small python script to hash the basicauth password:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ vim gen-pass.py
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ cat gen-pass.py
import getpass
import bcrypt
#sudo apt install python3-bcrypt
password = getpass.getpass("password: ")
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
print(hashed_password.decode())
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ sudo apt install python3-bcrypt -y
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ python3 gen-pass.py
password: P@SSW0RD
$2b$12$AZg14Yp.hvDLk/iaYk9.ReqXyfonW94cwqzzxewZDWzTdAQZFo3zy
</pre></code>
<p>now with the hashed password, we can write the web.yml config file that the node exporter will use:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ vim web.yml
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ cat web.yml
basic_auth_users:
admin: $2b$12$AZg14Yp.hvDLk/iaYk9.ReqXyfonW94cwqzzxewZDWzTdAQZFo3zy
</pre></code>
<p>And now finally we can pull the container image and activate it:</p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ docker-compose pull ; docker-compose up -d
</pre></code>
<p>Then we also make sure that the node-exporter port 9100 is accessible via the onion domain, as otherwise we can't access it while maintaining the serverside anonymity: </p>
<pre><code class="nim">
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ cat /etc/tor/torrc | grep 9100
HiddenServicePort 9100 127.0.0.1:9100
[ Wonderland ] [ /dev/pts/19 ] [/srv/node-exporter]
→ systemctl restart tor@default
</pre></code>
<p>And now from there the server should be monitored as intended.</p>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /white -->
<!-- +++++ Second Post +++++ -->
<div id="anon1">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<h2><b>Visualizing the result</b></h2> </br> </br>
<p>Now that the remote server's node-exporter is reachable via it's onion domain, we can see that the resource usage started to appear in the dashboard that we setup earlier:</p>
<img src="20.png" class="imgRz">
<p>In this case, in the event of the remote server being seized, the adversary would only be able to see that it has been queried to from a tor exit node IP. This is to make sure that your anonymous infrastructure is able to sustain having each of it's individual servers being taken down, as long as they are not all taken down at once.</p>
<p>And that's it ! We can now monitor our own remote servers, while maintaining serverside anonymity at the same time. </p>
</div>
</div><!-- /row -->
</div> <!-- /container -->
</div><!-- /white -->
<!-- +++++ Footer Section +++++ -->
<div id="anonb">
<div class="container">
<div class="row">
<div class="col-lg-4">
<h4>Nihilism</h4>
<p>
Until there is Nothing left.</p><p><a href="../../../../opsec/legal.html">Legal Disclaimer</a></p><p>Creative Commons Zero: <a href="../../../../opsec/runtheblog/index.html">No Rights Reserved</a></br><img src="\CC0.png">
</p>
</div><!-- /col-lg-4 -->
<div class="col-lg-4">
<h4>My Links</h4>
<p>
<a target="_blank" rel="noopener noreferrer" href="http://blog.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/rss/feed.xml">RSS Feed</a><br/><a target="_blank" rel="noopener noreferrer" href="http://nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/simplex.html">SimpleX Chatrooms</a><br/>
</p>
</div><!-- /col-lg-4 -->
<div class="col-lg-4">
<h4>About nihilist</h4>
<p style="word-wrap: break-word;"><u>Donate XMR:</u> 8AUYjhQeG3D5aodJDtqG499N5jXXM71gYKD8LgSsFB9BUV1o7muLv3DXHoydRTK4SZaaUBq4EAUqpZHLrX2VZLH71Jrd9k8</p></br>
</div><!-- /col-lg-4 -->
</div>
</div>
</div>
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
</body>
</html>
Reference in a new issue View git blame Copy permalink