Previous Page

Mulligan Security - 2025-02-07

Server Monitoring

What is server monitoring?

When deploying compute resources (bare-metal, VPSes or more abstract work units) you will have to manage a living system. This system will always have the following characteristics:

• Limited ressources: the amounts of RAM and CPU cycles, network bandwidth as well as storage space are neither infinite nor free.
• Evolving requirements: depending on how you use your services, how many concurrent users you have you might need more or less ressources than what you initially purchased
• Nominal operating parameters: range of RAM and CPU use, temperatures and so on in which your service performs as expected

The first item is fixed and only linked to your financial constraints. The other two are constantly evolving and thus must be monitored.

What if I don't?

If you don't properly monitor your infrastructure you will face the following consequences sooner or later:

• service instability: you won't notice when things start going awry
• costs overrun: you will end up paying more than you need to in order to deliver the same service
• undetected attacks: attacks that impact your services can go unnoticed when the cues (eg: high RAM consumption from a cryptojacking) are not picked up

How do I do it?

How you monitor your systems can vary based on your technical requirements. It can be as simple as logging in once a week, check the output of some diagnostic command and calling it a day.
This will give you a snapshot but you will miss a lot of important information.

You can also set up a complicated system that reports current metrics, trends and gives you capacity planning alerts based on the data obtained! You will have to find the middle-ground yourself, this article will propose one that you can tweak whichever way you need.

Risks of doing it improperly

Accessing your server for monitoring purposes is, from a risk perspective, pretty much the same as doing any other administration task or interacting with the services hosted therein. If done improperly (say logging in over the clearweb from your home address) you've just given anyone looking an undeniable link between your overt identity and your clandestine activities.

A fail-closed system is what you should strive for: opsec best practices should be the default and if there's a technical issue preventing you from following them (attack on tor, flaky network, client or server-side misconfiguration) the system should prevent access at all in order to keep you safe.

Basic tools

Let's look at a cryptojacked server... In this case the intruder did not take any precautions or try to hide their activity. This often happens with basic scripts that scan the internet in large-scale low-cost credential stuffing attacks.

glances

Here we will look at glances. Glances is a python tool that gives nice looking visuals with information about server status

Pros

• looks nice

Cons

• Requires python
• not part of the POSIX convention
• somewhat resource intensive for limited hardware
• Only shows point-in-time data, no history
• No alerting whatsoever

top

Now, an oldie but a goodie: top! Wherever you find a unix you'll find top, from MacOS to BSD...

Pros

• Lightweight
• POSIX Compliant

Cons

• Ugly
• limited ordering/filtering features compared to glances
• Only shows point-in-time data, no history
• No alerting whatsoever

Risks

Whenever you connect to your server, such as for monitoring or other administrative tasks, if you do so through the clearweb then you are liable to being recorded. Even when using SSH you will leave a trail of metadata all the way back to your access point. That might be enough to get your door busted down the line.

In the following part of the post we will look into how to set up advanced monitoring tools so you don't have to keep an eye on a bunch of tmux sessions with glances/top open in order to know the behaviour of your systems over time.

First, you need to read and understand how to connect to your server safely and anonymously (see this article), grok it and then come back here.

...
...
Done? Let's proceed.

Target Architecture

First, let's have a look at the network topology we'll be building:

Setting up the Server

First you want to set up your central monitoring server. For ease of use and better performance we are going to colocate the prometheus collector along with grafana.

Required installation

To get started we need the following software on the machine:

• Tor: anonymize traffic
• prometheus: aggregate metrics
• prometheus-node-exporter: export local server metrics
• docker: to run grafana

Tor Configuration

On the Client

run the following as root to create a hidden service for the prometheus collector

apt update
apt install prometheus-node-exporter tor
systemctl stop tor #stop the tor service

mkdir -p /var/lib/tor/onion/prometheus/authorized_clients #create the client auth keys folder to store our second layer of authentication
chmod 400 -R /var/lib/tor/prometheus #set restrictive file permissions

vi /etc/tor/torrc #edit the torrc file to add content

cat /etc/tor/torrc
AutomapHostsSuffixes .onion,.exit
DataDirectory /var/lib/tor
SOCKSPort 127.0.0.1:9050 IsolateDestAddr
HiddenServiceDir /var/lib/tor/onion/prometheus
HiddenServicePort 9100 127.0.0.1:9100

tor-client-auth-gen 
private_key=descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ #paste this key to your local machine as your prometheus node will need it
echo "descriptor:x25519:6HDNHLLKIFNU5Q6T75B6Q3GBYDO5ZF4SQUX7EYDEKWNLPQUWUBTA" > /var/lib/tor/onion/prometheus/0.auth

chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder

systemctl start tor #restart tor
systemctl status tor #check that everything works

cat /var/lib/tor/onion/prometheus/hostname
[clientaddr].onion

On the aggregator

The prometheus collector will only be accessed locally by grafana so it doesn't need to be accessible over tor. Grafana, on the other hand, does.
Let's start with tor's configuration, use the following commands as root:

sudo systemctl stop tor #stop the tor service

mkdir -p /var/lib/tor/auth_keys #create the client auth keys folder to store our second layer of authentication
mkdir -p /var/lib/tor/onion/grafana #create the client auth keys folder to store our second layer of authentication
chmod 400 -R /var/lib/tor/auth_keys #set restrictive file permissions

#line below will allow your aggregator to connect to your monitored server. Without it no requests can even reach it
echo "[prometheusclientaddr].onion:descriptor:x25519:DBQW3GP5FCN2KQBDKTDKDAQUQWBEGBZ5TFYJE4KTJFBUOJPKYZBQ" > /var/lib/tor/auth_keys/prometheus_server.auth_private

chmod 400 -R /var/lib/tor/onion #set restrictive file permissions

vi /etc/tor/torrc #edit the torrc file to add content

cat /etc/tor/torrc
AutomapHostsSuffixes .onion,.exit
DataDirectory /var/lib/tor
SOCKSPort 127.0.0.1:9050 IsolateDestAddr
HiddenServiceDir /var/lib/tor/onion/grafana
HiddenServicePort 80 127.0.0.1:3000
ClientOnionAuthDir /var/lib/tor/auth_keys

tor-client-auth-gen 
private_key=descriptor:x25519:YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ
public_key=descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ

echo "YCPURSYN4FL4QKQSXFTGLYNBHOVVRCQYRZLFHMZFCUFU5R6DCRMQ" > ~/mygrafana_auth_key
echo "descriptor:x25519:UUQW4LIO447WRQOSRSNDXEW5NZMSR3CYOP65ZIFWH6G2PUKWV5WQ" > /var/lib/tor/onion/grafana/0.auth

chown debian-tor:debian-tor -R /var/lib/tor # make tor owner of this folder

systemctl start tor #restart tor
systemctl status tor #check that everything works

And that's all you'll need! one hidden service for grafana.
You'll find your hostname in /var/lib/tor/onion/grafana/hostname.

Prometheus server configuration

clean and simple: we scrape our server every 10s for new data, configure a proxy URL so scraping happens over tor, using our socksport and configure ou scraping targets.

vi /etc/prometheus/prometheus.yml

cat /etc/prometheus/prometheus.yml

alerting:
  alertmanagers: []
global:
  scrape_interval: 10s
remote_read: []
remote_write: []
scrape_configs:
- job_name: remote-nodes
  proxy_url: socks5h://localhost:9050
  static_configs:
  - labels: {}
    targets:
    - [clientaddr].onion:9100
- job_name: local-node
  static_configs:
  - labels: {}
    targets:
    - localhost:9100

Grafana configuration

Let's start grafana and make it available: as root, as tor is already configured we can then access it through our torbrowser

    docker run -d -p 127.0.0.1:3000:3000 --name=grafana grafana/grafana

Connecting to our grafana instance

On your monitoring server you can find your hostname at /var/lib/tor/grafana/hostname. Use it in the tor browser to reach your instance. You will be prompted for your private key, you can find it where you saved it earlier ~/mygrafana_auth_key, paste the part after

Configuring the data sources

Next we need to tell grafana to use prometheus as a data source:


Now, let's configure it (specifying localhost:9090 as the API port)




And Voila! we have simple system monitoring over tor in a dashboard:

Conclusion

In this article we saw why and how you need to implement anonymous server monitoring for your infrastructure. If you are running hidden services with any form of sensitive data stored on them, having them under constant monitoring is a must but this monitoring must not compromise your identity or the rest of your infrastructure.