Previous Page

Mulligan Security - 2025-02-07

Server Monitoring

What is server monitoring?

When deploying compute resources (bare-metal, VPSes or more abstract work units) you will have to manage a living system. This system will always have the following characteristics:

• Limited ressources: the amounts of RAM and CPU cycles, network bandwidth as well as storage space are neither infinite nor free.
• Evolving requirements: depending on how you use your services, how many concurrent users you have you might need more or less ressources than what you initially purchased
• Nominal operating parameters: range of RAM and CPU use, temperatures and so on in which your service performs as expected

The first item is fixed and only linked to your financial constraints. The other two are constantly evolving and thus must be monitored.

What if I don't?

If you don't properly monitor your infrastructure you will face the following consequences sooner or later:

• service instability: you won't notice when things start going awry
• costs overrun: you will end up paying more than you need to in order to deliver the same service
• undetected attacks: attacks that impact your services can go unnoticed when the cues (eg: high RAM consumption from a cryptojacking) are not picked up

How do I do it?

How you monitor your systems can vary based on your technical requirements. It can be as simple as logging in once a week, check the output of some diagnostic command and calling it a day.
This will give you a snapshot but you will miss a lot of important information.

You can also set up a complicated system that reports current metrics, trends and gives you capacity planning alerts based on the data obtained! You will have to find the middle-ground yourself, this article will propose one that you can tweak whichever way you need.

Risks of doing it improperly

Accessing your server for monitoring purposes is, from a risk perspective, pretty much the same as doing any other administration task or interacting with the services hosted therein. If done improperly (say logging in over the clearweb from your home address) you've just given anyone looking an undeniable link between your overt identity and your clandestine activities.

A fail-closed system is what you should strive for: opsec best practices should be the default and if there's a technical issue preventing you from following them (attack on tor, flaky network, client or server-side misconfiguration) the system should prevent access at all in order to keep you safe.

Basic tools

Let's look at a cryptojacked server... In this case the intruder did not take any precautions or try to hide their activity. This often happens with basic scripts that scan the internet in large-scale low-cost credential stuffing attacks.

glances

Here we will look at glances. Glances is a python tool that gives nice looking visuals with information about server status

Pros

• looks nice

Cons

• Requires python
• not part of the POSIX convention
• somewhat resource intensive for limited hardware

top

Now, an oldie but a goodie: top! Wherever you find a unix you'll find top, from MacOS to BSD...

Pros

• Lightweight
• POSIX Compliant

Cons

• Ugly
• limited ordering/filtering features compared to glances

Risks

Whenever you connect to your server, such as for monitoring or other administrative tasks, if you do so through the clearweb then you are liable to being recorded. Even when using SSH you will leave a trail of metadata all the way back to your access point. That might be enough to get your door busted down the line.

In the following part of the post we will look into:

• How to connect to your server safely and anonymously
• How to set up advanced monitoring tools so you don't have to keep an eye on a bunch of tmux sessions with glances/top open

Power tools

Before getting started let's review our tools and reminds ourselves of the security implications of their use:

• Tor: if you're reading this, you already know what it is.
  Risks:
  • Information leakage: if you try to resolve "mysecretillegalhostingserver.onion" against your ISP's DNS server it will leave an incriminating log: unless your server is well-known and has a lot of traffic you can't really justify knowing it's onion address
• SSH: Secure SHell. This tools allows you to connect to a remote server with an encrypted tunnel, this providing you with confidentiality when doing administration tasks.
  Risks
  • Authentication: the first time you connect to a server you should check its host key fingerprint. This is NOT an issue in our case since tor will provide another couple of layers of authentication. If you connect on a clearweb server through tor though you will want to check the host key fingerprint to make sure your exit node isn't trying to MITM you.
  • Password security: Nefarious operators trawl through the web on a daily basis trying credential stuffing attacks (logging into your server with weak/well known passwords), if you set up root:toor as a login you will get compromised quickly.
  • Information leakage: instead of setting up a password you decide to do things more securely and use an ssh key as a mean of authentication. By default, the ssh client will try every key it has until succeeding when connecting to a server. Why is that bad? Say your cloud provider decides to log verbosely your VPS' ssh server connection. When you connect next they might get a bunch of public keys that you use on other services. If Leo decides to ask github if anyone is using any of those keys to, say, push code to repositories or deploy stuff through actions then they will have a link between your github account and your onion server. Let's hope you haven't set up a personal email with github, because if you did, you're toast.