mirror of
http://git.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/nihilist/blog-contributions.git
synced 2025-07-02 11:56:40 +00:00
447 lines
19 KiB
HTML
447 lines
19 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<meta name="description" content="Cloud provider threat model">
|
|
<meta name="author" content="MulliganSecurity">
|
|
<link rel="shortcut icon" href="../../../../../../assets/img/favicon.png">
|
|
|
|
<title>Introduction to Anonymous Server Monitoring</title>
|
|
|
|
<!-- Bootstrap core CSS -->
|
|
<link href="../../assets/css/bootstrap.css" rel="stylesheet">
|
|
<link href="../../assets/css/xt256.css" rel="stylesheet">
|
|
|
|
|
|
|
|
<!-- Custom styles for this template -->
|
|
<link href="../../assets/css/main.css" rel="stylesheet">
|
|
|
|
|
|
|
|
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
|
|
<!--[if lt IE 9]>
|
|
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
|
|
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
|
|
<![endif]-->
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<!-- Static navbar -->
|
|
<div class="navbar navbar-inverse-anon navbar-static-top">
|
|
<div class="container">
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand-anon" href="\index.html">The Nihilism Blog</a>
|
|
</div>
|
|
<div class="navbar-collapse collapse">
|
|
<ul class="nav navbar-nav navbar-right">
|
|
|
|
<li><a href="/about.html">About</a></li>
|
|
<li><a href="/blog.html">Categories</a></li>
|
|
<li><a href="https://blog.nowhere.moe/donate.html">Donate</a></li>
|
|
<li><a href="/contact.html">Contact</a></li>
|
|
</ul>
|
|
</div><!--/.nav-collapse -->
|
|
|
|
</div>
|
|
</div>
|
|
|
|
<!-- +++++ Posts Lists +++++ -->
|
|
<!-- +++++ First Post +++++ -->
|
|
<div id="anon2">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
<a href="../index.html">Previous Page</a></br></br><p><img src="../../assets/img/mulligan_sec.jpeg" width="50px" height="50px"> <ba>Mulligan Security - 2025-02-07 </a></p>
|
|
<p>
|
|
|
|
<h1><b>Server Monitoring</b></h1>
|
|
|
|
<h2>What is server monitoring?</h2>
|
|
|
|
When deploying compute resources (bare-metal, VPSes or more abstract work units) you will have to manage a living system. This system will <b>always</b> have the following characteristics:
|
|
<ul>
|
|
<li>Limited ressources: the amounts of RAM and CPU cycles, network bandwidth as well as storage space are neither infinite nor free.</li>
|
|
<li>Evolving requirements: depending on how you use your services, how many concurrent users you have you might need more or less ressources than what you initially purchased </li>
|
|
<li>Nominal operating parameters: range of RAM and CPU use, temperatures and so on in which your service performs as expected </li>
|
|
</ul>
|
|
|
|
<br><br>
|
|
|
|
The first item is fixed and only linked to your financial constraints. The other two are constantly evolving and thus must be <b>monitored</b>.
|
|
|
|
<h2>What if I don't?</h2>
|
|
|
|
If you don't properly monitor your infrastructure you will face the following consequences sooner or later:
|
|
|
|
<ul>
|
|
<li>service instability: you won't notice when things start going awry</li>
|
|
<li>costs overrun: you will end up paying more than you need to in order to deliver the same service</li>
|
|
<li>undetected attacks: attacks that impact your services can go unnoticed when the cues (eg: high RAM consumption from a cryptojacking) are not picked up</li>
|
|
</ul>
|
|
|
|
<h2>How do I do it?</h2>
|
|
How you monitor your systems can vary based on your technical requirements. It can be as simple as logging in once a week, check the output of some diagnostic command and calling it a day. <br>
|
|
This will give you a snapshot but you will miss a lot of important information.
|
|
|
|
<br><br>
|
|
You can also set up a complicated system that reports current metrics, trends and gives you capacity planning alerts
|
|
based on the data obtained!
|
|
|
|
You will have to find the middle-ground yourself, this article will propose one that you can tweak whichever way you need.
|
|
|
|
<h2>Risks of doing it improperly</h2>
|
|
|
|
Accessing your server for monitoring purposes is, from a risk perspective, pretty much the same as doing any other administration task or interacting with the services hosted therein. If done improperly (say logging in over the clearweb from your home address) you've just given anyone looking an undeniable link between your overt identity and your clandestine activities.<br><br>
|
|
|
|
|
|
A <b>fail-closed</b> system is what you should strive for: opsec best practices should be the default and if there's a technical issue preventing you from following them (attack on tor, flaky network, client or server-side misconfiguration) the system should prevent access at all in order to keep you safe.
|
|
</p>
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
|
|
<div id="anon3">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
<p>
|
|
<h1><b>Basic tools</b></h1>
|
|
|
|
Let's look at a cryptojacked server... In this case the intruder did not take any precautions or try to hide their activity. This often happens with basic scripts that scan the internet in large-scale low-cost credential stuffing attacks.
|
|
|
|
<h2>glances</h2>
|
|
Here we will look at glances. Glances is a python tool that gives nice looking visuals with information about server status
|
|
<img src="glances.png" class="imgRz">
|
|
|
|
<h3>Pros</h3>
|
|
<ul>
|
|
<li>looks nice</li>
|
|
</ul>
|
|
<h3>Cons</h3>
|
|
<ul>
|
|
<li>Requires python</li>
|
|
<li>not part of the POSIX convention</li>
|
|
<li>somewhat resource intensive for limited hardware</li>
|
|
<li><b>Only shows point-in-time data, no history</b></li>
|
|
<li><b>No alerting whatsoever</b></li>
|
|
</ul>
|
|
|
|
<h2>top</h2>
|
|
Now, an oldie but a goodie: top! Wherever you find a unix you'll find top, from MacOS to BSD...
|
|
<br>
|
|
|
|
<img src="top.png" class="imgRz">
|
|
|
|
<h3>Pros</h3>
|
|
<ul>
|
|
<li>Lightweight</li>
|
|
<li>POSIX Compliant</li>
|
|
</ul>
|
|
<h3>Cons</h3>
|
|
<ul>
|
|
<li>Ugly</li>
|
|
<li>limited ordering/filtering features compared to glances</li>
|
|
<li><b>Only shows point-in-time data, no history</b></li>
|
|
<li><b>No alerting whatsoever</b></li>
|
|
</ul>
|
|
|
|
|
|
</p>
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
<div id="anon2">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
<h1><b>Risks</b></h1>
|
|
Whenever you connect to your server, such as for monitoring or other administrative tasks, if you do so through the clearweb then you are liable to being recorded. Even when using SSH you will leave a trail of metadata all the way back to your access point. That might be enough to get your door busted down the line.
|
|
|
|
<br><br>
|
|
In the following part of the post we will look into how to set up advanced monitoring tools so you don't have to keep an eye on a bunch of tmux sessions with glances/top open in order to know the behaviour of your systems over time.
|
|
<br><br>
|
|
|
|
First, you need to read and understand how to connect to your server safely and anonymously (see <a href="/opsec/anonymousremoteserver/index.html">this article</a>), grok it and then come back here.
|
|
<br>
|
|
<br>
|
|
...
|
|
<br>
|
|
...
|
|
<br>
|
|
Done? Let's proceed.
|
|
<br>
|
|
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
|
|
|
|
<div id="anon3">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
<p>
|
|
<h1><b>Target Architecture</b></h1>
|
|
First, let's have a look at the network topology we'll be building:
|
|
<br>
|
|
<img src="architecture.png"/>
|
|
|
|
<h1><b>Setting up the Server</b></h1>
|
|
|
|
First you want to set up your central monitoring server. For ease of use and better performance we are going to colocate the prometheus collector along with grafana.
|
|
|
|
<h2>Tor Configuration</h2>
|
|
The prometheus collector will only be accessed locally by grafana so it doesn't need to be accessible over tor. Grafana, on the other hand, does.
|
|
<br>
|
|
|
|
Let's start with our torrc:<br>
|
|
<pre><code class="nim">
|
|
AutomapHostsSuffixes .onion,.exit
|
|
DataDirectory /var/lib/tor
|
|
SOCKSPort 127.0.0.1:9050 IsolateDestAddr
|
|
HiddenServiceDir /var/lib/tor/onion/grafana
|
|
HiddenServicePort 80 127.0.0.1:2700
|
|
ClientOnionAuthDir /var/lib/tor/auth_keys
|
|
</code></pre>
|
|
|
|
And that's all you'll need! one hiddn service for grafana. <br> You'll find your hostname in /var/lib/tor/onion/grafana/hostname.
|
|
|
|
<h2>Prometheus server configuration</h2>
|
|
clean and simple: we scrape our server every 10s for new data, configure a proxy URL so scraping happens over tor, using our socksport and configure ou scraping targets.
|
|
<br>
|
|
modify the prometheus.yml file (most likely located in /etc/prometheus)
|
|
|
|
<pre><code class="nim">
|
|
global:
|
|
scrape_interval: 10s
|
|
scrape_configs:
|
|
- job_name: nodes
|
|
proxy_url: socks5h://localhost:9050
|
|
static_configs:
|
|
- labels: {}
|
|
targets:
|
|
- [fill later with our client .onion address]:9002
|
|
</code></pre>
|
|
|
|
<h1>Setting up the client</h1>
|
|
On the client it's even easier.
|
|
|
|
<h2>Tor Configuration</h2>
|
|
Since prometheus works on a pull model, you will need to expose your node exporter, no need for a socks proxy either.
|
|
<br>
|
|
|
|
<pre><code class="nim">
|
|
AutomapHostsSuffixes .onion,.exit
|
|
DataDirectory /var/lib/tor
|
|
HiddenServiceDir /var/lib/tor/onion/prometheus
|
|
HiddenServicePort 9002 127.0.0.1:9002
|
|
</code></pre>
|
|
|
|
Next, you need to install the prometheus-node-exporter. Depending on your distribution of choice it's very likely it's in your package manager under that name.
|
|
<br>
|
|
|
|
and here is how we will start it in our unit file (created in /etc/systemd/system/prometheus-node-exporter.service) : <br>
|
|
<pre><code class="nim">
|
|
|
|
[Unit]
|
|
After=network.target
|
|
|
|
[Service]
|
|
CapabilityBoundingSet=
|
|
DeviceAllow=
|
|
DynamicUser=false
|
|
ExecStart=/bin/node_exporter \
|
|
--collector.systemd \
|
|
\
|
|
--web.listen-address 127.0.0.1:9002 --collector.ethtool --collector.softirqs --collector.tcpstat --collector.wifi
|
|
|
|
Group=node-exporter
|
|
LockPersonality=true
|
|
MemoryDenyWriteExecute=true
|
|
NoNewPrivileges=true
|
|
PrivateDevices=true
|
|
PrivateTmp=true
|
|
ProtectClock=false
|
|
ProtectControlGroups=true
|
|
ProtectHome=true
|
|
ProtectHostname=true
|
|
ProtectKernelLogs=true
|
|
ProtectKernelModules=true
|
|
ProtectKernelTunables=true
|
|
ProtectSystem=strict
|
|
RemoveIPC=true
|
|
Restart=always
|
|
RestrictAddressFamilies=AF_UNIX
|
|
RestrictAddressFamilies=AF_NETLINK
|
|
RestrictAddressFamilies=AF_INET
|
|
RestrictAddressFamilies=AF_INET6
|
|
RestrictNamespaces=true
|
|
RestrictRealtime=true
|
|
RestrictSUIDSGID=true
|
|
RuntimeDirectory=prometheus-node-exporter
|
|
SystemCallArchitectures=native
|
|
UMask=0077
|
|
User=node-exporter
|
|
WorkingDirectory=/tmp
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
</code></pre>
|
|
|
|
<br>
|
|
Do note that the name of the executable might change based on your distribution. What it does:
|
|
<br>
|
|
<ul>
|
|
<li> collect systemd data (services and so on)</li>
|
|
<li> collect internet throughput data</li>
|
|
<li> wifi information</li>
|
|
<li> cpu interrupts information </li>
|
|
</ul>
|
|
|
|
And make them available to your server.
|
|
<br><br>
|
|
<b>Right now, if an attacker could find your hidden service URL they could harvest this data about your server, you need to secure it by adding a key that will only allow your aggregator to connect</b><br>
|
|
|
|
Let's generate a keypair:
|
|
<pre><code class="nim">
|
|
user@computer$ tor-client-auth-gen
|
|
private_key=descriptor:x25519:3B6CE5X4I4XGXA5TDQWQONLLAJ6B5FQNPTBOFSF4AN6K6AJUXBOQ
|
|
public_key=descriptor:x25519:H7O5I7HUGLFM4IMPHNRN6L4S6TG4KJYDBXTYGOYJOUHH5NXVPJVA
|
|
</code></pre>
|
|
|
|
The private_key line must be copied to the following path on your prometheus aggregator: /var/lib/tor/auth_keys/prometheus.auth_private, prepended with your target onion address like this<br>
|
|
<pre><code>
|
|
mymonitoredserver.onion:descriptor:x25519:3B6CE5X4I4XGXA5TDQWQONLLAJ6B5FQNPTBOFSF4AN6K6AJUXBOQ
|
|
</code></pre>
|
|
|
|
The public_key must be added on the monitored server at the following path: /var/lib/tor/prometheus/authorized_clients/server.auth with the following content<br>
|
|
<pre><code>
|
|
descriptor:x25519:H7O5I7HUGLFM4IMPHNRN6L4S6TG4KJYDBXTYGOYJOUHH5NXVPJVA
|
|
</code></pre>
|
|
|
|
That way, only your monitoring server will be able to authenticate and scrape data from your monitored server.
|
|
<br><br>
|
|
Grafana has its own authentication system and database, still it reamins a critical service and it's not immune from 0 days and vulnerabilities that could be leveraged to obtain access. In order to apply a <b>defense in depth</b> principle we are going to do the same exercise for it:
|
|
|
|
<pre><code class="nim">
|
|
user@computer$ tor-client-auth-gen
|
|
private_key=descriptor:x25519:FD7NAZTGZAXA6CTXNXR3JCVSKAPW23EP5EQOUMXKRQCKACEVUJ7A
|
|
public_key=descriptor:x25519:OBIIXC3MWQ4VCEUS7Z6LOMOQG3CFP77SSWE45EDITP55WHVZFM6Q
|
|
</code></pre>
|
|
|
|
We'll put the public key on our monitoring server at /var/lib/tor/grafana/authorized_clients/admin.auth <br>
|
|
and our public key on our whonix workstation at /var/lib/tor/auth_keys/grafana.auth_private <br>
|
|
That way, even if an attacker discovers your grafana instance URL and has in their possession either your password or an exploit allowing them to do an authentication bypass
|
|
they still won't be able to get in unless they also break the encryption underpinning the tor network.
|
|
|
|
|
|
</p>
|
|
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
<div id="anon2">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
<h1> <b>Connecting to our grafana instance</b></h1>
|
|
On your monitoring server you can find your hostname at /var/lib/tor/grafana/hostname. Use it in the tor browser to reach your instance. You might be prompted for your private key if tor browser doesn't use your system's tor daemon. To avoid that you can change its connection settings.<br>
|
|
|
|
|
|
<img src="grafana_login.png"/>
|
|
<br>
|
|
|
|
<h1><b>Configuring the data sources</b></h1>
|
|
Next we need to tell grafana to use prometheus as a data source: <br>
|
|
|
|
<img src="add_datasource.png"/>
|
|
<br>
|
|
|
|
Now, let's configure it (specifying localhost:9001 as the API port)<br>
|
|
|
|
<img src="datasource_config.png"/>
|
|
<br>
|
|
|
|
And Voila! we have simple system monitoring over tor in a dashboard: <br>
|
|
|
|
<img src="example_dashboard.png"/>
|
|
|
|
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
|
|
<div id="anon3">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-8 col-lg-offset-2">
|
|
|
|
<h1><b>Conclusion</b></h1>
|
|
In this article we saw why and how you need to implement anonymous server monitoring for your infrastructure. If you are running hidden services with any form of sensitive data stored on them, having them under constant monitoring is a must but this monitoring must not compromise your identity or the rest of your infrastructure.
|
|
|
|
|
|
</div>
|
|
</div><!-- /row -->
|
|
</div> <!-- /container -->
|
|
</div><!-- /grey -->
|
|
|
|
|
|
<!-- +++++ Footer Section +++++ -->
|
|
|
|
<div id="anonb">
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-lg-4">
|
|
<h4>Nihilism</h4>
|
|
<p>
|
|
Until there is Nothing left.</p></br></br><p>Creative Commons Zero: <a href="../../../../opsec/runtheblog/index.html">No Rights Reserved</a></br><img src="\CC0.png">
|
|
|
|
</p>
|
|
</div><!-- /col-lg-4 -->
|
|
|
|
<div class="col-lg-4">
|
|
<h4>My Links</h4>
|
|
<p>
|
|
|
|
<a target="_blank" rel="noopener noreferrer" href="http://blog.nowhere.moe/rss/feed.xml">RSS Feed</a><br/><a target="_blank" rel="noopener noreferrer" href="https://simplex.chat/contact#/?v=2-7&smp=smp%3A%2F%2FL5jrGV2L_Bb20Oj0aE4Gn-m5AHet9XdpYDotiqpcpGc%3D%40nowhere.moe%2FH4g7zPbitSLV5tDQ51Yz-R6RgOkMEeCc%23%2F%3Fv%3D1-3%26dh%3DMCowBQYDK2VuAyEAkts5T5AMxHGrZCCg12aeKxWcpXaxbB_XqjrXmcFYlDQ%253D&data=%7B%22type%22%3A%22group%22%2C%22groupLinkId%22%3A%22c3Y-iDaoDCFm6RhptSDOaw%3D%3D%22%7D">SimpleX Chat</a><br/>
|
|
|
|
</p>
|
|
</div><!-- /col-lg-4 -->
|
|
|
|
<div class="col-lg-4">
|
|
<h4>About Mulligan Security</h4>
|
|
<p style="word-wrap: break-word;"><u>Donate XMR:</u><br>86NCojqYmjwim4NGZzaoLS2ozbLkMaQTnd3VVa9MdW1jVpQbseigSfiCqYGrM1c5rmZ173mrp8RmvPsvspG8jGr99yK3PSs</p></br><p><u>Contact:</u> mulligansecurity@riseup.net <br><a href="http://msec2nnqtbwh5c5yxpiswzwnqperok5k33udj7t6wmqcleu3ifj34sqd.onion">website</a><br><a href="https://simplex.chat/contact#/?v=2-7&smp=smp%3A%2F%2FiZJOs1BYKxD2nEndBtacHlBP-bNKv3gywICYPZZjXXE%3D%40chatnedvznvcnsovrm3e6jrgt6pkpai5i3rgslrrxlnv352ardboebid.onion%2FtT5R0tQWBzJPAkjvH-wai4PnpfTor89R%23%2F%3Fv%3D1-3%26dh%3DMCowBQYDK2VuAyEA_7oNMJAjBrt210CSc2LEIZJh5BFizPx7JUYFCmj8p1k%253D">SimpleX</a></p>
|
|
|
|
</div><!-- /col-lg-4 -->
|
|
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<!-- Bootstrap core JavaScript
|
|
================================================== -->
|
|
<!-- Placed at the end of the document so the pages load faster -->
|
|
|
|
</body>
|
|
</html>
|