get started on the article

simplexalerts/index.md

@@ -0,0 +1,238 @@
+---
+author: MulliganSecurity
+date: 2025-06-04
+gitea_url: "http://git.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/nihilist/blog-contributions/issues/223"
+xmr: 82htErcFXbSigdhK9tbfMoJngZmjGtDUacQxxUFwSvtb9MY8uPSuYSGAuN1UvsXiXJ8BR9BVUUhgFBYDPvhrSmVkGneb91j
+tags:
+  - Core Tutorial
+---
+
+# The case for alerting
+
+As you know, [monitoring](../anonymous_server_monitoring/index.md) is important when running any kind of operations, especially so
+for clandestine ones.
+
+
+## Alert Types
+
+There are basically two types of alerting mode:
+- Organic mode: you keep a running screen with visualization for your important metrics within your field of view all day, learn by osmosis what "normal" looks like so you can react when something abnormal appears
+  - pros
+    - easy to set up
+    - great for catching complex conditions, you subconscious does all the work
+  - cons
+    - messy and unreliable
+    - only works if you are in front of the screen
+- automated mode: you monitoring alerts you when some conditions are met
+  - pros
+    - reliable
+    - customizable: you define the exact context you want, how you want to receive alerts
+    - runs 24/7
+  - cons
+    - will only catch issues for conditions you explicitely defined
+    - can't come up with alert amelioration ideas
+
+## Associated Risks
+To paraphrase one of my favourite playwrights:
+
+
+To alert or not alert? That is the question. Whether 'tis nobler in the mind to unknowingly suffer she slings and arrows of outrageous fortune or to take arms against a sea of anomalies, fighting the advesary dwelling within...
+
+
+As your perimeter and infrastructure grows, as you add more servers your system complexity will shoot up exponentially. Simple organic alerting shows its limit when you have to correlate logs and behaviors across multiple systems.
+That's why you need alerting, if an adversary decides to stealthily probe at your infrastructure and you know what to look for you will see their attempt for what it is. Choosing to remain in the dark about it is foolish at best and irresponsible if you are part of an outfit as your laziness will put others in harm's way.
+
+### But alerting carries risk too!
+
+Indeed. Today we will keep building on the [monitoring](../anonymous_server_monitoring/index.md) tutorial.
+
+
+Alerts can be used to deanonymize you. If an adversary suspects that you or others are tied to a clandestine infrastructure, they might decide to trigger alerts (say, bruteforcing one of your endpoints) and see if you receive notifications through channels they control.
+
+
+Grafana supports a large number of possible alerts methods, most of them unfit to our purpose as those channels can be watched by the adversary.
+
+![](available_alerters.png)
+
+- Telegram: opaque and centralized, tied to phone numbers
+- AWS, Cisco, DingDing, Slack, Discord... same issue
+- Email: you can control it but it still is traceable and tied to a domain name and clearweb infrastructure with all the accompanying metadata
+- webhook: simply call an arbitrary URL with the alert message => **that's the way to go with the most control for ourselves**
+
+
+# Tools of the trade
+
+Let's start with webhooks. Webhooks are a great swiss-army knife when you need a system to act on its environment. They only need an http endpoint and the ability to make http request. They can easily be run as onion services too!
+
+
+For this tutorial, Mulligan Security open sourced the tool they use for their own infrastructure: the [grafana-simplex-alerter](https://github.com/MulliganSecurity/grafana-simplex-alerter).
+
+
+This will be used with the [simplex-chat](https://github.com/simplex-chat/simplex-chat) CLI client.
+
+
+## Target architecture
+
+![](architecture.png)
+
+
+
+## Simplex-chat
+
+You can install simplex-chat from your source repository or directly from the [simplex-chat release page](https://github.com/simplex-chat/simplex-chat/releases)
+
+
+Download the simplex-chat ubuntu release as shown:
+
+![](releases.png)
+
+
+### Initializing your Simplex client
+
+Run your client in server mode:
+
+
+    [user@devnode:~]$ simplex-chat -d clientDB -p 1337
+    No user profiles found, it will be created now.
+    Please choose your display name.
+    It will be sent to your contacts when you connect.
+    It is only stored on your device and you can change it later.
+    display name: myAlertBot
+    Current user: myAlertBot
+
+
+This command will:
+- create two databases that your client uses for reconnecting to groups and interacting with the simpleX Network
+- set this client display name (what you will see in your groups when receiving alerts)
+- open a websocket port that the alerter will use on 127.0.0.1:1337
+
+
+
+## Grafana-Simplex-alerter
+
+### What's this code and how can I trust it?
+
+Great question. This code runs a simple webserver using uvicorn, configures the simplex-chat client based on a yaml config file you provide it with invite links to the alert groups you want to configue.
+
+#### FOSS
+This code is FOSS. All of it. No secret, sauce. Furthermore it's pretty easy to analyze yourself as it's short
+
+#### Dependency pinning
+Everything that goes into making this code is cryptographically pinned and auditable:
+
+- flake.lock file => python version, build-tools, and so on
+- uv.lock => all libraries used by the code
+
+#### Opentelemetry-enabled
+Your alerting system itself should be monitored, this is can be done the following way:
+
+- prometheus metrics: the alerter exposes a special /metrics endpoint so you can collect telemetry data about it
+- continuous profiling: you have the option to connect it to a pyroscope server to get low-level per-function performance profile informations
+- opentelemetry integration: connecting it to an opentelemetry collector such as alloy will give you correlated logs, traces, metrics and profiles, allowing you to easily debug and monitor it
+
+
+And if you don't care about any of this: if you don't use those options no resources shall be consumed.
+
+### Building the alerter
+
+#### With nix
+
+If you have the nix package manager installed, you can simply run:
+
+
+    nix profile install github:MulliganSecurity/grafana-simplex-alerter
+
+
+You will then be able to run simplex-alerter:
+
+
+    [user@devnode:~]$ simplex-alerter --help
+    usage: simplex-alerter [-h] [-a ADDR] [-m PROMETHEUS_CONFIG] [-o OTEL_SERVER] [-f PYROSCOPE_SERVER] [-b BIND_ADDR] [-d] [-c CONFIG] [-g] [-e ENDPOINT]
+
+    options:
+      -h, --help            show this help message and exit
+      -o, --opentelemetry OTEL_SERVER
+                            opentelemetry server
+      -f, --profiling PYROSCOPE_SERVER
+                            pyroscope server address for profiling
+      -b, --bind-addr BIND_ADDR
+                            host:port to run the app on
+      -d, --debug           enable debug mode, increases pyroscope sampling rate if configured
+      -c, --config CONFIG   config file
+      -g, --generate-config
+                            generate config file with placeholder values
+      -e, --endpoint ENDPOINT
+                            simplex endpoint
+
+
+#### With Docker
+
+
+    [user@devnode:~]$ git clone https://github.com/MulliganSecurity/grafana-simplex-alerter.git
+    cd grafana-simplex-alerter
+    docker build . -t simplex-alerter
+
+
+For the rest of the tutorial I will show the docker commands. If you installed the alerter using nix, simply replace "docker run simplex-alerter" with simplex-alerter.
+Do take note of the following:
+
+- In docker run the "--rm" parameter will be used to automatically destroy the container after it's run. If using from nix you can disregard it.
+
+
+### Running the alerter
+
+#### Create your invite links
+
+Now, on your phone or on your desktop create a group to receive alerts.
+
+##### Create a group
+In this tutorial I will create the "clandestineInfraAlerts" group and invite anyone from within my organization that needs to receive those alerts.
+
+![](create_group.png)
+![](name_group.png)
+
+
+I will now need to create an invite link so the alerte can send messages there:
+
+![](group_link.png)
+
+
+#### Generate a basic config file
+
+To get started we need a basic config file to fill out with our alerter information:
+
+
+    [user@devnode:~]$ docker run --rm simplex-alerter -g > config.yml
+    [user@devnode:~]$ cat config.yml 
+    alert_groups:
+    - invite_link: https://simplex.chat/contact#/?v=2-7&sm...
+      name: alert_group0
+
+
+##### Configure the alerter
+
+
+Update your config file with the invite link you created earler and set the name to your group name
+
+    [user@devnode:~]$ vi config.yml
+
+so it looks like this:
+
+    [user@devnode:~]$ cat config.yml 
+    alert_groups:
+    - invite_link: https://simplex.chat/contact#/?v=2-7&smp=smp%3A%2F%2FUkMFNAXLXeAAe0beCa4w6X_zp18PwxSaSjY17BKUGXQ%3D%40smp12.simplex.im%2FlOgzQT8ZxfF3TV_x00c0mNLMFBDkl6gj%23%2F%3Fv%3D1-4%26dh%3DMCowBQYDK2VuAyEAypkpAgfmsShNThQBGvPXxjBk8O03vKe1x0311UHhK3I%253D%26q%3Dc%26srv%3Die42b5weq7zdkghocs3mgxdjeuycheeqqmksntj57rmejagmg4eor5yd.onion&data=%7B%22groupLinkId%22%3A%22EfuyLGxGhsc0iWkqr9NYvQ%3D%3D%22%7D
+      name: clandestineInfraAlerts
+
+##### Start the alerter
+
+Here using docker becomes a hassle, since we are going to have to make the simplex-chat port accessible. For security easons it runs on the loopback interface.
+
+
+let's create a socat tunnel with:
+
+    apt install socat
+    sudo socat TCP-LISTEN:31337,bind=172.17.0.1,reuseaddr,fork TCP:127.0.0.1:1337&
+
+Now run our container:
+
+