add pyroscope tracing bit and observlib blurb

This commit is contained in:
MulliganSecurity 2025-06-06 11:27:46 +02:00
parent a3e391e287
commit 1a4c1f95d2

View file

@ -128,11 +128,36 @@ Season with statistical threshold detection depending on how likely your adminis
{duration>=10s && .service.name="my-interactive-website"} {duration>=10s && .service.name="my-interactive-website"}
Do note that Pyroscope for continuous profiling should also be used, but this is highly application specific (eg: monitor critical functions for duration variation). You will want to create recording rules that will build prometheus metrics from your continuous profiling infrastructure so you can alert against those. Creation of recording rules is out of scope for this tutorial but they use the same language and tooling as alerting rules. Do note that Pyroscope for continuous profiling should also be used, but this is highly application-specific (eg: monitor critical functions for execution duration variation). You will want to create recording rules that will build prometheus metrics from your continuous profiling infrastructure so you can alert against those. Creation of recording rules is out of scope for this tutorial but they use the same language and tooling as alerting rules.
![](pyroscope_tracing.png) ![](pyroscope_tracing.png)
#### Futher recommendation
As part of a partnership, the [observlib](http://git.nowherejezfoltodf4jiyl6r56jnzintap5vyjlia7fkirfsnfizflqd.onion/Capably7710/observlib) library was produced fo internal use by MulliganSecurity and availability to our customers.
This library is a very simple opentelemetry distro geared toward small organization with secure anonymous monitoring needs.
##### What it does:
- Configure the opentelemetry pipeline for your python application in such a way an alloy aggregator can easily combine and contextualize all your telemetry data before shipping it over tor to your monitoring server
- set up the necessary configuration so our end users can also just spin up a prometheus metrics instance if they do not want to set up opentelemetry collection
- provide a *traced* decorator that can be added to any function and:
- measure execution time
- update success/failure counters based on the results with arbitrary labels and amounts
- ensure proper tracing configuration with span error status configuration for later debugging
##### Why use it
Because it makes monitoring integrated and easier. Using an integrated opentelemetry setup allows you to easily pivot from a prometheus alert to specific traces and execution flows using exemplars or logs using grafana's aggregator automatic contextualization. This gives you a way to easily go back in time to understand fully your system state, up to and including execution continuous profiling.
##### Using it in libraries
End users can also use observlib in their own libraries without requiring their third party users to configure anything: just don't call the configure_telemetry function and you will still be able to setup data collection.
Keep in mind that unless the application importing your lib sets up opentelemetry and/or prometheus exporting no data will be collected or resources consumed.
### But alerting carries risk too! ### But alerting carries risk too!
Indeed. Today we will keep building on the [monitoring](../anonymous_server_monitoring/index.md) tutorial. Indeed. Today we will keep building on the [monitoring](../anonymous_server_monitoring/index.md) tutorial.