Building a Multi-Tenant Observability Stack with GreptimeDB and Perses

When building a multi-tenant SaaS like MarkReply, monitoring isn’t just about CPU usage or memory leaks—it’s about understanding tenant behavior. We needed to track how many unique custom domains were hitting our Caddy edge router, map those requests to specific user accounts, and monitor active widget interactions in real-time.

Instead of paying for an expensive managed telemetry service, we built a blazing-fast, fully self-hosted observability platform. Here is how we wired together OpenTelemetry, GreptimeDB, and Perses to get deep, tenant-level insights.

📊 The Architecture

To capture data from the very edge of our network all the way down to the database layer, we rely on a three-tier telemetry pipeline:

1. The Producers: Our Caddy reverse proxy logs edge traffic and SSL challenges, while our Rust (Axum) backend instruments business logic (like comment submissions and widget loads).
2. The Pipeline: Both services push gRPC traces to a central OpenTelemetry Collector.
3. The Storage & Visualization: The Collector exports data to GreptimeDB, a time-series database optimized for metrics and logs. Finally, Perses hooks into GreptimeDB using standard SQL to render our dashboards.

📡 Centralizing with OpenTelemetry

The OpenTelemetry (OTel) Collector acts as our traffic cop. It receives raw telemetry data, batches it, and forwards it to our database.

Here is a simplified otel-collector-config.yaml to receive traces and ship them to GreptimeDB:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/greptimedb:
    endpoint: "greptimedb:4001"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/greptimedb]

🦀 Instrumenting the Rust Backend

To get tenant-specific data into our dashboards, we need our Rust application to attach custom attributes to our traces. By adding the opentelemetry-otlp crate, we can configure our tracing_subscriber to export data directly to the collector.

When a user interacts with a widget on a custom domain, we inject their tenant email and domain directly into the trace:

#[tracing::instrument(
    name = "widget_load",
    skip(state),
    fields(
        tenant.email = %account.email,
        tenant.domain = %domain_name
    )
)]
pub async fn serve_widget(state: State<AppState>, ...) {
    // ... handler logic
}

📈 Visualizing Multi-Tenant Data in Perses

With our data flowing safely into GreptimeDB, we spun up Perses—a GitOps-compatible, highly performant dashboarding tool.

Because GreptimeDB natively speaks SQL, we can write powerful queries directly in Perses without learning a proprietary query language like PromQL. For example, to generate a live table of our most active tenants and roll up all their active custom domains into a clean, comma-separated list, we use STRING_AGG:

SELECT
  "span_attributes.tenant.email" AS account_email,
  STRING_AGG(DISTINCT "span_attributes.tenant.domain", ', ') AS active_domains,
  COUNT(*) AS total_interactions,
  MAX(timestamp) AS last_seen
FROM opentelemetry_traces
WHERE service_name = 'comment-service'
  AND "span_attributes.tenant.email" IS NOT NULL
GROUP BY "span_attributes.tenant.email"
ORDER BY total_interactions DESC;

This single query powers our primary “Active Widgets” dashboard panel. We also run similar queries against the caddy service traces to audit raw edge requests and monitor Let’s Encrypt SSL challenge provisioning in real time.

By utilizing this open-source stack, we achieved total visibility into our multi-tenant architecture with zero vendor lock-in. The infrastructure remains incredibly lightweight, allowing us to keep our server costs low while providing enterprise-grade observability.