Observability · Service mesh

MeshWatch

Full-stack observability — metrics, logs, traces, and AI incident analysis — on a service mesh for $5.12/month, 60% under serverless.

$5.12/month
60%under serverless
15pods
The problem

Observability shouldn’t cost more than the thing it watches

Managed observability bills scale with ingestion, and for a homelab that’s absurd — the telemetry can cost more than the workloads. I wanted the full three pillars (metrics, logs, traces) plus automated incident triage, self-hosted, for pocket change.

The interesting part was making AI incident analysis run locally too, so a lag spike gets a plain-English first look without a cloud API in the loop.

Architecture

The mesh emits, the stack collects, a local LLM explains

Istio gives every service mTLS and uniform telemetry; Prometheus, Loki, and Tempo collect the three signals; Grafana renders them; and Ollama Phi-3 turns an incident into a first-pass explanation.

Key decisions

What I chose, and why

Decision

Self-hosted on k3s

over a managed observability SaaS

Ingestion-priced SaaS punishes curiosity; self-hosting turns a per-GB bill into a fixed, tiny electricity cost.

Decision

A local Phi-3 model

over a cloud LLM API

Incident summaries run on-box — no per-call cost, no data leaving the rack, and it doubles as a testbed for local inference.

Decision

Tempo for traces

over skipping tracing

Traces are the pillar people drop first; keeping them cheap meant I could actually see cross-service latency, not guess.

Measured outcomes

The results

$5.12Monthly cost
60%Cheaper vs serverless
15Pods
0.2%Error rate
45msAvg response