Docs
Snapshot freshness model
~6 min read · how Blackglass decides when host data is “current enough”
Drift detection is only useful if the snapshot you’re comparing against reality is actually current. This page explains how Blackglass keeps snapshots fresh in three different network shapes — direct SSH, push-agent, and a hybrid of the two — and how to tune the cadence for your fleet.
1. Two collection paths: SSH-pull and agent-push
Every host snapshot Blackglass evaluates comes from one of two paths. They have very different freshness profiles.
- SSH-pull (interactive). When you click Run scan, Blackglass tries to SSH directly into each host and capture the snapshot in real time. If SSH succeeds, the snapshot is “as of right now.”
- Agent-push (continuous). The push-agent installed on each host captures a snapshot on a timer (default 60 seconds) and posts it to the ingest API. Blackglass keeps the most recent push per host in an in-memory cache. If SSH fails (NAT, air-gap, key rotation, paid SSH gateway down), the collector falls back to the cached push.
For most fleets one path dominates: cloud VMs with public SSH live on the SSH-pull path; Droplets behind NAT, on-prem servers, and DigitalOcean App Platform → private hosts live on the agent-push path. Blackglass uses whichever path returns a usable snapshot first, with SSH-pull preferred.
2. The freshness pill on the dashboard
Next to the Run scan button on your dashboard, the freshness pill shows how recent the latest signal across the fleet is. The thresholds are:
- Green — latest snapshot ≤ 90 seconds old. Healthy.
- Amber — 90s < latest ≤ 5 min. The agent may be on the longer (legacy) cadence, or one host has dropped off briefly.
- Red — latest > 5 min. Investigate: agent timer disabled, hosts offline, or ingest credentials wrong.
The pill is a fleet-wide max — it summarises the freshest signal anywhere in your fleet. To see per-host freshness, open Hosts → (any host) → Last seen.
3. How “Run scan” waits for fresh data
The most common confusing case: you change a config on a host, click Run scan immediately, and want the new state — but SSH is unreachable and the cached agent snapshot is from 30 seconds before your change.
Blackglass handles this with a bounded wait. When the collector falls back to the agent cache and finds a snapshot older than your Run scan click, it polls the cache up to COLLECTOR_AGENT_FRESH_WAIT_MS(default 90s) for a newer push to arrive. While it waits, the scan-progress card shows a live countdown (“Waiting for fresh agent snapshot (47s remaining)…”) so you know the scan isn’t stalled.
Combined with the 60-second default agent cadence, this means the worst case for “Run scan after a manual change” on a blackholed host is roughly 60–90 seconds, not 5 minutes.
4. Tuning the cadence
Three knobs control freshness, all environment variables on the server (no agent restart required for the wait knob):
| Variable | Default | What it controls |
|---|---|---|
| Agent push interval | 60s | How often the host pushes a fresh snapshot. Configured on the host’s systemd timer (blackglass-agent.timer) or cron entry. |
| COLLECTOR_AGENT_FRESH_WAIT_MS | 90,000 | Max time the SSH-fail fallback waits for a newer push than the user’s Run scan click. Set to 0 to disable waiting and always use the freshest cached snapshot immediately. |
| COLLECTOR_AGENT_FRESH_POLL_MS | 1,500 | How often the wait loop re-checks the cache while waiting. Lower = snappier countdown, higher = less CPU on busy boxes. |
For the lowest possible “time-to-detect” on blackholed hosts, set the agent timer to 30 seconds and leave COLLECTOR_AGENT_FRESH_WAIT_MS at the default. Going below 30 seconds for the agent timer is rarely worth the extra ingest cost.
5. Troubleshooting stale snapshots
If the freshness pill stays amber or red:
- On the host, check the systemd timer:
systemctl status blackglass-agent.timer journalctl -u blackglass-agent.service -n 50 --no-pager - Verify the agent reaches your console:
curl -sI https://blackglasssec.com/api/agents/health # Expect: HTTP/2 200 - If the agent shows a 401 / 403, the per-host ingest secret has rotated. Re-run the install script — it’ll mint a new key and replace the old one.
- If you can SSH to the host but the agent isn’t pushing, the timer is most likely disabled (
systemctl enable --now blackglass-agent.timer).
Still seeing stale data? Get in touch and include your fleet size + a snippet of the agent journal — we’ll dig in.