Book a conversation
AdvisoryCoursesBlogContactBook a conversation

Blog · Essay · reliability engineering

Reliability Engineering: How a Real‑Time Asset Tracking Rollout Broke and What CEOs Must Do Next

1 Nov 2026 · Rohan Girdhani

When Yahoo Finance Singapore ran a piece titled *Agentic AI excitement runs ahead of results* — https://news.google.com/rss/articles/CBMiiAFBVV95cUxQR3lvRmFTUFRoSmRmYXdXTDBKVkQ1NGJmYkgtRVhhR1JmM0RlZmZMMTVIWVVNWE9HNjQxakp3YWVOd18zWmg3alZNdlVOakV5VlB6WUZrY3h3ZVRudm9jLTVCOUMtZmlHVF81SnF4ZGpVVU5VSGFDUzN2S0dOQkg5SmgtcWNTa0RV?oc=5 the conversation turned to whether hype can outpace delivery. The story of Cognosos, a Singapore‑based startup that raised $40.9 M to build a real‑time asset‑tracking platform, is a perfect counterpoint: the technology worked, but only after we survived a cascade of reliability failures that would have derailed any CEO’s timeline.

In this post I walk through what actually broke, how we diagnosed it, and what you can inspect and fix on Monday to avoid the same pitfalls. This is not a high‑level trend piece; it is a field‑report from the trenches of reliability engineering.

1. The original promise and the hidden assumptions

Cognosos promised its customers—logistics firms, construction sites, and equipment‑leasing companies—a sub‑second view of every tracked asset across a continent. The product stack was:

  • Edge devices (BLE‑enabled tags) sending telemetry every 500 ms.
  • MQTT broker (EMQX) in the cloud handling millions of concurrent connections.
  • Stream processing (Apache Flink) normalising, enriching and persisting data.
  • APIs for dashboards and downstream analytics.

The business case hinged on reliability: a missed ping could mean a misplaced crane or a delayed shipment, directly hitting revenue.

The hidden assumptions

  1. Uniform network quality – we assumed most devices would stay within Wi‑Fi range of a gateway.
  2. Linear traffic growth – we projected a 10 % month‑over‑month increase based on early adopters.
  3. Single‑region deployment – the architecture was built in one AWS region to keep latency low.

None of these assumptions held once the pilot moved to a national rollout.

2. The first failure: broker overload and silent message loss

What happened

Two weeks after onboarding a major construction client, the MQTT broker started rejecting connections. The logs showed `socket limit reached` errors, but the monitoring dashboard still reported a healthy broker because it only tracked CPU and memory, not socket count.

Why it mattered

The rejected connections meant hundreds of thousands of telemetry packets never entered the pipeline. The client’s dashboard showed “all assets online” while in reality half of them were silent.

How we diagnosed it

  1. Enabled socket‑level metrics on EMQX (via the built‑in Prometheus exporter).
  2. Correlated spikes with a new batch of edge devices that used a newer firmware version, which opened two simultaneous MQTT connections per tag for redundancy.
  3. Ran a load‑test replicating the new connection pattern, confirming the broker’s socket limit of 65 000 was being hit.

The fix (executed in 48 hours)

  • Increase the socket limit on the broker instances (Linux `ulimit -n`).
  • Deploy a second broker node behind a load balancer to distribute connections.
  • Patch the edge firmware to use a single connection, falling back to a retry after a back‑off instead of opening a second socket.

3. The second failure: downstream back‑pressure causing data loss in Flink

What happened

After the broker fix, the stream processor began to lag. Checkpoints were missing, and the exactly‑once guarantee was broken. The downstream PostgreSQL database started rejecting writes due to duplicate primary‑key errors.

Why it mattered

Data integrity is a core reliability metric. Duplicate rows meant the analytics team could not trust any KPI derived from the feed, and the client threatened to suspend the contract.

How we diagnosed it

  1. Enabled Flink’s back‑pressure metrics (`jobmanager.taskmanager.metric.backPressuredTimeMs`).
  2. Observed a sustained 30 % back‑pressure on the sink operator.
  3. Identified the bottleneck: the PostgreSQL write‑ahead log (WAL) was saturated because the transaction size had been increased to batch 10 000 rows per commit, a change made during the pilot to improve throughput.

The fix (executed in 72 hours)

  • Re‑tuned the sink to batch 1 000 rows per commit, reducing WAL pressure.
  • Added a side‑channel buffer (Kafka topic) between Flink and PostgreSQL to absorb spikes.
  • Implemented idempotent writes using a natural key and `ON CONFLICT DO NOTHING` to guarantee exactly‑once semantics even if retries occur.

4. The third failure: single‑region outage during a regional power cut

What happened

A scheduled maintenance window coincided with an unexpected grid failure in the AWS region hosting the broker and Flink cluster. All services went dark for 12 minutes. The client’s SLA demanded 99.9 % uptime, which translates to ≤ 43 minutes of downtime per month; we were already close to the limit.

Why it mattered

Beyond the SLA breach, the outage exposed a single point of failure in the architecture. The client’s risk‑management team demanded a multi‑region disaster‑recovery plan.

How we diagnosed it

  • Reviewed the architecture diagram and confirmed no cross‑region replication for MQTT or Flink state.
  • Checked CloudWatch alarms: none were set for region‑wide health, only for individual instance metrics.

The fix (executed in 1 week)

  • Deployed a secondary broker cluster in a different AWS region, using Global Accelerator to route devices to the healthiest endpoint.
  • Enabled Flink’s state‑backend to use S3** with cross‑region replication, allowing a warm standby.
  • Added region‑wide health alarms and a run‑book for a manual failover that can be triggered in under five minutes.

5. Monday checklist for CEOs and their CTOs

  1. Audit socket‑level limits on any message‑broker or load‑balancer you operate. Verify they exceed the projected peak connections by at least 30 %.
  2. Enable back‑pressure visibility in all stream‑processing jobs. Set alerts for > 15 % sustained back‑pressure.
  3. Validate idempotency in every downstream write path. If a retry can cause duplicate state, you have a reliability gap.
  4. Map single‑region dependencies. Any service that cannot survive a region outage should be duplicated or have a documented failover.
  5. Run a “traffic‑spike drill” quarterly: simulate a 2‑× increase in device connections and verify the end‑to‑end latency stays below your SLA.

These five items can be reviewed in a single 90‑minute meeting and turned into concrete tickets. The cost of the fixes is modest compared to the revenue at risk when a client experiences data loss.

6. Lessons beyond the technical fixes

  • Assumptions are liabilities – always surface them early and test them under realistic load.
  • Metrics must match failure modes – monitoring CPU alone missed the socket‑exhaustion problem.
  • Operational resilience is a product feature – your customers will judge reliability the same way they judge functionality.

By treating reliability engineering as a first‑class product requirement, you move from “nice‑to‑have” to “must‑have”, aligning engineering incentives with the CEO’s revenue goals.

FAQ

What is the difference between reliability engineering and traditional DevOps?

Reliability engineering focuses on system‑wide fault tolerance, quantifying failure modes and designing for graceful degradation. DevOps emphasizes delivery speed and automation. The two overlap, but reliability engineering adds rigorous failure‑mode analysis and SLA‑driven design.

How much budget should I allocate to reliability improvements?

A rule of thumb is 5–10 % of the total product budget for reliability‑related tooling, testing, and redundancy. The ROI is measured in avoided SLA penalties and retained customer revenue.

Can I adopt these fixes without a full architecture rewrite?

Yes. Most of the actions—socket limit adjustments, back‑pressure alerts, idempotent writes, and multi‑region replication—are incremental and can be rolled out in stages.

How do I convince my board that reliability engineering is worth the spend?

Present a risk‑adjusted cost‑benefit analysis: quantify potential revenue loss from downtime versus the modest incremental cost of the fixes. Use real incidents (like the 12‑minute outage) as concrete examples.

Where can I find more practical examples of reliability engineering in AI‑driven products?

My blog contains several case studies, including the adoption‑versus‑transformation series and the judgment‑under‑constraint edge‑AI rollout. Visit the blog index for the full list.

If you’re a CEO who wants to ensure your AI‑powered services survive the real‑world stresses that break most pilots, let’s talk. I can help you map the hidden reliability gaps in your stack and prioritize fixes that protect revenue. Book a short discovery call here: https://calendly.com/rohan-girdhani/discovery-call.

*Related reading:*

In the market

Headlines this post is responding to — not invented stats.