When a device raises an alert, someone is waiting for it. A fuel pump reporting a fault, a freezer warming up, a door left open. The question that matters is simple: how long did it take from the moment it happened to the moment a person knew?
Most teams cannot answer that with confidence. They have one number from one place, usually an average, and it describes the system on a good day. This guide covers how to measure latency end to end, so you know where the time goes and how slow your slowest alerts really are.
Where the time goes
A message from a connected device passes through several systems before it reaches anyone. Each one adds a little time, and each one can be the place where a delay starts.
- The device takes the reading and may wait to batch it with others.
- A phone or gateway receives it and forwards it, sometimes after reconnecting first.
- The broker, often MQTT on a service such as AWS IoT Core, accepts it and routes it.
- Rules and backend code decide whether it needs an alert.
- The alert goes out by push notification, SMS or email, each with its own delivery time.
Measure only the total, and a slow gateway looks exactly like a slow notification service. Measure hop by hop, and the slow one is obvious.
Put a timestamp on every hop
The method is straightforward. The device writes the time it took the reading into the message itself. Every system the message passes through adds its own time as it handles it: when the broker received it, when the backend made its decision, when the alert was handed to the delivery service. Store those times with the message, and the delay of every hop becomes a subtraction.
Two rules keep this honest. Start from the time the event happened on the device, never the time the message arrived. And keep every timestamp in one fixed format, in UTC, with milliseconds, so times from different systems compare without conversion errors.
Keep the clocks honest
Comparing a device time with a cloud time only works if both clocks agree. Device clocks drift, and a device that has been offline or restarted may not know the time at all.
- Sync the device clock over NTP when it connects, and record whether each reading was taken with a synced clock.
- Measure durations inside one system with a monotonic clock, which never jumps when the time is corrected.
- Flag messages whose device time is in the future or far in the past, and fall back to the time they were received.
- Keep cloud services on one time source, which managed cloud services already do.
With synced clocks, the gap between device and cloud is accurate to a few milliseconds, far finer than any delay a person would notice.
Report percentiles, not averages
Latency is never spread evenly. Most messages are quick and a small share are slow, sometimes very slow, because of a weak signal, a reconnect or a busy service. An average blends them together and lands on a figure that describes almost no real message.
Percentiles answer better questions. The median is the time half your messages beat. The 95th percentile is the time all but one in twenty beat, and the 99th percentile all but one in a hundred. A fleet that sends ten thousand alerts a month has a hundred above its 99th percentile, and those are the ones users remember.
Set a budget and alert on it
Once the hops are measured, give each one a share of the total delay you can accept. That split is a latency budget. A fault alert that must reach someone within ten seconds might allow two seconds on the device and gateway, one in the broker, two in the backend and five for delivery. The numbers differ for every product. Writing them down is what matters.
What to watch once the fleet is live
- The 95th and 99th percentile of total latency, every day.
- Each hop against its share of the budget.
- How many messages arrive without a synced device time.
- Every alert that took longer than the budget, listed one by one so each can be traced.
Treat latency like any other health signal. When a percentile moves past its budget, someone should hear about it before a customer does.
This is how we measure the systems we build at CoreFinity Tech. Every message is timestamped from the device to the alert, and the slow ones are tracked as closely as the typical ones. If you want to know how quickly your own alerts really arrive, send us a short description of the system and we will tell you what to measure first.