24/7 Infrastructure Monitoring and Incident Alerting: Service Levels and Escalation

Cloud systems rarely fail in a single dramatic moment. They decay: a node quietly fills its disk, a connection pool saturates, a nightly job runs longer than its window, and eventually a customer picks up the phone. Monitoring exists to shorten the gap between the moment a system starts drifting and the moment a human being knows about it.

The trouble is that the phrase 24/7 monitoring covers wildly different products. It can mean a dashboard that is always on and never watched. It can mean a bot posting alerts into a channel nobody staffs after 6pm. Or it can mean a named person who is accountable, a committed response window, and a documented escalation path. The three cost different amounts and they are not remotely equivalent in value.

Monitoring, alerting and on-call are three separate products

Monitoring is collection and retention: metrics, logs, traces, health check results. It answers what state the system is in now and, just as importantly, what state it was in three weeks ago. Without history, every incident investigation degenerates into guesswork.

Alerting is the logic layer on top. When a condition holds, a message is delivered somewhere specific. An alert is only worth having if it leads to an action. An alert whose recipient does not know what to do with it is a design defect, and if it fires often enough it teaches everyone to ignore the whole channel.

On-call is the human layer: who is holding the phone at 2am, what access that person has to actually fix something, and who picks up if they do not answer. Vendors usually sell all three in one bundle. When you evaluate a proposal, separate them and ask about each independently. Plenty of contracts are strong on the first layer, adequate on the second, and effectively empty on the third.

Four layers worth instrumenting

Infrastructure

CPU, memory, disk capacity and IOPS, network throughput and packet loss, instance and node state, availability zone health. This is the easiest layer to monitor and the one that tells you least about user experience. A host at twenty percent CPU can be returning 500s to every customer. Keep this layer for diagnosis, not as the main source of pages.

Platform and managed services

Database latency and error rates, replication lag, open connections against the ceiling, queue depth, cache hit ratio, scheduled job status and duration, TLS certificate and domain expiry, service quota headroom. A large share of long-running incidents originate here rather than in the infrastructure beneath, mostly because these limits stay invisible right up until you hit one.

Application and user experience

Error rate per endpoint, response time at the 95th and 99th percentile rather than the mean, throughput, and the success rate of the journeys that matter commercially: sign-in, checkout, order placement, search. Synthetic checks running on a schedule from outside your own network are the cheapest way to know whether the system is genuinely serving users, even when every internal graph is green.

Security and operational discipline

IAM permission changes, administrative logins from unusual locations or at unusual hours, security group and firewall edits, configuration changed outside the change process, port scanning patterns, resources created outside approved regions. Most of these signals should not wake anyone at midnight, but they do need to be captured completely and reviewed on a fixed cadence.

Turning signals into alerts

The number of alerts is not a measure of monitoring quality. The useful measure is the proportion of alerts that lead to a worthwhile action. A few principles keep that ratio tolerable:

• Alert on symptoms users can feel, not on technical causes. A rising error rate on checkout is a good alert. A busy CPU on one node usually is not.
• Every alert needs a runbook covering how to confirm it, how to mitigate, and what closing it looks like. If nobody can write the runbook, the alert is not mature enough to enter the rotation.
• Tie thresholds to agreed service level objectives rather than to round numbers that feel comfortable. If the target is 99.9 percent successful requests per month, the threshold should reflect how fast the error budget is being consumed.
• Evaluate over a window rather than a single data point, so a thirty second spike does not generate a 3am phone call.
• Group alerts into incidents. When a database struggles, every dependent service fails; the responder needs one incident, not forty notifications.
• Alerts must auto-resolve when the condition clears. A long tail of stale open alerts destroys the ability to notice a new one.

Severity grading and what each level means

Severity is where the two sides align expectations. If it is not written down in advance, every incident becomes top priority to the customer and lowest priority to the provider. The table below is a common reference structure; the specifics should be rewritten around the business the system supports.

Level	Definition	Example	Notification path
P1	Service down or data at risk, no workaround available	Site unreachable; primary database unresponsive	Phone call to wake the responder, incident bridge opened
P2	Severe degradation affecting many users or one core business flow	Payments failing for twenty percent of attempts; latency up fivefold	Phone call within the committed window, notice to the shared channel
P3	Limited impact with a workaround, but escalates if ignored	Disk at fifteen percent free; a replica falling behind	Priority ticket, handled in business hours
P4	No service impact, needs recording and scheduled work	Certificate expiring in thirty days; configuration drift	Standard ticket, folded into the maintenance cycle

The numbers that belong in a service level agreement

Treat the items below as a checklist of things to make every bidder quote, not as commitments from anyone. The point is that two proposals are only comparable when both are forced to answer the same list:

• Time to acknowledge, measured from alert firing to a human confirming they are on it, stated separately per severity and separately for business hours and out of hours.
• Target time to mitigate, with an explicit distinction between a best-effort target and a contractual commitment with remedies. Very few providers will commit hard restoration times for software they did not write.
• The deadline for a written post-incident report at each severity.
• Availability of the monitoring platform itself, metric collection lag, and retention periods for logs, metrics and traces.
• Coverage by time zone and public holidays, and the working language of the rotation.

The first sixty minutes

Most of the damage in an incident happens in the first hour, and so does most of the confusion. A simple sequence that has been rehearsed beats a detailed one nobody has read.

1. The alert reaches the person on the current rota, not a shared mailbox.
2. That person acknowledges within the committed window. If they do not, the system automatically moves to the backup.
3. Establish impact before cause: how many users, which function, since when.
4. Prioritise restoring service over identifying the root cause. Rollback, traffic shifting, scaling up and disabling a feature behind a flag are all legitimate first moves, and they should be pre-authorised in writing.
5. Appoint an incident coordinator separate from whoever has their hands on the keyboard, as soon as three or more people are involved.
6. Communicate to the customer against an agreed template: what is happening, what the impact is, what is being done, when the next update lands. Do not promise a resolution time before the cause is understood.
7. Keep the timeline as you go. Reconstructing it afterwards is always expensive and always partly wrong.

Escalation: when, and to whom

Escalation is not an admission of failure; it is a designed mechanism, and it needs two directions. The technical direction moves an incident toward deeper expertise or higher privilege, from the duty engineer to a database specialist, and onward to the platform provider if the problem appears to sit on their side. The management direction moves information to someone empowered to accept a trade-off, such as losing a short window of data in exchange for a faster restore, or suspending a revenue-generating feature.

Three things need to be settled before you need them: the automatic escalation clock for each severity, a contact list per tier with backup numbers, and what each tier is actually authorised to decide. Review the list whenever staff change, because a wrong phone number in an escalation matrix is the kind of defect that only surfaces at the worst possible moment.

After the incident

A useful post-incident report contains the timeline, the measured impact, the direct cause and the contributing factors, what helped detection or delayed it, and a list of actions with named owners and dates. The hard part is not writing the report; it is tracking those actions to closure.

Review the alert configuration itself on a fixed cycle, quarterly for instance. Any alert that fired without leading to an action needs its threshold fixed or needs deleting. Any incident that occurred without a single alert firing needs new instrumentation. Skip this and the alert set inflates year on year and quietly stops working.

How HiTechCloud can help

HiTechCloud is an AWS partner and a Google Premier Partner, and operates infrastructure located in Vietnam. On monitoring and incident response, the team can work on: reviewing current coverage against the four signal layers described above; setting up metric, log and trace collection with a retention policy; building an alert set tied to agreed service level objectives, with matching runbooks; designing the rota, escalation matrix and incident communication templates; and joining post-incident reviews so findings turn into concrete configuration changes.

Scope, service level targets and commercial model are agreed per system. If you are comparing several proposals, use the checklist above as a common question set for everyone bidding.