LoveHeartBeat

—
Current organization
Organization settings Members
Multi tenant
Notifications
Jessie Samson
💬Mentioned you in a comment.10m
10:41 AMAugust 7,2021
Mark as unread
J
Jane Foster
📅Created an event.20m
10:20 AMAugust 7,2021
Mark as unread
Jessie Samson
👍Liked your comment.1h
9:30 AMAugust 7,2021
Mark as unread
Kiera Anderson
💬Mentioned you in a comment.
9:11 AMAugust 7,2021
Mark as unread
Herman Carter
👤Tagged you in a comment.
10:58 PMAugust 7,2021
Mark as unread
Benjamin Button
👍Liked your comment.
10:18 AMAugust 7,2021
Mark as unread
No notifications yet.
Notification history
Not signed in
—
Sign out Sign in

Escalation & Approval

Who gets woken when nobody answers, and what a machine is not allowed to do without a human saying yes

At L1

18

first responder

At L2

5

escalated once

At L3

2

escalated twice

Ack overdue

4

past window

Triage queue

L1 → L2 → L3: each tier gets the routing rule's window to acknowledge before the next one is paged

Alert	Tier	Team	Owner	Waiting / window	Status
p99 latency spike on /v1/checkout L3 · Infrastructure Team	L3	Infrastructure Team	unassigned	48m / 15m	Overdue
Glue job customer_sync failed L2 · Data Eng	L2	Data Eng	unassigned	22m / 15m	Overdue
ChatGPT spend threshold L1 · FinOps	L1	FinOps	FinOps owner	8m / 30m	Open
Apigee rate-limit storm L1 · API Platform	L1	API Platform	API on-call	12m / 30m	Acknowledged

View all

Alert notifications

What was sent, to which channels, and why

Alert	Tier	Channels	Recipient	Reason	Sent
p99 latency spike on /v1/checkout alert.escalated	L3	PagerDuty, SMS	Infrastructure Team	No acknowledgement in 15m — escalated to L3	3m ago
Glue job customer_sync failed alert.escalated	L2	Slack, Email	Data Eng	No acknowledgement in 15m — escalated to L2	22m ago
EC2 cost anomaly alert.resolved	L1	Slack	Cloud Ops	Resolved: reserved-instance drift, rightsized	1h ago

View all

Pending approvals

Actions the automation engine has proposed and is holding until a human decides

Proposed action	Requested by	Confidence	Impact	Waiting	Priority
Restart user-service in production Production Restart Approval	Automation engine	92%	user-service, auth-service · ~3m downtime	5m	High
Scale orders connection pool 450 → 600 Database Performance Approval	Database monitor	87%	orders-db · no downtime	12m	Medium
Scale checkout-api to 6 replicas Capacity Approval	Predictive scaler	81%	checkout-api · +$34/day	21m	Medium
Rotate the expiring Databricks PAT Security Policy Changes	Credential monitor	99%	Data Observability, ETL collection	2h	Low

View all

Escalation paths

Each level is given its timeout to acknowledge before the next one is paged

Path	Trigger	Levels	Success	Status
Critical Infrastructure On-call → Team lead → Eng manager → VP	severity=critical AND category=infrastructure	4 · 15m/30m/60m/120m	94%	Active
Security Incident Analyst → Security lead → CISO → Exec	category=security	4 · 10m/20m/30m/60m	98%	Active
Database Performance DBA → Senior DBA → Architect	source=database AND response_time>5s	3 · 20m/45m/90m	91%	Active
ETL Freshness Data engineer → Platform lead	category=etl AND freshness_breached	2 · 30m/90m	88%	Active

The routing rule a path belongs to carries its first timeout —Alert Managementis where that number is set.

Approval workflows

What automation may not do on its own

Workflow	Approvers	Timeout	Pending
Production Restart Emergency override allowed	Team lead (required), Eng manager	30m	3
Database Schema Changes No override	DBA, Architect, Eng manager — all required	60m	1
Security Policy Changes Emergency override allowed	Security lead, CISO — both required	45m	0
Cost-increasing scale-up Auto-approves under $50/day	FinOps owner	120m	0

Escalation analytics

How far up the chain incidents actually travelled, last 5 days

Day	Level 1	Level 2	Level 3	Level 4	Escaped L1
Mon	45	12	3	1	27%
Tue	52	15	4	0	29%
Wed	38	8	2	1	21%
Thu	41	11	3	0	27%
Fri	47	13	2	0	28%

An incident escaping level 1 usually means the first responder was already busy, not that the alert was hard. That ratio is the one worth watching.

MacBook Air - 13″

MacBook Pro - 13″

Carry Anna

John Smith

Jessie Samson

Jane Foster

Jessie Samson

Kiera Anderson

Herman Carter

Benjamin Button

Escalation & Approval

Who gets woken when nobody answers, and what a machine is not allowed to do without a human saying yes

At L1

18

At L2

5

At L3

2

Ack overdue

4

Triage queue

p99 latency spike on /v1/checkout

Glue job customer_sync failed

ChatGPT spend threshold

Apigee rate-limit storm

Alert notifications

p99 latency spike on /v1/checkout

Glue job customer_sync failed

EC2 cost anomaly

Pending approvals

Restart user-service in production

Scale orders connection pool 450 → 600

Scale checkout-api to 6 replicas

Rotate the expiring Databricks PAT

Escalation paths

Critical Infrastructure

Security Incident

Database Performance

ETL Freshness

Approval workflows

Production Restart

Database Schema Changes

Security Policy Changes

Cost-increasing scale-up

Escalation analytics

Theme Customizer

Color Scheme

RTL

Support Chat

Navigation Type

Vertical Navbar Appearance

Horizontal Navbar Shape

Horizontal Navbar Appearance