DevSecOps Program · Part 4 — Runtime & Response: Detect and React

This is Part 4 of the DevSecOps Program — the phase where the program turns to face the running system. Everything Parts 1 through 3 built protects code before it ships. Part 2 gated your source, dependencies and secrets in CI. Part 3 secured the two ends of the build: it scanned infrastructure-as-code before provisioning, generated an SBOM for every build, and signed artefacts so a cluster could verify it was running the object you actually built. All of that is preventive — it hardens what you deploy and proves what is inside it. But prevention is never complete. A dependency that was clean at build time becomes vulnerable the moment a new CVE lands. A misconfiguration slips a gate. A credential leaks through a path no scanner watched. Part 4 accepts that reality and builds the other half of the program: the ability to see what your running system is doing, notice when it deviates from normal, and respond in a way that is fast, rehearsed and provable. The through-line is one sentence: assume something will get through, and be ready to detect it and react before it becomes a breach.

Scope of this phase

Every part of this program is scope-bounded so you know exactly when you are done and are not tempted to boil the ocean.

• In scope: runtime monitoring and detection (centralised logging, a cloud-native SIEM or Falco watching runtime behaviour, and a working definition of "normal" tied to the asset inventory from Part 1); runtime vulnerability management (continuously scanning running images and hosts, not just the build); alerting that respects attention (high-signal alerts routed to a channel a human watches, with severity-based routing and alert fatigue treated as a security failure); and incident response (a written, rehearsed IR plan with defined roles, quarterly tabletop exercises, and an understanding of the NIS2 / DORA / GDPR reporting clocks).
• Explicitly out of scope (the final part): continuous control monitoring, security metrics (MTTR, coverage, mean time to detect), audit-evidence aggregation and the maturity/governance layer that ties the whole program together — all of that is Part 5. Here you build the operational capability; Part 5 measures and governs it.
• Definition of done: the exit checklist at the end of this part. When every box is ticked, your runtime emits centralised logs, a detection layer flags anomalous behaviour, running images are re-scanned against new CVEs, high-severity alerts reach a watched channel without drowning in noise, and you have a written IR plan you have actually rehearsed against the clock.

The order matters. Runtime detection is only meaningful once the build is trustworthy. If you cannot vouch for what you deploy, every runtime alert starts from "is this even our code?" With Parts 1–3 in place, the signed, inventoried artefacts you ship are exactly what this phase observes — and when an incident does happen, the SBOM and provenance from Part 3 are what let you trace it back to a specific build in minutes.

From protecting the build to watching the system

Parts 1–3 all share a stance: inspect something before it runs and block it if it is dangerous. That stance has a hard limit — it can only catch what is knowable at build time. Three things routinely escape it:

• Time. A container image can pass every SCA check on the day it is built and become vulnerable the next morning when a CVE is disclosed against a library it contains. The artefact did not change; the world's knowledge of it did.
• Behaviour. Static analysis reasons about code; it cannot see what a process actually does once it is live — the outbound connection to an IP it has never contacted, the shell spawned inside a container that should only ever run one binary, the privilege escalation attempt.
• The unknown. No gate catches everything. A novel misconfiguration, a leaked credential, a supply-chain compromise upstream of your scanners — some fraction always gets through, and the only place left to catch it is at runtime.

Part 4 is the layer that addresses all three. It is the transition from protecting the build to watching the system, and it is what makes the whole program resilient rather than merely careful.

Read the loop as a cycle, not a checklist: you observe the running system through centralised telemetry, detect deviations from normal, alert the right human with enough signal to act, respond through a rehearsed plan, and learn by feeding every incident back into detections, gates and the plan itself. Each stage in this phase maps onto one turn of that loop — and the loop only tightens if the "learn" step actually closes back onto the earlier parts of the program.

Runtime monitoring and detection

You cannot detect what you cannot see, so the foundation is centralised logging. Application logs, container and orchestrator logs, cloud audit trails (CloudTrail, GCP Audit Logs), authentication events and network flow logs all need to land in one queryable place — a cloud-native SIEM, a managed log platform, or an open stack. Scattered logs on individual hosts are worthless during an incident, when the one machine you need is the one that has been wiped. Centralisation is also what turns "we think something happened" into a timeline you can reconstruct.

On top of the logs sits detection. Two complementary approaches:

• A SIEM correlates events across sources and fires rules against them — a burst of failed logins followed by a success, an IAM policy widened to *:*, a storage bucket made public, a login from an impossible-travel location. This is the control-plane and identity view.
• Falco watches the runtime behaviour of your workloads. Using eBPF to observe kernel-level syscalls, it detects things a log aggregator never sees: a shell spawned inside a container, an unexpected outbound connection, a write to a sensitive path, a process escaping its expected profile, container drift where a running container diverges from its immutable image. This is the workload view.

Detection depends entirely on knowing what "normal" looks like — which is where the asset inventory from Part 1 pays off again. The inventory told you which services exist, what they talk to, and who owns them. That is the baseline. An outbound connection is only "unexpected" relative to a service whose normal egress you have characterised; a shell in a container is only alarming because you know that container should run exactly one process. Detection without a baseline is just noise generation.

A Falco rule is a small, readable statement of "this should never happen in a healthy system":

- rule: Shell spawned in container
  desc: An interactive shell was started inside a running container
  condition: >
    spawned_process
    and container
    and shell_procs
    and proc.tty != 0
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
     image=%container.image.repository proc=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

The rule reads almost like the sentence you would say out loud: if an interactive shell starts inside a container, tell me — because in an immutable, single-purpose container, nobody should be opening a shell in production, and an attacker who has gained a foothold almost always does.

Runtime vulnerability management

Part 2's SCA scanning answered a question at a single moment: at build time, does this image contain known-vulnerable dependencies? That is necessary but time-bound. The set of "known" vulnerabilities grows every single day, and your running fleet does not rebuild every day. An image that was genuinely clean when it passed the pipeline last month may be carrying a critical CVE right now, disclosed after it shipped, still running in production untouched.

Runtime vulnerability management closes that gap by continuously re-scanning what is actually running — the images deployed to your clusters and the hosts underneath them — against the current vulnerability databases, not the ones that existed at build time. Tools such as Trivy (running as an in-cluster operator), Grype, or your cloud provider's native image scanning re-evaluate deployed workloads on a schedule and flag newly-disclosed CVEs in things you already shipped.

The contrast with Part 2 is the whole point:

The two are not redundant; they cover different windows in time. Build-time SCA stops you shipping something known-bad today. Runtime scanning tells you when something you already shipped became known-bad. Feed runtime findings back through the same triage and remediation path — the SBOMs from Part 3 make "which running services contain the affected package?" a query rather than an investigation — and a fresh critical CVE becomes a scoped patch-and-redeploy instead of a fire drill.

Alerting that respects attention

A detection layer that fires on everything is worse than no detection layer, because it trains your team to ignore it. Alert fatigue is a security failure mode, not an operational annoyance: when the channel is full of low-value noise, the one alert that matters scrolls past unseen, and you have engineered a system that looks like it is watching while nobody actually is. The 2013 breaches that famously did trigger alerts nobody acted on are the canonical lesson — the tool worked; the human attention around it did not.

The discipline is to route by severity and protect the high-signal channel:

• Critical / high — a confirmed or high-confidence indication of compromise (Falco flags a shell in a production container, a new critical CVE lands on an internet-facing service, an anomalous privilege escalation) goes to a channel a human is guaranteed to see: a paging tool, an on-call rotation, a dedicated Slack channel that is only used for things requiring action.
• Medium — worth review but not worth waking someone: a ticket, a daily digest, a dashboard reviewed each morning.
• Low / informational — logged and queryable, never pushed. It exists for investigation, not interruption.

Severity routing is easy to express as code, which keeps it reviewable and versioned alongside everything else:

route:
  receiver: default-digest
  group_by: [alertname, service]
  routes:
    - matchers: [severity="critical"]
      receiver: pagerduty-oncall     # wakes a human
      continue: false
    - matchers: [severity="warning"]
      receiver: slack-review          # watched, not paging
receivers:
  - name: pagerduty-oncall
    pagerduty_configs: [{ service_key: "<key>" }]
  - name: slack-review
    slack_configs: [{ channel: "#sec-review" }]
  - name: default-digest
    webhook_configs: [{ url: "https://digest.internal/collect" }]

The goal is not fewer alerts for their own sake — it is a channel where every message genuinely warrants a human's attention, so that attention is still there on the day it counts. Tune aggressively: every false positive that reaches the paging channel should either be fixed at the rule or demoted, treated as a bug in the detection system.

Incident response — a plan you have rehearsed

Detection tells you something is wrong. Incident response is what you do next — and the single biggest predictor of how an incident goes is whether the plan existed before it started. An IR plan invented at 2 a.m. during a live compromise is not a plan; it is improvisation under the worst possible conditions.

A written IR plan does not need to be long, but it must be concrete. The core of it:

1. 1Roles. Who is the incident lead (makes decisions, owns the timeline)? Who handles technical containment? Who owns communications — internal, customer, regulator? Name the roles and who fills them, including backups. In a small team one person may wear several hats — write that down too.
2. 2Detection & triage. How does an incident get declared, and by whom? What are the severity levels, and what does each trigger?
3. 3Containment. Stop the bleeding without destroying evidence: isolate the affected host or revoke the credential, but snapshot before you wipe.
4. 4Eradication. Remove the root cause — the vulnerable version, the leaked key, the backdoored dependency — not just the visible symptom.
5. 5Recovery. Restore to known-good, verify integrity (the signatures from Part 3 matter here), and watch closely for recurrence.
6. 6Post-mortem. A blameless write-up: what happened, the timeline, what worked, what didn't, and concrete follow-up actions. This is the "learn" stage of the loop — it feeds new detections, new gates, and plan improvements back into the program.

A plan on paper is a hypothesis. Quarterly tabletop exercises are how you test it: gather the people, walk through a realistic scenario ("a critical CVE just dropped on our public API gateway and we are seeing anomalous outbound traffic"), and follow the plan out loud. Tabletops reliably surface the gaps that matter — the on-call engineer who cannot revoke a production credential, the missing contact for the DPO, the assumption that someone else was monitoring a system nobody owned. Fixing those in a conference room is free; fixing them mid-incident is not.

The regulatory clock as a forcing function

For many startups and SMEs, a real breach comes with a countdown that is now written into law — and those deadlines are a useful forcing function precisely because they are unforgiving:

• NIS2 (for in-scope essential and important entities) requires an early warning within 24 hours of becoming aware of a significant incident, followed by a fuller incident notification within 72 hours, and a final report within one month.
• DORA (for EU financial entities and many of their ICT providers) sets its own incident-reporting obligations for major ICT-related incidents, with an initial notification followed by intermediate and final reports on a defined timeline.
• GDPR requires notifying the relevant supervisory authority of a personal-data breach within 72 hours of becoming aware of it, where the breach is likely to result in a risk to individuals.

You cannot assemble the who, what and when of an incident inside a 24-hour window if you are starting from scratch — the reporting clock only works if your logging, detection and IR plan are already in place. Treat these deadlines not as compliance box-ticking but as design requirements: they tell you how fast your detect-and-respond loop actually has to run. Rehearsing against the clock in a tabletop is how you find out whether it can.

The output of this phase is evidence

Everything in this phase produces exactly the operational evidence that ISO 27001 and SOC 2 expect — and, unlike a policy document, it is evidence generated by the system doing its job rather than assembled the week before an audit:

• Centralised, retained logs demonstrate the logging and monitoring controls (ISO 27001 A.8.15/A.8.16; SOC 2 CC7.2).
• Alert and detection records show that monitoring is not just configured but acted upon.
• Incident records and post-mortems are the artefacts an auditor asks for under incident-management controls (ISO 27001 A.5.24–A.5.28; SOC 2 CC7.3–CC7.5) — proof you can detect, respond and learn.
• Tabletop exercise notes evidence that the plan is tested, not shelf-ware.

Stand this phase up properly and "we monitor our systems and we can respond to incidents" stops being an assertion and becomes a stack of timestamped, attributable records the pipeline and the runtime produce on their own.

Definition of done — Runtime & Response exit checklist

You are ready for Part 5 when every one of these is true:

• Centralised logging: application, container, cloud-audit, auth and network logs land in one queryable, retained store — not scattered on hosts.
• Runtime detection in place: a SIEM correlates control-plane and identity events, and Falco (or an equivalent) watches workload runtime behaviour for shells, unexpected egress, privilege escalation and container drift.
• "Normal" is defined: detections are baselined against the Part 1 asset inventory, so an anomaly is measured against a known-good picture rather than guessed at.
• Running workloads re-scanned: deployed images and hosts are continuously scanned against current CVE data, catching vulnerabilities disclosed after the build shipped — with SBOMs making impact a query.
• Severity-routed alerting: critical alerts page a human on a watched channel; medium and low are digested or logged; the paging channel is kept high-signal and false positives are treated as bugs.
• A written IR plan exists: roles, triage, containment, eradication, recovery and a blameless post-mortem step are documented and accessible.
• The plan is rehearsed: at least one tabletop exercise has run, against the clock, with gaps captured and assigned.
• The clock is understood: the team knows the NIS2 (24h/72h), DORA and GDPR (72h) obligations that apply to it and has designed the response loop to meet them.

Tick every box and you can now state, and prove, that you will notice when something goes wrong in production and respond to it in a rehearsed, time-bounded, evidenced way — the operational half of the program that prevention alone can never provide.

What's next

Part 5 — Govern & Mature is the final part, and it turns from doing the work to proving and improving it. Everything across Parts 1 through 4 now produces signals — merge gates, scan results, SBOMs, signatures, logs, alerts, incident records. Part 5 is where those become a governed program: continuous control monitoring, the metrics that tell you whether it is actually working (mean time to detect, mean time to respond, coverage, gate pass rates), audit-evidence aggregation mapped to frameworks, and the maturity model that takes you from "we have controls" to "we can demonstrate a managed, improving security program." It ships next month and closes the saga.

Runtime is also where security stops being a point-in-time claim and becomes a continuous, evidenced practice — which is exactly the shape GRC frameworks reward. ISMShed ingests your logs-and-alerts posture, runtime scan findings, incident records and tabletop evidence and maps them to controls across ISO 27001, ENS, NIS2, DORA, SOC 2 and GDPR — so "we can detect and respond" is backed by machine records and dated exercise notes, not a policy PDF. If you would like experienced hands to stand up centralised logging and detection, tune alerting so it stays high-signal, and run a first tabletop against the NIS2 and DORA clocks with your own team, Axelia's DevSecOps, Red Team and GRC consultants run the program — and rehearse the response — alongside you. Detect well and respond calmly, and the final phase is simply proving, at scale, what you already do.