Monitoring, Detection & OT Incident Response

Ask an IT security operations center how it watches a network, and the toolkit is familiar: agents on every endpoint, periodic vulnerability scans that actively probe every host, and a SIEM that assumes every log source keeps reasonably accurate time. Ask an OT team the same question, and each of those defaults breaks in a specific, predictable way. This lesson is an SME-level walkthrough of why monitoring and incident response look different in an industrial control system (ICS), and what a defensible OT security operations program does instead — not because OT teams are behind, but because the equipment and the consequences are genuinely different.

Four constraints drive everything in this lesson:

• You often can't safely ask a device a question. An active vulnerability scan sends unexpected packets to find out what's listening. Many legacy PLCs and RTUs were never tested against malformed or out-of-sequence packets, and some will hang, reboot, or drop into a fault state when they receive one — during a live process.
• Most field devices don't know how to tell you what they're doing. A PLC's engineering port frequently has no logging capability at all. The absence of a log entry is not the absence of an event — it's a coverage gap, and pretending otherwise creates a false sense of visibility.
• Correlating events across sources requires everyone to agree on the time. A SIEM's entire value proposition — "this login happened, then two minutes later this setpoint changed" — collapses if the two systems' clocks disagree by more than a few seconds, and isolated OT networks are exactly where clock drift goes unnoticed.
• The standard incident-response move, isolate the host, can itself be the hazard. Pulling the network cable on a compromised Windows laptop is a Tuesday. Pulling the network cable on a PLC actively controlling a live batch, or a safety system with no local fallback, can leave a physical process in an undefined and dangerous state.

Learning objectives

By the end of this 4-hour session, you will be able to:

1. Explain why passive monitoring is preferred over active scanning on an OT network, citing the specific risk active scanning poses to legacy field devices.
2. Identify syslog coverage gaps common to Level 0/1 field devices and describe at least two compensating telemetry sources that can substitute for a missing device log.
3. Explain the role of time synchronization in event correlation and describe the consequence of clock drift during incident reconstruction.
4. Triage, in simulation, a queue of security alerts by weighing asset criticality, safety impact, and confidence rather than alert volume alone.
5. Evaluate a containment decision in which isolating the affected host risks a safety or production consequence, and select a defensible compensating action.
6. Design an OT-aware incident response playbook section that names who decides on containment and what happens when full isolation isn't an option the plant can accept.

Active scanning discovers and characterizes network hosts by sending them unexpected traffic — port probes, crafted packets, sometimes exploit-style test payloads — and interpreting the response. It's the IT default because most modern operating systems are built and tested to survive exactly that kind of traffic without incident. Passive monitoring discovers and characterizes the same network by observing traffic that's already flowing, without ever originating a packet toward the devices being watched. It's slower to build a complete picture and it can miss a device that never talks, but it cannot itself cause the outage it's trying to detect.

The reason OT defaults to passive monitoring isn't caution for its own sake — it's a documented failure mode. Many Level 0/1 field devices (PLCs, RTUs, protective relays) run embedded firmware with a minimal or nonexistent network stack, built decades ago against an assumption that only trusted engineering traffic would ever reach the port. An unexpected packet — even a benign, well-formed one the device just wasn't built to expect — can cause the device to hang, drop its current control logic, or in rarer cases reboot. On a device that's actively holding a valve position or running a safety interlock, that reboot is not an inconvenience; it's a process event.

Worked example — deploying a passive sensor at Cascade Summit Electric, Substation 12

Cascade Summit Electric (fictional) wants visibility into which devices are communicating on Substation 12's control LAN without touching any of the relays or RTUs directly. The design:

• A network tap or a switch's SPAN/mirror port is configured to copy all traffic on the control-LAN switch to a dedicated monitoring port — the copy is one-way; nothing sent to that port can reach back onto the live network.
• A passive monitoring sensor plugged into that mirrored port parses the copied traffic — Modbus TCP function codes, DNP3 object groups, ARP and DHCP chatter — to build an asset inventory and a baseline of "who normally talks to whom."
• The sensor never originates a single packet toward the relays or RTUs themselves. It only listens.
• Over roughly two to four weeks, the sensor's baseline stabilizes: which IEDs poll which RTU, on what cadence, using which function codes. Traffic that doesn't fit that baseline — a new IP address, an unfamiliar function code, a device polling at ten times its normal rate — becomes the anomaly signal Section 4 builds on.

Two failure modes to watch for: first, treating a passive sensor's asset list as complete on day one — a device that only talks once a month (a backup RTU, for example) won't appear until it actually communicates, so early asset counts should be treated as a floor, not a ceiling. Second, assuming a mirrored SPAN port on an oversubscribed switch reliably copies every packet — under heavy load, switches can silently drop mirrored traffic before it reaches the sensor, so mirrored coverage should be spot-checked against a known traffic pattern, not assumed.

Syslog is the standard protocol and message format most IT infrastructure uses to forward event records — logins, configuration changes, errors — to a central collector for retention and analysis. It's the backbone assumption of a typical SIEM deployment: every meaningful device generates a log, and every log gets forwarded somewhere central. That assumption holds reasonably well at Level 2 and above in an OT network — HMIs, engineering workstations, jump hosts, and OT firewalls generally run modern operating systems that support syslog forwarding out of the box.

It breaks down hard at Level 0/1. Most PLCs, RTUs, and protective relays have no syslog client, no concept of a forwarded event log, and in many cases no persistent event storage at all beyond a small internal fault buffer that overwrites itself. The device that is, arguably, the single most important thing to monitor — because it's the one actually touching the physical process — is frequently the one contributing nothing to the log stream.

Worked example — instrumenting a legacy PLC with no logging capability at Ironwood

Ironwood Precision Manufacturing (fictional) runs a 2009-vintage PLC controlling a mixing line. The PLC has no syslog client and no forwarded event capability. Rather than accept a total blind spot, the identity of "what happened on this PLC" gets reconstructed from three compensating sources instead of one device log:

• Passive protocol parsing. The Section 1 passive sensor, already watching the wire, decodes the Modbus function codes flowing to and from the PLC — a write to a holding register, a mode change from Run to Program — and timestamps each one, even though the PLC itself never generated a log entry.
• Upstream device logs. The engineering workstation used to connect to the PLC does support logging — its own login events, and which project file was opened and when, become a proxy for "who was working on this PLC and when," even without a log from the PLC confirming what they actually changed.
• Physical process correlation. The historian recording the mixer's live tag values (temperature, agitator speed, setpoint) shows exactly when a physical parameter changed — which, cross-referenced against the passive sensor's Modbus writes, tells the investigator which write actually took effect.

Correlating events across multiple sources — "this remote login happened, then ninety seconds later this setpoint changed" — only works if every source's timestamp reflects the same clock, or close enough to it that the sequence of events is unambiguous. Network Time Protocol (NTP) is the standard mechanism for keeping a fleet of devices synchronized to a common time reference, typically a small number of authoritative time servers, often themselves disciplined by GPS.

OT networks are exactly where clock drift goes unnoticed longest, for a structural reason: many control-network devices are intentionally isolated from the internet and therefore from the public NTP pool most IT infrastructure syncs against by default. Left with no time source at all, a device's internal clock drifts — sometimes by seconds per day, occasionally by more — and nobody notices, because the device's own operation doesn't depend on knowing the correct time. It only becomes a problem the moment someone tries to correlate that device's log against a different system's log during an investigation.

Worked example — the two-minute gap that wasn't

During a tabletop review at a fictional water utility, investigators initially concluded that a suspicious remote login to a jump host occurred two full minutes after an unauthorized setpoint change on a downstream RTU — which would have made the login irrelevant to the setpoint change, since it happened afterward. A closer look found the RTU's local clock, never synchronized to any time source, had drifted nearly three minutes fast over the eleven months since its last manual clock set. Correcting for the drift put the login before the setpoint change, by about fifty seconds — reversing the entire causal story the investigation had been building.

Where a device genuinely can't run an NTP client — a real possibility at Level 0/1 — the compensating practice is a documented, periodic manual clock check against the authoritative internal source, logged so an investigator later knows exactly how much drift to expect and correct for, rather than discovering the gap mid-investigation the way the water-utility example did.

A security information and event management (SIEM) platform aggregates events from many sources — passive network sensors, syslog from Level 2+ devices, firewall logs, authentication systems — into one searchable stream, and applies correlation rules to flag combinations worth a human's attention. A SIEM doesn't decide what matters; it decides what's worth surfacing. Deciding what matters is alert triage, and doing it well in OT requires weighing factors a generic IT triage rubric doesn't fully capture.

A batch changeover on a packaging line can generate dozens of legitimate "anomalies" in a single hour — new setpoints, valves cycling in an unusual order, a brief spike in a flow rate — all completely normal for that specific operating mode. A single unexplained write to a safety PLC's configuration register, by contrast, might generate exactly one alert all week. A triage process that ranks by volume or by generic severity score will bury the second alert under the first. OT alert triage has to weigh at least three things together, not volume alone: asset criticality (what does this device actually control, and what happens if it's wrong), safety impact (could this plausibly affect a person's safety, not just production), and confidence (how well does this match a known bad pattern versus a merely unfamiliar one).

Worked example — two alerts, one shift

Most incident-response frameworks describe a similar lifecycle: Preparation, Detection & Analysis, Containment, Eradication, Recovery, and Post-Incident Activity. The first two phases work almost identically in OT and IT — the differences this lesson has already covered (passive monitoring, syslog gaps, time sync) live inside detection and analysis. It's containment — stopping the incident from getting worse — where OT and IT genuinely diverge, and where a well-meaning IT-trained responder can cause real harm by applying the wrong default.

In IT, the default containment move for a compromised host is to isolate it — pull the network cable, disable the switch port, quarantine the endpoint — while investigation continues elsewhere. That default assumes the host can be safely disconnected without an immediate physical consequence, which is true of nearly every IT endpoint. It is not automatically true of an OT asset. A PLC actively holding a valve position, running a safety interlock, or mid-cycle on a batch process may not have a safe "disconnected" state at all — depending on the specific control architecture, cutting its network connection could leave it holding its last command indefinitely, freeze it in an unsafe intermediate state, or in some designs trigger a fail-safe shutdown that itself has a physical consequence (a line stopping mid-batch, a boiler tripping) that has to be managed, not just accepted as a side effect of "containing" the incident.

Worked example — a suspicious write to a live batch controller

At Ironwood Precision Manufacturing (fictional), the Section 4 triage process flags an unexplained write to a mixer PLC that is, at that exact moment, mid-cycle on an active batch. The incident-response playbook does not default to "isolate the PLC." Instead, it walks a decision the playbook itself has to spell out in advance:

1. Confirm current process state with the shift operator before any network action — is this PLC actively controlling something right now, and what happens physically if its connection drops?
2. Weigh compensating containment options that don't require full isolation: restricting the specific command/register the write targeted at the firewall, increasing monitoring resolution on that PLC specifically, or placing an operator in direct visual/manual oversight of the process while investigation continues.
3. Escalate the isolation decision itself to a named role who can authorize it — typically a controls engineer or plant operations lead jointly with the security lead — rather than letting a security analyst unilaterally sever a live control loop.
4. If isolation is chosen, do it in coordination with operations so the process can be brought to a safe, planned state first, rather than as a unilateral, unannounced network action.

None of this means OT incidents never get contained by isolation — sometimes isolation is exactly the right call, and delaying it out of excessive caution is its own failure mode. The point is that the decision requires an extra input IT containment doesn't: current physical process state, confirmed with the people who can actually see it, before the network action happens.

The lifecycle phases are identical on paper. What changes is which inputs each phase requires before it's safe to act — and in OT, several of those inputs are physical, not just digital.

⚠ Simulation only — no real network traffic, no real SIEM, no live equipment of any kind is touched or represented with real data.

This builder models a single alert arriving against the fictional Cascade Summit Electric, Substation 12 network. Choose the asset the alert concerns, where the alert came from, how confident the detection is, whether it carries a safety implication, and what the affected asset is doing physically right now. Select Triage this alert and the tool reasons through the combination the way a triage analyst and a controls engineer would together — recommending a priority level and a containment action, and explaining why full isolation is or isn't the right first move.

A written incident-response playbook section for a specific asset class is the short document that answers, before the alert ever fires, exactly what telemetry substitutes for a missing device log, who gets consulted before any containment action, and what happens when full isolation isn't an option the plant can accept. Without one, every incident gets negotiated from scratch under time pressure — usually while a process is running and nobody wants to be the person who guesses wrong.

Standards, guidance, and video