SIEM and Log Management

How logs are collected, normalised, retained and correlated in a SIEM, with a worked detection example, Sigma rules, tuning, and common pitfalls.

Almost every attack leaves traces in logs, but those traces are scattered across hundreds of systems in different formats. Log management gathers and stores them; a SIEM (Security Information and Event Management) system adds normalisation, correlation and alerting on top so analysts can answer "what happened, where, and to whom" from one place.


Log Sources

CategoryExamples
Operating systemWindows Event Logs (Security, System, Sysmon), Linux auth.log, syslog, journald, auditd
NetworkFirewall, router/switch, proxy, DNS, DHCP, VPN, IDS/IPS, NetFlow
ApplicationWeb server access/error logs, database audit logs, email gateway
CloudAWS CloudTrail, Azure Activity / Entra sign-in logs, Google Cloud audit logs, SaaS audit logs
IdentityActive Directory, SSO/IdP, MFA, PAM, authentication servers
Endpoint securityEDR, antivirus, DLP

Prioritise sources that answer security questions: authentication, process creation, network egress, privileged actions, and changes to configuration.


Collection

text
Sources --> [Agent / forwarder / syslog / API] --> [Collector] --> [Parser] --> [Index/Storage] --> [Search, Rules, Dashboards]

MethodNotes
Agent or forwarderInstalled on the host (for example Splunk Universal Forwarder, Elastic Agent/Beats, Wazuh agent); reliable, can buffer and filter
SyslogStandard protocol for network devices and Linux; UDP is lossy, prefer TCP/TLS
API / pullCloud and SaaS services expose logs through APIs or event streams
Agentless (WMI/WEF)Windows Event Forwarding sends events to a central collector without a third-party agent

Normalisation and Parsing

Raw logs differ wildly in format. Parsing extracts fields (timestamp, source IP, user, action, result) and normalisation maps them to a common schema, so one query such as user=alice works across firewall, VPN and application logs. Bad parsing is a silent failure: rules look at fields that are empty. Keep the original raw event alongside the parsed fields.


Retention and Integrity

  • Retention is driven by investigation needs (attackers are often present for months) and by regulation or policy. Common practice is a short "hot" period for fast search and a longer, cheaper "cold/archive" tier. Do not invent a number: use your policy and compliance requirements, see Data Maintenance.
  • Integrity matters because attackers delete logs. Forward logs off the host immediately, restrict who can modify the log store, use write-once or append-only storage, and hash archives. This also supports evidence use, see Digital Forensics.

Correlation, Use Cases and Detections

A single event is rarely conclusive; correlation links events across time and sources. A use case is a defined threat scenario plus the data, logic and response needed to detect it.

Example: brute force followed by success

text
IF   count(failed_login WHERE user = X AND source_ip = Y) >= 10  within 5 minutes
AND  a successful_login for user X from source_ip Y               within 10 minutes after
THEN raise alert "Possible password guessing - account compromise"
     severity = high if user is privileged, else medium
     enrich with: geo-location of IP, is the account an admin, is IP on a threat feed

The same pattern covers many scenarios: impossible travel, new admin account followed by privilege use, large outbound transfer after a malware alert. Map use cases to frameworks such as MITRE ATT&CK to see coverage gaps.

Tuning

ProblemFix
Too many false positivesRaise thresholds, exclude known-good (vulnerability scanners, service accounts), add context conditions
Missed attacksAdd sources, loosen over-tight rules, test with simulated attacks
Rule rotReview rules regularly; log changes under change management

UEBA

User and Entity Behaviour Analytics builds a baseline of normal behaviour per user or host and flags deviations (login at unusual hours, unusual data volume). It complements fixed rules and catches unknown patterns but needs good data and tuning. See also Security Technologies.


Sigma: Vendor-Neutral Rules

Sigma is an open, YAML-based format for describing detection logic independent of any SIEM. A rule is written once (log source, field conditions, severity, ATT&CK tags) and converted by tooling into Splunk, Elastic, Sentinel and other query languages. This makes community detections shareable and portable. Concept only here; the exact syntax is in the Sigma project documentation.


SOAR

A SIEM tells you something happened; SOAR (Security Orchestration, Automation and Response) acts on it: enrich the IP, create a ticket, disable the account, isolate the host, page the on-call analyst. Automate repetitive, low-risk steps first and keep human approval for disruptive actions. See Security Operations Center.


Common Products

ProductNotes
SplunkCommercial; powerful search language (SPL), widely used
Elastic Security (ELK)Elasticsearch, Logstash, Kibana; open core, flexible
Microsoft SentinelCloud-native SIEM/SOAR on Azure
WazuhOpen-source SIEM/XDR with host agents, file integrity monitoring and rules

Pitfalls

  • Time sync: if hosts disagree on the clock, events cannot be ordered or correlated. Use NTP everywhere and store timestamps in UTC.
  • Volume and cost: licensing and storage often scale with ingestion. Collect what supports a detection or investigation; filter noise (for example debug logs) at the source.
  • Blind spots: unmonitored systems, logging disabled by default, or logs not forwarded.
  • Parsing drift: software updates change log formats and break rules unnoticed.
  • No owner: rules without a person responsible for tuning decay into noise.

Key Points

  • Logs without correlation are noise; a SIEM turns them into detections.
  • Collect, parse, store safely, then write use cases tied to real threats.
  • Tune continuously; measure false-positive rate.
  • Accurate, synchronised time and protected log storage underpin everything.

Hands-on