Almost every attack leaves traces in logs, but those traces are scattered across hundreds of systems in different formats. Log management gathers and stores them; a SIEM (Security Information and Event Management) system adds normalisation, correlation and alerting on top so analysts can answer "what happened, where, and to whom" from one place.
Log Sources
| Category | Examples |
|---|---|
| Operating system | Windows Event Logs (Security, System, Sysmon), Linux auth.log, syslog, journald, auditd |
| Network | Firewall, router/switch, proxy, DNS, DHCP, VPN, IDS/IPS, NetFlow |
| Application | Web server access/error logs, database audit logs, email gateway |
| Cloud | AWS CloudTrail, Azure Activity / Entra sign-in logs, Google Cloud audit logs, SaaS audit logs |
| Identity | Active Directory, SSO/IdP, MFA, PAM, authentication servers |
| Endpoint security | EDR, antivirus, DLP |
Prioritise sources that answer security questions: authentication, process creation, network egress, privileged actions, and changes to configuration.
Collection
Sources --> [Agent / forwarder / syslog / API] --> [Collector] --> [Parser] --> [Index/Storage] --> [Search, Rules, Dashboards]
| Method | Notes |
|---|---|
| Agent or forwarder | Installed on the host (for example Splunk Universal Forwarder, Elastic Agent/Beats, Wazuh agent); reliable, can buffer and filter |
| Syslog | Standard protocol for network devices and Linux; UDP is lossy, prefer TCP/TLS |
| API / pull | Cloud and SaaS services expose logs through APIs or event streams |
| Agentless (WMI/WEF) | Windows Event Forwarding sends events to a central collector without a third-party agent |
Normalisation and Parsing
Raw logs differ wildly in format. Parsing extracts fields (timestamp, source IP, user, action, result) and normalisation maps them to a common schema, so one query such as user=alice works across firewall, VPN and application logs. Bad parsing is a silent failure: rules look at fields that are empty. Keep the original raw event alongside the parsed fields.
Retention and Integrity
- Retention is driven by investigation needs (attackers are often present for months) and by regulation or policy. Common practice is a short "hot" period for fast search and a longer, cheaper "cold/archive" tier. Do not invent a number: use your policy and compliance requirements, see Data Maintenance.
- Integrity matters because attackers delete logs. Forward logs off the host immediately, restrict who can modify the log store, use write-once or append-only storage, and hash archives. This also supports evidence use, see Digital Forensics.
Correlation, Use Cases and Detections
A single event is rarely conclusive; correlation links events across time and sources. A use case is a defined threat scenario plus the data, logic and response needed to detect it.
Example: brute force followed by success
IF count(failed_login WHERE user = X AND source_ip = Y) >= 10 within 5 minutes
AND a successful_login for user X from source_ip Y within 10 minutes after
THEN raise alert "Possible password guessing - account compromise"
severity = high if user is privileged, else medium
enrich with: geo-location of IP, is the account an admin, is IP on a threat feed
The same pattern covers many scenarios: impossible travel, new admin account followed by privilege use, large outbound transfer after a malware alert. Map use cases to frameworks such as MITRE ATT&CK to see coverage gaps.
Tuning
| Problem | Fix |
|---|---|
| Too many false positives | Raise thresholds, exclude known-good (vulnerability scanners, service accounts), add context conditions |
| Missed attacks | Add sources, loosen over-tight rules, test with simulated attacks |
| Rule rot | Review rules regularly; log changes under change management |
UEBA
User and Entity Behaviour Analytics builds a baseline of normal behaviour per user or host and flags deviations (login at unusual hours, unusual data volume). It complements fixed rules and catches unknown patterns but needs good data and tuning. See also Security Technologies.
Sigma: Vendor-Neutral Rules
Sigma is an open, YAML-based format for describing detection logic independent of any SIEM. A rule is written once (log source, field conditions, severity, ATT&CK tags) and converted by tooling into Splunk, Elastic, Sentinel and other query languages. This makes community detections shareable and portable. Concept only here; the exact syntax is in the Sigma project documentation.
SOAR
A SIEM tells you something happened; SOAR (Security Orchestration, Automation and Response) acts on it: enrich the IP, create a ticket, disable the account, isolate the host, page the on-call analyst. Automate repetitive, low-risk steps first and keep human approval for disruptive actions. See Security Operations Center.
Common Products
| Product | Notes |
|---|---|
| Splunk | Commercial; powerful search language (SPL), widely used |
| Elastic Security (ELK) | Elasticsearch, Logstash, Kibana; open core, flexible |
| Microsoft Sentinel | Cloud-native SIEM/SOAR on Azure |
| Wazuh | Open-source SIEM/XDR with host agents, file integrity monitoring and rules |
Pitfalls
- Time sync: if hosts disagree on the clock, events cannot be ordered or correlated. Use NTP everywhere and store timestamps in UTC.
- Volume and cost: licensing and storage often scale with ingestion. Collect what supports a detection or investigation; filter noise (for example debug logs) at the source.
- Blind spots: unmonitored systems, logging disabled by default, or logs not forwarded.
- Parsing drift: software updates change log formats and break rules unnoticed.
- No owner: rules without a person responsible for tuning decay into noise.
Key Points
- Logs without correlation are noise; a SIEM turns them into detections.
- Collect, parse, store safely, then write use cases tied to real threats.
- Tune continuously; measure false-positive rate.
- Accurate, synchronised time and protected log storage underpin everything.
Hands-on
- Introduction to SIEM - log sources, ingestion, SIEM features
- Logs Fundamentals - Windows event logs and web access logs
- Did you SIEM? (Splunk basics) - hunting through logs in Splunk
Related
- Security Operations Center
- Wireshark and tcpdump - network capture
- Incident Response Process