Web Application Firewall Logging and Troubleshooting
Envoy Logs
Request logs come out in the default-envoy-proxy logs in the envoy-gateway-system namespace. Look in:
- Individual
default-envoy-proxy-abcdefghij-12345pod logs - Use
sternto follow all of the envoy logs:stern . -n envoy-gateway-system
Successful Requests
HTTP Response Log
Purpose
This log is generated by Envoy and records the final outcome of the request.
Example
{
":authority": "your-appname.<business_unit>-nonlive.container-platform.service.justice.gov.uk",
"duration":4,
"method": "GET",
"response_code": 200,
"response_code_details": "via_upstream",
"upstream_cluster":"httproute/your-namespace/your-appname/rule/0",
"upstream_host": "1.2.3.14:8080"
}
What to Look For
HTTP Status Code
"response_code":200
Confirms that the client was sent a successful response code
Response Details
"response_code_details":"via_upstream"
Indicates that the response was generated by the upstream application/service and not by Envoy itself.
Upstream Host
"upstream_host": "1.2.3.14:8080"
The IP Address and Port of the upstream application/service
Upstream Cluster
"upstream_cluster":"httproute/your-namespace/your-appname/rule/0"
The Envoy cluster that traffic would be forwarded to after matching the route. This includes:
- The namespace the request was routed to
- The application the request was routed to
Response Time
Duration of the request in milliseconds
"duration": 4
Determining Gateway Information
The WAF and Envoy access logs do not directly contain the Gateway name or Gateway namespace. To identify which Gateway was responsible for processing the request, inspect the associated HTTPRoute resource.
Command
kubectl get httproute your-appname -n your-namespace -o yaml
Example output
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: ListenerSet
name: default-listenerset
namespace: envoy-gateway-system
sectionName: https
status:
parents:
- conditions:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: ListenerSet
name: default-listenerset
namespace: envoy-gateway-system
sectionName: https
In this example:
| Field | Value |
|---|---|
| Parent Kind | ListenerSet |
| Parent Name | default-listenerset |
| Parent Namespace | envoy-gateway-system |
Listener
The listener handling the request is attached to the HTTPS listener:
sectionName: https
Finding the Gateway
When using Envoy Gateway, the HTTPRoute may reference a ListenerSet rather than a Gateway directly.
To determine which Gateway owns the ListenerSet:
kubectl describe listenerset default-listenerset -n envoy-gateway-system
The resulting resource contains the references back to the Gateway from which the ListenerSet was generated.
Name: default-listenerset
Namespace: envoy-gateway-system
Spec:
Parent Ref:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: default
Namespace: envoy-gateway-system
Blocked Requests
You should expect to see three stages:
- Threat Detection - The WAF identifies a malicious request.
- Blocking Decision - The request exceeds the configured anomaly threshold and is denied.
- HTTP Response - The HTTP response code is returned to the user.
These log entries are tied together using a single ID, output as unique_id in logs 1 and 2, and as x-request-id in log 3.
Log 1: Threat Detection
Purpose
This log indicates that Coraza detected a request matching a known SQL injection pattern.
Example
{
"level": "error",
"msg": "[client \"198.51.100.10\"] Coraza: Warning. SQL Injection Attack Detected via libinjection",
"id": "942100",
"severity": "critical",
"uri": "https://your-appname.<business_unit>-nonlive.container-platform.service.justice.gov.uk/?id=1%27+OR+%271%27%3D%271"
}
What to Look For
Unique Request ID
[unique_id "52eb1cf3-0d45-4ef5-a1f6-31d4d1edc924"]
This is the unique ID for this request, also output as x-request-id in Log 3.
Rule ID
[id "942100"]
This is the OWASP CRS rule responsible for detecting SQL injection attempts using libinjection.
Severity
[severity "critical"]
Indicates the rule has classified the activity as a high-risk attack.
Matched Data
Example:
Matched Data: ... found within ARGS:id
This identifies:
- The parameter that contained the attack (
id) - The value that triggered the rule
Source IP
Example:
[client "198.51.100.10"]
This identifies the originating client.
Log 2: Blocking Decision
Purpose
This log records the blocking decision made by the OWASP CRS anomaly scoring system. If this log entry is not present this may indicate that the anomaly threshold was not exceeded.
Example
{
"level": "info",
"msg": "[client \"198.51.100.10\"] Coraza: Access denied (phase 2). Inbound Anomaly Score Exceeded (Total Score: 5)",
"id": "949110"
}
What to Look For
Unique Request ID
[unique_id "52eb1cf3-0d45-4ef5-a1f6-31d4d1edc924"]
This is the unique ID for this request, also output as x-request-id in Log 3.
Rule ID
[id "949110"]
This is the OWASP CRS blocking evaluation rule.
Access Denied
Coraza: Access denied
This confirms the request has been blocked.
Total Score
Total Score: 5
This is the accumulated anomaly score assigned during request inspection.
Log 3: HTTP 403 Response Returned
Purpose
This log is generated by Envoy and records the final outcome of the request.
Example
{
":authority": "your-appname.<business_unit>-nonlive.container-platform.service.justice.gov.uk",
"duration":4,
"method": "GET",
"response_code": 403,
"response_code_details": "waf_request_body_blocked",
"upstream_host": null
"x-request-id":"52eb1cf3-0d45-4ef5-a1f6-31d4d1edc924"
}
What to Look For
Unique Request ID
"x-request-id":"52eb1cf3-0d45-4ef5-a1f6-31d4d1edc924"
This is the unique ID for this request, also output as unique_id in Logs 1 and 2.
HTTP Status Code
"response_code":403
Confirms that the client was forbidden from accessing the application.
Response Details
"response_code_details":"waf_request_body_blocked"
Indicates that the block originated from the WAF.
Upstream Host
"upstream_host": null
If upstream_host is null, the request never reached the backend application.
Response Time
Duration of the request in milliseconds. Typically very low, for a blocked request.
"duration": 2
This indicates the request was rejected immediately by the WAF layer.
Expected Sequence
A successful WAF block should typically appear as:
1. Rule 942100
SQL Injection Attack Detected
2. Rule 949110
Inbound Anomaly Score Exceeded
Access Denied
3. Envoy Access Log
HTTP 403 Returned # A failure response code was sent to the user
upstream_host = null # The reuqest was never forwarded onto the application layer
Web Application Firewall Troubleshooting
TODO Note: Add log-search instructions here once the WAF logging integration is in place.
Investigate false positives
A false positive occurs when the WAF identifies legitimate application traffic as malicious.
If you believe a WAF rule is blocking legitimate traffic:
- configure your
HTTPRouteto useDetectionOnly - reproduce representative application traffic
- review the WAF events
- identify the rule ID causing the match
- determine why the legitimate request triggered the rule
- add an application-specific exclusion if required
- test the change
- change
SecRuleEngineback toOn
Avoid disabling the entire WAF when a specific rule exclusion can resolve the issue.