Log and event storage best practices

Best practices for log and event data storage depend on the data compliance regulations you must meet, such as PCI and HIPAA. Also consider optimizing the use of your database. Storing too much data may affect database performance and size requirements.

If you're storing too much data in your database, these symptoms may occur:

  • Error messages that systems may be experiencing loss of database activity
  • Inability to import software updates
  • General slow-down in Deep Security

To avoid those symptoms:

  1. Store system events according to the compliance standard requirement.

  2. Forward system and security events to external storage. See Forward Deep Security events to a Syslog or SIEM server. Then you can reduce how long events are kept in the local database.

  3. Set thresholds in the log inspection module for event storage or event forwarding. Severity clipping allows you to send events to a Syslog server (if enabled) or to store events based on the severity level of the log inspection rule. See Configure log inspection event forwarding and storage.

Default local storage settings are in the table below. To change these settings, go to Administration > System Settings > Storage. To delete software versions or older rule updates, go to Administration > Updates > Software > Local or Administration > Updates > Security > Rules.

To reduce database disk space usage, forward events to an external Syslog server or SIEM and reduce the local event retention time. Only keep counters locally.
Data type settings Data pruning default setting
Automatically delete Anti-Malware Events older than 7 Days
Automatically delete Web Reputation Events older than: 7 Days
Automatically delete Firewall Events older than: 7 Days
Automatically delete Intrusion Prevention Events older than: 7 Days
Automatically delete Integrity Monitoring Events older than: 7 Days
Automatically delete Log Inspection Events older than: 7 Days
Automatically delete Application Control Events older than: 7 Days
Automatically delete System Events older than: 53 Weeks
Automatically delete Server Logs older than: 7 Days
Automatically delete Counters older than: 13 Weeks
Number of older software versions to keep per platform:* 5
Number of older Rule Updates to keep: 10

* If multi-tenancy is enabled, this setting will not be available.

If using a PostgreSQL database, old events might not be pruned immediately. PostgreSQL maintenance jobs periodically remove the old events' database partitions. Pruning will occur during the next scheduled job.

Events are records of individual events. They populate the Events pages.

Counters are the number of times individual events have occurred. They populate the dashboard widgets (number of firewall events over the last 7 days, etc.) and the reports.

Server log files are from Deep Security Manager's web server. They don't include event logs from agents installed on your network's web servers.

Troubleshooting

During troubleshooting, it may be useful to increase the logging level and record more detailed events.

Increased logging can significantly increase disk space usage. Reduce the logging level again when you have finished troubleshooting.
  1. Open the Computer or Policy editorClosedYou can change these settings for a policy or for a specific computer. To change the settings for a policy, go to the Polices page and double-click the policy that you want to edit (or select the policy and click Details). To change the settings for a computer, go to the Computers page and double-click the computer that you want to edit (or select the computer and click Details)..
  2. Go to Settings > General > Logging Level.
  3. Choose whether to inherit the logging override settings from the policy assigned to this computer (Inherited), to not override logging settings (Do Not Override), to log all triggered firewall rules (Full Firewall Event Logging), to log all triggered intrusion prevention rules (Full Intrusion Prevention Event Logging), or to log all triggered rules (Full Logging).
  4. Click Save .

Limit log file sizes

You can set the maximum size of each individual log file and how many of the most recent files are kept. Event log files will be written to until they reach the maximum allowed size, at which point a new file will be created and written to until it reaches the maximum size and so on. Once the maximum number of files is reached, the oldest will be deleted before a new file is created. Event log entries usually average around 200 bytes in size and so a 4 MB log file will hold about 20,000 log entries. How quickly your log files fill up depends on the number of rules in place.

  1. Open the Computer or Policy editorClosedYou can change these settings for a policy or for a specific computer. To change the settings for a policy, go to the Polices page and double-click the policy that you want to edit (or select the policy and click Details). To change the settings for a computer, go to the Computers page and double-click the computer that you want to edit (or select the computer and click Details). for the policy that you want to configure.
  2. Go to Settings > Advanced > Events.

  3. Configure these properties:

    • Maximum size of the event log files (on Agent/Appliance): Maximum size that the log file can reach before a new log file is created.
    • Number of event log files to retain (on Agent/Appliance): Maximum number of log files that will be kept. Once the maximum number of log files is reached, the oldest file will be deleted before a new one is created.
    • Do Not Record Events with Source IP of: This option is useful if you don't want Deep Security to make record events for traffic from certain trusted computers.
      The following three settings let you fine tune event aggregation. To save disk space, Deep Security Agents and Appliances will take multiple occurrences of identical events and aggregate them into a single entry and append a "repeat count", a "first occurrence" timestamp, and a "last occurrence" timestamp. To aggregate event entries, Deep Security Agents and Appliances need to cache the entries in memory and then write them to disk.
    • Cache Size: Determines how many types of events to track at any given time. Setting a value of 10 means that 10 types of events will be tracked (with a repeat count, first occurrence timestamp, and last occurrence timestamp). When a new type of event occurs, the oldest of the 10 aggregated events will be flushed from the cache and written to disk.
    • Cache Lifetime: Determines how long to keep a record in the cache before flushing it to disk. If this value is 10 minutes and nothing else causes the record to be flushed, any record that reaches an age of 10 minutes gets flushed to disk.
    • Cache Stale time: Determines how long to keep a record whose repeat count has not been recently incremented. If Cache Lifetime is 10 minutes and Cache Staletime is 2 minutes, an event record which has gone 2 minutes without being incremented will be flushed and written to disk.
      Regardless of the above settings, the cache is flushed whenever events are sent to the Deep Security Manager.
  4. Click Save.

Event logging tips