Log and event storage best practices

Best practices for log and event data storage depend upon the data compliance regulations you must meet, for example PCI and HIPAA. As well, you need to consider optimizing the use of your database. Storing too much data may affect database performance and size requirements.

Symptoms that you may be storing too much data for your database are the following: error messages that systems may be experiencing loss of database activity, an inability to import software updates, or just a general slow-down working in Deep Security.

1. Set system events storage to the compliance standard requirement.

2. Set up forwarding of system and module events to a syslog server or SIEM, see Forward Deep Security events to an external syslog or SIEM server . This will allow you to lower your retention time on the Storage tab, if necessary.

3. Set up thresholds in the log inspection module for event storage or event forwarding. Referred to as "severity pruning" in the Deep Security documentation, this allows you to send events to a syslog server (if enabled) or to store events based on the severity level of the log inspection rule. See Configure log inspection event forwarding and storage.

Deep Security Manager provides you with a default data retention setting of seven days for almost all events, with the exception of system events, which is set to "Never".

The table below shows defaults for storage. To view and update these settings, go to Administration > System Settings > Storage.

Data type	Data pruning default setting
Anti-malware events	7 days
Web reputation events	7 days
Firewall events	7 days
Intrusion prevention events	7 days
Integrity monitoring events	7 days
Log inspection events	7 days
Application control events	7 days
System events	13 weeks
Server logs	7 days
Counters	13 weeks
Software versions **	5 versions
Older rule updates **	10 rule updates

**Note: To delete Software Versions or Older Rule Updates, go to Administration > Updates > Software > Local or Administration > Updates > Security > Rules.

Troubleshooting

Increase the logging level and record more events for troubleshooting purposes. Exercise caution because increased logging can significantly increase the total size of your event logs.

1. Open the Computer or Policy editorYou can change these settings for a policy or for a specific computer. To change the settings for a policy, go to the Polices page and double-click the policy that you want to edit (or select the policy and click Details). To change the settings for a computer, go to the Computers page and double-click the computer that you want to edit (or select the computer and click Details). to configure.
2. Go to Settings > General > Logging Level.
3. Choose whether to inherit the logging override settings from the policy assigned to this computer (Inherited), to not override logging settings (Do Not Override), to log all triggered firewall rules (Full Firewall Event Logging), to log all triggered intrusion prevention rules (Full Intrusion Prevention Event Logging), or to log all triggered rules (Full Logging).
4. Click Save to apply the changes.

Limit log file sizes

You can set the maximum size of each individual log file and how many of the most recent files are kept. Event log files will be written to until they reach the maximum allowed size, at which point a new file will be created and written to until it reaches the maximum size and so on. Once the maximum number of files is reached, the oldest will be deleted before a new file is created. Event log entries usually average around 200 bytes in size and so a 4MB log file will hold about 20,000 log entries. How quickly your log files fill up depends on the number of rules in place.

1. Open the Computer or Policy editorYou can change these settings for a policy or for a specific computer. To change the settings for a policy, go to the Polices page and double-click the policy that you want to edit (or select the policy and click Details). To change the settings for a computer, go to the Computers page and double-click the computer that you want to edit (or select the computer and click Details). for the policy to configure.
2. Go to Settings > Advanced > Events.
3. Configure the following properties:
   • Maximum size of the event log files (on Agent/Appliance): Maximum size that the log file can reach before a new log file is created.
   • Number of event log files to retain (on Agent/Appliance): Maximum number of log files that will be kept. Once the maximum number of log files is reached, the oldest file will be deleted before a new one is created.
     Events are records of individual events. Counters are a record of the number of times individual events have occurred. Events are used to populate the Events pages. Counters are used to populate the Dashboard Widgets (number of firewall events over the last 7 days, etc.) and the reports. You might want to collect only counters if, for example, you are using syslog for event collection; events can potentially take up a lot of disk space and you may not want to store the data twice.
   • Do Not Record Events with Source IP of: This option is useful if you want Deep Security to not make record events for traffic from certain trusted computers.
     The following three settings let you fine tune event aggregation. To save disk space, Deep Security agents/appliances will take multiple occurrences of identical events and aggregate them into a single entry and append a "repeat count", a "first occurrence" timestamp, and a "last occurrence" timestamp. To aggregate event entries, Deep Security agents/appliances need to cache the entries in memory while they are being aggregated before writing them to disk.
   • Cache Size: Determines how many types of events to track at any given time. Setting a value of 10 means that 10 types of events will be tracked (with a repeat count, first occurrence timestamp, and last occurrence timestamp). When a new type of event occurs, the oldest of the 10 aggregated events will be flushed from the cache and written to disk.
   • Cache Lifetime: Determines how long to keep a record in the cache before flushing it to disk. If this value is 10 minutes and nothing else causes the record to be flushed, any record that reaches an age of 10 minutes gets flushed to disk.
   • Cache Stale time: Determines how long to keep a record whose repeat count has not been recently incremented. If Cache Lifetime is 10 minutes and Cache Staletime is two minutes, an event record which has gone two minutes without being incremented will be flushed and written to disk.
     Regardless of the above settings, the cache is flushed whenever events are sent to the Deep Security Manager.
4. Click Save .

Event logging tips

• On computers that are less important, modify the amount of logs collected. This can be done in the Events and Advanced Network Engine Options areas on the Computer or Policy editorYou can change these settings for a policy or for a specific computer. To change the settings for a policy, go to the Polices page and double-click the policy that you want to edit (or select the policy and click Details). To change the settings for a computer, go to the Computers page and double-click the computer that you want to edit (or select the computer and click Details). > Settings > Advanced tab.
• Consider reducing the event logging of firewall rule activity by disabling the event logging options in the firewall stateful configuration. (For example, if you disable UDP logging, it will eliminate unsolicited UDP log entries.)
• For intrusion prevention rules, the best practice is to log only dropped packets. If you log packet modifications, it may cause too many log entries.
• For intrusion prevention rules, only include packet data (an option in the intrusion prevention rule's Properties window) when you are interested in examining the behavior of a specific attack. Packet data increases log sizes, so it shouldn't be used for everything.