Configuring the disk space on the Log Forwarder

If the incoming logs are cached faster than they are sent to Insight, then a back pressure arises.

The following formula can be used to calculate the disk space on the Log Forwarder. The formula requires the estimated audit rate and time to sustain the audit rate, without logs being sent to Insight. Modify the values in this example as required. The default value of the disk space is 256 MB.

Disk Space in Mega bytes = (Audit Rate X Time in Seconds X 5.9 ) / 1024.

• Audit Rate = Number of policy audits generated per second
• Time in Seconds = Time duration for which the disk can sustain the audit rate without the logs being sent to Insight.

If the default or the configured value of the storage.total_limit_size setting is reached, then the Log Forwarder discards the oldest audits to create disk space for new audits.

Perform the following steps to configure the storage.total_limit_size setting in the out.conf file on the protector machine.

1. Log in and open a CLI on the protector machine.

2. Navigate to the config.d directory using the following command.

   cd /opt/protegrity/logforwarder/data/config.d
   

   Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/data/config.d path. Use the /opt/protegrity/fluent-bit/data/config.d path for protectors v9.1.0.0 and earlier.

3. Back up the existing out.conf file using the following command.

   cp out.conf out.conf_backup
   

4. Open the out.conf file using a text editor.

5. Update the value of storage.total_limit_size setting in the output blocks. The default value of the storage.total_limit_size is 256 MB. The following snippet shows the extract of the code.

   [OUTPUT]
       Name opensearch 
       Match logdata 
       Retry_Limit False
       Index pty_insight_audit
       Type  _doc
       Time_Key ingest_time_utc
       Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
       storage.total\_limit\_size 256M
   
   [OUTPUT]
       Name opensearch 
       Match flulog
       Retry_Limit 1
       Index pty_insight_audit
       Type  _doc
       Time_Key ingest_time_utc
       Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
       storage.total\_limit\_size 256M
   
   [OUTPUT]
       Name opensearch 
       Match errorlog
       Retry_Limit 1
       Index pty_insight_audit
       Type  _doc
       Time_Key ingest_time_utc
       Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
       storage.total\_limit\_size 256M
   

   Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/data/config.d path. Use the /opt/protegrity/fluent-bit/data/config.d path for protectors v9.1.0.0 and earlier.

6. Save and close the file.

7. Restart the Log Forwarder on the protector using the following commands.

   /opt/protegrity/logforwarder/bin/logforwarderctrl stop
   /opt/protegrity/logforwarder/bin/logforwarderctrl start
   

   Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/bin path. Use the /opt/protegrity/fluent-bit/bin path for protectors v9.1.0.0 and earlier.

8. If required, complete the configurations on the remaining protector machines.