Configuring the disk space on the Log Forwarder

The Log Forwarder collects logs from the protectors and forwards them to the Audit Store. If the Audit Store is not reachable due to network issues, then the Log Forwarder caches the undelivered logs locally on the hard disk.

If the incoming logs are cached faster than they are sent to the Audit Store, then a back pressure arises.

The following formula can be used to calculate the disk space on the Log Forwarder. The formula requires the estimated audit rate and time to sustain the audit rate, without logs being sent to the Audit Store. Modify the values in this example as required. The default value of the disk space is 256 MB.

Disk Space in Mega bytes = (Audit Rate X Time in Seconds X 5.9 ) / 1024.

  • Audit Rate = Number of policy audits generated per second
  • Time in Seconds = Time duration for which the disk can sustain the audit rate without the logs being sent to the Audit Store.

If the default or the configured value of the storage.total_limit_size setting is reached, then the Log Forwarder discards the oldest audits to create disk space for new audits.

Perform the following steps to configure the storage.total_limit_size setting in the out.conf file on the protector machine.

  1. Log in and open a CLI on the protector machine.

  2. Navigate to the config.d directory using the following command.

    cd /opt/protegrity/logforwarder/data/config.d
    

    Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/data/config.d path. Use the /opt/protegrity/fluent-bit/data/config.d path for protectors v9.1.0.0 and earlier.

  3. Back up the existing out.conf file using the following command.

    cp out.conf out.conf_backup
    
  4. Open the out.conf file using a text editor.

  5. Update the value of storage.total_limit_size setting in the output blocks. The default value of the storage.total_limit_size is 256 MB. The following snippet shows the extract of the code.

    [OUTPUT]
        Name opensearch 
        Match logdata 
        Retry_Limit False
        Index pty_insight_audit
        Type  _doc
        Time_Key ingest_time_utc
        Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
        **storage.total\_limit\_size 256M**
    
    [OUTPUT]
        Name opensearch 
        Match flulog
        Retry_Limit 1
        Index pty_insight_audit
        Type  _doc
        Time_Key ingest_time_utc
        Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
        **storage.total\_limit\_size 256M**
    
    [OUTPUT]
        Name opensearch 
        Match errorlog
        Retry_Limit 1
        Index pty_insight_audit
        Type  _doc
        Time_Key ingest_time_utc
        Upstream /opt/protegrity/logforwarder/data/config.d/upstream.cfg
        **storage.total\_limit\_size 256M**
    

    Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/data/config.d path. Use the /opt/protegrity/fluent-bit/data/config.d path for protectors v9.1.0.0 and earlier.

  6. Save and close the file.

  7. Restart the Log Forwarder on the protector using the following commands.

    /opt/protegrity/logforwarder/bin/logforwarderctrl stop
    /opt/protegrity/logforwarder/bin/logforwarderctrl start
    

    Protectors v9.2.0.0 and later use the /opt/protegrity/logforwarder/bin path. Use the /opt/protegrity/fluent-bit/bin path for protectors v9.1.0.0 and earlier.

  8. If required, complete the configurations on the remaining protector machines.