Log Forwarder Performance

Log Forwarder Performance

Log forwarder architecture is optimized to minimize the amount of connections and reduce the overall network bandwidth required to send audit logs to ESA. This is achieved with batching and aggregation taking place on two levels. The first level is in protector function instances, where audit logs from consecutive requests to an instance are batched and aggregated. The second level of batching and aggregation takes place in the log forwarder function before audit logs are forwarded to ESA. This section shows how to configure the deployment to accommodate different patterns of anticipated audit log stream. It also shows how to monitor deployment resources to detect problems before audit records are lost.

• Protector Function Terraform configuration:

  • audit_log_flush_interval: Determines the minimum amount of time audit logs are aggregated for before they are sent to Pub/Sub topic. Default value is 30 seconds. Changing flush interval may affect the level of aggregation and it will affect the backlog of audit logs buffered in the function. The protector function features multithreaded processing which means that multiple requests can be handled at the same time, which in turn can contribute to large backlog of audit logs waiting to be sent to Pub/Sub. The protector function is hosted on Cloud Run containerized environment where each instance of the function is shut down after specific amount of time when there is no more requests to be handled. If the flush interval is too long, the function might be shut down before all of the audit log backlog is send to Pub/Sub. This can be avoided by lowering the interval value.

                          

• Log Forwarder Function Terraform configuration

  • audit_log_flush_interval: Determines the minimum amount of time audit logs are aggregated for before they are sent to ESA audit log store. Default value is 10 seconds. Changing flush interval may affect the level of aggregation and it will affect the backlog of audit logs buffered in the function. The forwarder function features multithreaded processing which means that multiple requests can be handled at the same time, which in turn can contribute to large backlog of audit logs waiting to be sent to ESA. The forwarder function is hosted on Cloud Run containerized environment where each instance of the function is shut down after specific amount of time when there is no more requests to be handled. If the flush interval is too long, the function might be shut down before all of the audit log backlog is send to ESA. This can be avoided by lowering the interval value. On the other hand if the interval is too short, forwarder function might end up sending to many requests to ESA, which in some extreme cases may result in messages being sent to dead letter queue.
  • gen2_available_cpu: Increasing the Function CPU count allows setting higher concurrency, which in turn allows processing more messages from the Pub/Sub in parallel. The high CPU count will effectively lower the number of forwarder function instances which will lower number of connections to ESA.
  • gen2_container_concurrency: See bullet point above.
  • audit_log_dead_letter_topic: Dead-letter Pub/Sub topic name. This topic will be used by Log Forwarder in case ESA is temporarily unavailable. Messages from DLQ topic can be re-processed by another instance of Log Forwarder either manually or on schedule once ESA connectivity is restored.

                          

• Monitoring Log Forwarder Resources

  • Protector Function Logs: If protector function is unable to send logs to Pub/Sub, it will log the following message:

    Failed to send x/y audit logs to GCP Pub/Sub.
    

    See the description of ‘audit_log_flush_interval’ in the protector function configuration section above to learn about potential mitigation.

  • Pub/Sub DLQ Topic Metrics: Any positive value in count aggregator on ’topic/message_sizes’ metric indicates that not all audit logs are being delivered to ESA. Review whether connection to ESA is set up in Install Log Forwarder Function via Terraform Scripts

  • Log Forwarder Function Logs: If log forwarder function is unable to send logs to ESA, it will log the following message:

    [/jenkins/workspace/iaplambda_release_3.1/src/iap/logging/fluent-bit-external-sink.cpp:225] opensearch.0: Dropped records: x/y.
    

    See the description of ‘audit_log_flush_interval’ in the log forwarder configuration section above to learn about potential mitigation.

    Note

    When the error message above occurs, the dropped audit records will be sent to a dead-letter Pub/Sub topic for later manual or automated re-processing.