Sending logs to an external security information and event management (SIEM)

To forward logs to the external SIEM:

1. Open the CLI Manager on the Primary ESA.

   1. Log in to the CLI Manager of the Primary ESA where the td-agent was configured in Setting Up td-agent to Receive Protector Logs.

   2. Navigate to Administration > OS Console.

   3. Enter the root password and select OK.

2. Navigate to the /products/uploads directory using the following command.

   cd /products/uploads
   

3. Obtain the required plugin file using one of the following commands based on the setup.

   • If the appliance has Internet access, then run the following command.

     wget https://rubygems.org/downloads/fluent-plugin-syslog-tls-2.0.0.gem
     

   • If the appliance does not have Internet access, then complete the following steps.

     1. Download the fluent-plugin-syslog-tls-2.0.0.gem set up file from a system that has Internet and copy it to the appliance in the /products/uploads directory.
     2. Ensure that the file downloaded has the execute permission.

4. Prepare the required plugin file using the following commands.

   1. Assign the required ownership permissions to the installer using the following command.

      chown td-agent *.gem
      

   2. Assign the required permissions to the software installation directory using the following command.

      chmod -R 755 /opt/td-agent/lib/ruby/gems/3.2.0/
      

   3. Assign ownership of the software installation directory to the required users using the following command.

      chown -R td-agent:plug /opt/td-agent/lib/ruby/gems/3.2.0/
      

5. Install the required plugin file using one of the following commands based on the setup.

   • If the appliance has Internet access, then run the following command.

     sudo -u td-agent /opt/td-agent/bin/fluent-gem install fluent-plugin-syslog-tls
     

   • If the appliance does not have Internet access, then run the following command.

     sudo -u td-agent /opt/td-agent/bin/fluent-gem install --local /products/uploads/fluent-plugin-syslog-tls-2.0.0.gem
     

6. Copy the required certificates on the ESA or the appliance.

   1. Log in to the ESA or the appliance and open the CLI Manager.

   2. Create a directory for the certificates using the following command.

      mkdir -p /opt/protegrity/td-agent/new_certs
      

   3. Update the ownership of the directory using the following command.

      chown -R td-agent:plug /opt/protegrity/td-agent/new_certs
      

   4. Log in to the remote SIEM system.

   5. Using a command prompt, navigate to the directory where the certificates are located. For example, cd /etc/pki/tls/certs.

   6. Connect to the ESA or appliance using a file transfer manager. For example, sftp root@ESA_IP.

   7. Copy the CA and client certificates to the /opt/Protegrity/td-agent/new_certs directory using the following command.

      put CA.pem /opt/protegrity/td-agent/new_certs
      put client.* /opt/protegrity/td-agent/new_certs
      

   8. Update the permissions of the certificates using the following command.

      chmod -r 744 /opt/protegrity/td-agent/new_certs/CA.pem
      chmod -r 744 /opt/protegrity/td-agent/new_certs/client.pem
      chmod -r 744 /opt/protegrity/td-agent/new_certs/client.key
      

7. Update the configuration files using the following steps.

   1. Navigate to the config.d directory using the following command.

      cd /opt/protegrity/td-agent/config.d
      

   2. Back up the existing output file using the following command.

      cp OUTPUT.conf OUTPUT.conf_backup
      

   3. Open the OUTPUT.conf file using a text editor.

   4. Update the following contents in the OUTPUT.conf file.

      • Update the match tag in the file to <match *.*.* logdata flulog>.

      • Add the following code in the match tag in the file:

          <store>
            @type relabel
            @label @syslogtls
          </store>
        

      The final OUTPUT.conf file with the updated content is shown here:

      <filter **>
        @type elasticsearch_genid
        # to avoid duplicate logs
        # https://github.com/uken/fluent-plugin-elasticsearch#generate-hash-id
        hash_id_key _id    # storing generated hash id key (default is _hash)
      </filter>
      
      <match *.*.* logdata flulog>
        @type copy
      
        <store>
            @type opensearch
            hosts <Hostname of the ESA>
            port 9200
            index_name pty_insight_audit
            type_name _doc
            pipeline logs_pipeline
            # adds new data - if the data already exists (based on its id), the op is skipped.
            # https://github.com/uken/fluent-plugin-elasticsearch#write_operation
            write_operation create
            # By default, all records inserted into Elasticsearch get a random _id. This option allows to use a field in the record as an identifier.
            # https://github.com/uken/fluent-plugin-elasticsearch#id_key
            id_key _id
            scheme https
            ssl_verify true
            ssl_version TLSv1_2
            ca_file /etc/ksa/certificates/plug/CA.pem
            client_cert /etc/ksa/certificates/plug/client.pem
            client_key /etc/ksa/certificates/plug/client.key
            request_timeout 300s # defaults to 5s https://github.com/uken/fluent-plugin-elasticsearch#request_timeout
            <buffer>
              @type file
              path /opt/protegrity/td-agent/es_buffer
              retry_forever true           # Set 'true' for infinite retry loops.
              flush_mode interval
              flush_interval 60s
              flush_thread_count 8  # parallelize outputs https://docs.fluentd.org/deployment/performance-tuning-single-process#use-flush_thread_count-parameter
              retry_type periodic
              retry_wait 10s
            </buffer>
        </store>
      
        <store>
            @type relabel
            @label @triggering_agent
        </store>
      
      <store>
          @type relabel
          @label @syslogtls
        </store>
      </match>
      

      Ensure that there are no trailing spaces or line breaks at the end of the file.

   5. Save and close the file.

   6. Create and open the OUTPUT_syslogTLS.conf file using a text editor.

   7. Perform the steps from one of the following solution as per the requirement.

      • Solution 1: Forward all logs to the external syslog server:

        Add the following contents to the OUTPUT_syslogTLS.conf file.

        <label @syslogtls>
          <filter *.*.* logdata flulog>
            @type record_transformer
            enable_ruby true
            <record>
            severity "${
                  case record['level']
                  when 'Error'
                    'err'
                  when 'ERROR'
                    'err'
                  else
                    'info'
                   end
                 }"
        
            #local0 -Protection
            #local1 -Application
            #local2 -System
            #local3 -Kernel
            #local4 -Policy
            #local5 -User Defined
            #local6 -User Defined
            #local7 -Others
            #local5 and local6 can be defined as per the requirement
        
            facility "${
                  case record['logtype']
                  when 'Protection'
                    'local0'
                  when 'Application'
                    'local1'
                  when 'System'
                     'local2'
                  when 'Kernel'
                     'local3'
                  when 'Policy'
                     'local4'
                  else
                     'local7'
                  end
                       }"
        
            #noHostName - can be changed by customer
            hostname ${record["origin"] ? (record["origin"]["hostname"] ? record["origin"]["hostname"] : "noHostName") : "noHostName" }
            </record>
          </filter>
        
          <match *.*.* logdata flulog>
            @type copy
            <store>
              @type syslog_tls
              host <IP_of_the_rsyslog_server_host>
              port 601
              client_cert /opt/protegrity/td-agent/new_certs/client.pem
              client_key /opt/protegrity/td-agent/new_certs/client.key
              ca_cert /opt/protegrity/td-agent/new_certs/CA.pem
              verify_cert_name true
              severity_key severity
              facility_key facility
              hostname_key hostname
              <format>
              @type json
              </format>
            </store>
          </match>
        </label>
        

        Ensure that there are no trailing spaces or line breaks at the end of the file.

      • Solution 2: Forward only the protection logs to the external syslog server:

        Add the following contents to the OUTPUT_syslogTLS.conf file.

        <label @syslogtls>
          <filter logdata>
            @type record_transformer
            enable_ruby true
            <record>
            severity "${
                  case record['level']
                  when 'Error'
                    'err'
                  when 'ERROR'
                    'err'
                  else
                    'info'
                   end
                 }"
        
            #local0 -Protection
            #local1 -Application
            #local2 -System
            #local3 -Kernel
            #local4 -Policy
            #local5 -User Defined
            #local6 -User Defined
            #local7 -Others
            #local5 and local6 can be defined as per the requirement
        
            facility "${
                  case record['logtype']
                  when 'Protection'
                    'local0'
                  when 'Application'
                    'local1'
                  when 'System'
                     'local2'
                  when 'Kernel'
                     'local3'
                  when 'Policy'
                     'local4'
                  else
                     'local7'
                  end
                       }"
        
            #noHostName - can be changed by customer
            hostname ${record["origin"] ? (record["origin"]["hostname"] ? record["origin"]["hostname"] : "noHostName") : "noHostName" }
            </record>
          </filter>
        
          <match logdata>
            @type copy
            <store>
              @type syslog_tls
              host <IP_of_the_rsyslog_server_host>
              port 601
              client_cert /opt/protegrity/td-agent/new_certs/client.pem
              client_key /opt/protegrity/td-agent/new_certs/client.key
              ca_cert /opt/protegrity/td-agent/new_certs/CA.pem
              verify_cert_name true
              severity_key severity
              facility_key facility
              hostname_key hostname
              <format>
              @type json
              </format>
            </store>
          </match>
        </label>
        

      Ensure that the <IP_of_the_rsyslog_server_host> is specified in the file.

      For more information about the formatting the output, navigate to https://docs.fluentd.org/configuration/format-section.

      The logs are formatted using the rfc 3164 format that is commonly used.

      For more information about the rfc format, navigate to https://datatracker.ietf.org/doc/html/rfc3164.

      Ensure that there are no trailing spaces or line breaks at the end of the file.

   8. Save and close the file.

   9. Update the permissions for the file using the following commands.

      chown td-agent:td-agent OUTPUT_syslogTLS.conf
      chmod 700 OUTPUT_syslogTLS.conf
      

8. Restart the td-agent service.

   1. Log in to the ESA Web UI.

   2. Navigate to System > Services > Misc > td-agent,

   3. Restart the td-agent service.

9. Check the status and restart the rsyslog server on the remote SIEM system using the following commands.

   systemctl status rsyslog
   systemctl restart rsyslog
   

The logs are now sent to Insight on the ESA and the external SIEM over TLS.

When logs are sent to Insight and the external SIEM over TLS and the domain name for the system is updated, then update the bind address for the system.

For more information about updating the bind address, refer here.