Troubleshooting

• 1: Policy and Key Audit logs
• 2: Known issues for the Audit Store
• 3: ESA Error Handling
• 3.1: ESA Upgrade Readiness Patch Error Messages and Resolutions
• 3.2: Common ESA Logs
• 3.3: Common ESA Errors
• 3.4: Understanding the Insight indexes
• 3.5: Understanding the index field values
• 3.6: Index entries
• 3.7: Log return codes
• 3.8: Policy Management Errors
• 3.9: Protectors security log codes
• 3.10: Additional log information
• 4: Known Issues for the td-agent
• 5: Known Issues for Protegrity Analytics
• 6: Known Issues for the Log Forwarder
• 7: Deprecations

1 - Policy and Key Audit logs

The policy audit logs generated for policy-related operations are sent to ESA. You can view them in Discover. Log in to the ESA, navigating to Audit Store > Dashboard > Open in new tab. Select Discover from the menu and select a time period such as Last 30 days.

Note:

• The policy and key audit log codes are similar to the previous version.
• The log descriptions in v10.2.0 are revised for policy and key audits. These changes may impact automated systems, alerts, and parsing logic in production environments. We recommend to review and update any dependent tools or queries.

event_status Field

In the ESA v10.2.0, a new field event_status has been introduced for all policy and key related audits. This field captures the outcome of each policy operation:

• Success: Indicates the action was completed successfully.

• Failure: Indicates the action was unsuccessful due to an error.

• Other: Indicates the event_status cannot be classified as neither a success nor a failure. The other value is usually used for logs providing information about an operation performed.

Example: Master Key Rotation – Success and Failure

Success Scenario

Imagine you are performing a routine rotation of the Master Key to maintain cryptographic hygiene. The following logs would indicate a successful operation:

These logs confirm that the key lifecycle was handled properly. For instance, a new key was created, the old one was deactivated, and the system remains secure and compliant.

Failure Scenario

Now, suppose the Master Key rotation fails due to a service outage, for example the kmgw service has stopped. You might see logs like:

These logs indicate that the rotation process was interrupted. No new key was created and the old key remains active. This could pose a security risk if not resolved promptly.

Other Scenario

Table: Policy and Key Audit logs

To view the policy audit logs:

1. Log in to the ESA.
2. Navigate to Audit Store > Dashboard.
3. From the menu, select Discover.
4. Select index pty_insight_analytics*policy_log_* from Index patterns and a time period such as Today.
   The list of policy audit logs appear.
   For more information about the Insight Indexes, refer to Understanding the Insight indexes.

2 - Known issues for the Audit Store

• Known Issue: The Audit Store node security remains uninitialized and the message Audit Store Security is not initialized. appears on the Audit Store Cluster Management page.

  Resolution:

  Run the following steps to resolve the issue.

  1. From the ESA Web UI, navigate to System > Services > Audit Store.
  2. Ensure that the Audit Store Repository service is running.
  3. Open the ESA CLI.
  4. Navigate to Tools.
  5. Run Apply Audit Store Security Configs.

• Known Issue: Logs sent to the Audit Store do not get saved and errors might be displayed.

  Issue:

  The Audit Store cannot receive and store logs when the disk space available on the ESA is low. In this case, errors or warnings similar to high disk watermark [90%] exceeded are displayed in the logs.

  Resolution:

  Perform one of the following steps to resolve the issue:

  • Delete old indices that are not required using ILM in Analytics.
  • Increase the disk space on all nodes.
  • Add new nodes to the cluster.

• Known Issue: The Audit Store Repository fails to start after updating the IP address, hostname, or domain name.

  Issue:

  After updating the IP address, hostname, or domain name, the Audit Store Repository service fails to start.

  Resolution:

  After updating the IP address, complete the steps provided in Updating the IP address of the ESA.

  After updating the hostname, complete the steps provided in Updating the hostname of the ESA.

  After updating the domain name, complete the steps provided in Updating the domain name of the ESA.

• Known Issue: The Upgrade cannot continue as the cluster health status is red. Check out the Troubleshooting Guide for info on how to proceed. error message appears.

  Issue: A cluster status in red color means that at least one primary shard and its replicas are not allocated to a node, that is, there are indices with the index health status in red color in the Audit Store cluster.

  Workaround

  Complete the following steps on a node where the Audit Store is available to resolve the cluster health with the red status.

  1. From the Web UI of the ESA, navigate to System > Services > Audit Store.

  2. Ensure that the Audit Store Repository service is running.

  3. Log in to the CLI Manager of the Appliance.

  4. Navigate to Administration > OS Console.

  5. Identify the indices with the health status as red using the following command.

     wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key -O - https://<ESA_HOSTNAME_IP>:9201/_cat/indices | grep red
     

     Ensure that you update the variables before running the command. An example of the command is provided here.

     wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key -O - https://protegrity-esa123.protegrity.com:9201/_cat/indices | grep red
     

     A list of indices containing the health status as red appears as shown in the following example.

     red    open   pty_insight_audit_vx.x-xxxx.xx.xx-000014             dxmEWom8RheqOhnaFeM3sw   1   1
     

     In the example, pty_insight_audit_vx.x-xxxx.xx.xx-000014 is the index having a red index health status where the index’s primary shard and replicas are not available or allocated to any node in the cluster.

  6. Identify the reason for unassigned shards using the following command.

     wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key -O - https://<ESA_HOSTNAME_IP>:9201/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
     

     Ensure that you update the variables before running the command. An example of the command is provided here.

     wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key -O - https://protegrity-esa123.protegrity.com:9201/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
     

     The reasons for the shards being unassigned appear. This example shows one of the reasons for the unassigned shard.

         `pty_insight_audit_vx.x-xxxx.xx.xx-000014             0 p UNASSIGNED NODE_LEFT`
     
         `pty_insight_audit_vx.x-xxxx.xx.xx-000014             0 r UNASSIGNED NODE_LEFT`
     

     In the example, the 0th p and r shards of the pty_insight_audit_vx.x-xxxx.xx.xx-000014 index are unassigned due to the NODE_LEFT reason, that is, because the node left the Audit Store cluster. The p indicates a primary shard and the r indicates a replica shard.

  7. Retrieve the details for the shard being unassigned using the following command.

     wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key --header='Content-Type:application/json' --method=GET --body-data='{ "index": "<Index_name>", "shard": <Shard_ID>, "primary":<true or false> }' -O - https://<ESA_HOSTNAME_IP>:9201/_cluster/allocation/explain?pretty
     

     Ensure that you update the variables before running the command. An example of the command with the index name as pty_insight_audit_vx.x-xxxx.xx.xx-000014, shard ID as 0, and primary shard as true is provided here.

     wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key --header='Content-Type:application/json' --method=GET --body-data='{ "index": "pty_insight_audit_vx.x-xxxx.xx.xx-000014", "shard": 0, "primary": true }' -O - https://protegrity-esa123.protegrity.com:9201/_cluster/allocation/explain?pretty
     

     The details of the unassigned shard appears. This example shows one of the reasons for the unassigned shard.

     {
             "index": "pty_insight_audit_vx.x-xxxx.xx.xx-000014",
             "shard": 0,
             "primary": true,
             "current_state": "unassigned",
             "unassigned_info": {
                 "reason": "NODE_LEFT",
                 "at": "2022-03-28T05:05:25.631Z",
                 "details": "node_left [gJ38FzlDSEmTAPcP0yw57w]",
                 "last_allocation_status": "no_valid_shard_copy"
             },
             "can_allocate": "no_valid_shard_copy",
             **"allocate\_explanation": "cannot allocate because all found copies of the shard are either stale or corrupt",**
             "node_allocation_decisions": [
                 {
                     "node_id": "3KXS1w9HTOeMH1KbDShGIQ",
                     "node_name": "ESA1",
                     "transport_address": "xx.xx.xx.xx:9300",
                     "node_attributes": {
                         "shard_indexing_pressure_enabled": "true"
                     },
                     "node_decision": "no",
                     "store": {
                         "in_sync": false,
                         "allocation_id": "HraOWSZlT3KNXxOHDhZL5Q"
                     }
                 }
             ]
         }
     

     In this example, the shard is not allocated because all found copies of the shard are either stale or corrupt. There are no valid shard copies that can be allocated for this index. This is a data loss scenario, where the data is unavailable because the node or nodes that had the data have disconnected from the cluster. In such a scenario, if the disconnected nodes are brought back in the cluster, then the cluster can reconstruct itself and become healthy again. If bringing the nodes back is not possible, then deleting indices with the red index health status is the only way to fix a red cluster health status.

  8. Complete one of the following two steps to stabilize the cluster.

     • Troubleshoot the cluster:

       1. Verify that the Audit Store services are running. Restart any Audit Store service that is in the stopped state.
       2. Ensure that the disconnected nodes are running.
       3. Try to add any disconnected nodes back to the cluster.
       4. Restart the system or restore the system from a backup.

     • Delete the index:

       Delete the indices with the index health status as red using the following command. Execute the command from any one node of Audit Store node which is running.

       wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key --header='Content-Type:application/json' --method=DELETE -O - https://<ESA_HOSTNAME_IP>:9201/<Index_name>
       

       Ensure that you update the variables before running the command. An example of the command to delete the pty_insight_audit_vx.x-xxxx.xx.xx-000014 index is provided here.

       wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key --header='Content-Type:application/json' --method=DELETE -O - https://protegrity-esa123.protegrity.com:9201/pty_insight_audit_vx.x-xxxx.xx.xx-000014
       

       CAUTION:

       This command deletes the index and must be used carefully.

• Known Issue: The Authentication failure for the user during JWT token generation while downloading ssh key from node error appears while performing any Audit Store cluster-related operation.

  Issue: The Can create JWT token permission is required for a role to perform Audit Store cluster-related operations. The error appears if the permission is not assigned to the user.

  Workaround

  Use a user with the appropriate permissions for performing Audit Store cluster-related operations. Alternatively, verify and add the Can create JWT token permission. To verify and add the Can Create JWT Token permission, from the ESA Web UI, navigate to Settings > Users > Roles.

• Known Issue: The pty_insight_analytics_miscellaneous_* index on the Discover page loads with errors.

  Issue:

  The index fails to load correctly because the index_time_utc field must be defined in the index pattern.

  Resolution:

  Complete the following steps to update the index pattern.

  1. Log in to the ESA Web UI.
  2. Navigate to Audit Store > Dashboard.
  3. From menu on the Audit Store Dashboards page, navigate to Management > Dashboard Management > Index patterns.
  4. Click pty_insight_analytics_miscellaneous_* and click the Remove index pattern icon.
  5. Click Delete to confirm.
  6. Click Create index pattern.
  7. Specify pty_insight_analytics_miscellaneous_* for the index pattern name.
  8. Click Next step.
  9. Select index_time_utc for the Time field.
  10. Click Create index pattern.
  11. Navigate to the Discover page to view the required logs.

3 - ESA Error Handling

ESA appliance collects all logs that come from different Protegrity Servers. The following section explains the logs that you may find and the errors that you may encounter on the ESA.

3.1 - ESA Upgrade Readiness Patch Error Messages and Resolutions

This article provides detailed information on error messages and their resolutions related to the ESA Upgrade Readiness Patch for version 10.2.0. It covers various issues such as insufficient disk space, version compatibility, unicast hosts file errors, analytics initialization, and service status checks. Each error message is accompanied by high-level steps for resolving the issue, ensuring a smooth upgrade process to ESA version 10.2.0.

Use the information provided in this section for the detailed steps to resolve the issues shown by the readiness patch.

Issue: Insufficient Disk Space

ERROR: The required disk space is insufficient for the following partitions: 

/ - available: <available_space>%, required: 40% 

/opt - available: {opt_available_gb}GB, required: 10GB

/var/log/ - available: <available_space>%, required: 20%

Resolution

The space used in the OS(/) partition should not be more than 60%. If the space used is more than 60%, then you must clean up the OS(/) partition before proceeding with the patch installation process. For more information about cleaning up the OS(/) partition, refer here.

For increasing the /opt partition, perform the steps using Increasing the Appliance Disk Size.

For increasing the size of /var/log directory, perform the steps using Extending the Logs Volume.

Issue: Expected ESA version

ERROR: Expecting appliance version 10.1.x/10.0.x/9.2.0.x/9.1.0.x. Current ESA version {appliance_version}

Resolution

This patch must be applied on any of the following versions:

• 9.1.0.x
• 9.2.0.x
• 10.0.x
• 10.1.x

Issue: Missing IP address in Unicast hosts file

ERROR: Unicast hosts file does not contain the IP addresses of the nodes and is empty.  Run the Audit Store Management Unicast Hosts scheduled task.

Applicable Version: 9.1.0.x

Resolution

To run the Audit Store Management Unicast Hosts scheduled task:

1. Log in to the ESA WebUI.

2. Navigate to System > Task Scheduler.

3. Select the Audit Store Management Update Unicast Hosts task.

4. Click Run now!.

   In some ESA versions, the task is named as Audit Store Management - Cluster Config - Sync.

Issue: Unicast hosts file verification failed

ERROR: Unicast IP verification failed. IPs in cluster: {node_ips} IPs in unicast hosts file: {ips}. Verify the entries in the {UNICAST_HOSTS_FILE} file.

ERROR: Unicast hostname verification failed. Retrieved hostname: {hostname}. Verify the entries in the {UNICAST_HOSTS_FILE} file.

ERROR: Unicast hosts file is empty. Please add the IP and hostname entries for all the nodes in the Audit Store cluster in the Unicast hosts file /opt/protegrity/auditstore/config/unicast_hosts.txt

ERROR: Unicast hosts file not found. {UNICAST_HOSTS_FILE}. Create the file with the relevant node IPs. 

Applicable Version: 9.2.0.x, 10.x

Resolution

Update the unicast hosts file:

1. Obtain the IP and hostname entries of all the ESA nodes in the Audit Store cluster.
2. Log in to the CLI Manager on the ESA.
3. Navigate to Administration > OS Console.
4. Enter the root password.
5. Navigate to /opt/protegrity/auditstore/config.
6. Open the unicast_hosts.txt file using a text editor.
7. Add and verify the node entries in the file.
8. Save and close the file.

Issue: Analytics is not initialized

ERROR: Analytics is not initialized. Please initialize analytics.

Resolution:

Initialize Analytics.

1. Log in to the ESA Web UI.
2. Verify that the Audit Store services are running by navigating to System > Services > Audit Store.
3. Navigate to Analytics. For ESA v10.x, navigate to Audit Store > Initialize Analytics.
4. Click Initialize Analytics.

Issue: Insight services not running

ERROR: Analytics service is not running. Please start the service.

ERROR: Audit Store Dashboards service is not running. Please start the service.

ERROR: Audit Store Management service is not running. Please start the service.

ERROR: Audit Store Repository service is not running. Please start the service.

ERROR: Audit Store connection failed, cluster verification cannot proceed.

ERROR: Audit Store connection failed, unicast file verification cannot proceed.

ERROR: Audit Store connection failed. Audit Store Repository service is not running. Please start it to proceed.

ERROR: Audit Store connection failed. Audit Store Repository took too long to respond (00:00:30). <exception-Connection Error>

ERROR: td-agent service is not running. Please start the service.

Resolution:

Enable the Run the following steps to resolve the issue.

1. From the ESA Web UI, navigate to System > Services > Audit Store.
2. Ensure that the Audit Store Repository service is running.
3. Open the ESA CLI.
4. Navigate to Tools.
5. Run Apply Audit Store Security Configs.

Issue: Audit Store connection failed due to certificates

ERROR: Audit Store connection failed. Certificated expired. Please upload valid certificate.

Resolution:

Upload valid certificates using the steps provided in Uploading Certificates and Insight Certificates.

Issue: The master or repository keys are expired

Validate that master/repository keys are not expired with respect to OUP

"ERROR: Call to check key state failed. Please verify that all services are running" (when hubcontroller is down)

"ERROR: Master or repository key is expired, please rotate the key(s)"

Resolution:

• If the security keys, such as, master key or repository key have expired or are due to expire within 30 days, then the upgrade fails. Hence, the keys must be rotated before performing the upgrade. Additionally, ensure that the keys are active and in running state.

  For more information about rotating keys, refer to Working with Keys.

• If you are using an HSM, ensure that the HSM is accessible and running.

  For more information about HSM, refer to the corresponding HSM vendor document.

If the prerequisites are not met, the ESA upgrade process fails. In such a case, it is required to restore the ESA to its previous stable version.

Issue: The active Key Store health check failed

Validate that Key Store is healthy

"ERROR: Call to check HSM health failed. Please verify that all services are running" (when kmgw/ soft hsm service is down)

"ERROR: The health check for active HSM GW instance is failing, please make sure HSM is accessible"

Resolution:

Test the Key Store connection to validate where the Key Store can be accessed using the steps from Key Store Management.

Issue: The license is invalid or expired

Validate that license is valid (expired/invalid)

"ERROR: License is not valid"

"ERROR: Failed to check license. Please verify that Policy Management services are running" (when hubcontroller is down)

Resolution:

Before upgrading the ESA, ensure that the license is not expired or invalid.

An expired or invalid license blocks policy services on the ESA and Devops API’s. A new or existing protector will not receive any policies until a valid license is applied.

For more information about the license, refer Protegrity Data Security Platform Licensing.

Warning: List of unsupported protectors connected with the ESA

Check for unsupported protectors
Installation pre-checks have detected protector versions that under specific circumstances pose a potential risk

"ERROR: Failed to list unsupported protectors" (When hubcontroller is down)

Resolution:

The ESA v10.2.0 only supports protectors having the PEP server version 1.2.2+42 and later.

During the upgrade, the process checks if any unsupported protector is registered in the system.
If the Installation pre-check script detects any protector that is not supported by the ESA v10.2.0, then the Warning dialog box appears.

You can either terminate the Upgrade process or you can continue with unsupported protectors.

It is recommended to upgrade the protectors to a supported version before proceeding with the Upgrade process.

Perform the following steps to identify the PEP server version of the protector:

1. Log in to the ESA.
2. Navigate to Policy Management > Nodes.
3. View the Version field for all the protectors.

If the protector version is unsupported, perform one of the following actions:

• Uninstall the unsupported protector and delete the node from the list of registered nodes.
• Upgrade the unsupported protector to a supported version. For most of the protectors, this process involves uninstalling and installing the protectors. However, some protectors, such as Data Security Gateway (DSG) and Big Data Protector (BDP), might support the upgrade process. The new installed protector updates the registered node entry.

Issue: DTP/DTP2 data elements are present

Validate that no DTP/DTP2 data elements are present

"ERROR: Found unsupported DTP data elements"

"ERROR: Failed to establish a new connection to the HubController service"

Resolution:

If the DTP/DTP2 is present in the algorithm property of a data element while upgrading from the ESA v9.1.0.x or later versions to ESA v10.2.0, then the upgrade script fails. The following error message appears:

ERROR: Found unsupported DTP data elements

Perform the following actions:

1. Reprotect data with a new data element that is supported.
2. Remove the data elements that contain DTP/DTP2 algorithm.

This prevents the data loss that occurs during the upgrade. The DTP/DTP2 data elements are now unsupported.

For more information about the data elements to be used, contact Protegrity Support.

Issue: FPE data elements are present

Validate that no FPE data elements with left/right > 0 are present

"ERROR: FPE Data Element(s) with characters in clear ('From Left' / 'From Right') are no longer supported on the target version. Please consult the documentation or Protegrity staff for guidance. Data Element(s) affected: "

"ERROR: Failed to list FPE Data Elements"

Resolution:

If the format-preserving encryption (FPE) data elements with Left and Right settings are present while upgrading from the ESA v9.1.0.x or later versions to ESA v10.2.0, then the upgrade script fails. The following error message appears:

ERROR: FPE Data Element(s) with characters in clear ('From Left' / 'From Right') are no longer supported on the target version.  
   Please consult the documentation or Protegrity staff for guidance. 
   Data Element(s) affected: <List of affected data elements>

Perform the following actions:

1. Reprotect data with a new FPE data element that does not have Left and Right settings or a new data element that is supported.
2. Remove the data elements that contain Left and Right settings.

This prevents the data loss that occurs during the upgrade. The FPE data elements with Left and Right settings are now unsupported.

For more information about the data elements to be used, contact Protegrity Support.

3.2 - Common ESA Logs

3.3 - Common ESA Errors

Table: ESA Common Errors

Couldn't find device with uuid <ID> 
Cannot change VG <volume group> while PVs are missing

/etc/init.d/appliance-queues server restart

/etc/opt/scripts/after-backup.sh

Failed to fetch data from External Groups. Try refreshing the page

Unauthorised User

Error! Failed to leave cluster: LeaveCluster <IP address>: The node cannot leave the cluster as it has existing associated tasks.

Login failure - 'failed to authenticate user: Insufficient privileges'

SNMP Service started failed

SNMP Service started failed

                           Cloud gateway logrorate failed with error: error renaming temp state file /var/lib/logrotate/status.tmp

WARNING: Failed to find a usable hardware address from the network 
interfaces; using random bytes: 1b:1f:ff:64:9b:b6:ea:ce

Anti-Virus database update has failed.

Failed to apply all the changes. 
Please accept all the changes from the Web UI

Call to check key state failed. Please verify that all services are running.

3.4 - Understanding the Insight indexes

All the Appliances and Protectors send logs to Insight. The logs from the Audit Store are displayed on the Discover screen of the Audit Store Dashboards. Here, you can view the different fields logged. In addition to viewing the data, these logs serve as input for Insight to analyze the health of the system and to monitor the system for providing security. These logs are stored in the Audit index with the name, such as, pty_insight_analytics_audit_9.2-*. To refer to old and new audit indexes, the alias pty_insight_*audit_* is used.

The /var/log/asdashboards.log file is empty. The init.d logs for the Audit Store Dashboards are available in /var/log/syslog. The container-related logs are available in /var/log/docker/auditstore_dashboards.log.

You can view the Discover screen by logging into the ESA and navigating to Audit Store > Dashboard > Open in new tab, select Discover from the menu, and select a time period such as Last 30 days. The Discover screen appears.

The following table lists the various indexes and information about the data contained in the index. You can view the index list by logging into the ESA, and navigating to Audit Store > Cluster Management > Overview > Indices. Indexes can be created or deleted. However, deleting an index will lead to a permanent loss of data in the index. If the index was not backed up earlier, then the logs from the index deleted cannot be recreated or retrieved.

3.5 - Understanding the index field values

Common Logging Information

These logging fields are common with the different log types generated by Protegrity products.

Note: These common fields are used across all log types.

Additional_Info

These descriptions are used for all types of logs.

Process

This section describes the properties of the process that created the log. For example, the protector or the rputils.

Origin

This section describes the origin of the log, that is, from where the log came from and when it was generated.

Protector

This section describes the Protector that generated the log. For example, the vendor and the version of the Protector.

Note: For more information about the Protector vendor, family, and version, refer here.

Protection

This section describes the protection that was done, what was done, the result of the operation, where it was done, and so on.

Client

This section describes from where the log came from.

Policy

This section describes the information about the policy.

Metering

This section describes the metering log information.

Note: These fields are applicable for Protectors up to v7.2.1. If you upgraded your ESA from v7.2.1 to v9.1.0.0 and migrated the metering audits, then these fields contain data.

Metering is not supported for Protectors v8.0.0.0 and above and these are fields will be blank.

Signature

This section handles the signing of the log. The key that was used to sign the log and the actual checksum that was generated.

Verification

This section describes the log information generated for a failed signature verification job.

3.6 - Index entries

Audit index

The log types of protection, metering, audit, and security are stored in the audit index. These log are generated during security operations. The logs generated by protectors are stored in the audit index with the name as shown in the following table for the respective version.

The following parameters are configured for the index rollover in v10.2.0:

• Index age: 30 days
• Document count: 200,000,000
• Index size: 5 GB

Protection logs

These logs are generated by protectors during protecting, unprotecting, and reprotecting data operations. These logs are generated by protectors, such as, DSG.

Use the following query in Discover to view these logs.

logtype:protection

A sample log is shown here:

 {
    "process": {
      "thread_id": "1227749696",
      "module": "coreprovider",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "4190",
      "user": "user4",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "level": "SUCCESS",
    "signature": {
      "key_id": "11a8b7d9-1621-4711-ace7-7d71e8adaf7c",
      "checksum": "43B6A4684810383C9EC1C01FF2C5CED570863A7DE609AE5A78C729A2EF7AB93A"
    },
    "origin": {
      "time_utc": "2024-09-02T13:55:17.000Z",
      "hostname": "hostname1234",
      "ip": "10.39.3.156"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "pcc_version": "3.6.0.1",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1"
    },
    "protection": {
      "dataelement": "TE_A_S13_L1R2_Y",
      "datastore": "DataStore",
      "audit_code": 6,
      "operation": "Protect",
      "policy_user": "user1"
    },
    "index_node": "protegrity-esa399/10.39.1.23",
    "tiebreaker": 210,
    "logtype": "Protection",
    "additional_info": {
      "description": "Data protect operation was successful"
    },
    "index_time_utc": "2024-09-02T13:55:24.766355224Z",
    "ingest_time_utc": "2024-09-02T13:55:17.678Z",
    "client": {},
    "correlationid": "cm0f1jlq700gbzb19cq65miqt"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-02T13:55:17.000Z"
    ],
    "index_time_utc": [
      "2024-09-02T13:55:24.766Z"
    ],
    "ingest_time_utc": [
      "2024-09-02T13:55:17.678Z"
    ]
  },
  "sort": [
    1725285317000
  ]

The above example contains the following information:

• additional_info
• origin
• protector
• protection
• process
• client
• protector
• signature

For more information about the various fields, refer here.

Metering logs

These logs are generated by protectors of prior to 8.0.0.0. These logs are not generated by latest protectors.

Use the following query in Discover to view these logs.

logtype:metering

For more information about the various fields, refer here.

Audit logs

These logs are generated when the rule set of the DSG protector gets updated.

Use the following query in Discover to view these logs.

logtype:audit

A sample log is shown here:

 {
   "additional_info.description": "User admin modified default_80 tunnel successfully ",
   "additional_info.title": "Gateway : Tunnels : Tunnel 'default_80' Modified",
   "client.ip": "192.168.2.20",
   "cnt": 1,
   "index_node": "protegrity-esa746/192.168.1.10",
   "index_time_utc": "2024-01-24T13:30:17.171646Z",
   "ingest_time_utc": "2024-01-24T13:29:35.000000000Z",
   "level": "Normal",
   "logtype": "Audit",
   "origin.hostname": "protegrity-cg406",
   "origin.ip": "192.168.2.20",
   "origin.time_utc": "2024-01-24T13:29:35.000Z",
   "process.name": "CGP",
   "process.user": "admin",
   "tiebreaker": 2260067,
   "_id": "ZTdhNzFmMTUtMWZlOC00MmY4LWJmYTItMjcwZjMwMmY4OGZh",
   "_index": "pty_insight_audit_v9.1-2024.01.23-000006"
 }

This example includes data from each of the following groups defined in the index:

• additional_info
• client
• origin
• process

For more information about the various fields, refer here.

Security logs

These logs are generated by security events of the system.

Use the following query in Discover to view these logs.

logtype:security

For more information about the various fields, refer here.

Troubleshooting index

The log types of application, kernel, system, and verification logs are stored in the troubleshooting index. These logs helps you understand the working of the system. The logs stored in this index are essential when the system is down or has issues. This is the pty_insight_analytics_troubleshooting index. The index pattern for viewing these logs in Discover is pty_insight_*troubleshooting_*.

The following parameters are configured for the index rollover:

• Index age: 30 days
• Document count: 200,000,000
• Index size: 5 GB

Application Logs

These logs are generated by Protegrity servers and Protegrity applications.

Use the following query in Discover to view these logs.

logtype:application

A sample log is shown here:

{
    "process": {
      "name": "hubcontroller"
    },
    "level": "INFO",
    "origin": {
      "time_utc": "2024-09-03T10:02:34.597000000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16916,
    "logtype": "Application",
    "additional_info": {
      "description": "GET /dps/v1/deployment/datastores | 304 | 127.0.0.1 | Protegrity Client | 8ms | "
    },
    "index_time_utc": "2024-09-03T10:02:37.314521452Z",
    "ingest_time_utc": "2024-09-03T10:02:36.262628342Z",
    "correlationid": "cm0m9gjq500ig1h03zwdv6kok"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:02:34.597Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:02:37.314Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:02:36.262Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Application@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357754597
  ]

The above example contains the following information:

• additional_info
• origin
• process

For more information about the various fields, refer here.

Kernel logs

These logs are generated by the kernel and help you analyze the working of the internal system. Some of the modules that generate these logs are CRED_DISP, KERNEL, USER_CMD, and so on.

Use the following query in Discover to view these logs.

logtype:Kernel

For more information and description about the components that can generate kernel logs, refer here.

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "CRED_DISP"
    },
    "origin": {
      "time_utc": "2024-09-03T10:02:55.059999942Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": "1",
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16964,
    "logtype": "Kernel",
    "additional_info": {
      "module": "pid=38236",
      "description": "auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:setcred grantors=pam_rootok acct=\"rabbitmq\" exe=\"/usr/sbin/runuser\" hostname=? addr=? terminal=? res=success'\u001dUID=\"root\" AUID=\"unset\"",
      "procedure": "uid=0"
    },
    "index_time_utc": "2024-09-03T10:02:59.315734771Z",
    "ingest_time_utc": "2024-09-03T10:02:55.062254541Z"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:02:55.059Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:02:59.315Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:02:55.062Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Kernel@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357775059
  ]

This example includes data from each of the following groups defined in the index:

• additional_info
• origin
• process

For more information about the various fields, refer here.

System logs

These logs are generated by the operating system and help you analyze and troubleshoot the system when errors are found.

Use the following query in Discover to view these logs.

logtype:System

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

 {
    "process": {
      "name": "ESAPAP",
      "version": "10.0.0+2412",
      "user": "admin"
    },
    "level": "Low",
    "origin": {
      "time_utc": "2024-09-03T10:00:34.000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": "1",
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16860,
    "logtype": "System",
    "additional_info": {
      "description": "License is due to expire in 30 days. The validity of license has been acknowledged by the user. (web-user 'admin' , IP: '10.87.2.32')",
      "title": "Appliance Info : License is due to expire in 30 days. The validity of license has been acknowledged by the user. (web-user 'admin' , IP: '10.87.2.32')"
    },
    "index_time_utc": "2024-09-03T10:01:10.113708469Z",
    "client": {
      "ip": "10.37.4.12"
    },
    "ingest_time_utc": "2024-09-03T10:00:34.000000000Z"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:00:34.000Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:01:10.113Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:00:34.000Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@System@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357634000
  ]

This example includes data from each of the following groups defined in the index:

• additional_info
• origin
• process

For more information about the various fields, refer here.

Verification logs

These log are generated by Insight on the ESA when a signature verification fails.

Use the following query in Discover to view these logs.

logtype:Verification

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "insight.pyc",
      "id": 45277
    },
    "level": "Info",
    "origin": {
      "time_utc": "2024-09-03T10:14:03.120342Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 17774,
    "logtype": "Verification",
    "additional_info": {
      "module": ".signature.job_executor",
      "description": "",
      "procedure": "__log_failure"
    },
    "index_time_utc": "2024-09-03T10:14:03.128435514Z",
    "ingest_time_utc": "2024-09-03T10:14:03.120376Z",
    "verification": {
      "reason": "SV_VERIFY_RESPONSES.INVALID_CHECKSUM",
      "job_name": "System Job",
      "job_id": "9Vq1opEBYpV14mHXU9hW",
      "index_name": "pty_insight_analytics_audits_10.0-2024.08.30-000001",
      "doc_id": "JI5bt5EBMqY4Eog-YY7C"
    }
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:14:03.120Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:14:03.128Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:14:03.120Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Verification@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725358443120
  ]

This example includes data from each of the following groups defined in the index:

• additional_info
• process
• origin
• verification

For more information about the various fields, refer here.

Policy log index

The log type of policy is stored in the policy log index. They include logs for the policy-related operations, such as, when the policy is updated. The index pattern for viewing these logs in Discover is pty_insight_*policy_log_*.

The following parameters are configured for the policy log index:

• Index age: 30 days
• Document count: 200,000,000
• Index size: 5 GB

Use the following query in Discover to view these logs.

logtype:policyLog

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "hubcontroller",
      "user": "service_admin",
      "version": "1.8.0+6.g5e62d8.1.8"
    },
    "level": "Low",
    "origin": {
      "time_utc": "2024-09-03T08:29:14.000000000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 10703,
    "logtype": "Policy",
    "additional_info": {
      "description": "Data element created. (Data Element 'TE_LASCII_L2R1_Y' created)"
    },
    "index_time_utc": "2024-09-03T08:30:31.358367506Z",
    "client": {
      "ip": "10.87.2.32",
      "username": "admin"
    },
    "ingest_time_utc": "2024-09-03T08:29:30.017906235Z",
    "correlationid": "cm0m64iap009r1h0399ey6rl8",
    "policy": {
      "severity": "Low",
      "audit_code": 150
    }
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T08:29:14.000Z"
    ],
    "index_time_utc": [
      "2024-09-03T08:30:31.358Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T08:29:30.017Z"
    ]
  },
  "highlight": {
    "additional_info.description": [
      "(Data Element '@opensearch-dashboards-highlighted-field@DE@/opensearch-dashboards-highlighted-field@' created)"
    ]
  },
  "sort": [
    1725352154000
  ]

The example contains the following information:

• additional_info
• origin
• policy
• process

For more information about the various fields, refer here.

Policy Status Dashboard index

The policy status dashboard index contains information for the Policy Status Dashboard. It holds the policy and trusted application deployment status information. The index pattern for viewing these logs in Discover is pty_insight_analytics*policy_status_dashboard_*.

{
    "logtype": "Status",
    "process": {
      "thread_id": "2458884416",
      "module": "rpstatus",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "2852",
      "user": "root",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "origin": {
      "time_utc": "2024-09-03T10:24:19.000Z",
      "hostname": "ip-10-49-2-49.ec2.internal",
      "ip": "10.49.2.49"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "datastore": "DataStore",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0"
    },
    "ingest_time_utc": "2024-09-03T10:24:19.510Z",
    "status": {
      "core_correlationid": "cm0f1jlq700gbzb19cq65miqt",
      "package_correlationid": "cm0m1tv5k0019te89e48tgdug"
    },
    "policystatus": {
      "type": "TRUSTED_APP",
      "application_name": "APJava_sample",
      "deployment_or_auth_time": "2024-09-03T10:24:19.000Z",
      "status": "WARNING"
    }
  },
  "fields": {
    "policystatus.deployment_or_auth_time": [
      "2024-09-03T10:24:19.000Z"
    ],
    "origin.time_utc": [
      "2024-09-03T10:24:19.000Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:24:19.510Z"
    ]
  },
  "sort": [
    1725359059000
  ]

The example contains the following information:

• additional_info
• origin
• protector
• policystatus
• policy
• process

Protectors status index

The protector status logs generated by protectors of v10.0.0 are stored in this index. The index pattern for viewing these logs in Discover is pty_insight_analytics_protectors_status_*.

The following parameters are configured for the index rollover:

• Index age: 30 days
• Document count: 200,000,000
• Index size: 5 GB

Use the following query in Discover to view these logs.

logtype:status

A sample log is shown here:

{
   "logtype":"Status",
   "process":{
      "thread_id":"2559813952",
      "module":"rpstatus",
      "name":"java",
      "pcc_version":"3.6.0.1",
      "id":"1991",
      "user":"root",
      "version":"10.0.0.2.91.5ec4b8b",
      "core_version":"2.1.0-alpha+24.g7fc71.2.1",
      "platform":"Linux_x64"
   },
   "origin":{
      "time_utc":"2024-07-30T07:22:41.000Z",
      "hostname":"ip-10-39-3-218.ec2.internal",
      "ip":"10.39.3.218"
   },
   "cnt":1,
   "protector":{
      "vendor":"Java",
      "datastore":"ESA-10.39.2.7",
      "family":"sdk",
      "version":"10.0.0.2.91.5ec4b8b"
   },
   "ingest_time_utc":"2024-07-30T07:22:41.745Z",
   "status":{
      "core_correlationid":"clz79lc2o004jmb29neneto8k",
      "package_correlationid":"clz82ijw00037k790oxlnjalu"
   }
}

The example contains the following information:

• additional_info
• origin
• policy
• protector

Protector Status Dashboard index

The protector status dashboard index contains information for the Protector Status Dashboard. It holds the protector status information. The index pattern for viewing these logs in Discover is pty_insight_analytics*protector_status_dashboard_.

A sample log is shown here:

{
    "logtype": "Status",
    "process": {
      "thread_id": "2458884416",
      "module": "rpstatus",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "2852",
      "user": "root",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "origin": {
      "time_utc": "2024-09-03T10:24:19.000Z",
      "hostname": "ip-10-49-2-49.ec2.internal",
      "ip": "10.49.2.49"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "datastore": "DataStore",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0"
    },
    "ingest_time_utc": "2024-09-03T10:24:19.510Z",
    "status": {
      "core_correlationid": "cm0f1jlq700gbzb19cq65miqt",
      "package_correlationid": "cm0m1tv5k0019te89e48tgdug"
    },
    "protector_status": "Warning"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:24:19.000Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:24:19.510Z"
    ]
  },
  "sort": [
    1725359059000
  ]

The example contains the following information:

• additional_info
• origin
• protector
• process

DSG transaction metrics

The table in this section lists the details for the various parameters generated by DSG transactions. The DSG transaction logs are stored in the pty_insight_analytics_dsg_transaction_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_transaction_metrics_*. The following parameters are configured for the index rollover:

• Index age: 1 day
• Document count: 10,000,000
• Index size: 1 GB

This index stores the following fields.

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

For more information about the transaction metric logs, refer to the section Transaction Metrics Logging in the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

DSG usage metrics

This section describes the codes associated with the following DSG usage metrics:

• Tunnels usage data (Version 0)
• Service usage data (Version 0)
• Profile usage data (Version 0)
• Rules usage data (Version 0)

The table in this sub sections lists the details for the various parameters generated while using the DSG. The DSG usage metrics logs are stored in the pty_insight_analytics_dsg_usage_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_usage_metrics_*. The following parameters are configured for the index rollover:

• Index age: 1 day
• Document count: 3,500,000
• Index size: 1 GB

For more information about the usage metrics, refer to the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

Tunnels usage data

The table in this section describes the usage metric for Tunnels.

A sample is provided here:

{"metrics_type":"Tunnel","version":0,"tunnel_type":"HTTP","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","tunnel_id":"140361619513360","checksum":"4139677074","uptime":620.8048927783966,"bytes_processed":401,"frontend_bytes_processed":401,"backend_bytes_processed":0,"previous_bytes_processed":401,"previous_frontend_bytes_processed":401,"previous_backend_bytes_processed":0,"total_bytes_processed":1203,"total_frontend_bytes_processed":1203,"total_backend_bytes_processed":0,"message_count":1,"previouse_message_count":1,"total_message_count":3}

Services usage data

The table in this section describes the usage metric for Services.

A sample is provided here:

{"metrics_type":"Service","version":0,"service_type":"REST-API","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"}, "previous_timestamp":"2023-04-13T12:28:08Z", "service_id":"140361548704016","checksum":"3100121694","tunnel_checksum":"4139677074","calls":401,"frontend_calls":401,"backend_calls":0,"previous_calls":401,"previous_frontend_calls":401,"previous_backend_calls":0,"total_calls":1203,"total_frontend_calls":1203,"total_backend_calls":0,"bytes_processed":2,"frontend_bytes_processed":1,"backend_bytes_processed":1,"previous_bytes_processed":2,"previous_frontend_bytes_processed":1,"previous_backend_bytes_processed":1,"total_bytes_processed":6,"total_frontend_bytes_processed":3,"total_backend_bytes_processed":3}

Profile usage data

The table in this section describes the usage metric for Profile.

A sample is provided here:

{"metrics_type":"Profile","version":0,"cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","profile_id":"140361548999248","checksum":"3504922421","parent_checksum":"3100121694","calls":2,"previous_calls":2,"total_calls":6,"profile_reference_count":0,"previous_profile_reference_count":0,"total_profile_reference_count":0,"bytes_processed":802,"previous_bytes_processed":802,"total_bytes_processed":2406,"elapsed_time_sample_count":2,"elapsed_time_average":221078.5,"total_elapsed_time_sample_count":6,"total_elapsed_time_sample_average":245797.0}

Rules usage data

The table in this section describes the usage metric for Rules.

A sample is provided here:

{"metric_type":"Rule","version":0,"rule_type":"Extract","codec":"Set User Identity","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","broken":false,"domain_name_rewrite":false,"rule_id":"140361553016464","rule_checksum":"932129179","parent_checksum":"3504922421","calls":1,"previous_calls":1,"total_calls":3,"profile_reference_count":0,"previous_profile_reference_count":0,"total_profile_reference_count":0,"bytes_processed":1,"previous_bytes_processed":1,"total_bytes_processed":3,"elapsed_time_sample_count":1,"elapsed_time_sample_average":406842.0,"total_elapsed_time_sample_count":3,"total_elapsed_time_sample_average":451163.6666666667}

DSG error metrics

The table in this section lists the details for the various parameters generated for the DSG Error Metrics. The DSG Error Metrics logs are stored in the pty_insight_analytics_dsg_error_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_error_metrics_*. The following parameters are configured for the index rollover:

• Index age: 1 day
• Document count: 3,500,000
• Index size: 1 GB

This index stores the following fields.

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

For more information about the error metric logs, refer to the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

Miscellaneous index

The logs that are not added to the other indexes are captured and stored in the miscellaneous index. The index pattern for viewing these logs in Discover is pty_insight_analytics_miscellaneous_*.

This index should not contain any logs. If any logs are visible in this index, then kindly contact Protegrity support.

The following parameters are configured for the index rollover:

• Index age: 7 days
• Document count: 3,500,000
• Index size: 200 mb

Use the following query in Discover to view these logs.

logtype:miscellaneous;

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

3.7 - Log return codes

3.8 - Policy Management Errors

Nodes Connectivity Status of the nodes is displayed as Error under Policy Management > Data Stores in ESA Web UI

Issue : In a multi-site ESA configuration, if the protectors are at or below v9.1.0.0, then the Node Connectivity Status on the Primary site ESAs might display Error status. This behavior is observed for all the protector nodes after performing failover and fail back operations between Primary and the Disaster Recovery sites.

Description : This may occur because the PEP server attempts to send status using a Node ID which is not present in the ESA’s repository. This repository is responsible for maintaining the status of all the registered pep server nodes.

Additionally, the following warning log in PEP server logs appears
(WARNING) Failed to send node status: The requested URL was not found on the server
To access PEP server logs, navigate to Discover, by logging into the ESA and navigating to Audit Store > Dashboard > Open in new tab, select Discover from the menu and select a time period.

Workaround : Perform the following steps to reset the node’s status to Green ("OK").

1. Log in to ESA Web UI of the Primary ESA.
2. Navigate to Policy Management > Data Stores
3. Select nodes showing status as Red ("Error") and click on delete button to remove entry.

If there are many PEP server nodes registered, ensure to delete the nodes in a batch of 200. After deleting the registered nodes successfully, the PEP server nodes are re-registered with ESA and status updates to Green ("OK").

The following section provides information about the resultant errors when trying to fetch the members from a member source.

3.9 - Protectors security log codes

The security logging level can be configured when a data security policy is created in the Policy management in ESA. If logging level is set to audit successful and audit failed, then both successful and failed Unprotect/Protect/Reprotect/Delete operations will be logged.

You can define the server where these security audit logs will be sent to. You can do that by modifying the Log Server configuration section in pepserver.cfg file.

If you configure to send protector security logs to ESA, you will be able view them in Discover, by logging into the ESA and navigating to Audit Store > Dashboard > Open in new tab, select Discover from the menu, and select a time period such as Last 30 days. The following table displays the logs sent by protectors.

3.10 - Additional log information

These are values for understanding the values that are displayed in the log records.

Log levels

Most events on the system generate logs. The level of the log helps you understand whether the log is just an information message or denotes some issue with the system. The log message and the log level allows you to understand more about the working of the system and also helps you identify and troubleshoot any system issues.

Protection logs: These logs are generated for Unprotect, Reprotect, and Protect (URP) operations.

• SUCCESS: This log is generated for a successful URP operation.
• WARNING: This log is generated if a user does not have access and the operation is unprotect.
• EXCEPTION: This log is generated if a user does not have access, the operation is unprotect, and the return exception property is set.
• ERROR: This log is generated for all other issues.

Application logs: These logs are generated by the application. The log level denotes the severity level of the log, however, levels 1 and 6 are used for the log configuration.

• 1: OFF. This level is used to turn logging off.
• 2: SEVERE. This level indicates a serious failure that prevents normal program execution.
• 3: WARNING. This level indicates a potential problem or an issue with the system.
• 4: INFO. This level is used to display information messages about the application.
• 5: CONFIG. This level is used to display static configuration information that is useful during debugging.
• 6: ALL. This level is used to log all messages.

Policy logs: These logs are used for the policy logs.

• LOWEST
• LOW
• NORMAL
• HIGH
• CRITICAL
• N/A

Protector information

The information displayed in the Protector-related fields of the audit log are listed in the table.

Protectors having CORE version 1.2.2+42.g01eb3.1.2 and higher are compatible with ESA v10.2.0. For more version-related information, refer to the Product Compatibility on My.Protegrity. The protector family might display the process.name for some protectors. This will be fixed in a later release.

Modules and components and the log type

Some of the components and modules and the logtype that they generate are provided in the following table.

For more information and description about the components that can generate kernel logs, refer here.

Kernel logs

This section lists the various kernel logs that are generated.

Note: This list is compiled using information from https://pmhahn.github.io/audit/.

User and group account management:

• ADD_USER: A user-space user account is added.
• USER_MGMT: The user-space management data.
• USER_CHAUTHTOK: A user account attribute is modified.
• DEL_USER: A user-space user is deleted.
• ADD_GROUP: A user-space group is added.
• GRP_MGMT: The user-space group management data.
• GRP_CHAUTHTOK: A group account attribute is modified.
• DEL_GROUP: A user-space group is deleted.

User login live cycle events:

• CRYPTO_KEY_USER: The cryptographic key identifier used for cryptographic purposes.
• CRYPTO_SESSION: The parameters set during a TLS session establishment.
• USER_AUTH: A user-space authentication attempt is detected.
• LOGIN: The user log in to access the system.
• USER_CMD: A user-space shell command is executed.
• GRP_AUTH: The group password is used to authenticate against a user-space group.
• CHUSER_ID: A user-space user ID is changed.
• CHGRP_ID: A user-space group ID is changed.
• Pluggable Authentication Modules (PAM) Authentication:
  • USER_LOGIN: A user logs in.
  • USER_LOGOUT: A user logs out.
• PAM account:
  • USER_ERR: A user account state error is detected.
  • USER_ACCT: A user-space user account is modified.
  • ACCT_LOCK: A user-space user account is locked by the administrator.
  • ACCT_UNLOCK: A user-space user account is unlocked by the administrator.
• PAM session:
  • USER_START: A user-space session is started.
  • USER_END: A user-space session is terminated.
• Credentials:
  • CRED_ACQ: A user acquires user-space credentials.
  • CRED_REFR: A user refreshes their user-space credentials.
  • CRED_DISP: A user disposes of user-space credentials.

Linux Security Model events:

• DAC_CHECK: The record discretionary access control (DAC) check results.
• MAC_CHECK: The user space Mandatory Access Control (MAC) decision is made.
• USER_AVC: A user-space AVC message is generated.
• USER_MAC_CONFIG_CHANGE:
• SELinux Mandatory Access Control:
  • AVC_PATH: dentry and vfsmount pair when an SELinux permission check.
  • AVC: SELinux permission check.
  • FS_RELABEL: file system relabel operation is detected.
  • LABEL_LEVEL_CHANGE: object’s level label is modified.
  • LABEL_OVERRIDE: administrator overrides an object’s level label.
  • MAC_CONFIG_CHANGE: SELinux Boolean value is changed.
  • MAC_STATUS: SELinux mode (enforcing, permissive, off) is changed.
  • MAC_POLICY_LOAD: SELinux policy file is loaded.
  • ROLE_ASSIGN: administrator assigns a user to an SELinux role.
  • ROLE_MODIFY: administrator modifies an SELinux role.
  • ROLE_REMOVE: administrator removes a user from an SELinux role.
  • SELINUX_ERR: internal SELinux error is detected.
  • USER_LABELED_EXPORT: object is exported with an SELinux label.
  • USER_MAC_POLICY_LOAD: user-space daemon loads an SELinux policy.
  • USER_ROLE_CHANGE: user’s SELinux role is changed.
  • USER_SELINUX_ERR: user-space SELinux error is detected.
  • USER_UNLABELED_EXPORT: object is exported without SELinux label.
  • AppArmor Mandatory Access Control:
    • APPARMOR_ALLOWED
    • APPARMOR_AUDIT
    • APPARMOR_DENIED
    • APPARMOR_ERROR
    • APPARMOR_HINT
    • APPARMOR_STATUS APPARMOR

Audit framework events:

• KERNEL: Record the initialization of the Audit system.
• CONFIG_CHANGE: The Audit system configuration is modified.
• DAEMON_ABORT: An Audit daemon is stopped due to an error.
• DAEMON_ACCEPT: The auditd daemon accepts a remote connection.
• DAEMON_CLOSE: The auditd daemon closes a remote connection.
• DAEMON_CONFIG: An Audit daemon configuration change is detected.
• DAEMON_END: The Audit daemon is successfully stopped.
• DAEMON_ERR: An auditd daemon internal error is detected.
• DAEMON_RESUME: The auditd daemon resumes logging.
• DAEMON_ROTATE: The auditd daemon rotates the Audit log files.
• DAEMON_START: The auditd daemon is started.
• FEATURE_CHANGE: An Audit feature changed value.

Networking related:

• IPSec:
  • MAC_IPSEC_ADDSA
  • MAC_IPSEC_ADDSPD
  • MAC_IPSEC_DELSA
  • MAC_IPSEC_DELSPD
  • MAC_IPSEC_EVENT: The IPSec event, when one is detected, or when the IPSec configuration changes.
• NetLabel:
  • MAC_CALIPSO_ADD: The NetLabel CALIPSO DoI entry is added.
  • MAC_CALIPSO_DEL: The NetLabel CALIPSO DoI entry is deleted.
  • MAC_MAP_ADD: A new Linux Security Module (LSM) domain mapping is added.
  • MAC_MAP_DEL: An existing LSM domain mapping is added.
  • MAC_UNLBL_ALLOW: An unlabeled traffic is allowed.
  • MAC_UNLBL_STCADD: A static label is added.
  • MAC_UNLBL_STCDEL: A static label is deleted.
• Message Queue:
  • MQ_GETSETATTR: The mq_getattr and mq_setattr message queue attributes.
  • MQ_NOTIFY: The arguments of the mq_notify system call.
  • MQ_OPEN: The arguments of the mq_open system call.
  • MQ_SENDRECV: The arguments of the mq_send and mq_receive system calls.
• Netfilter firewall:
  • NETFILTER_CFG: The Netfilter chain modifications are detected.
  • NETFILTER_PKT: The packets traversing Netfilter chains.
• Commercial Internet Protocol Security Option:
  • MAC_CIPSOV4_ADD: A user adds a new Domain of Interpretation (DoI).
  • MAC_CIPSOV4_DEL: A user deletes an existing DoI.

Linux Cryptography:

• CRYPTO_FAILURE_USER: A decrypt, encrypt, or randomize cryptographic operation fails.
• CRYPTO_IKE_SA: The Internet Key Exchange Security Association is established.
• CRYPTO_IPSEC_SA: The Internet Protocol Security Association is established.
• CRYPTO_LOGIN: A cryptographic officer login attempt is detected.
• CRYPTO_LOGOUT: A cryptographic officer logout attempt is detected.
• CRYPTO_PARAM_CHANGE_USER: A change in a cryptographic parameter is detected.
• CRYPTO_REPLAY_USER: A replay attack is detected.
• CRYPTO_TEST_USER: The cryptographic test results as required by the FIPS-140 standard.

Process:

• BPRM_FCAPS: A user executes a program with a file system capability.
• CAPSET: Any changes in process-based capabilities.
• CWD: The current working directory.
• EXECVE; The arguments of the execve system call.
• OBJ_PID: The information about a process to which a signal is sent.
• PATH: The file name path information.
• PROCTITLE: The full command-line of the command that was used to invoke the analyzed process.
• SECCOMP: A Secure Computing event is detected.
• SYSCALL: A system call to the kernel.

Special system calls:

• FD_PAIR: The use of the pipe and socketpair system calls.
• IPC_SET_PERM: The information about new values set by an IPC_SET control operation on an Inter-Process Communication (IPC) object.
• IPC: The information about a IPC object referenced by a system call.
• MMAP: The file descriptor and flags of the mmap system call.
• SOCKADDR: Record a socket address.
• SOCKETCALL: Record arguments of the sys_socketcall system call (used to multiplex many socket-related system calls).

Systemd:

• SERVICE_START: A service is started.
• SERVICE_STOP: A service is stopped.
• SYSTEM_BOOT: The system is booted up.
• SYSTEM_RUNLEVEL: The system’s run level is changed.
• SYSTEM_SHUTDOWN: The system is shut down.

Virtual Machines and Container:

• VIRT_CONTROL: The virtual machine is started, paused, or stopped.
• VIRT_MACHINE_ID: The binding of a label to a virtual machine.
• VIRT_RESOURCE: The resource assignment of a virtual machine.

Device management:

• DEV_ALLOC: A device is allocated.
• DEV_DEALLOC: A device is deallocated.

Trusted Computing Integrity Measurement Architecture:

• INTEGRITY_DATA: The data integrity verification event run by the kernel.
• INTEGRITY_EVM_XATTR: The EVM-covered extended attribute is modified.
• INTEGRITY_HASH: The hash type integrity verification event run by the kernel.
• INTEGRITY_METADATA: The metadata integrity verification event run by the kernel.
• INTEGRITY_PCR: The Platform Configuration Register (PCR) invalidation messages.
• INTEGRITY_RULE: A policy rule.
• INTEGRITY_STATUS: The status of integrity verification.

Intrusion Prevention System:

• Anomaly detected:
  • ANOM_ABEND
  • ANOM_ACCESS_FS
  • ANOM_ADD_ACCT
  • ANOM_AMTU_FAIL
  • ANOM_CRYPTO_FAIL
  • ANOM_DEL_ACCT
  • ANOM_EXEC
  • ANOM_LINK
  • ANOM_LOGIN_ACCT
  • ANOM_LOGIN_FAILURES
  • ANOM_LOGIN_LOCATION
  • ANOM_LOGIN_SESSIONS
  • ANOM_LOGIN_TIME
  • ANOM_MAX_DAC
  • ANOM_MAX_MAC
  • ANOM_MK_EXEC
  • ANOM_MOD_ACCT
  • ANOM_PROMISCUOUS
  • ANOM_RBAC_FAIL
  • ANOM_RBAC_INTEGRITY_FAIL
  • ANOM_ROOT_TRANS
• Responses:
  • RESP_ACCT_LOCK_TIMED
  • RESP_ACCT_LOCK
  • RESP_ACCT_REMOTE
  • RESP_ACCT_UNLOCK_TIMED
  • RESP_ALERT
  • RESP_ANOMALY
  • RESP_EXEC
  • RESP_HALT
  • RESP_KILL_PROC
  • RESP_SEBOOL
  • RESP_SINGLE
  • RESP_TERM_ACCESS
  • RESP_TERM_LOCK

Miscellaneous:

• ALL: Matches all types.
• KERNEL_OTHER: The record information from third-party kernel modules.
• EOE: An end of a multi-record event.
• TEST: The success value of a test message.
• TRUSTED_APP: The record of this type can be used by third-party application that require auditing.
• TTY: The TTY input that was sent to an administrative process.
• USER_TTY: An explanatory message about TTY input to an administrative process that is sent from the user-space.
• USER: The user details.
• USYS_CONFIG: A user-space system configuration change is detected.
• TIME_ADJNTPVAL: The system clock is modified.
• TIME_INJOFFSET: A Timekeeping offset is injected to the system clock.

4 - Known Issues for the td-agent

• Known Issue: The Buffer overflow error appears in the /var/log/td-agent/td-agent.log file.

  Description: When the total size of the files in td-agent buffer /opt/protegrity/td-agent/es_buffer directory reaches the default maximum limit of 64 GB, then the Buffer overflow error appears.

  Resolution:

  Add the total_limit_size parameter to increase the buffer limit in the OUTPUT.conf file using the following steps.

  1. Log in to the ESA Web UI.

  2. Navigate to System > Services.

  3. Under Misc, stop the td-agent service.

  4. Log in to the CLI Manager of the ESA node.

  5. Navigate to Administration > OS Console.

  6. Navigate to the /opt/protegrity/td-agent/config.d directory.

  7. Open the OUTPUT.conf file.

  8. Add the total_limit_size parameter in the buffer section of the OUTPUT.conf file.

     In this example, the total_limit_size is doubled to 128 GB.

     

     

  9. Save the file.

  10. Log in to the ESA Web UI.

  11. Navigate to System > Services.

  12. Under Misc, start the td-agent service.

• Known Issue: The Too many open files error appears in the /var/log/td-agent/td-agent.log file.

  Description: When the total number of files in the td-agent buffer /opt/protegrity/td-agent/es_buffer directory reaches the maximum limit, then the Too many open files error appears.

  Resolution:

  Change the limit for the maximum number of open files for the td-agent service in the /etc/init.d/td-agent file using the following steps.

  1. Log in to the ESA Web UI.

  2. Navigate to System > Services.

  3. Under Misc, stop the td-agent service.

  4. Log in to the CLI Manager of the ESA node.

  5. Navigate to Administration > OS Console.

  6. Navigate to the /etc/init.d directory.

  7. Open the td-agent file.

  8. Change the ulimit.

     In this example, the ulimit is increased to 120000.

     

     

  9. Save the file.

  10. Log in to the ESA Web UI.

  11. Navigate to System > Services.

  12. Under Misc, start the td-agent service.

5 - Known Issues for Protegrity Analytics

• Known Issue: Client side validation is missing on the Join an existing Audit Store Cluster page.

  Issue:

  Log in to the ESA Web UI and navigate to the Audit Store > Cluster Management > Overview page >Join Cluster. When you specify an invalid IP, enter a username or password more than the 36-character limit that is accepted on the Appliance, and click Join Cluster, then no errors are displayed and the request is processed.

  Observation:

  The Join an existing Audit Store Cluster page request is processed without any client-side validation, hence, an invalid IP address or a username or password more than 36 characters does not display any error.

• Known Issue: High memory usage on the ESA.

  Issue:

  When using the Audit Store, the memory usage is high on the ESA.

  Workaround:

  Reduce the memory usage by updating the memory allocated to the Audit Store on the ESA to 4 GB using the Set Audit Store Repository Total Memory CLI option.

  For more information about the Set Audit Store Repository Total Memory CLI option, refer to Setting the total memory for the Audit Store Repository.

• Known Issue: In Analytics, on the Index Lifecycle Management page, after exporting, importing, or deleting an index one of the following scenarios occurs:

  • The index operation performed does not appear in the other operation lists. For example, an exported index does not appear in the import index list.
  • Performing the same operation on the same index again displays an error message.
  • If the index appears in another operation list, performing the operation displays an error message. For example, an exported index appears in the delete index list and deleting the index displays an error.

  Issue: After performing an operation, the index list for the export, import, and delete operations does not refresh automatically.

  Workaround: Refresh the Index Lifecycle Management page after performing an export, import, or delete operation.

• Known Issue: The Specifying types in document update requests is deprecated, use the endpoint /{index}/_update/{id} instead. error message might appear during an upgrade.

  Issue: While upgrading, the error trace is logged while disabling the rollover index.

  Resolution: Ignore this message. It is a deprecation message and does not affect the upgrade or the functionality of the product.

• Known Issue: The Info: During upgrade, ignore errors related to unicast hosts file being empty. message might appear during an upgrade.

  Issue: While upgrading, the following errors might appear:

  • ERROR: Unicast hosts file does not contain the IP addresses of the nodes and is empty. Run the Audit Store Management Unicast Hosts scheduled task.
  • ERROR: Unicast hosts file is empty. Please add the IP and hostname entries for all the nodes in the Audit Store cluster in the Unicast hosts file /opt/protegrity/config/unicast_hosts.txt

  Resolution: Ignore this message. It does not affect the functionality of the product. The upgrade fixes the entries in the unicast hosts file and the upgrade completes successfully.

• Known Issue: The Info: While upgrading nodes in an Audit Store cluster, if the checks passed on one of the nodes, then ignore errors for the Audit Store cluster verification on the other nodes. message might appear during an upgrade.

  Issue: While upgrading, the following errors might appear:

  • ERROR: Audit Store version of the node across the cluster are inconsistent: {version}. Ensure the nodes are on the same ESA version.
  • ERROR: Expecting appliance version {Version number}.
  • ERROR: Expecting appliance version 10.1.X/10.0.X/9.2.0.X/9.1.0.X got {appliance_version}.
  • ERROR: Cluster health check failed. Current status: {cluster_status}.Resolve the cluster health to stabilize the cluster.
  • ERROR: Initializing shards check failed. Current value: {initializing_shards}. Wait till the shards are initialized and the value reaches zero.
  • ERROR: Relocating shards check failed. Current value: {relocating_shards}. Wait till the shards are automatically rebalanced and the value reaches zero.
  • ERROR: Unassigned shards check failed. Current value : {unassigned_shards}. Wait till the shards are processed and the value reaches zero.

  Resolution: If the checks passed on one of the nodes while upgrading nodes in an Audit Store cluster, then ignore the message if it appears on the other nodes. It does not affect the functionality of the product. The upgrade of the muilti-cluster nodes will complete successfully.

6 - Known Issues for the Log Forwarder

Known Issue: The Protector is unable to reconnect to a Log Forwarder after it is restarted.

Description: This issue occurs whenever you have a Proxy server between a Protector and a Log Forwarder. When the Log Forwarder is stopped, the connection between the Protector and the Proxy server is still open, even though the connection between the Proxy server and the Log Forwarder is closed. As a result, the Protector continues sending audit files to the Proxy server. This results in loss of the audit files. Whenever the Log Forwarder is restarted, the Protector is unable to reconnect to the Log Forwarder.

This issue is applicable to all the Protectors where the Log Forwarder is not running on the local host machine. For example, this issue is applicable to AIX or z/OS protectors because the Log Forwarder is not running on the same machine where the Protectors have been installed. This issue also occurs if you have a Load Balancer or a Firewall between the Protector and the Log Forwarder, instead of a Proxy server.

Resolution: Remove the Proxy server or ensure that you configure the Proxy server in a way that the connection between the Protector and the Proxy server is stopped as soon as the Log Forwarder is stopped. This ensures that whenever the Log Forwarder is restarted, the Protector reconnects with the Log Forwarder and continues to send the audits to the Log Forwarder without any data loss.

For more information about configuring the Proxy server, contact your IT administrator.

7 - Deprecations

Deprecated Functions

Extract KeyID from Data  getCurrentKeyId getDefaultDataElement Byte API

Database Protector

• native database access control

Deprecated Products

• Protection Server
• File Protector Gateway 
• File Protector: Linux Kernel Implementation 
• File Protector:  Windows Bases Volume Encryption 
• File Protector for AIX 
• XC Client, XC Server, XC Lite 
• Solaris operating system on all the Protectors
• HAWQ Big Data Component 
• Linux Based Volume Encryption

Deprecated Capabilities

• HDFS FP
• ESA OS Based High Availability 
• Samba 
• Protegrity Storage Unit (PSU)
• Change in Insight behavior (from v9.0.0.0)
• Jasper Reporting 
• Metering 
• MapReduce 
• HMAC API in BDP 
• Gate in Upgrade

Deprecated Data Elements

Deprecated data elements can still be used. However, users will be unable to create these data elements using the GUI. Users can use the DevOps APIs to create these data elements. In this case, the system triggers a warning and create a log entry. This is to indicate that the data element is deprecated. In addition, the system triggers a notification if the policy contains any of the deprecated datatypes. The system triggers this notification while the hubcontroller service starts.

• Printable Characters Tokenization 
• Unicode Gen 1 Tokenization 
• Date Tokenization 
• DTP 2 Encryption 
• 3DES Encryption 
• CUSP 3DES Encryption 
• SHA1 Hashing
• Date tokens (DATE-YYYY-MM-DD, DATE-DD/MM/YYYY, DATE-MM/DD/YYYY )
• UNICODE tokens
• PRINTABLE tokens
• UNICODE-Base 64 tokens
• SHA1
• 3DES

Removed Data Elements

The support for the Format Preserving Encryption (FPE) data element was started in the ESA v7.2.x. If the FPE left/right data element is created in any version of the ESA, from 7.2.x to 9.2.x, then there is a risk of data loss.
The data encoding can have an effect on the resulting token when the Left and Right properties are in use. This means that FPE tokens with this property cannot always be moved “as-is” to other systems where the encoding of the data changes.

In ESA v10.0.1,

• The upgrade is blocked if the FPE left/right data element is created.
• A new solution is created to support the FPE left/right data element. This solution works only when the protectors of version 10.0.0 or newer are used.