Known issues for the Audit Store

A list of known issues with their solution or workaround are provided here. The steps provided to resolve the known issues ensure that your product does not display errors or crash.

Known Issue: The Audit Store node security remains uninitialized and the message Audit Store Security is not initialized. appears on the Audit Store Cluster Management page.
Resolution:
Run the following steps to resolve the issue.
1. From the ESA Web UI, navigate to System > Services > Audit Store.
2. Ensure that the Audit Store Repository service is running.
3. Open the ESA CLI.
4. Navigate to Tools.
5. Run Apply Audit Store Security Configs.
Known Issue: The Audit Store Repository service stops when the user logs out from the ESA admin console after changing the hostname from the admin console tools menu.
Issue:
The Audit Store Repository service stops when the user rotates Insight certificate, finalizes an appliance (as in the case of cloud environments), changes the hostname, or starts or stops the audit store repository service, by logging into admin console and then logging out from admin console. The Audit Store Repository service also stops when the user logs out of the admin console after starting or stopping the Audit Store Repository service from the admin console.
Resolution:
Manually start the Audit Store Repository service from System > Services on the ESA Web UI.
Known Issue: Logs sent to the Audit Store do not get saved and errors might be displayed.
Issue:
The Audit Store cannot receive and store logs when the disk space available on the ESA is low. In this case, errors or warnings similar to high disk watermark [90%] exceeded are displayed in the logs.
Resolution:
Perform one of the following steps to resolve the issue:
- Delete old indices that are not required using ILM in Analytics.
- Increase the disk space on all nodes.
- Add new nodes to the cluster.
Known Issue: The Audit Store Repository fails to start after updating the domain name.
Issue:
After updating the domain name, the Audit Store Repository service fails to start.
Resolution:
The Audit Store Repository depends on the domain name. However, the domain name configuration in the Audit Store does not get updated automatically when there is a change in the domain name.
Perform the following steps to apply the updated domain name to the Audit Store:
1. Log in to the CLI Manager of the ESA or the Appliance.
2. Navigate to Administration > OS Console.
3. Enter the root password and select OK.
4. Navigate to the /opt/protegrity/auditstore/config directory using the following command.
```
cd **/opt/protegrity/auditstore/config**
```
5. Open the opensearch.yml file in a text editor.
6. Replace the existing domain name attribute with your domain name for the following configuration attributes.
  - network.host
  - http.host Consider the following example where the domain name of protegrity.com is updated to example.com.
  Existing configuration:
```
network.host: ["localhost","127.0.0.1","192.168.1.120","protegrity-esa123","protegrity-esa123.**protegrity.com**"]
```
  Updated configuration:
```
network.host: ["localhost","127.0.0.1","192.168.1.120","protegrity-esa123","protegrity-esa123.**example.com**"]
```
7. Save and close the file.
8. Rotate the Insight certificates.
  For more information about rotating certificates, refer here.
9. Log in to the CLI Manager of the ESA or the Appliance.
10. Navigate to Administration > OS Console.
11. Enter the root password and select OK.
12. Run the following commands to refresh the network settings.
```
/etc/opt/scripts/on-hostname-set/90_update_asrepository.sh
/etc/opt/scripts/on-hostname-set/91_update_cluster_user_ssh_keys.sh
```

Known Issue: The Upgrade cannot continue as the cluster health status is red. Check out the Troubleshooting Guide for info on how to proceed. error message appears.

Issue: A cluster status in red color means that at least one primary shard and its replicas are not allocated to a node, that is, there are indices with the index health status in red color in the Audit Store cluster.

Workaround

Complete the following steps to resolve the cluster health with the red status.

From the Web UI of the Appliance, navigate to System > Services > Audit Store.
Ensure that the Audit Store Repository service is running.
Log in to the CLI Manager of the Appliance.
Navigate to Administration > OS Console.

Identify the indices with the health status as red using the following command.

wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key -O - https://<Appliance_IP>:9200/_cat/indices | grep red

Ensure that you update the variables before running the command. An example of the command is provided here.

wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key -O - https://localhost:9200/_cat/indices | grep red

A list of indices containing the health status as red appears as shown in the following example.

red    open   pty_insight_audit_vx.x-xxxx.xx.xx-000014             dxmEWom8RheqOhnaFeM3sw   1   1

In the example, pty_insight_audit_vx.x-xxxx.xx.xx-000014 is the index having a red index health status where the index’s primary shard and replicas are not available or allocated to any node in the cluster.

Identify the reason for unassigned shards using the following command.

wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key -O - https://<Appliance_IP>:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

Ensure that you update the variables before running the command. An example of the command is provided here.

wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key -O - https://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

The reasons for the shards being unassigned appear. This example shows one of the reasons for the unassigned shard.

    `pty_insight_audit_vx.x-xxxx.xx.xx-000014             0 p UNASSIGNED NODE_LEFT`

    `pty_insight_audit_vx.x-xxxx.xx.xx-000014             0 r UNASSIGNED NODE_LEFT`

In the example, the 0th p and r shards of the pty_insight_audit_vx.x-xxxx.xx.xx-000014 index are unassigned due to the NODE_LEFT reason, that is, because the node left the Audit Store cluster. The p indicates a primary shard and the r indicates a replica shard.

Retrieve the details for the shard being unassigned using the following command.

wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key --header='Content-Type:application/json' --method=GET --body-data='{ "index": "<Index_name>", "shard": <Shard_ID>, "primary":<true or false> }' -O - https://<Appliance_IP>:9200/_cluster/allocation/explain?pretty

Ensure that you update the variables before running the command. An example of the command with the index name as pty_insight_audit_vx.x-xxxx.xx.xx-000014, shard ID as 0, and primary shard as true is provided here.

wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key --header='Content-Type:application/json' --method=GET --body-data='{ "index": "pty_insight_audit_vx.x-xxxx.xx.xx-000014", "shard": 0, "primary": true }' -O - https://localhost:9200/_cluster/allocation/explain?pretty

The details of the unassigned shard appears. This example shows one of the reasons for the unassigned shard.

{
        "index": "pty_insight_audit_vx.x-xxxx.xx.xx-000014",
        "shard": 0,
        "primary": true,
        "current_state": "unassigned",
        "unassigned_info": {
            "reason": "NODE_LEFT",
            "at": "2022-03-28T05:05:25.631Z",
            "details": "node_left [gJ38FzlDSEmTAPcP0yw57w]",
            "last_allocation_status": "no_valid_shard_copy"
        },
        "can_allocate": "no_valid_shard_copy",
        **"allocate\_explanation": "cannot allocate because all found copies of the shard are either stale or corrupt",**
        "node_allocation_decisions": [
            {
                "node_id": "3KXS1w9HTOeMH1KbDShGIQ",
                "node_name": "ESA1",
                "transport_address": "xx.xx.xx.xx:9300",
                "node_attributes": {
                    "shard_indexing_pressure_enabled": "true"
                },
                "node_decision": "no",
                "store": {
                    "in_sync": false,
                    "allocation_id": "HraOWSZlT3KNXxOHDhZL5Q"
                }
            }
        ]
    }

In this example, the shard is not allocated because all found copies of the shard are either stale or corrupt. There are no valid shard copies that can be allocated for this index. This is a data loss scenario, where the data is unavailable because the node or nodes that had the data have disconnected from the cluster. In such a scenario, if the disconnected nodes are brought back in the cluster, then the cluster can reconstruct itself and become healthy again. If bringing the nodes back is not possible, then deleting indices with the red index health status is the only way to fix a red cluster health status.

Complete one of the following two steps to stabilize the cluster.
- Troubleshoot the cluster:
  1. Verify that the Audit Store services are running. Restart any Audit Store service that is in the stopped state.
  2. Ensure that the disconnected nodes are running.
  3. Try to add any disconnected nodes back to the cluster.
  4. Restart the system or restore the system from a backup.
- Delete the index:
  Delete the indices with the index health status as red using the following command. Execute the command from any one node of Audit Store node which is running.
```
wget -q --ca-cert=<Path_to_CA_certificate>/CA.pem --certificate=<Path_to_client_certificate>/client.pem --private-key=<Path_to_client_key>/client.key --header='Content-Type:application/json' --method=DELETE -O - https://<Appliance_IP>:9200/<Index_name>
```
  Ensure that you update the variables before running the command. An example of the command to delete the pty_insight_audit_vx.x-xxxx.xx.xx-000014 index is provided here.
```
wget -q --ca-cert=/etc/ksa/certificates/as_cluster/CA.pem --certificate=/etc/ksa/certificates/as_cluster/client.pem --private-key=/etc/ksa/certificates/as_cluster/client.key --header='Content-Type:application/json' --method=DELETE -O - https://localhost:9200/pty_insight_audit_vx.x-xxxx.xx.xx-000014
```
  CAUTION:
  This command deletes the index and must be used carefully.

Known Issue: The Authentication failure for the user during JWT token generation while downloading ssh key from node error appears while performing any Audit Store cluster-related operation.
Issue: The Can create JWT token permission is required for a role to perform Audit Store cluster-related operations. The error appears if the permission is not assigned to the user.
Workaround
Use a user with the appropriate permissions for performing Audit Store cluster-related operations. Alternatively, verify and add the Can create JWT token permission. To verify and add the Can Create JWT Token permission, from the ESA Web UI, navigate to Settings > Users > Roles.

←Previous
Next→

Last modified February 7, 2025