Index lifecycle management (ILM)

The Protegrity Data Security Platform enforces security policies at many protection points throughout an enterprise and sends logs to the Audit Store. The logs are stored in a log repository, in this case the Audit Store. Manage the log repository using the Index Lifecycle Management (ILM). These logs are then available for reporting.

In the earlier versions of the ESA, the UI for Index Lifecycle Management was named as Information Lifecycle Management.

The following figure shows the ILM system components and the workflow.

The ILM log repository is divided into the following parts:

  • Active logs that may be required for immediate reporting. These logs are accessed regularly for high frequency reporting.
  • Logs that are pushed to Short Term Archive (STA). These logs are accessed occasionally for moderate reporting frequency.
  • Logs that are pushed to Long Term Archive (LTA). These logs are accessed rarely for low reporting frequency. The logs are stored where they can be backed up by the backup mechanism used by the enterprise.

The ILM feature in Protegrity Analytics is used to archive the log entries from the index. The logs generated for the ILM operations appear on this page. Only logs generated by ILM operation on the ESA v9.2.0.0 and above appear on the page after upgrading to the latest version of the ESA. For ILM logs generated on an earlier version of the ESA, navigate to Audit Store > Dashboard > Open in new tab, select Discover from the menu, select the time period, and search for the ILM logs using keywords for the additional_info.procedure field, such as, export, process_post_export_log, or scroll_index_for_export.

Use the search bar to filter logs. Click the Reset Search () icon to clear the search filter and view all the entries. To search for the ILM logs using the origin time, specify the Origin Time(UTC) term within double quotes.

Move entries out of the index when not required and import them back into the index when required using the export and import feature. Only one operation can be run at a time for each node for exporting logs or importing logs. The ILM screen is shown in the following figure.

The Viewer role user or a user with the viewer role can only view data on the ILM screen. Admin rights are required to use the import, export, migrate, and delete features of the ILM.

Use the ILM for managing indexes, such as, the audit index, the policy log index, the protector status index, and the troubleshooting index. The Audit Store Dashboard has the ISM feature for managing the other indexes. Using the ISM feature might result in a loss of logs and it is not advised to use the ILM feature where possible.

Exporting logs

As log entries fill the Audit Store, the size of the log index increases. This slows down log operations for searching and retrieving log entries. To speed up these operations, export log entries out of the index and store them in an external file. If required, import the entries again for audit and analysis.

Moving index entries out of the index file, removes the entries from the index file and places them in a backup file. This backup file is the STA and reduces the load and processing time for the main index. The backup file is created in the /opt/protegrity/insight/archive/ directory. To store the file at a different location, mount the destination in the /opt/protegrity/insight/archive/ directory. In this case, specify the directory name, for example, /opt/protegrity/insight/archive/. Also, ensure that the specified already exists inside the archive directory.

If the location is on the same drive or volume as the main index, then the size of the index would reduce. However, this would not be an effective solution for saving space on the current volume. To save space, move the backup file to a remote system or into LTA.

Only one export operation can be run at a time. Empty indexes cannot de exported and must be manually deleted.

  1. On the ESA, navigate to Audit Store > Analytics > Index Lifecycle Management.

  2. Click Export.

    The Export Data screen appears.

  3. Complete the fields for exporting the log data from the default index.

    The available fields are:

    • From Index: Select the index to export data from.
    • Password: Specify the password for securing the backup file.
    • Confirm Password: Specify the password again for reconfirmation.
    • Directory (optional): Specify the location to save the backup file. If a value is not specified, then the default directory /opt/protegrity/insight/archive/ is used.
  4. Click Export.

  5. Specify the root password.

  6. Click Submit.

The log entries are extracted, then copied to the backup file, and protected using the password. After a successful export, the exported index will be deleted from the Audit Store database.

After the export is complete, move the backup file to a different location till the log entries are required. Import the entries in the index again for analysis or audit.

Importing logs

The exported log entires and secondary indexes are stored in a separate file. If these entries are required for analysis, then import them back into the Audit Store. To be able to import, the archive file should be inside the archive directory or within a directory inside the archive directory.

Keep the passwords handy, in case the log entries were exported and protected using password protection. Do not rename the default index file name for this feature to work. Imported indexes are excluded and are not exported when the auto-export task is run from the scheduler.

  1. On the ESA, navigate to Audit Store > Analytics > Index Lifecycle Management.

  2. Click Import.

    The Import Data screen appears.

  3. Complete the fields for importing the log data to the default index or secondary index.

    The available fields are:

    • File Name: Select the file name of the backup file.
    • Password: Specify the password for the backup file.
  4. Click Import.

Data will be imported to an index that is named using the file name or the index name. When importing a file which was exported in version 8.0.0.0 or later, then the new index name will be the date range of the entries in the index file using the format pty_insight_audit_ilm_(from_date)-(to_date). For example, pty_insight_audit_ilm_20191002_113038-20191004_083900.

Deleting indexes

Use the Delete option to delete indexes that are not required. Only delete custom indexes that are created and listed in the Source list. Deleting the index will lead to a permanent loss of data in the index. If the index was not archived earlier, then the logs from the index deleted cannot be recreated or retrieved.

  1. On the ESA, navigate to Audit Store > Analytics > Index Lifecycle Management.

  2. Click Delete.

    The Delete Index screen appears.

  3. Select the index to delete from the Source list.

  4. Select the Data in the selected index will be permanently deleted. This operation cannot be undone. check box.

  5. Click Delete.

    The Authentication screen appears.

  6. Enter the root password.

  7. Click Submit.