Using the scheduler

An administrator can execute tasks for ILM, reporting, and signature verification. These tasks that need to be executed regularly or after a fixed interval can be converted to a scheduled task. This ensures that the task is processed regularly at the set time leaving the administrator free to work on other more important tasks.

To view the list of tasks that are scheduled, from the Analytics screen, navigate to Scheduler > Tasks. The viewer role user or a user with the viewer role can only view logs and history related to the Scheduler. You need admin rights to create or modify schedules.

The following tasks are available by default:

TaskDescription
Export Troubleshooting IndicesScheduled task for exporting logs from the troubleshooting index.
Export Policy Log IndicesScheduled task for exporting logs from the policy index.
Export Protectors Status IndicesScheduled task for exporting logs from the protector status index.
Delete Miscellaneous IndicesScheduled task for deleting old versions of the miscellaneous index that are rolled over.
Delete DSG Error IndicesScheduled task for deleting old versions of the DSG error index that are rolled over.
Delete DSG Usage IndicesScheduled task for deleting old versions of the DSG usage matrix index that are rolled over.
Delete DSG Transaction IndicesScheduled task for deleting old versions of the DSG transaction matrix index that are rolled over.
Signature VerificationScheduled task for performing signature verification of log entries.
Export Audit IndicesScheduled task for exporting logs from the audit index.
Rollover IndexScheduled task for performing an index rollover.

Ensure that the scheduled tasks are disabled on all the nodes before upgrading the ESA.

The scheduled task values on a new installation and an upgraded machine might differ. This is done to preserve any custom settings and modifications for the scheduled task. After upgrading the ESA, revisit the scheduled task parameters and modify them if required.

The list of scheduled tasks are displayed. You can create tasks, view, edit, enable or disable, and modify scheduled task properties from this screen. The following columns are available on this screen.

ColumnDescription
NameA unique name for the scheduled task.
ScheduleThe frequency set for executing the task.
Task TemplateThe task template for creating the schedule.
Priority IPsA list of IP addresses of the machines on which the task must be run.
ParamsThe parameters for the task that must be executed.
EnabledUse this toggle switch to enable or disable the task from running as per the schedule.
ActionThe actions that can be performed on the scheduled task.

The available action options are:

  • Click the Edit icon () to update the task.
  • Click the Delete icon () to delete the task.

Creating a Scheduled Task

Use the repository scheduler to create scheduled tasks. You can set a scheduled task to run after a fixed interval, every day at a particular time, a fixed day every week, or a fixed day of the month.

Complete the following steps to create a scheduled task.

  1. From the Analytics screen, navigate to Scheduler > Tasks.

  2. Click Add New Task.

    The New Task screen appears.

  3. Complete the fields for creating a scheduled task.

    The following fields are available:

    • Name: Specify a unique name for the task.
    • Schedule: Specify the template and time for running the command using cron. The date and time when the command will be run appears in the area below the Schedule field. The following settings are available:
      • Select Template: Select a template from the list. The following templates are available:

        • Custom: Specify a custom schedule for executing the task.
        • Every Minute: Set the task to execute every minute.
        • Every 5 Minutes: Set the task to execute after every 5 minutes.
        • Every 10 Minutes: Set the task to execute after every 10 minutes.
        • Every Hour: Set the task to execute every hour.
        • Every 2 Hours: Set the task to execute every 2 hours.
        • Every 5 Hours: Set the task to execute every 5 hours.
        • Every Day: Set the task to execute every day at 12 am.
        • Every Alternate Day: Set the task to execute every alternate day at 12 am.
        • Every Week: Set the task to execute once every week on Sunday at 12 am.
        • Every Month: Set the task to execute at 12 am on the first day of every month.
        • Every Alternate Month: Set the task to execute at 12 am on the first day of every alternate month.
        • Every Year: Set the task to execute at 12 am on the first of January every year.

If a template is selcted and the date and time settings are modified, then the Custom template is used.

        The scheduler runs only one instance of a particular task. If the task is already running, then the scheduler skips running the task again. For example, if a task is set to run every 1 minute, and the earlier instance is not complete, then the scheduler skips running the task. The scheduled task will be run again at the scheduled time after the current task is complete.

    -   Date and time: Specify the date and the time when the command must be executed. The following fields are available:

        -   **Min**: Specify the time settings in minutes for executing the command.
        -   **Hrs**: Specify the time settings in hours for executing the command.
        -   **DOM**: Specify the day of the month for executing the command.
        -   **Mon**: Specify the month for executing the command.
        -   **DOW**: Specify the day of the week for executing the command.
        Some of the fields also accept the special syntax. For the special syntax, refer [here](#special-syntax).

-   **Task Template**: Select a task template to view and specify the parameters for the scheduled task. The following task templates are available:
    -   **ILM Multi Delete** 
    -   **ILM Multi Export**
    -   **Audit index Rollover**
    -   **Signature Verification**
-   **Priority IPs**: Specify a list of the ESA IP addresses in the order of priority for execution. The task is executed on the first IP address that is specified in this list. If the IP is not available to execute the task, then the job is executed on the next prioritized IP address in the list.
-   **Use Only Priority IPs**: Enable this toggle switch to only execute the task on any one node from the list of the ESA IP addresses specified in the priority field. If this toggle switch is disabled, then the task execution is first attempted on the list of IPs specified in the **Priority IPs** field. If a machine is not available, then the task is run on any machine that is available on the Audit Store cluster which might not be mentioned in the **Priority IPs** field.
-   **Multi node Execution**: If disabled, then the task is run on a single machine. Enable this toggle switch to run the task on all available machines.
-   **Enabled**: Use this toggle switch to enable or disable the task from running as per the schedule.
  1. Specify the parameters for the scheduled task and click Save. The parameters are based on the OR condition. The task is run when any one of the conditions specified is satisfied.

The scheduled task is created and enabled. The job executes on the date and time set.

ILM Multi Delete:

This task is used for automatically deleting indexes when the criteria specified is fulfilled. It displays the required fields for specifying the criteria parameters for deleting indexes. You can use a regex expression for the index pattern.

  • Index Pattern: A regex pattern for specifying the indexes that must be monitored.
  • Max Days: The maximum number of days to retain the index after which they must be deleted. The default is 365 (365 days).
  • Max Docs: The maximum document limit for the index. If the number of docs exceeds this number, then the index is deleted. The default is 1000000000 (1 Billion).
  • Max MB(size): The maximum size of the index in MB. If the size of the index exceeds this number, then the index is deleted. The default is 150000 (150 GB).

Specify one or multiple options for the parameters.

The fields for ILM entries is shown in the following figure.

ILM Multi Export:

This task is used for automatically exporting logs when the criteria specified is fulfilled. It displays the required fields for specifying the criteria parameters for exporting indexes. This task is disabled by default after it is created. Enable the Use Only Priority IPs and specify specific ESA machines in the Priority IPs field this task is created to improve performance. Any indexes imported into ILM are not exported using this scheduled task.

This task is available for processing the audit, troubleshooting, policy log, and protector status indexes.

  • Index Pattern: The pattern for the indexes that must be exported. Use regex to specify multiple indexes.
  • Max Days: The number of days to store indexes. Any index beyond this age is exported. The default age specified is 365 days.
  • Max Docs: The maximum docs present over all the indexes. If the number of docs exceeds this number, then the indexes are exported. The default is 1000000000 (1 Billion).
  • Max MB(size): The maximum size of the index in MB. If the size of the index exceeds this number, then the index is exported. The default is 150000 (150 GB).
  • File password: The password for the exported file. The password is hidden. Keep the password safe. A lost password cannot be retrieved.
  • Retype File password: The password confirmation for the exported file.
  • Dir Path: The directory for storing the exported index in the default path. The default path specified is /opt/protegrity/insight/archive/. You can specify and create nested folders using this parameter. Also, if the directory specified does not exist, then the directory is created in the /opt/protegrity/insight/archive/ directory.

You can specify one or multiple options for the Max Days, Max Docs, and Max MB(size) parameters.

The fields for the entries is shown in the following figure.

Audit Index Rollover:

This task performs an index rollover on the index referred by the alias when any of the specified conditions are fulfilled. The conditions are index age, number of documents in the index, or the index size crosses the specified value.

This task is available for processing the audit, troubleshooting, policy log, protector status, and DSG-related indexes.

  • Max Age: The maximum age after which the index must be rolled over. This default is 30d, that is 30 days. The values supported are, y for years, M for months, w for weeks, d for days, h or H for hours, m for minutes, and s for seconds.
  • Max Docs: The maximum number of docs that an index can contain. An index rollover is performed when this limit is reached. The default is 200000000, that is 200 million.
  • Max Size: The maximum index size of the index that is allowed. An index rollover is performed when the size limit is reached. The default is 5gb. The units supported are, b for bytes, kb for kilobytes, mb for megabytes, gb for gigabytes, tb for terabytes, and pb for petabytes.

The fields for the Audit Index Rollover entries is shown in the following figure.

Signature Verification:

This task runs the signature verification tasks after the time interval that is set. It runs the default signature-related job and the ad-hoc jobs created on the Signature Verification tab.

  • Max Job Idle Time Minutes: The maximum time to keep the jobs idle. After the jobs are idle for the time specified, the idle jobs are cleared and re-queued. The default specified is 2 minutes.
  • Max Parallel Jobs Per Node: The maximum number of signature verification jobs to run in parallel on each system. If number of jobs specified here is reached, then new scheduled jobs are not started. This default is 4 jobs. For example, if 10 jobs are queued to run on 2 ESAs, then 4 jobs are started on the first ESA, 4 jobs are started on the second ESA, and 2 jobs will be queued to run till an ESA job slot gets free to accept and run the queued job.

The fields for the Manage Signature Verification Jobs entries is shown in the following figure.

Working with scheduled tasks

After creating a scheduled task, specify whether the task must be enabled or disabled for running. You can edit the task to modify the commands or the task schedule.

Complete the following steps to modify a task.

  1. From the Analytics screen, navigate to Scheduler > Tasks.

    The list of scheduled tasks appears.

Use the search field to search for a specific task from the list.

  1. Click the Enabled toggle switch to enable or disable the task for running as per the schedule.

    Alternatively, clear the Enabled toggle switch to prevent the task from running as per the schedule.

  2. Click the Edit icon () to update the task.

    The Edit Task page is displayed.

  3. Update the task as required and click Save.

The task is saved and run as per the defined schedule.

Viewing the scheduler monitor

The Monitor screen shows a list of all the scheduled tasks. It also displays whether the task is running or was executed successfully. You can also stop a running task or restart a stopped task from this screen.

Complete the following steps to monitor the tasks.

  1. From the Analytics screen, navigate to Scheduler > Monitor.

    The list of scheduled tasks appears.

The Tail option can be set from the upper-right corner of the screen. Setting the Tail option to ON updates the scheduler history list with the latest scheduled tasks that are run.

You can use the search field to search for specific tasks from the list.
  1. Scroll to view the list of scheduled tasks executed. The following information appears:

    • Name: This is the name of the task that was executed.
    • IP: This is the host IP of the system that executed the task.
    • Start Time: This is the time when the scheduled task started executing.
    • End Time: This is the end time when the scheduled task finished executing.
    • Elapsed Time: This is the execution time in seconds for the scheduled task.
    • State: This is the state displayed for the task. The available states are:
      • : Running. The task is running. You can click Stop from Actions to stop the task.

      • : Queued to stop. The task processing will stop soon.

      • : Stopped. The task has been stopped. The job might take about 20 seconds to stop the process.

        If an ILM Multi Export job is stopped, then the next ILM Multi Export job cannot be started within 2 minutes of stopping a previous running job.

        If a signature verification scheduler job is stopped from the Scheduler > Monitor page, then the status might be updated on this page after about 5 minutes.

      • : Completed. The task is complete.

    • Action: Click Stop to abort the running task. This button is only displayed for tasks that are running.

Using the Index State Management

Use the scheduler and the Analytics ILM for managing indexes. The Index State Management can be used to manage indexes not supported by the scheduler or ILM. However, it is not recommended to use the Index State Management for managing indexes. The Index State Management provides configurations and settings for rotating the index.

Perform the following steps to configure the index:

  1. Log in to the ESA Web UI.
  2. Navigate to Audit Store > Dashboard. The Audit Store Dashboards appears. If a new tab does not automatically open, click Open in a new tab.
  3. Update the index definition.
    1. From the menu, navigate to Index Management.
    2. Click the required index entry.
    3. Click Edit.
    4. Select JSON editor.
    5. Click Continue.
    6. Update the required configuration under rollover.
    7. Click Update.
  4. Update the policy definition for the index.
    1. From the menu, navigate to Index Management.
    2. Click Policy managed indexes.
    3. Select the check box for the index that was updated.
    4. CLick Change Policy.
    5. Select the index from the Managed indices list.
    6. From the State filter, select Rollover.
    7. Select the index from the New policy list.
    8. Ensure that the Keep indices in their current state after the policy takes effect option is selected.
    9. Click Change.

Special syntax

The special syntax for specifying the schedule is provided in the following table.

CharacterDefinitionFieldsExample
,Specifies a list of values.All1, 2, 5, 6.
-Specifies a range of values.All3-5 specifies 3, 4, 5.
/Specifies the values to skip.All*/4 specifies 0, 4, 8, and so on.
*Specifies all values.All* specifies all the values in the field where it is used.
?Specifies no specific value.DOM, DOW4 in the day-of-month field and ? in the day-of-week field specifies to run on the 4th day of the month.
#Specifies the nth day of the month.DOW2#4 specifies 2 for Monday and 4 for 4th week in the month.
LSpecifies the last day in the week or month.DOM, DOW7L specifies the last Saturday in the month.
WSpecifies the weekday closest to the specified day.DOM12W specifies to run on the 12th of the month. If 12 is a Saturday, then run on Friday the 11th. If 12th is a Sunday, then run on Monday the 13th.