Working with alerts

Use alerting to keep track of the different activities that take place on the system. The alerting ecosystem consists of the monitor, trigger, action, and channels.

Viewing alerts

Generated alerts are displayed on the Insight Dashboard. View and acknowledge the alerts from the alerting dashboard by navigating to OpenSearch Plugins > Alerting > Alerts.

For more information about working with Monitors, Alerts, and Notifications, refer to Monitors in OpenSearch Dashboards.

Creating notifications

Create notification channels to receive alerts as per individual requirements. The alerts are sent to the destination specified in the channel.

Creating a custom webhook notification

A webhook notification sends the alerts generated by a monitor to a destination, such as, a web page.

Perform the following steps to configure the notification channel for generating webhook alerts:

  1. Log in to the Web UI.

  2. From the menu, navigate to Management > Notifications > Channels.

  3. Click Create channel.

  4. Specify the following information under Name and Description:

    • Name: Http_webhook
    • Description: For generating http webhook alerts.

  5. Specify the following information under Configurations:

    • Channel type: Custom webhook
    • Method: POST
    • Define endpoints by: Webhook URL
    • Webhook URL: Specify the URL that receives the alert. For example https://webhook.site/9385a259-3b82-4e99-ad1e-1eb875f00734.
    • Webhook headers: Specify the key value pairs for the webhook.

  6. Click Send test message to send a message to the email recipients.

  7. Click Create to create the channel.

    The webhook is set up successfully.

  8. Create a monitor and attach the channel created using the steps from the section Creating the monitor.

Creating email alerts using custom webhook

An email notification sends alerts generated by a monitor to an email address. It is also possible to configure the SMTP channel for sending an email alert. The email alerts can be encrypted or non-encrypted. Accordingly, the required SMTP settings for email notifications must be configured.

Ensure that the following is configured as per the requirement:

  1. Ensure that the following prerequisites are met.

    • Outbound SMTP access is enabled.
    • Required SMTP port is open, for example, 587 for STARTTLS.
    • Firewall and routing configurations allow SMTP traffic.

  2. Log in to the CLI to configure the email service. For more information about using the CLI commands, refer to Administrator Command Line Interface (CLI) Reference.

  3. Verify if any email service is already configured.

admin get email
  1. Configure the email service.
admin set email -h "email_provider" -p <port> --use-tls -u "<username>" -w "<password>"
  1. Send a test email message.
admin test email -f "<senders_email>" -t "<receivers_email>" -s "Test" -b "This is a test."

  1. Log in to the Web UI.

  2. From the menu, navigate to OpenSearch Plugins > Notifications > Channels.

  3. Click Create channel.

  4. Specify the following information under Name and Description:

    • Name: send_email_with_certs_alerts
    • Description: For secure SMTP alerts.

  5. Specify the following information under Configurations:

-   **Channel type**: **Custom webhook**
-   **Webhook URL**: `http://pty-smtp-service.email-service.svc.cluster.local:8000/api/v1/email/send`
  1. Under Webhook headers, click Add header and specify the following information:
-   **Key**: **Pty-Username**
-   **Value**: `%internal_scheduler;`
  1. Under Webhook headers, click Add header and specify the following information:
-   **Key**: **Pty-Roles**
-   **Value**: **auditstore_admin**

  1. Click Create to save the channel configuration.

    Caution: Do not click Send test message because the configuration for the channel is not complete.

    The success message appears and the channel is created. The webhook for the email alerts is set up successfully.

  2. Create a monitor and attach the channel created using the steps from the section Creating the monitor.

Forwarding alerts to a local file

Complete the configuration provided in this section to send the logs to the alerting module. The logs are saved in the \fluentd\log directory.

  1. Log in to the jumpbox.
  2. Navigate to a directory for working with configuration files.
  3. Run the following command to update the fluent.conf file.
kubectl get configmap standalone-fluentd-config -n pty-insight -o jsonpath='{.data.fluent\.conf}' > fluent.conf
  1. Update the following code at the start of the file.
<source>
    @type http
    bind "0.0.0.0"
    port 24284
    <parse>
        @type "json"
    </parse>
</source>
  1. Locate the following code.
<match *.*.* logdata flulog>
  1. Replace the text identified in the earlier step with the following code to process all the data.
<match **>
  1. Add the following code before the closing </match>tag to output the content to a file.
<store>
@type "file"
path "/fluentd/log/buffer"
append true
<buffer time>
    path "/fluentd/log/buffer"
</buffer>
</store>
  1. Run the following command to load the new configuration.
kubectl create configmap standalone-fluentd-config  -n pty-insight     --from-file=fluent.conf     --dry-run=client -o yaml > standalone-fluentd-config-new.yaml
  1. Run the following command to load the configuration.
kubectl replace -f standalone-fluentd-config-new.yaml -n pty-insight
  1. Run the following command to generate the standalone-fluentd-deployment.yaml file.
kubectl get deployment standalone-fluentd -n pty-insight -o yaml > standalone-fluentd-deployment.yaml
  1. Open the standalone-fluentd-deployment.yaml file.
  2. Locate the following code.
spec:
  containers:
  - args:
    - |
    export GEM_HOME="$HOME/.local/gems" && \
    export PATH="$GEM_HOME/bin:$PATH" && \
    gem install fluent-plugin-opensearch --no-document --user-install && \
    fluentd -c /fluentd/etc/fluent.conf -v
  1. Add the following fluent-plugin-http file in the code to install the required .gem file.
spec:
  containers:
  - args:
    - |
    export GEM_HOME="$HOME/.local/gems" && \
    export PATH="$GEM_HOME/bin:$PATH" && \
    gem install fluent-plugin-opensearch fluent-plugin-http --no-document --user-install && \
    fluentd -c /fluentd/etc/fluent.conf -v
  1. Add the following code to the volumeMounts: parameter. Append the mount path at the end retaining the current volume mounts.
        volumeMounts:
        - mountPath: /fluentd/etc/
          name: standalone-fluentd-config
  1. Locate the following code.
     volumes:
      - configMap:
          defaultMode: 420
          name: standalone-fluentd-config
        name: standalone-fluentd-config
      - name: tls-for-insight-key-pair
        secret:
          defaultMode: 420
          secretName: tls-for-insight-key-pair
  1. Update the code to add the directory details to the configuration file.
     volumes:
      - configMap:
          defaultMode: 420
          name: standalone-fluentd-config
        name: standalone-fluentd-config
      - name: tls-for-insight-key-pair
        secret:
          defaultMode: 420
          secretName: tls-for-insight-key-pair
      - emptyDir: {}
        name: fluentd-log
  1. Apply the configurations using the following command.
kubectl apply -f standalone-fluentd-deployment.yaml
  1. Process the configurations using the following command.
kubectl rollout restart deployment standalone-fluentd -n pty-insight
  1. Verify that the pods are running.
kubectl get pods -n pty-insight
  1. Proceed to create a monitor using the steps from Creating the monitor.

Creating the monitor

A monitor tracks the system and sends an alert when a trigger is activated. Triggers cause actions to occur when certain criteria are met. Those criteria are set when a trigger is created. For more information about monitors, actions, and triggers, refer to Alerting.

Perform the following steps to create a monitor. The configuration specified here is just an example. For real use, create whatever configuration is needed per individual requirements:

  1. Ensure that a notification is created using the steps from Creating notifications.

  2. From the menu, navigate to OpenSearch Plugins > Alerting > Monitors.

  3. Click Create Monitor.

  4. Specify a name for the monitor.

  5. For the Monitor defining method, select Extraction query editor.

  6. For the Schedule, select 30 Minutes.

  7. For the Index, select the required index.

  8. Specify the following query for the monitor. Modify the query as per the requirement.

    {
    "size": 0,
    "query": {
        "match_all": {
            "boost": 1
        }
    }
}

  9. Click Add trigger and specify the information provided here.

    1. Specify a trigger name.

    2. Specify a severity level.

    3. Specify the following code for the trigger condition:

      ctx.results[0].hits.total.value > 0

  10. Click Add action.

  11. From the Channels list, select the required channel.

  12. Add the following code in the Message field. The default message displayed might not be formatted properly. Update the message by replacing the Line spaces with the n escape code. The message value is a JSON value. Use escape characters to structure the email properly using valid JSON syntax.

```
{
"message": "Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}",
"subject": "Monitor {{ctx.monitor.name}} just entered alert status"
}
```
>	**Note:** The **message** value is a JSON value. Be sure to use escape characters to structure the email properly using valid JSON syntax. The default message displayed might not be formatted properly. Update the message by replacing the Line spaces with the **\\n** escape code.
  1. Select the Preview message check box to view the formatted email message.
  2. Click Send test message and verify the recipient’s inbox for the message.
  3. Click Save to update the configuration.

Last modified : April 06, 2026