Working with Logging Metrics

• 1: Transaction Metrics Logging
• 2: Usage Metrics Logging
• 3: Error Metrics Logging

1 - Transaction Metrics Logging

The transaction metrics allows the user to view the detailed information of the operations performed by the DSG. The Transaction metrics logging feature can be enabled at the service level.

For more information about enabling the transaction metrics logging feature, refer to the Table: Service Fields.

The transaction metrics are logged in the gateway.log file in JSON format.

The sample transaction metrics for a REST request is as seen in the following snippet.

The following table describes the parameters available in the transaction metrics for different services.

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

By default, the normalize-time-labels flag is configured in the features.json file. If the normalize-time-labels flag is configured, then it converts the default timestamp parameters to normalized timestamp parameters, as shown in the Table: Default and Normalized timestamp parameters.

To access the features.json file, navigate to Settings > System > Files, and under the Cloud Gateway - Settings area, access the features.json file.

The following table shows the default timestamp parameters and the normalized timestamp parameters.

Default and Normalized timestamp parameters

Forwarding Transaction Metrics to Audit Store

The transaction metrics are also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

• The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information on the Audit Store Dashboards.

  For more information about initializing Analytics, refer Initializing Analytics on the ESA.

  For more information about the audit indexes, refer to the section Understanding the audit index fields.

• The logs are forwarded to the Audit Store.

  For more information about forwarding the logs, refer to the section Forwarding Audit Logs to Insight.

The following figure shows the sample transaction metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.

The DSG transaction logs are stored in the pty_insight_analytics_dsg_transaction_metrics_9.2 index file. It is recommended to enable the scheduled task to free up the space used by old index files that you do not require. For transaction metrics, edit the Delete DSG Transaction Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler.

Total Time Breakdown for HTTP Request

This section describes the total time taken for processing the HTTP request.

The total_time value is calculated by adding the time taken by the following parameters:

• time_pre_processing: The time an HTTP or REST request waited before it was processed.
• processing_time_request: The time taken for the ruleset to process the request data.
• processing_time_downstream: The time taken to send a request to a downstream system and receive a response from the client.
• processing_time_response: The time taken for the ruleset to process the response data.

The following chart depicts the breakdown of the total time taken for an HTTP request.

The processing_time_downstream value is the difference between the start time of processing the response and the end time of processing a request. The processing_time_dowstream is calculated by considering the time taken by any the following parameters:

• http_outbound_time_queue: The time that the request spent in the queue before being processed.
• http_outbound_namelookup: The time taken to resolve the name.
• http_outbound_time_connect: The time taken to connect to the remote host.
• http_outbound_time_appconnect: The time taken to complete the SSH/TLS handshake.
• http_outbound_time_pretransfer: The time from the start until before the first byte is sent.
• http_outbound_time_starttransfer: The time taken from the start of the request until the first byte was received from the server.
• http_outbound_time_total: Total time that the client library took to process the HTTP request.
• http_outbound_time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer, and transfer before the final transaction was started.

The following chart depicts the processing time downstream for an HTTP request.

2 - Usage Metrics Logging

Usage metrics provide information about the usage of tunnels, services, profiles, and rules. By default, the usage metrics feature is enabled in the gateway.json file.

For more information about the gateway.json file, refer to the section Gateway.json file.

The following snippet shows how the usage metrics feature is enabled in the gateway.json file.

"stats": {
    "enabled": true
}

The metrics are recorded in CSV format in the gateway.log file, and then parsed to JSON and sent to Insight.

For more information about viewing the usage metrics on Insight, refer to the section Forwarding Usage Metrics to Insight.

The logs are emitted at a default interval of 120 seconds. During this 120-second window, all requests processed by the gateway process will be recorded. To modify the time window, configure the usageLogInterval parameter in the stats setting in gateway.json file.

Note:

The time interval is calculated once the gateway restarts.

The usage metrics will only be logged when there is a transaction.

The following table describes the usage metrics for Tunnels.

* -The origin_time_utc parameter will only be displayed on the Insight Dashboards.

The following table describes the usage metrics for Services.

* -The origin_time_utc parameter will only be displayed on the Insight Dashboards.

The following table describes the usage metrics for Profile.

* -The origin_time_utc parameter will only be displayed on the Insight Dashboards.

The following table describes the usage metrics for Rules.

* -The origin_time_utc parameter will only be displayed on the Insight Dashboards.

Forwarding Usage Metrics to the Audit Store

The usage metrics is also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

• The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information on the Audit Store Dashboards.

  For more information about initializing Analytics, refer Initializing Analytics on the ESA.

  For more information about the audit indexes, refer to the section Understanding the audit index fields.

• The logs are forwarded to Insight.

  For more information about forwarding the logs, refer to the section Forwarding Audit Logs to Insight.

The following figure shows the sample usage metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.

The DSG usage metrics logs are stored in the pty_insight_analytics_dsg_usage_metrics_9.2 index file. You can configure and enable the scheduled task to free up the space used by old index files that you do not require. For usage metrics, edit the Delete DSG Usage Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler.

3 - Error Metrics Logging

Error Metrics allow a user to view details about errors, which are encountered while processing a file. The error metrics logging feature can be enabled at the service level.

When the Protegrity Data Protection Transform Rule is configured, certain error conditions can occur during processing. The following are some of the error conditions:

• The input is too short or long for a particular data element
• Invalid Email ID
• Invalid Data Type
• Invalid Credit Card Details

When these error conditions occur, the rule will stop processing. The permissive error handling feature is hence used to handle the errors and process the erroneous input file.

For more information about permissive error handling, refer to the Table: Protegrity Data Protection Method.

If there are a lot of erroneous data in an input file, it can be difficult to identify and categorize errors. In that situation, the error metrics can be used to understand the total number of errors, the offset of where the error was encountered, the reasons why the error was encountered, the ruleset details, and so on.

Error metrics is written to the gateway.log file and the Log Viewer screen in the JSON format.

In the case of NFS and CIFS protocols, a lock file is created for each file that is to be processed. Error metrics will also be appended to the lock files alongside the update details for each file. For example, if there are ten files to be processed, namely, Test1 to Test10, then ten respective lock files will be created, namely, Test1.lock to Test10.lock. If there are any errors encountered in the Test1 file, then the error metrics for this file will be appended to the Test1.lock file.

Important: The error metrics is only supported for the following payloads:

• CSV Payload
• Fixed Width
  For more information about the CSV and Fixed Width payloads, refer to the sections CSV Payload and Fixed Width.

Important: The error metrics support is only available for the following services:

• HTTP
• REST
• NFS
• CIFS

Important: The following conditions should be met to use the error metrics logging feature:

• Users must use the Protegrity Data Protection method to transform the data.
• Permissive error handling should be configured at the transform rule.
• Error Metrics Logging field must be enabled at the service level.

Note: If the permissive error handling is disabled or a different transformation method is used, then the total_error_count will be 1 in the error metrics.

The sample error metrics for the REST request is as seen in the following log file snippet:

The following table describes the parameters available in the error metrics for different services.

"column_info": {
"1": {
"column_start": 3,
"column_width": 17
}
},

{"column_info":{"2":"first_name"}

reasons":[{"reason":"The input is too short (returnCode 0)","rulesets":[{"ruleset":"CSV Rule","offset":[{"columns":[{"2":{"rows":[2,5,8,11,14,17,20],"trimmed":false}}]}],"error_count":7}]}]

Forwarding Error Metrics to Insight

The error metrics is also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

• The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information in the Audit Store Dashboards.

  For more information about initializing Analytics, refer Initializing Analytics on the ESA.

  For more information about the audit indexes, refer to the section Understanding the audit index fields.

• The logs are forwarded to Insight.

  For more information about forwarding the logs, refer to the section Forwarding Audit Logs to Insight.

The following figure shows the sample error metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.

The DSG error metrics logs are stored in the pty_insight_analytics_dsg_error_metrics_9.2 index file. You can configure and enable the scheduled task to free up the space used by old index files that you do not require. For error metrics, edit the Delete DSG Error Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide 9.2.0.0.

Error Metrics with Non-Permissive Errors

This section describes how the non-permissive errors are logged in the error metrics.

When the permissive error handling is disabled and an error is encountered while transforming data, it will capture the error once and stop processing the entire file. The error could be in the input file, ruleset, or any other configuration. In such scenarios, the error metrics will always be logged with a total_error_count = 1.

The sample error metrics with non-permissive errors are as seen in the following log file snippet:

Configuring the HTTP Status Codes

This section describes how to configure the HTTP status codes for the errors that may occur while processing a file.

When errors are encountered and the user wants to handle them permissively, then different HTTP status codes can be configured in the Error Code field from the DSG Web UI. At the service level of the RuleSet page, an Error Code field is added for the HTTP and REST protocols to handle errors permissively.

Note: The Error Code field is not supported for the NFS and CIFS protocols.

The following are the HTTP status codes that can be configured from the Web UI:

• 200 OK
• 201 Created
• 202 Accepted
• 203 Non-Authoritative Information
• 205 Reset Content
• 206 Partial Content
• 400 Bad Request
• 401 Unauthorized
• 403 Forbidden
• 422 Unprocessable Entity
• 500 Internal Server Error
• 503 Service Unavailable

Note: By default, the Error Code is set to 200 OK.

The error metrics options for different protocols are as seen in the following figures: