Error metrics logging

Describes error metrics logging in DSG

Error Metrics allow a user to view details about errors, which are encountered while processing a file. The error metrics logging feature can be enabled at the service level.

When the Protegrity Data Protection Transform Rule is configured, certain error conditions can occur during processing. The following are some of the error conditions:

  • The input is too short or long for a particular data element
  • Invalid Email ID
  • Invalid Data Type
  • Invalid Credit Card Details

When these error conditions occur, the rule will stop processing. The permissive error handling feature is hence used to handle the errors and process the erroneous input file.

For more information about permissive error handling, refer to the Table: Protegrity Data Protection Method.

If there are a lot of erroneous data in an input file, it can be difficult to identify and categorize errors. In that situation, the error metrics can be used to understand the total number of errors, the offset of where the error was encountered, the reasons why the error was encountered, the ruleset details, and so on.

Error metrics is written to the gateway.log file and the Log Viewer screen in the JSON format.

In the case of NFS and CIFS protocols, a lock file is created for each file that is to be processed. Error metrics will also be appended to the lock files alongside the update details for each file. For example, if there are ten files to be processed, namely, Test1 to Test10, then ten respective lock files will be created, namely, Test1.lock to Test10.lock. If there are any errors encountered in the Test1 file, then the error metrics for this file will be appended to the Test1.lock file.

Important: The error metrics is only supported for the following payloads:

  • CSV Payload
  • Fixed Width
    For more information about the CSV and Fixed Width payloads, refer to the sections CSV Payload and Fixed Width.

Important: The error metrics support is only available for the following services:

  • HTTP
  • REST
  • NFS
  • CIFS

Important: The following conditions should be met to use the error metrics logging feature:

  • Users must use the Protegrity Data Protection method to transform the data.
  • Permissive error handling should be configured at the transform rule.
  • Error Metrics Logging field must be enabled at the service level.

Note: If the permissive error handling is disabled or a different transformation method is used, then the total_error_count will be 1 in the error metrics.

The sample error metrics for the REST request is as seen in the following log file snippet:

Error Metrics logged

The following table describes the parameters available in the error metrics for different services.

ParameterServices SupportedData Type in DSGData Type in InsightsDescriptionExample
column_infoHTTP, REST, NFS, CIFSinteger, stringinteger, stringFor the CSV payload, a list of column numbers and column names will be logged when the errors are encountered.
Note
The Header will be taken from the CSV or the Fixed Width extract rule.
While configuring the CSV extract rule, if the Header is set to -1, then the column_info parameter will not be logged in the error metrics.
For the Fixed Width payload, a list of column numbers, the column start value, and the column width will be logged when the errors are encountered.
The following snippets shows the column_info parameter for the Fixed Width payload:
"column_info": {
"1": {
"column_start": 3,
"column_width": 17
}
},
{"column_info":{"2":"first_name"}
columnsHTTP, REST, NFS, CIFSintegerlongThe column numbers where the error is encountered.2
error_countHTTP, REST, NFS, CIFSintegerlongThe total number of errors encountered for a particular reason.1
file_nameNFS, CIFSstringstringThe name of the file that is being processed by the DSG.Sample_NFS.csv
idHTTP, REST, NFS, CIFSstringstringA unique ID to identify the transaction.a272d51b14df435eb67fdf46d2ecff83
logtype*AllNAstringThe value to identify type of metric such as dsg_metrics_error.dsg_metrics_error
node_hostnameHTTP, REST, NFS, CIFSstringstringThe hostname of the DSG.protegrity-cg123
node_pidHTTP, REST, NFS, CIFSintegerintegerThe Process ID of the gateway process that processed request.5577
origin_time_utc*AllNAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.416
reasonsAllobjectobjectThe object representing the set of error reason, ruleset details, offset of columns or rows, and error count.
reasons":[{"reason":"The input is too short (returnCode 0)","rulesets":[{"ruleset":"CSV Rule","offset":[{"columns":[{"2":{"rows":[2,5,8,11,14,17,20],"trimmed":false}}]}],"error_count":7}]}]
reasonHTTP, REST, NFS, CIFSstringstringThe reason for a particular error will be displayed.The input is too short (returnCode 0)
request_uriHTTP, RESTstringstringThe URI of the request being processed by the DSG.http://testservice:8081/echo
rowsHTTP, REST, NFS, CIFSintegerintegerThe row numbers where the error is encountered.0
rulesetHTTP, REST, NFS, CIFSstringstringThe traversal of the transform rule, which induces the error.Text Protection/Word by word data extraction/Data Protection
service_nameHTTP, REST, NFS, CIFSstringstringThe name of the service processing the request.REST Demo Service
time_endHTTP, REST, NFS, CIFSstringdateThe timestamp representing when a request was completed.2024-02-28T11:48:26.318685532+00:00
total_error_countHTTP, REST, NFS, CIFSintegerlongThe total number of errors encountered while processing a request.1
time_startHTTP, REST, NFS, CIFSstringdateThe timestamp when the DSG received a request.2024-02-28T11:48:18.148773909+00:00
total_timeHTTP, REST, NFS, CIFSfloatdoubleThe difference in seconds between the time_end and time_start parameters.8.17
trimmedHTTP, REST, NFS, CIFSbooleanbooleanTrue indicates that the error metrics is trimmed. The trimming of an error metrics depend on the checkErrorLogAfterCount parameter, that is configurable in the gateway.json file.
For more information about the checkErrorLogAfterCount parameter, refer to the Table: gateway.json configurations in the Protegrity Data Security Gateway User Guide 3.2.0.0.
false
tunnel_nameHTTP, REST, NFS, CIFSstringstringThe name of the tunnel processing the request.Tunnel_8081

Forwarding Error Metrics to the Audit Store

The error metrics is also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

  • The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information in the Audit Store Dashboards.

    For more information about initializing the Analytics, refer to the section Initializing analytics on the ESA in the Protegrity Installation Guide.

    For more information about the audit indexes, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.

  • The logs are forwarded to the Audit Store.

    For more information about forwarding the logs, refer to the section Forwarding Audit Logs to the Audit Store.

The following figure shows the sample error metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.
For more information about these parameters, refer to the section Understanding the audit index fields in the Protegrity Insight Guide 9.2.0.0.

The DSG error metrics logs are stored in the pty_insight_analytics_dsg_error_metrics_9.2 index file. You can configure and enable the scheduled task to free up the space used by old index files that you do not require. For error metrics, edit the Delete DSG Error Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide 9.2.0.0.

Error Metrics with Non-Permissive Errors

This section describes how the non-permissive errors are logged in the error metrics.

When the permissive error handling is disabled and an error is encountered while transforming data, it will capture the error once and stop processing the entire file. The error could be in the input file, ruleset, or any other configuration. In such scenarios, the error metrics will always be logged with a total_error_count = 1.

The sample error metrics with non-permissive errors are as seen in the following log file snippet:

Error Metrics logged

Configuring the HTTP Status Codes

This section describes how to configure the HTTP status codes for the errors that may occur while processing a file.

When errors are encountered and the user wants to handle them permissively, then different HTTP status codes can be configured in the Error Code field from the DSG Web UI. At the service level of the RuleSet page, an Error Code field is added for the HTTP and REST protocols to handle errors permissively.

Note: The Error Code field is not supported for the NFS and CIFS protocols.

The following are the HTTP status codes that can be configured from the Web UI:

  • 200 OK
  • 201 Created
  • 202 Accepted
  • 203 Non-Authoritative Information
  • 205 Reset Content
  • 206 Partial Content
  • 400 Bad Request
  • 401 Unauthorized
  • 403 Forbidden
  • 422 Unprocessable Entity
  • 500 Internal Server Error
  • 503 Service Unavailable

Note: By default, the Error Code is set to 200 OK.

The error metrics options for different protocols are as seen in the following figures: