Error metrics logging
Error Metrics allow a user to view details about errors, which are encountered while processing a file. The error metrics logging feature can be enabled at the service level.
When the Protegrity Data Protection Transform Rule is configured, certain error conditions can occur during processing. The following are some of the error conditions:
- The input is too short or long for a particular data element
- Invalid Email ID
- Invalid Data Type
- Invalid Credit Card Details
When these error conditions occur, the rule will stop processing. The permissive error handling feature is hence used to handle the errors and process the erroneous input file.
For more information about permissive error handling, refer to the Table: Protegrity Data Protection Method.
If there are a lot of erroneous data in an input file, it can be difficult to identify and categorize errors. In that situation, the error metrics can be used to understand the total number of errors, the offset of where the error was encountered, the reasons why the error was encountered, the ruleset details, and so on.
Error metrics is written to the gateway.log file and the Log Viewer screen in the JSON format.
In the case of NFS and CIFS protocols, a lock file is created for each file that is to be processed. Error metrics will also be appended to the lock files alongside the update details for each file. For example, if there are ten files to be processed, namely, Test1 to Test10, then ten respective lock files will be created, namely, Test1.lock to Test10.lock. If there are any errors encountered in the Test1 file, then the error metrics for this file will be appended to the Test1.lock file.
Important: The error metrics is only supported for the following payloads:
- CSV Payload
- Fixed Width
For more information about the CSV and Fixed Width payloads, refer to the sections CSV Payload and Fixed Width.
Important: The error metrics support is only available for the following services:
- HTTP
- REST
- NFS
- CIFS
Important: The following conditions should be met to use the error metrics logging feature:
- Users must use the Protegrity Data Protection method to transform the data.
- Permissive error handling should be configured at the transform rule.
- Error Metrics Logging field must be enabled at the service level.
Note: If the permissive error handling is disabled or a different transformation method is used, then the total_error_count will be 1 in the error metrics.
The sample error metrics for the REST request is as seen in the following log file snippet:
The following table describes the parameters available in the error metrics for different services.
Parameter | Services Supported | Data Type in DSG | Data Type in Insights | Description | Example |
column_info | HTTP, REST, NFS, CIFS | integer, string | integer, string | For the CSV payload, a list of column numbers and column
names will be logged when the errors are encountered.NoteThe
Header will be taken from the CSV or the Fixed Width extract
rule. While configuring the CSV extract rule, if the Header is
set to -1, then the column_info parameter
will not be logged in the error metrics.The following snippets shows the
column_info parameter for the Fixed Width
payload:"column_info": { "1": { "column_start": 3, "column_width": 17 } }, | {"column_info":{"2":"first_name"} |
columns | HTTP, REST, NFS, CIFS | integer | long | The column numbers where the error is encountered. | 2 |
error_count | HTTP, REST, NFS, CIFS | integer | long | The total number of errors encountered for a particular reason. | 1 |
file_name | NFS, CIFS | string | string | The name of the file that is being processed by the DSG. | Sample_NFS.csv |
id | HTTP, REST, NFS, CIFS | string | string | A unique ID to identify the transaction. | a272d51b14df435eb67fdf46d2ecff83 |
logtype* | All | NA | string | The value to identify type of metric such as
dsg_metrics_error . | dsg_metrics_error |
node_hostname | HTTP, REST, NFS, CIFS | string | string | The hostname of the DSG. | protegrity-cg123 |
node_pid | HTTP, REST, NFS, CIFS | integer | integer | The Process ID of the gateway process that processed request. | 5577 |
origin_time_utc* | All | NA | date | The time in UTC at which this log is ingested. | Feb 26, 2024 @ 03:51:54.416 |
reasons | All | object | object | The object representing the set of error reason, ruleset details, offset of columns or rows, and error count. | reasons":[{"reason":"The input is too short (returnCode 0)","rulesets":[{"ruleset":"CSV Rule","offset":[{"columns":[{"2":{"rows":[2,5,8,11,14,17,20],"trimmed":false}}]}],"error_count":7}]}] |
reason | HTTP, REST, NFS, CIFS | string | string | The reason for a particular error will be displayed. | The input is too short (returnCode 0) |
request_uri | HTTP, REST | string | string | The URI of the request being processed by the DSG. | http://testservice:8081/echo |
rows | HTTP, REST, NFS, CIFS | integer | integer | The row numbers where the error is encountered. | 0 |
ruleset | HTTP, REST, NFS, CIFS | string | string | The traversal of the transform rule, which induces the error. | Text Protection/Word by word data extraction/Data Protection |
service_name | HTTP, REST, NFS, CIFS | string | string | The name of the service processing the request. | REST Demo Service |
time_end | HTTP, REST, NFS, CIFS | string | date | The timestamp representing when a request was completed. | 2024-02-28T11:48:26.318685532+00:00 |
total_error_count | HTTP, REST, NFS, CIFS | integer | long | The total number of errors encountered while processing a request. | 1 |
time_start | HTTP, REST, NFS, CIFS | string | date | The timestamp when the DSG received a request. | 2024-02-28T11:48:18.148773909+00:00 |
total_time | HTTP, REST, NFS, CIFS | float | double | The difference in seconds between the
time_end and time_start
parameters. | 8.17 |
trimmed | HTTP, REST, NFS, CIFS | boolean | boolean | True indicates that the error metrics is trimmed. The
trimming of an error metrics depend on the
checkErrorLogAfterCount parameter, that is
configurable in the gateway.json file.For
more information about the
checkErrorLogAfterCount parameter, refer to
the Table: gateway.json configurations in the
Protegrity Data Security Gateway User Guide
3.2.0.0. | false |
tunnel_name | HTTP, REST, NFS, CIFS | string | string | The name of the tunnel processing the request. | Tunnel_8081 |
Forwarding Error Metrics to the Audit Store
The error metrics is also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.
Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:
The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information in the Audit Store Dashboards.
For more information about initializing the Analytics, refer to the section Initializing analytics on the ESA in the Protegrity Installation Guide.
For more information about the audit indexes, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.
The logs are forwarded to the Audit Store.
For more information about forwarding the logs, refer to the section Forwarding Audit Logs to the Audit Store.
The following figure shows the sample error metrics on the Discover screen of the Audit Store Dashboards.
Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.
For more information about these parameters, refer to the section Understanding the audit index fields in the Protegrity Insight Guide 9.2.0.0.
The DSG error metrics logs are stored in the pty_insight_analytics_dsg_error_metrics_9.2 index file. You can configure and enable the scheduled task to free up the space used by old index files that you do not require. For error metrics, edit the Delete DSG Error Indices task and enable the task. The scheduled task can be set to n days based on your preference.
For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide 9.2.0.0.
Error Metrics with Non-Permissive Errors
This section describes how the non-permissive errors are logged in the error metrics.
When the permissive error handling is disabled and an error is encountered while transforming data, it will capture the error once and stop processing the entire file. The error could be in the input file, ruleset, or any other configuration. In such scenarios, the error metrics will always be logged with a total_error_count = 1.
The sample error metrics with non-permissive errors are as seen in the following log file snippet:
Configuring the HTTP Status Codes
This section describes how to configure the HTTP status codes for the errors that may occur while processing a file.
When errors are encountered and the user wants to handle them permissively, then different HTTP status codes can be configured in the Error Code field from the DSG Web UI. At the service level of the RuleSet page, an Error Code field is added for the HTTP and REST protocols to handle errors permissively.
Note: The Error Code field is not supported for the NFS and CIFS protocols.
The following are the HTTP status codes that can be configured from the Web UI:
- 200 OK
- 201 Created
- 202 Accepted
- 203 Non-Authoritative Information
- 205 Reset Content
- 206 Partial Content
- 400 Bad Request
- 401 Unauthorized
- 403 Forbidden
- 422 Unprocessable Entity
- 500 Internal Server Error
- 503 Service Unavailable
Note: By default, the Error Code is set to
200 OK
.
The error metrics options for different protocols are as seen in the following figures: