Transaction Metrics Logging
The transaction metrics allows the user to view the detailed information of the operations performed by the DSG. The Transaction metrics logging feature can be enabled at the service level.
For more information about enabling the transaction metrics logging feature, refer to the Table: Service Fields.
The transaction metrics are logged in the gateway.log file in JSON format.
The sample transaction metrics for a REST request is as seen in the following snippet.
The following table describes the parameters available in the transaction metrics for different services.
Parameter | Service Supported | Data Type in DSG | Data Type in the Audit Store | Description | Examples |
---|---|---|---|---|---|
auth_cache_hit | HTTP, REST | boolean | boolean | The credential cache status. True indicates that the basic authentication credentials were cached and False indicates that the credentials were not cached. | False |
auth_end_time | HTTP, REST | string | string | The timestamp when the basic authentication was completed. | 2024-02-28T11:26:17.482491732+00:00 |
auth_start_time | HTTP, REST | string | string | The timestamp when the basic authentication was started. | 2024-02-28T11:26:17.466345072+00:00 |
auth_total_time | HTTP, REST | float | double | The difference in seconds between the auth_time_end and auth_time_start parameters. | 0.016147 |
auth_user_name | HTTP, REST | string | string | The username used for basic authentication. | admin |
bucket_name | S3 Out-of-Band | string | string | The name of the S3 bucket from where the DSG reads the object to be processed. | dsg-s3/incoming |
client_correlation_handle | All | string | string | The ID used to uniquely identify a request. It is usually a UUID. This parameter is optional. | 31373039313139363333353837 |
client_ip | All | string | string | The IP address of the client that sent the request to the DSG. | 192.168.1.10 |
data_element_name | All | string | string | The name of the data element used to transform the sensitive data. | PTY_Unicode |
data_protection | All | object | object | The object representing the Protegrity Data Protection transformation rule. | {"data_protection":{"data_elements":[{"data_element_name":"TE_A_N_S13_L1R3_N","num_unprotect":20,"len_unprotect":428}]}} |
dsg_version | All | string | string | The version number of the gateway process. | 3.1.0.0.103 |
file_name | S3 Out-of-Band, Mounted Out-ofBand | string | string | The name of the file that has been processed. | Sample_S3.csv |
http_method | HTTP, REST | string | string | The HTTP method associated with request. | POST |
http_outbound_available_clients | HTTP | integer | long | The number of outbound HTTP clients available for the requests. | 100 |
http_outbound_count_new_connections | HTTP | integer | long | The number of new connections created to process the request.
| 1 |
http_outbound_count_redirect | HTTP | integer | long | The number of redirects encountered while processing a request. | 0 |
http_outbound_local_port | HTTP | integer | integer | The local port used for the outbound connection. | 60084 |
http_outbound_response_code | HTTP | integer | integer | The HTTP status response code from downstream system. | 200 |
http_outbound_size_download | HTTP | float | double | The size of the data received from the downstream system in bytes. | 76.00 |
http_outbound_size_queue | HTTP | float | double | The number of requests waiting to be sent to downstream systems. | 0 |
http_outbound_size_upload | HTTP | float | double | The size of data sent to downstream system in bytes. | 76.00 |
http_outbound_speed_download | HTTP | float | double | Average download speed. Bytes per second. | 4.00 |
http_outbound_speed_upload | HTTP | float | double | Average upload speed. Bytes per second. | 75697.00 |
http_outbound_time_appconnect | HTTP | float | double | The time taken to complete the SSH/TLS handshake. | 0.000000000 |
http_outbound_time_connect | HTTP | float | double | The time taken to connect to the remote host. | 0.000374000 |
http_outbound_time_namelookup | HTTP | float | double | The time taken to resolve the name. | 0.000161000 |
http_outbound_time_pretransfer | HTTP | float | double | The time from the start until before the first byte is sent. | 0.000397000 |
http_outbound_time_queue | HTTP | float | double | The time that the requests spent in the queue before being processed. | 0.000008821 |
http_outbound_time_request | HTTP | float | double | The time from when the request was popped off the queue to be processed to the time a response was sent back to the caller. | 0.001168013 |
http_outbound_time_starttransfer | HTTP | float | double | The time taken from the start of the request until the first byte was received from the server. | 0.000398000 |
http_outbound_time_total | HTTP | float | double | Total time that the client library took to process the HTTP request. | 0.001004000 |
http_outbound_url | HTTP | string | string | The destination URL used for the outbound request. | http://tornadoserver:8889/passthrough |
http_reason_phrase | HTTP, REST | string | string | The reason phrase associated with the HTTP status code. | OK |
http_status_code | HTTP, REST | integer | integer | The HTTP status code sent to the HTTP client. | 200 |
input_etag | S3 Out-of-Band | string | string | The Etag of the input object processed by the DSG. | a0b00e60cc87fff8537e68827c3f329a |
input_size | S3 Out-of-Band | integer | long | The size of the input object, in bytes, processed by the DSG. | 81 |
learn_mode_enabled | All | boolean | boolean | Indicates if the Learn mode is enabled. | false |
len_protect | All | integer | long | The length of the sensitive data that is protected. | 30 |
local_port | HTTP, REST | integer | integer | The local port used for the inbound connection, can be used with the open_connections parameter to identify new and unique connections. | 43004 |
logtype* | All | NA | string | The value to identify type of metric, such as, dsg_metrics_transaction. | dsg_metrics_transaction |
method | SFTP | string | string | The SFTP method associated with the request. The method can be either GET or PUT. | download |
node_hostname | All | string | string | The hostname of the DSG. | protegrity-cg123 |
node_pid | All | integer | integer | The process id of the gateway process that processed the request. | 56532 |
num_protect | All | integer | long | The number of protect operations performed. | 3 |
num_replace | All | integer | long | The number of regex replace performed. | 2 |
open_connections | HTTP, REST | integer | long | The number of open connections associated with the tunnel in a process. | 1 |
origin_time_utc* | All | NA | date | The time in UTC at which this log is ingested. | Feb 26, 2024 @ 03:51:54.416 |
output_bucket_name | S3 Out-of-Band | string | string | The name of S3 bucket where the DSG writes the processed object. | dsg-s3/incoming |
output_etag | S3 Out-of-Band | string | string | Etag of the output object processed by the DSG. | a0b00e60cc87fff8537e68827c3f329a |
output_file_name | S3 Out-of-Band | string | string | The name of the object that is written to the new S3 bucket (i.e. The value of output_bucket_name parameter) by the DSG. | Sample_s3.csv |
output_size | S3 Out-of-Band | integer | long | The size of the object, in bytes, written to the output S3 bucket. | 81 |
processing_time_downstream | HTTP, SMTP, SFTP | float | double | The time is the difference between the start time of processing a response and the end time of processing a request. | 0.003696442 |
processing_time_request | All | float | double | The time taken for the ruleset to process the request data. | 0.001080275 |
processing_time_response | HTTP, SMTP, SFTP, S3 Out-of-Band | float | double | The time taken for the ruleset to process the response data. It is only applicable to the protocols where a response is expected from a downstream system. | 0.000162601 |
regex_replace | All | object | object | The object representing the Regex Replace transformation rule. | {"regex_replace":{"replace_rules":[{"rule_name":"Hello -> HELLO","num_replace":6},{"rule_name":"World -> dlroW","num_replace":6}]}} |
request_uri | HTTP, REST | string | string | The URI of the request being processed by the DSG. | http://httpservice/passthrough |
rule_name | All | string | string | The name of the rule used to transform the sensitive data. | Sample Rule1 |
server_ip | SFTP | string | string | The IP address orhostname of the SFTP server that the DSG is communicating with. | sftp.server.com |
service_name | All | string | string | The name of the service processing the request. | Passthrough |
service_type | All | string | string | The type of the service processing the request. | HTTP-GW |
time_pre_processing | HTTP, REST | float | double | The time an HTTP or REST request waited before it was processed. | 0.010870 |
time_start | All | date | date | The timestamp when the DSG received a request. | 2024-02-28T11:27:13.515926838+00:00 |
time_end | All | date | date | The timestamp representing when a request was completed. | 2024-02-28T11:27:13.519971132+00:00 |
time_lock | S3 Out-of-Band | float | double | The time taken to process the file from the time the lock was created. | 1708963670.43 |
time_total | All | float | double | The difference, in seconds, between the time_end and time_start parameters. | 0.005429983 |
transformations | All | object | object | The object representing the Regex Replace and Protegrity Data Protection transformation rules. | "transformations":{"data_protection":{"data_elements":[{"data_element_name":"TE_A_N_S13_L1R3_N","num_unprotect":20,"len_unprotect":428}]}} |
tunnel_name | All | string | string | The name of the tunnel processing the request. | default_80 |
user_name | All | string | string | The username used for the protection, unprotection, or reprotection. | jack123 |
* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.
By default, the normalize-time-labels flag is configured in the features.json file. If the normalize-time-labels flag is configured, then it converts the default timestamp parameters to normalized timestamp parameters, as shown in the Table: Default and Normalized timestamp parameters.
To access the features.json file, navigate to Settings > System > Files, and under the Cloud Gateway - Settings area, access the features.json file.
The following table shows the default timestamp parameters and the normalized timestamp parameters.
Default Timestamp Parameters | Normalized Timestamp Parameters |
---|---|
auth_end_time | auth_time_end |
auth_start_time | auth_time_start |
auth_total_time | auth_time_total |
end_time | time_end |
start_time | time_start |
total_time | time_total |
pre_processing_time | time_pre_processing |
Forwarding Transaction Metrics to Insight
The transaction metrics is also forwarded to Insight and can be viewed on the Audit Store Dashboards.
Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:
The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information on the Audit Store Dashboards.
For more information about initializing the Analytics, refer to the section Initializing analytics on the ESA in the Protegrity Installation Guide.
For more information about the audit indexes, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.
The logs are forwarded to the Audit Store.
For more information about forwarding the logs, refer to the section Forwarding Audit Logs to Insight.
The following figure shows the sample transaction metrics on the Discover screen of the Audit Store Dashboards.
Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.
For more information about these parameters, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.
The DSG transaction logs are stored in the pty_insight_analytics_dsg_transaction_metrics_9.2 index file. It is recommended to enable the scheduled task to free up the space used by old index files that you do not require. For transaction metrics, edit the Delete DSG Transaction Indices task and enable the task. The scheduled task can be set to n days based on your preference.
For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide.
Total Time Breakdown for HTTP Request
This section describes the total time taken for processing the HTTP request.
The total_time value is calculated by adding the time taken by the following parameters:
- time_pre_processing: The time an HTTP or REST request waited before it was processed.
- processing_time_request: The time taken for the ruleset to process the request data.
- processing_time_downstream: The time taken to send a request to a downstream system and receive a response from the client.
- processing_time_response: The time taken for the ruleset to process the response data.
The following chart depicts the breakdown of the total time taken for an HTTP request.
The processing_time_downstream value is the difference between the start time of processing the response and the end time of processing a request. The processing_time_dowstream is calculated by considering the time taken by any the following parameters:
- http_outbound_time_queue: The time that the request spent in the queue before being processed.
- http_outbound_namelookup: The time taken to resolve the name.
- http_outbound_time_connect: The time taken to connect to the remote host.
- http_outbound_time_appconnect: The time taken to complete the SSH/TLS handshake.
- http_outbound_time_pretransfer: The time from the start until before the first byte is sent.
- http_outbound_time_starttransfer: The time taken from the start of the request until the first byte was received from the server.
- http_outbound_time_total: Total time that the client library took to process the HTTP request.
- http_outbound_time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer, and transfer before the final transaction was started.
The following chart depicts the processing time downstream for an HTTP request.