Transaction Metrics Logging

Describes Transaction Metrics Logging in DSG

The transaction metrics allows the user to view the detailed information of the operations performed by the DSG. The Transaction metrics logging feature can be enabled at the service level.

For more information about enabling the transaction metrics logging feature, refer to the Table: Service Fields.

The transaction metrics are logged in the gateway.log file in JSON format.

The sample transaction metrics for a REST request is as seen in the following snippet.

The following table describes the parameters available in the transaction metrics for different services.

ParameterService SupportedData Type in DSGData Type in the Audit StoreDescriptionExamples
auth_cache_hitHTTP, RESTbooleanbooleanThe credential cache status. True indicates that the basic authentication credentials were cached and False indicates that the credentials were not cached.False
auth_end_timeHTTP, RESTstringstringThe timestamp when the basic authentication was completed.2024-02-28T11:26:17.482491732+00:00
auth_start_timeHTTP, RESTstringstringThe timestamp when the basic authentication was started.2024-02-28T11:26:17.466345072+00:00
auth_total_timeHTTP, RESTfloatdoubleThe difference in seconds between the auth_time_end and auth_time_start parameters.0.016147
auth_user_nameHTTP, RESTstringstringThe username used for basic authentication.admin
bucket_nameS3 Out-of-BandstringstringThe name of the S3 bucket from where the DSG reads the object to be processed.dsg-s3/incoming
client_correlation_handleAllstringstringThe ID used to uniquely identify a request. It is usually a UUID. This parameter is optional.31373039313139363333353837
client_ipAllstringstringThe IP address of the client that sent the request to the DSG.192.168.1.10
data_element_nameAllstringstringThe name of the data element used to transform the sensitive data.PTY_Unicode
data_protectionAllobjectobjectThe object representing the Protegrity Data Protection transformation rule.{"data_protection":{"data_elements":[{"data_element_name":"TE_A_N_S13_L1R3_N","num_unprotect":20,"len_unprotect":428}]}}
dsg_versionAllstringstringThe version number of the gateway process.3.1.0.0.103
file_nameS3 Out-of-Band, Mounted Out-ofBandstringstringThe name of the file that has been processed.Sample_S3.csv
http_methodHTTP, RESTstringstringThe HTTP method associated with request.POST
http_outbound_available_clientsHTTPintegerlongThe number of outbound HTTP clients available for the requests.100
http_outbound_count_new_connectionsHTTPintegerlongThe number of new connections created to process the request.
  • A single request per client can be processed. Whenever a connection is created, it is cached and it can be reused.
  • If an existing connection is used, then the value of this parameter is 0.
  • If new connections are created, then the value of this parameter will be greater than 0.
1
http_outbound_count_redirectHTTPintegerlongThe number of redirects encountered while processing a request.0
http_outbound_local_portHTTPintegerintegerThe local port used for the outbound connection.60084
http_outbound_response_codeHTTPintegerintegerThe HTTP status response code from downstream system.200
http_outbound_size_downloadHTTPfloatdoubleThe size of the data received from the downstream system in bytes.76.00
http_outbound_size_queueHTTPfloatdoubleThe number of requests waiting to be sent to downstream systems.0
http_outbound_size_uploadHTTPfloatdoubleThe size of data sent to downstream system in bytes.76.00
http_outbound_speed_downloadHTTPfloatdoubleAverage download speed. Bytes per second.4.00
http_outbound_speed_uploadHTTPfloatdoubleAverage upload speed. Bytes per second.75697.00
http_outbound_time_appconnectHTTPfloatdoubleThe time taken to complete the SSH/TLS handshake.0.000000000
http_outbound_time_connectHTTPfloatdoubleThe time taken to connect to the remote host.0.000374000
http_outbound_time_namelookupHTTPfloatdoubleThe time taken to resolve the name.0.000161000
http_outbound_time_pretransferHTTPfloatdoubleThe time from the start until before the first byte is sent.0.000397000
http_outbound_time_queueHTTPfloatdoubleThe time that the requests spent in the queue before being processed.0.000008821
http_outbound_time_requestHTTPfloatdoubleThe time from when the request was popped off the queue to be processed to the time a response was sent back to the caller.0.001168013
http_outbound_time_starttransferHTTPfloatdoubleThe time taken from the start of the request until the first byte was received from the server.0.000398000
http_outbound_time_totalHTTPfloatdoubleTotal time that the client library took to process the HTTP request.0.001004000
http_outbound_urlHTTPstringstringThe destination URL used for the outbound request.http://tornadoserver:8889/passthrough
http_reason_phraseHTTP, RESTstringstringThe reason phrase associated with the HTTP status code.OK
http_status_codeHTTP, RESTintegerintegerThe HTTP status code sent to the HTTP client.200
input_etagS3 Out-of-BandstringstringThe Etag of the input object processed by the DSG.a0b00e60cc87fff8537e68827c3f329a
input_sizeS3 Out-of-BandintegerlongThe size of the input object, in bytes, processed by the DSG.81
learn_mode_enabledAllbooleanbooleanIndicates if the Learn mode is enabled.false
len_protectAllintegerlongThe length of the sensitive data that is protected.30
local_portHTTP, RESTintegerintegerThe local port used for the inbound connection, can be used with the open_connections parameter to identify new and unique connections.43004
logtype*AllNAstringThe value to identify type of metric, such as, dsg_metrics_transaction.dsg_metrics_transaction
methodSFTPstringstringThe SFTP method associated with the request. The method can be either GET or PUT.download
node_hostnameAllstringstringThe hostname of the DSG.protegrity-cg123
node_pidAllintegerintegerThe process id of the gateway process that processed the request.56532
num_protectAllintegerlongThe number of protect operations performed.3
num_replaceAllintegerlongThe number of regex replace performed.2
open_connectionsHTTP, RESTintegerlongThe number of open connections associated with the tunnel in a process.1
origin_time_utc*AllNAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.416
output_bucket_nameS3 Out-of-BandstringstringThe name of S3 bucket where the DSG writes the processed object.dsg-s3/incoming
output_etagS3 Out-of-BandstringstringEtag of the output object processed by the DSG.a0b00e60cc87fff8537e68827c3f329a
output_file_nameS3 Out-of-BandstringstringThe name of the object that is written to the new S3 bucket (i.e. The value of output_bucket_name parameter) by the DSG.Sample_s3.csv
output_sizeS3 Out-of-BandintegerlongThe size of the object, in bytes, written to the output S3 bucket.81
processing_time_downstreamHTTP, SMTP, SFTPfloatdoubleThe time is the difference between the start time of processing a response and the end time of processing a request.0.003696442
processing_time_requestAllfloatdoubleThe time taken for the ruleset to process the request data.0.001080275
processing_time_responseHTTP, SMTP, SFTP, S3 Out-of-BandfloatdoubleThe time taken for the ruleset to process the response data. It is only applicable to the protocols where a response is expected from a downstream system.0.000162601
regex_replaceAllobjectobjectThe object representing the Regex Replace transformation rule.{"regex_replace":{"replace_rules":[{"rule_name":"Hello -> HELLO","num_replace":6},{"rule_name":"World -> dlroW","num_replace":6}]}}
request_uriHTTP, RESTstringstringThe URI of the request being processed by the DSG.http://httpservice/passthrough
rule_nameAllstringstringThe name of the rule used to transform the sensitive data.Sample Rule1
server_ipSFTPstringstringThe IP address orhostname of the SFTP server that the DSG is communicating with.sftp.server.com
service_nameAllstringstringThe name of the service processing the request.Passthrough
service_typeAllstringstringThe type of the service processing the request.HTTP-GW
time_pre_processingHTTP, RESTfloatdoubleThe time an HTTP or REST request waited before it was processed.0.010870
time_startAlldatedateThe timestamp when the DSG received a request.2024-02-28T11:27:13.515926838+00:00
time_endAlldatedateThe timestamp representing when a request was completed.2024-02-28T11:27:13.519971132+00:00
time_lockS3 Out-of-BandfloatdoubleThe time taken to process the file from the time the lock was created.1708963670.43
time_totalAllfloatdoubleThe difference, in seconds, between the time_end and time_start parameters.0.005429983
transformationsAllobjectobjectThe object representing the Regex Replace and Protegrity Data Protection transformation rules."transformations":{"data_protection":{"data_elements":[{"data_element_name":"TE_A_N_S13_L1R3_N","num_unprotect":20,"len_unprotect":428}]}}
tunnel_nameAllstringstringThe name of the tunnel processing the request.default_80
user_nameAllstringstringThe username used for the protection, unprotection, or reprotection.jack123

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

By default, the normalize-time-labels flag is configured in the features.json file. If the normalize-time-labels flag is configured, then it converts the default timestamp parameters to normalized timestamp parameters, as shown in the Table: Default and Normalized timestamp parameters.

To access the features.json file, navigate to Settings > System > Files, and under the Cloud Gateway - Settings area, access the features.json file.

The following table shows the default timestamp parameters and the normalized timestamp parameters.

Default Timestamp ParametersNormalized Timestamp Parameters
auth_end_timeauth_time_end
auth_start_timeauth_time_start
auth_total_timeauth_time_total
end_timetime_end
start_timetime_start
total_timetime_total
pre_processing_timetime_pre_processing

Forwarding Transaction Metrics to Insight

The transaction metrics is also forwarded to Insight and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

  • The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information on the Audit Store Dashboards.

    For more information about initializing the Analytics, refer to the section Initializing analytics on the ESA in the Protegrity Installation Guide.

    For more information about the audit indexes, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.

  • The logs are forwarded to the Audit Store.

    For more information about forwarding the logs, refer to the section Forwarding Audit Logs to Insight.

The following figure shows the sample transaction metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.
For more information about these parameters, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.

The DSG transaction logs are stored in the pty_insight_analytics_dsg_transaction_metrics_9.2 index file. It is recommended to enable the scheduled task to free up the space used by old index files that you do not require. For transaction metrics, edit the Delete DSG Transaction Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide.

Total Time Breakdown for HTTP Request

This section describes the total time taken for processing the HTTP request.

The total_time value is calculated by adding the time taken by the following parameters:

  • time_pre_processing: The time an HTTP or REST request waited before it was processed.
  • processing_time_request: The time taken for the ruleset to process the request data.
  • processing_time_downstream: The time taken to send a request to a downstream system and receive a response from the client.
  • processing_time_response: The time taken for the ruleset to process the response data.

The following chart depicts the breakdown of the total time taken for an HTTP request.

The processing_time_downstream value is the difference between the start time of processing the response and the end time of processing a request. The processing_time_dowstream is calculated by considering the time taken by any the following parameters:

  • http_outbound_time_queue: The time that the request spent in the queue before being processed.
  • http_outbound_namelookup: The time taken to resolve the name.
  • http_outbound_time_connect: The time taken to connect to the remote host.
  • http_outbound_time_appconnect: The time taken to complete the SSH/TLS handshake.
  • http_outbound_time_pretransfer: The time from the start until before the first byte is sent.
  • http_outbound_time_starttransfer: The time taken from the start of the request until the first byte was received from the server.
  • http_outbound_time_total: Total time that the client library took to process the HTTP request.
  • http_outbound_time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer, and transfer before the final transaction was started.

The following chart depicts the processing time downstream for an HTTP request.

Last modified February 7, 2025