Usage Metrics Logging

Describes usage metrics logging in DSG

Usage metrics provide information about the usage of tunnels, services, profiles, and rules. By default, the usage metrics feature is enabled in the gateway.json file.

For more information about the gateway.json file, refer to the section Gateway.json file.

The following snippet shows how the usage metrics feature is enabled in the gateway.json file.

"stats": {
    "enabled": true
}

The metrics are recorded in CSV format in the gateway.log file, and then parsed to JSON and sent to Insight.

For more information about viewing the usage metrics on Insight, refer to the section Forwarding Usage Metrics to the Audit Store.

The logs are emitted at a default interval of 120 seconds. During this 120-second window, all requests processed by the gateway process will be recorded. To modify the time window, configure the usageLogInterval parameter in the stats setting in gateway.json file.

Note:

The time interval is calculated once the gateway restarts.

The usage metrics will only be logged when there is a transaction.

The following table describes the usage metrics for Tunnels.

ParameterData Type in DSGData Type in InsightDescriptionExample
metrics_typestringstringThe metric type is displayed as tunnel.Tunnel
versionintegerintegerA version for the tunnel metric type.0
tunnel_typestringstringThe type of tunnel used to process a request is displayed.HTTP
logtypestringstringThe value to identify type of metric such as dsg_metrics_usage_tunnel.dsg_metrics_usage_tunnel
log_timeepoch_millisdateThe time when the usage is reported.1707242725766
log_intervalintegerlongThe time difference between the current and previous logs.30003
tunnel_idstringstringThe unique ID of the tunnel.t-808715e0-b725-4781-8bf3-429220dd46d5
uptimefloatdoubleThe time in seconds since the tunnel loaded.22670.46733236313
bytes_processedintegerlongThe frontend and backend bytes the tunnel processed since the last time usage was reported.38
frontend_bytes_processedintegerlongThe frontend bytes the tunnel has processed since the last time usage was reported.38
backend_bytes_processedintegerlongThe backend bytes the tunnel has processed since the last time usage was reported.0
total_bytes_processedintegerlongThe total number of frontend and backend bytes the tunnel has processed during the time the tunnel has been loaded.38
total_frontend_bytes_processedintegerlongThe total number of frontend bytes the tunnel has processed during the time the tunnel has been loaded.38
total_backend_bytes-_processedintegerlongThe total number of backend bytes the tunnel has processed during the time the tunnel has been loaded.0
message_countintegerlongThe number of requests the tunnel received since the last time usage was reported.1
total_message_countintegerlongThe total number of requests the tunnel received during the time the tunnel has been loaded.1
origin_time_utc*NAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.416

* -The origin_time_utc parameter will only be displayed on the Audit Store Dashboards.

The following table describes the usage metrics for Services.

ParameterData Type in DSGData Type in InsightDescriptionExample
metrics_typestringstringThe metric type is displayed as Service.Service
versionintegerintegerA version for the service metric type.0
service_typestringstringThe type of service used to process a request is displayed.REST-API
logtypestringstringThe value to identify type of metric such as dsg_metrics_usage_service.dsg_metrics_usage_service
log_timeepoch_millisdateThe time when the usage is reported.1707242 725766
log_intervalintegerlongThe time difference between the current and previous logs.30003
service_idstringstringThe unique ID of the service.s-62a3a161-6bd7-42fa-a9c6-6357d77824ca
parent_idstringstringThe unique ID of the tunnel rule.t-808715e0-b725-4781-8bf3-429220dd46d5
callsintegerlongThe number of times the service processed frontend and backend requests since the time the usage was last reported.38
frontend_callsintegerlongThe number of times the service processed frontend requests since the time the usage was last reported.38
backend_callsintegerlongThe number of times the service processed backend requests since the time the usage was last reported.0
total_callsintegerlongThe total number of times the service processed frontend and backend requests since the service has been loaded.38
total_frontend_callsintegerlongThe total number of times the service processed frontend and backend requests since the service has been loaded.38
total_backend_callsintegerlongThe total number of times the service processed frontend and backend requests since the service has been loaded.0
bytes_processedintegerlongThe frontend and backend bytes the service processed since the last time usage was reported.2
frontend_bytes_processedintegerlongThe frontend bytes the tunnel processed since the last time usage was reported.1
backend_bytes_processedintegerlongThe backend bytes the tunnel processed since the last time usage was reported.1
total_bytes_processedintegerlongThe total number of frontend and backend bytes the service has processed during the time the service has been loaded.2
total_frontend_bytes_processedintegerlongThe total number of frontend bytes the tunnel has processed during the time the tunnel has been loaded.1
total_backend_bytes_processedintegerlongThe total number of backend bytes the tunnel has processed during the time the tunnel has been loaded.1
origin_time_utc*NAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.416

* -The origin_time_utc parameter will only be displayed on the Audit Store Dashboards.

The following table describes the usage metrics for Profile.

ParameterData Type in DSGData Type in InsightDescriptionExample
metrics_typestringstringThe metric type is displayed as Profile.Profile
versionintegerintegerA version for the profile metric type.0
log_timeepoch_millisdateThe time when the usage is reported.1707242725766
log_intervalintegerlongThe time difference between the current and previous logs.339439
logtypestringstringThe value to identify type of metric such as dsg_metrics_usage_profile.dsg_metrics_usage_profile
parent_idstringstringThe unique ID of the service rule.s-62a3a161-6bd7-42fa-a9c6-6357d77824ca
profile_idstringstringThe unique ID of the profile.p-b335795f-8e77-4b15-9ba0-06002cc29bb9
callsintegerlongThe number of times the profile processed a request since the time usage was last reported.1
total_callsintegerlongThe total number of times the profile processed a request since profile has been loaded.1
profile_reference_countintegerlongThe number of times this profile has been called through a profile reference since the time the usage was last reported.0
total_profile_reference_countintegerlongThe total number of times this profile has been called through a profile reference since the profile has been loaded.0
bytes_processedintegerlongThe bytes the profile processed since the last time the usage was reported.38
total_bytes_processedintegerlongThe total bytes the profile processed since the profile has been loaded.38
elapsed_time_sample_countintegerlongThe number of times the profile was sampled since the last time the usage was reported.1
elapsed_time_averageintegerlongThe average amount of time in nano-seconds it took to process a request based on elapsed-time-sample-count.13172454
total_elapsed_time_sample_countintegerlongThe number of times the profile was sampled since the profile has been loaded.1
total_elapsed_time_sample_averageintegerlongThe average amount of time in nano-seconds it took to process a request based on total-elapsed-time-sample-count.13172454
origin_time_utc*NAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.420

* -The origin_time_utc parameter will only be displayed on the Audit Store Dashboards.

The following table describes the usage metrics for Rules.

NameData Type in DSGData Type in InsightDescriptionExample
metrics versionintegerintegerA version for the rule metric type.0
rule-typestringstringThe type of rule used to process a request is displayed.Extract
codecstringstringIt will display the type of payload extracted or the method used for data transformation.Text
logtypestringstringThe value to identify type of metric such as dsg_metrics_usage_rule.dsg_metrics_usage_rule
log_timeepoch_millisdateThe time when the usage is reported.1707242725766
log_intervaldatedateThe time difference between the current and previous logs.22670424
brokenbooleanbooleanIt indicates whether the rule is broken or not.false
domain_name_rewritebooleanbooleanIt indicates whether the rule is domain name rewrite or not.false
rule_idstringstringThe unique ID of the rule.r-38b72f16-f838-4602-81aa-cd881a76e418
parent_idstringstringThe unique ID of the profile rule.p-b335795f-8e77-4b15-9ba0-06002cc29bb9
callsintegerlongThe number of times the rule processed a request since the time the usage was last reported.1
total_callsintegerlongThe total number of times the rule processed a request since rule has been loaded.1
profile_reference_countintegerlongThe number of times this rule has been called via a profile reference since the time the usage was last reported.0
total_profile_reference_countintegerlongThe total number of times this rule has been called via a profile reference since the rule has been loaded.0
bytes_processedintegerlongThe bytes the rule processed since the last time the usage was reported.1
total_bytes_processedintegerlongThe total bytes the rule processed since the rule has been loaded.1
elapsed_time_sample_countintegerlongThe number of times the rule was sampled since the last time usage was reported.1
elapsed_time_sample_averageintegerlongThe average amount of time in nano-seconds it took to process a data based on elapsed-time-sample-count.13137378
total_elapsed_time_sample_countintegerlongThe number of times the rule was sampled since the rule has been loaded.1
total_elapsed_time_sample_averageintegerlongThe average amount of time in nano-seconds it took to process a data based on total-elapsed-time-sample-count.13137378
origin_time_utc*NAdateThe time in UTC at which this log is ingested.Feb 26, 2024 @ 03:51:54.420

* -The origin_time_utc parameter will only be displayed on the Audit Store Dashboards.

Forwarding Usage Metrics to the Audit Store

The usage metrics is also forwarded to the Audit Store and can be viewed on the Audit Store Dashboards.

Ensure that the following prerequisites are met before you view the logs on the Audit Store Dashboards:

  • The Analytics component is initialized on the ESA. The initialization of Analytics is required for displaying the Audit Store information in the Audit Store Dashboards.

    For more information about initializing the Analytics, refer to the section Initializing analytics on the ESA in the Protegrity Installation Guide.

    For more information about the audit indexes, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.

  • The logs are forwarded to the Audit Store.

    For more information about forwarding the logs, refer to the section Forwarding Audit Logs to the Audit Store.

The following figure shows the sample usage metrics on the Discover screen of the Audit Store Dashboards.

Note: The index_node, tiebreaker, and index_time_utc parameters are only logged on the Audit Store Dashboards.
For more information about these parameters, refer to the section Understanding the audit index fields in the Protegrity Insight Guide.

The DSG usage metrics logs are stored in the pty_insight_analytics_dsg_usage_metrics_9.2 index file. You can configure and enable the scheduled task to free up the space used by old index files that you do not require. For usage metrics, edit the Delete DSG Usage Indices task and enable the task. The scheduled task can be set to n days based on your preference.

For more information about scheduled tasks, refer to the section Using the scheduler in the Protegrity Insight Guide 9.2.0.0.