Index entries

The index configuration, samples, and entry descriptions describe help identify and analyze the log entries in the indexes.

Audit index

The log types of protection, metering, audit, and security are stored in the audit index. These log are generated during security operations. The logs generated by protectors are stored in the audit index with the name as shown in the following table for the respective version.

ESA versionIndex patternDescriptionExample
ESA v10.1.0pty_insight_analytics_*audits*Use in the Audit Store Dashboards for viewing v10.1.0 logs on the dashboard.pty_insight_analytics_audits_10.0-2024.08.30-000001
v9.2.0.0 and earlierpty_insight_*audit_*Use in the Audit Store Dashboards for viewing older release logs on the dashboard.pty_insight_analytics_audit_9.2-2024.08.07-000001, pty_insight_audit_v9.1-2028.02.10-000019, pty_insight_audit_v2.0-2022.02.19-000006, pty_insight_audit_v1.1-2021.02.17-000001, pty_insight_audit_v1-2020.12.21-000001
v8.0.0.0 and abovepty_insight_*audit*Use in the Audit Store Dashboards for viewing all logs.pty_insight_analytics_audits_10.0-2024.08.30-000001, pty_insight_analytics_audit_9.2-2024.08.07-000001, pty_insight_audit_v9.1-2028.02.10-000019, pty_insight_audit_v2.0-2022.02.19-000006, pty_insight_audit_v1.1-2021.02.17-000001, pty_insight_audit_v1-2020.12.21-000001

The following parameters are configured for the index rollover in v10.1.0:

  • Index age: 30 days
  • Document count: 200,000,000
  • Index size: 5 GB

Protection logs

These logs are generated by protectors during protecting, unprotecting, and reprotecting data operations. These logs are generated by protectors, such as, DSG.

Use the following query in Discover to view these logs.

logtype:protection

A sample log is shown here:

 {
    "process": {
      "thread_id": "1227749696",
      "module": "coreprovider",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "4190",
      "user": "user4",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "level": "SUCCESS",
    "signature": {
      "key_id": "11a8b7d9-1621-4711-ace7-7d71e8adaf7c",
      "checksum": "43B6A4684810383C9EC1C01FF2C5CED570863A7DE609AE5A78C729A2EF7AB93A"
    },
    "origin": {
      "time_utc": "2024-09-02T13:55:17.000Z",
      "hostname": "hostname1234",
      "ip": "10.39.3.156"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "pcc_version": "3.6.0.1",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1"
    },
    "protection": {
      "dataelement": "TE_A_S13_L1R2_Y",
      "datastore": "DataStore",
      "audit_code": 6,
      "operation": "Protect",
      "policy_user": "user1"
    },
    "index_node": "protegrity-esa399/10.39.1.23",
    "tiebreaker": 210,
    "logtype": "Protection",
    "additional_info": {
      "description": "Data protect operation was successful"
    },
    "index_time_utc": "2024-09-02T13:55:24.766355224Z",
    "ingest_time_utc": "2024-09-02T13:55:17.678Z",
    "client": {},
    "correlationid": "cm0f1jlq700gbzb19cq65miqt"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-02T13:55:17.000Z"
    ],
    "index_time_utc": [
      "2024-09-02T13:55:24.766Z"
    ],
    "ingest_time_utc": [
      "2024-09-02T13:55:17.678Z"
    ]
  },
  "sort": [
    1725285317000
  ]

The above example contains the following information:

  • additional_info
  • origin
  • protector
  • protection
  • process
  • client
  • protector
  • signature

For more information about the various fields, refer here.

Metering logs

These logs are generated by protectors of prior to 8.0.0.0. These logs are not generated by latest protectors.

Use the following query in Discover to view these logs.

logtype:metering

For more information about the various fields, refer here.

Audit logs

These logs are generated when the rule set of the DSG protector gets updated.

Use the following query in Discover to view these logs.

logtype:audit

A sample log is shown here:

 {
   "additional_info.description": "User admin modified default_80 tunnel successfully ",
   "additional_info.title": "Gateway : Tunnels : Tunnel 'default_80' Modified",
   "client.ip": "192.168.2.20",
   "cnt": 1,
   "index_node": "protegrity-esa746/192.168.1.10",
   "index_time_utc": "2024-01-24T13:30:17.171646Z",
   "ingest_time_utc": "2024-01-24T13:29:35.000000000Z",
   "level": "Normal",
   "logtype": "Audit",
   "origin.hostname": "protegrity-cg406",
   "origin.ip": "192.168.2.20",
   "origin.time_utc": "2024-01-24T13:29:35.000Z",
   "process.name": "CGP",
   "process.user": "admin",
   "tiebreaker": 2260067,
   "_id": "ZTdhNzFmMTUtMWZlOC00MmY4LWJmYTItMjcwZjMwMmY4OGZh",
   "_index": "pty_insight_audit_v9.1-2024.01.23-000006"
 }

This example includes data from each of the following groups defined in the index:

  • additional_info
  • client
  • origin
  • process

For more information about the various fields, refer here.

Security logs

These logs are generated by security events of the system.

Use the following query in Discover to view these logs.

logtype:security

For more information about the various fields, refer here.

Troubleshooting index

The log types of application, kernel, system, and verification logs are stored in the troubleshooting index. These logs helps you understand the working of the system. The logs stored in this index are essential when the system is down or has issues. This is the pty_insight_analytics_troubleshooting index. The index pattern for viewing these logs in Discover is pty_insight_*troubleshooting_*.

The following parameters are configured for the index rollover:

  • Index age: 30 days
  • Document count: 200,000,000
  • Index size: 5 GB

Application Logs

These logs are generated by Protegrity servers and Protegrity applications.

Use the following query in Discover to view these logs.

logtype:application

A sample log is shown here:

{
    "process": {
      "name": "hubcontroller"
    },
    "level": "INFO",
    "origin": {
      "time_utc": "2024-09-03T10:02:34.597000000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16916,
    "logtype": "Application",
    "additional_info": {
      "description": "GET /dps/v1/deployment/datastores | 304 | 127.0.0.1 | Protegrity Client | 8ms | "
    },
    "index_time_utc": "2024-09-03T10:02:37.314521452Z",
    "ingest_time_utc": "2024-09-03T10:02:36.262628342Z",
    "correlationid": "cm0m9gjq500ig1h03zwdv6kok"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:02:34.597Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:02:37.314Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:02:36.262Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Application@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357754597
  ]

The above example contains the following information:

  • additional_info
  • origin
  • process

For more information about the various fields, refer here.

Kernel logs

These logs are generated by the kernel and help you analyze the working of the internal system. Some of the modules that generate these logs are CRED_DISP, KERNEL, USER_CMD, and so on.

Use the following query in Discover to view these logs.

logtype:Kernel

For more information and description about the components that can generate kernel logs, refer here.

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "CRED_DISP"
    },
    "origin": {
      "time_utc": "2024-09-03T10:02:55.059999942Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": "1",
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16964,
    "logtype": "Kernel",
    "additional_info": {
      "module": "pid=38236",
      "description": "auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:setcred grantors=pam_rootok acct=\"rabbitmq\" exe=\"/usr/sbin/runuser\" hostname=? addr=? terminal=? res=success'\u001dUID=\"root\" AUID=\"unset\"",
      "procedure": "uid=0"
    },
    "index_time_utc": "2024-09-03T10:02:59.315734771Z",
    "ingest_time_utc": "2024-09-03T10:02:55.062254541Z"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:02:55.059Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:02:59.315Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:02:55.062Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Kernel@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357775059
  ]

This example includes data from each of the following groups defined in the index:

  • additional_info
  • origin
  • process

For more information about the various fields, refer here.

System logs

These logs are generated by the operating system and help you analyze and troubleshoot the system when errors are found.

Use the following query in Discover to view these logs.

logtype:System

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

 {
    "process": {
      "name": "ESAPAP",
      "version": "10.0.0+2412",
      "user": "admin"
    },
    "level": "Low",
    "origin": {
      "time_utc": "2024-09-03T10:00:34.000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": "1",
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 16860,
    "logtype": "System",
    "additional_info": {
      "description": "License is due to expire in 30 days. The validity of license has been acknowledged by the user. (web-user 'admin' , IP: '10.87.2.32')",
      "title": "Appliance Info : License is due to expire in 30 days. The validity of license has been acknowledged by the user. (web-user 'admin' , IP: '10.87.2.32')"
    },
    "index_time_utc": "2024-09-03T10:01:10.113708469Z",
    "client": {
      "ip": "10.37.4.12"
    },
    "ingest_time_utc": "2024-09-03T10:00:34.000000000Z"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:00:34.000Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:01:10.113Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:00:34.000Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@System@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725357634000
  ]

This example includes data from each of the following groups defined in the index:

  • additional_info
  • origin
  • process

For more information about the various fields, refer here.

Verification logs

These log are generated by Insight on the ESA when a signature verification fails.

Use the following query in Discover to view these logs.

logtype:Verification

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "insight.pyc",
      "id": 45277
    },
    "level": "Info",
    "origin": {
      "time_utc": "2024-09-03T10:14:03.120342Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 17774,
    "logtype": "Verification",
    "additional_info": {
      "module": ".signature.job_executor",
      "description": "",
      "procedure": "__log_failure"
    },
    "index_time_utc": "2024-09-03T10:14:03.128435514Z",
    "ingest_time_utc": "2024-09-03T10:14:03.120376Z",
    "verification": {
      "reason": "SV_VERIFY_RESPONSES.INVALID_CHECKSUM",
      "job_name": "System Job",
      "job_id": "9Vq1opEBYpV14mHXU9hW",
      "index_name": "pty_insight_analytics_audits_10.0-2024.08.30-000001",
      "doc_id": "JI5bt5EBMqY4Eog-YY7C"
    }
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:14:03.120Z"
    ],
    "index_time_utc": [
      "2024-09-03T10:14:03.128Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:14:03.120Z"
    ]
  },
  "highlight": {
    "logtype": [
      "@opensearch-dashboards-highlighted-field@Verification@/opensearch-dashboards-highlighted-field@"
    ]
  },
  "sort": [
    1725358443120
  ]

This example includes data from each of the following groups defined in the index:

  • additional_info
  • process
  • origin
  • verification

For more information about the various fields, refer here.

Policy log index

The log type of policy is stored in the policy log index. They include logs for the policy-related operations, such as, when the policy is updated. The index pattern for viewing these logs in Discover is pty_insight_*policy_log_*.

The following parameters are configured for the policy log index:

  • Index age: 30 days
  • Document count: 200,000,000
  • Index size: 5 GB

Use the following query in Discover to view these logs.

logtype:policyLog

For a list of components and modules and the type of logs they generate, refer here.

A sample log is shown here:

{
    "process": {
      "name": "hubcontroller",
      "user": "service_admin",
      "version": "1.8.0+6.g5e62d8.1.8"
    },
    "level": "Low",
    "origin": {
      "time_utc": "2024-09-03T08:29:14.000000000Z",
      "hostname": "protegrity-esa503",
      "ip": "10.37.4.12"
    },
    "cnt": 1,
    "index_node": "protegrity-esa503/10.37.4.12",
    "tiebreaker": 10703,
    "logtype": "Policy",
    "additional_info": {
      "description": "Data element created. (Data Element 'TE_LASCII_L2R1_Y' created)"
    },
    "index_time_utc": "2024-09-03T08:30:31.358367506Z",
    "client": {
      "ip": "10.87.2.32",
      "username": "admin"
    },
    "ingest_time_utc": "2024-09-03T08:29:30.017906235Z",
    "correlationid": "cm0m64iap009r1h0399ey6rl8",
    "policy": {
      "severity": "Low",
      "audit_code": 150
    }
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T08:29:14.000Z"
    ],
    "index_time_utc": [
      "2024-09-03T08:30:31.358Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T08:29:30.017Z"
    ]
  },
  "highlight": {
    "additional_info.description": [
      "(Data Element '@opensearch-dashboards-highlighted-field@DE@/opensearch-dashboards-highlighted-field@' created)"
    ]
  },
  "sort": [
    1725352154000
  ]

The example contains the following information:

  • additional_info
  • origin
  • policy
  • process

For more information about the various fields, refer here.

Policy Status Dashboard index

The policy status dashboard index contains information for the Policy Status Dashboard. It holds the policy and trusted application deployment status information. The index pattern for viewing these logs in Discover is pty_insight_analytics*policy_status_dashboard_*.

{
    "logtype": "Status",
    "process": {
      "thread_id": "2458884416",
      "module": "rpstatus",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "2852",
      "user": "root",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "origin": {
      "time_utc": "2024-09-03T10:24:19.000Z",
      "hostname": "ip-10-49-2-49.ec2.internal",
      "ip": "10.49.2.49"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "datastore": "DataStore",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0"
    },
    "ingest_time_utc": "2024-09-03T10:24:19.510Z",
    "status": {
      "core_correlationid": "cm0f1jlq700gbzb19cq65miqt",
      "package_correlationid": "cm0m1tv5k0019te89e48tgdug"
    },
    "policystatus": {
      "type": "TRUSTED_APP",
      "application_name": "APJava_sample",
      "deployment_or_auth_time": "2024-09-03T10:24:19.000Z",
      "status": "WARNING"
    }
  },
  "fields": {
    "policystatus.deployment_or_auth_time": [
      "2024-09-03T10:24:19.000Z"
    ],
    "origin.time_utc": [
      "2024-09-03T10:24:19.000Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:24:19.510Z"
    ]
  },
  "sort": [
    1725359059000
  ]

The example contains the following information:

  • additional_info
  • origin
  • protector
  • policystatus
  • policy
  • process

Protectors status index

The protector status logs generated by protectors of v10.0.0 are stored in this index. The index pattern for viewing these logs in Discover is pty_insight_analytics_protectors_status_*.

The following parameters are configured for the index rollover:

  • Index age: 30 days
  • Document count: 200,000,000
  • Index size: 5 GB

Use the following query in Discover to view these logs.

logtype:status

A sample log is shown here:

{
   "logtype":"Status",
   "process":{
      "thread_id":"2559813952",
      "module":"rpstatus",
      "name":"java",
      "pcc_version":"3.6.0.1",
      "id":"1991",
      "user":"root",
      "version":"10.0.0.2.91.5ec4b8b",
      "core_version":"2.1.0-alpha+24.g7fc71.2.1",
      "platform":"Linux_x64"
   },
   "origin":{
      "time_utc":"2024-07-30T07:22:41.000Z",
      "hostname":"ip-10-39-3-218.ec2.internal",
      "ip":"10.39.3.218"
   },
   "cnt":1,
   "protector":{
      "vendor":"Java",
      "datastore":"ESA-10.39.2.7",
      "family":"sdk",
      "version":"10.0.0.2.91.5ec4b8b"
   },
   "ingest_time_utc":"2024-07-30T07:22:41.745Z",
   "status":{
      "core_correlationid":"clz79lc2o004jmb29neneto8k",
      "package_correlationid":"clz82ijw00037k790oxlnjalu"
   }
}

The example contains the following information:

  • additional_info
  • origin
  • policy
  • protector

Protector Status Dashboard index

The protector status dashboard index contains information for the Protector Status Dashboard. It holds the protector status information. The index pattern for viewing these logs in Discover is pty_insight_analytics*protector_status_dashboard_.

A sample log is shown here:

{
    "logtype": "Status",
    "process": {
      "thread_id": "2458884416",
      "module": "rpstatus",
      "name": "java",
      "pcc_version": "3.6.0.1",
      "id": "2852",
      "user": "root",
      "version": "10.0.0-alpha+13.gef09.10.0",
      "core_version": "2.1.0+17.gca723.2.1",
      "platform": "Linux_x64"
    },
    "origin": {
      "time_utc": "2024-09-03T10:24:19.000Z",
      "hostname": "ip-10-49-2-49.ec2.internal",
      "ip": "10.49.2.49"
    },
    "cnt": 1,
    "protector": {
      "vendor": "Java",
      "datastore": "DataStore",
      "family": "sdk",
      "version": "10.0.0-alpha+13.gef09.10.0"
    },
    "ingest_time_utc": "2024-09-03T10:24:19.510Z",
    "status": {
      "core_correlationid": "cm0f1jlq700gbzb19cq65miqt",
      "package_correlationid": "cm0m1tv5k0019te89e48tgdug"
    },
    "protector_status": "Warning"
  },
  "fields": {
    "origin.time_utc": [
      "2024-09-03T10:24:19.000Z"
    ],
    "ingest_time_utc": [
      "2024-09-03T10:24:19.510Z"
    ]
  },
  "sort": [
    1725359059000
  ]

The example contains the following information:

  • additional_info
  • origin
  • protector
  • process

DSG transaction metrics

The table in this section lists the details for the various parameters generated by DSG transactions. The DSG transaction logs are stored in the pty_insight_analytics_dsg_transaction_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_transaction_metrics_*. The following parameters are configured for the index rollover:

  • Index age: 1 day
  • Document count: 10,000,000
  • Index size: 1 GB

This index stores the following fields.

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

For more information about the transaction metric logs, refer to the section Transaction Metrics Logging in the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

DSG usage metrics

This section describes the codes associated with the following DSG usage metrics:

  • Tunnels usage data (Version 0)
  • Service usage data (Version 0)
  • Profile usage data (Version 0)
  • Rules usage data (Version 0)

The table in this sub sections lists the details for the various parameters generated while using the DSG. The DSG usage metrics logs are stored in the pty_insight_analytics_dsg_usage_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_usage_metrics_*. The following parameters are configured for the index rollover:

  • Index age: 1 day
  • Document count: 3,500,000
  • Index size: 1 GB

For more information about the usage metrics, refer to the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

Tunnels usage data

The table in this section describes the usage metric for Tunnels.

PositionNameData TypeDescription
0metrics typeinteger0 for Tunnels
1metrics versioninteger0
2tunnel-typestringthe tunnel type CIFS, HTTP, NFS, S3, SFTP, SMTP
3timestampstringtime usage is reported
4tunnel-idstringaddress of tunnel instance will be unique id generated when tunnel is created.
5uptimefloattime in seconds since the tunnel loaded
6bytes-processedintegerfrontend and backend bytes the tunnel processed since the last time usage was reported
7frontend-bytes-processedintegerfrontend bytes the tunnel has processed since the last time usage was reported
8backend-bytes-processedintegerbackend bytes the tunnel has processed since the last time usage was reported
9total-bytes-processedintegertotal number of frontend and backend bytes the tunnel has processed during the time the tunnel has been loaded
10frontend-bytes-processedintegertotal number of frontend bytes the tunnel has processed during the time the tunnel has been loaded
11backend-bytes-processedintegertotal number of backend bytes the tunnel has processed during the time the tunnel has been loaded
12message-countintegernumber of requests the tunnel received since the last time usage was reported
13total-message-countintegertotal number of requests the tunnel received during the time the tunnel has been loaded
14ingest_time_utcstringTime in UTC at which this log is ingested
15logtypestringValue to identify type of metric –> dsg_metrics_usage_tunnel

A sample is provided here:

{"metrics_type":"Tunnel","version":0,"tunnel_type":"HTTP","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","tunnel_id":"140361619513360","checksum":"4139677074","uptime":620.8048927783966,"bytes_processed":401,"frontend_bytes_processed":401,"backend_bytes_processed":0,"previous_bytes_processed":401,"previous_frontend_bytes_processed":401,"previous_backend_bytes_processed":0,"total_bytes_processed":1203,"total_frontend_bytes_processed":1203,"total_backend_bytes_processed":0,"message_count":1,"previouse_message_count":1,"total_message_count":3}

Services usage data

The table in this section describes the usage metric for Services.

PositionNameData TypeDescription
0metrics typeinteger1 for Services
1metrics versioninteger0
2service-typestringthe service type HTTP-GW, MOUNTED-OOB, REST-API, S3-OOB, SMTP-GW, SFTP-GW, WS-GW
3timestampstringtime usage is reported
4service-idstringUUID of service name
5tunnel-idstringUUID of tunnel name
6callsintegernumber of times service processed frontend and backend requests since the time usage was last reported
7frontend-callsintegernumber of times service processed frontend requests since the time usage was last reported
8backend-callsintegernumber of times service processed backend requests since the time usage was last reported
9total-callsintegertotal number number of times service processed frontend and backend requests since the service has been loaded
10total-frontend-callsintegertotal number number of times service processed frontend and backend requests since the service has been loaded
11total-backend-callsintegertotal number number of times service processed frontend and backend requests since the service has been loaded
12bytes-processedintegerfrontend and backend bytes the service processed since the last time usage was reported
13frontend-bytes-processedintegerfrontend bytes the tunnel processed since the last time usage was reported
14backend-bytes-processedintegerbackend bytes the tunnel processed since the last time usage was reported
15total-bytes-processedintegertotal number of frontend and backend bytes the service has processed during the time the service has been loaded
16total-frontend-bytes-processedintegertotal number of frontend bytes the tunnel has processed during the time the tunnel has been loaded
17total-backend-bytes-processedintegertotal number of backend bytes the tunnel has processed during the time the tunnel has been loaded
18ingest_time_utcstringTime in UTC at which this log is ingested
19logtypestringValue to identify type of metric –> dsg_metrics_usage_service

A sample is provided here:

{"metrics_type":"Service","version":0,"service_type":"REST-API","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"}, "previous_timestamp":"2023-04-13T12:28:08Z", "service_id":"140361548704016","checksum":"3100121694","tunnel_checksum":"4139677074","calls":401,"frontend_calls":401,"backend_calls":0,"previous_calls":401,"previous_frontend_calls":401,"previous_backend_calls":0,"total_calls":1203,"total_frontend_calls":1203,"total_backend_calls":0,"bytes_processed":2,"frontend_bytes_processed":1,"backend_bytes_processed":1,"previous_bytes_processed":2,"previous_frontend_bytes_processed":1,"previous_backend_bytes_processed":1,"total_bytes_processed":6,"total_frontend_bytes_processed":3,"total_backend_bytes_processed":3}

Profile usage data

The table in this section describes the usage metric for Profile.

PositionNameData TypeDescription
0metrics typeinteger2 for Profile
1metrics versioninteger0
2timestampstringtime usage is reported
3prev-timestampstringthe previous time usage was reported
4profile-idstringaddress of profile instance will be unique id generated when profile is created
5parent-idstringchecksum of profile or service calling this profile
6callsintegernumber of times the profile processed a request since the time usage was last reported
7total-callsintegertotal number of times the profile processed a request since profile has been loaded
8profile-ref-countintegerthe number of times this profile has been called via a profile reference since the time usage was last reported
9prev-profile-ref-countintegerthe number of times this profile has been called via a profile reference the last time usage was last reported
10total-profile-ref-countintegertotal number of times this profile has been called via a profile reference since the profile has been loade
11bytes-processedintegerbytes the profile processed since the last time usage was reported
12total-bytes-processedintegertotal bytes the profile processed since the profile has been loaded
13elapsed-time-sample-countintegerthe number of times the profile was sampled since the last time usage was reported
14elapsed-time-meanintegerthe average amount of time in nano-seconds it took to process a request based on elapsed-time-sample-count
15total-elapsed-time-sample-countintegerthe number of times the profile was sampled since the profile has been loaded
16total-elapsed-time-sample-meanintegerthe average amount of time in nano-seconds it took to process a request based on total-elapsed-time-sample-count
17ingest_time_utcstringTime in UTC at which this log is ingested
18logtypestringValue to identify type of metric –> dsg_metrics_usage_profile

A sample is provided here:

{"metrics_type":"Profile","version":0,"cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","profile_id":"140361548999248","checksum":"3504922421","parent_checksum":"3100121694","calls":2,"previous_calls":2,"total_calls":6,"profile_reference_count":0,"previous_profile_reference_count":0,"total_profile_reference_count":0,"bytes_processed":802,"previous_bytes_processed":802,"total_bytes_processed":2406,"elapsed_time_sample_count":2,"elapsed_time_average":221078.5,"total_elapsed_time_sample_count":6,"total_elapsed_time_sample_average":245797.0}

Rules usage data

The table in this section describes the usage metric for Rules.

PositionNameData TypeDescription
0metrics typeinteger3 for Rules
1metrics versioninteger0
2rule-typestringrule is one of Dynamice Injection, Error, Exit, Extract, Log, Profile Reference, Set Context Variable, Set User Identity, Transform
3codecstringonly applies to Extract
4timestampstringtime usage is reported
5flagbooleanBroken rule or is domain name rewrite
6rule-idstringaddress of rule instance will be unique id generated when rule is created.
7parent-idstringchecksum of rule or profile calling this rule
8callsintegernumber of times the rule processed a request since the time usage was last reported
9total-callsintegertotal number of times the rule processed a request since rule has been loaded
10profile-ref-countintegerthe number of times this rule has been called via a profile reference since the time usage was last reported
11prev-profile-ref-countintegerthe number of times this rule has been called via a profile reference the last time usage was last reported
12total-profle-ref-countintegertotal number of times this rule has been called via a profile reference since the rule has been loaded
13bytes-processedintegerbytes the rule processed since the last time usage was reported
14total-bytes-processedintegertotal bytes the rule processed since the rule has been loaded
15elapsed-time-sample-countintegerthe number of times the rule was sampled since the last time usage was reported
16elapsed-time-sample-meanintegerthe average amount of time in nano-seconds it took to process a data based on elapsed-time-sample-count
17total-elapsed-time-sample-countintegerthe number of times the rule was sampled since the rule has been loaded
18total-elapsed-time-sample-meanintegerthe average amount of time in nano-seconds it took to process a data based on total-elapsed-time-sample-count
19ingest_time_utcstringTime in UTC at which this log is ingested
20logtypestringValue to identify type of metric –> dsg_metrics_usage_rule

A sample is provided here:

{"metric_type":"Rule","version":0,"rule_type":"Extract","codec":"Set User Identity","cnt":1,"logtype":"Application","origin":{"time_utc":"2023-04-13T12:28:18Z"},"previous_timestamp":"2023-04-13T12:28:08Z","broken":false,"domain_name_rewrite":false,"rule_id":"140361553016464","rule_checksum":"932129179","parent_checksum":"3504922421","calls":1,"previous_calls":1,"total_calls":3,"profile_reference_count":0,"previous_profile_reference_count":0,"total_profile_reference_count":0,"bytes_processed":1,"previous_bytes_processed":1,"total_bytes_processed":3,"elapsed_time_sample_count":1,"elapsed_time_sample_average":406842.0,"total_elapsed_time_sample_count":3,"total_elapsed_time_sample_average":451163.6666666667}

DSG error metrics

The table in this section lists the details for the various parameters generated for the DSG Error Metrics. The DSG Error Metrics logs are stored in the pty_insight_analytics_dsg_error_metrics_9.2 index file. The index pattern for viewing these logs in Discover is pty_insight_analytics_dsg_error_metrics_*. The following parameters are configured for the index rollover:

  • Index age: 1 day
  • Document count: 3,500,000
  • Index size: 1 GB

This index stores the following fields.

* -The origin_time_utc and logtype parameters will only be displayed on the Audit Store Dashboards.

For more information about the error metric logs, refer to the Protegrity Data Security Gateway User Guide 3.2.0.0.

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

Miscellaneous index

The logs that are not added to the other indexes are captured and stored in the miscellaneous index. The index pattern for viewing these logs in Discover is pty_insight_analytics_miscellaneous_*.

This index should not contain any logs. If any logs are visible in this index, then kindly contact Protegrity support.

The following parameters are configured for the index rollover:

  • Index age: 7 days
  • Document count: 3,500,000
  • Index size: 200 mb

Use the following query in Discover to view these logs.

logtype:miscellaneous;

Scheduled tasks are available for deleting this index. You can configure and enable the scheduled task to free up the space used by old index files that you do not require.

For more information about scheduled tasks, refer here.

Last modified February 7, 2025