Protegritydocs on

Protegrity-specific term definitions

Mon, 01 Jan 0001 00:00:00 +0000

Application Data Security

Application data security is the process of protecting sensitive data used within an application during processing, storage, or transmission. Security measures such as encryption, tokenization, or data masking are applied to ensure that sensitive information remains secure from unauthorized access or data breaches. Application data security is important for ensuring that personal or financial information is protected in enterprise applications.

Application Programming Interface Security

Application programming interface security refers to the practice of protecting an Application Programming Interface (API) from unauthorized access, data breaches, or misuse. This involves applying security measures such as authentication, encryption, and access control. These measures secure the interactions between systems that communicate through APIs. API security is essential in modern applications where APIs are used to integrate services across cloud environments.

Docker Compose Deployment

Tue, 20 Feb 2024 00:00:00 +0000

Prerequisites

The Deployment Package provided by Protegrity is obtained from the portal and extracted.
Docker CLI version greater than or equal to 28.3.0 is installed. This is required for managing Docker containers.
Docker Compose version greater than or equal to 2.37.3 is installed. This is required for local containerized deployments.

Docker Compose v2 that uses the docker compose command syntax. Ensure that the the installation supports this version.

For Apple Macbook users, refer Additional Notes.

Obtaining the Deployment Package

Tue, 20 Feb 2024 00:00:00 +0000

Run the following steps to download the artifacts.

Log in to the Customer Portal.
Navigate to the page and download the DataDiscovery_RHUBI-9-64_x86-64_Generic.K8S_1.1.0.tar.gz file on the system.
Extract this package. The following files are available:
- README - File containing the instructions to deploy the product.
- docker_compose - Deploying the product on a local developer setup.
- eks-terraform-helm - Deploying the product on a scalable deployment on Amazon EKS. Terraform is used for deploying the infrastructure and Helm Charts are used for the application components.

Prerequisites

Tue, 20 Feb 2024 00:00:00 +0000

Before deploying Data Discovery on Amazon EKS, ensure that following requirements are met for a smooth deployment process.

Tools and Permissions

The following tools must be installed and properly configured on your local machine:

AWS CLI of version 2.28.3 is installed. It us a command-line interface for AWS services. Must be configured with valid credentials having EKS cluster creation and management permissions. For more information about the configuration details, refer to Configuration and credentials precedence.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

In an era where data privacy is paramount, safeguarding sensitive information in unstructured data has become critical—especially for organizations leveraging AI and machine learning technologies. Data Discovery is a powerful, developer-friendly product designed specifically to address this challenge.

Data Discovery’s Classification Service specializes in the detection of Personally Identifiable Information (PII), Protected Health Information (PHI), Payment Card Information (PCI) within free-text (unstructured) and table-based (structured. CSV) inputs. Unlike traditional data tools, it excels in dynamic, unstructured environments such as chatbot conversations, call transcripts, and Generative AI (Gen AI) outputs.

Configuring Environment Variables

Tue, 20 Feb 2024 00:00:00 +0000

Run the following steps to edit the environment variables:

Navigate to the docker_compose directory.
Open the .env file and set the following variables as required:

Variable	Description	Required
DOCKER_CLASSIFICATION_IMAGE	Repository path where the docker image of Classification Service is stored.	Yes
DOCKER_PATTERN_PROVIDER_IMAGE	Repository path where the docker image of Pattern classification Service is stored.	Yes
DOCKER_CONTEXT_PROVIDER_IMAGE	Repository path where the docker image of Context clarification Service is stored.	Yes
DOCKER_NETWORK_NAME	Name of the Docker network.	No
PATTERN_PROVIDER_LOGGING_CONFIG	a valid JSON python logging configuration for the Pattern Classification Provider.	No
CONTEXT_PROVIDER_LOGGING_CONFIG	a valid JSON python logging configuration for the Context Classification Provider.	No
CLASSIFICATION_LOGGING_CONFIG	a valid JSON python logging configuration for the Classification Service.	No
ENABLE_ALL_SECURITY_CONTROLS	Controls whether security mitigations are enabled. Accepted values: true (default) or false	No

Save the changes.

EKS Deployment Architecture

Tue, 20 Feb 2024 00:00:00 +0000

The following architectural diagram illustrates the main components in the deployment of the product on EKS.

Component	Description
Ingress Controller	The Ingress controller acts as the single point of entry for the requests provided by a user.
Ingress rule	The Ingress rule routes the requests to the Classification service.
Classification pods	Classification service pods that act as the main entry point and the aggregator of the responses provided by the service providers.
Context and Pattern service providers	Pattern and the Context service providers pods that perform that task of identifying the sensitive data.

What's New

Mon, 01 Jan 0001 00:00:00 +0000

Feature	Description	References (if any)
Structured Data Classification	Classify data in CSV content by analyzing and assigning classifications to each column.	Classify CSV API
Harmonize Classification Responses	Standardize classification outputs from multiple providers by mapping them to a unified set of conventional categories.	Harmonize Responses
Transformation - Label	Replace sensitive text with corresponding entity labels (e.g., , <CREDIT_CARD>) based on classified data types.	Label Text API
Terraform and Helm Deployment	Support deployment on EKS using Terraform and Helm Charts.	EKS Deployment
Registry Hosted Product Images	Product images used in this deployment are available on Protegrity’s public Image Registry at `registry.protegrity.com`.	Obtaining Package
Performance Improvement	General performance improvements.	NA
Accuracy Improvements	General accuracy improvements.	NA
Bug Fixes	Various bug fixes.	NA

General Architecture

Tue, 20 Feb 2024 00:00:00 +0000

The main components of the Protegrity Data Discovery product are as follows:

Classification service: The Classification Service serves as the primary access point for all classification-related interactions. It orchestrates various back-end components known as Providers, which are responsible for executing the actual classification tasks.
Pattern and Context classification providers: The Providers function as specialized modules in identifying and classifying Personally Identifiable Information (PII). They analyze input data to detect, classify, and locate sensitive information.

Viewing Application Logs

Tue, 20 Feb 2024 00:00:00 +0000

The application logs can be viewed using the following commands:

docker logs -f classification_service
docker logs -f context_provider
docker logs -f pattern_provider

Setting the Log Level and other logging configuration

The log level and other valid Python Logging configuration can be set in the .env file using JSON.

Run the following steps to set the overall logging level.

Navigate to the docker_compose directory.
Edit the .env file.
Uncomment the required logging configuration and set the logging level to one of the following:

Handling Overlapping Conflicts

Tue, 20 Feb 2024 00:00:00 +0000

While classifying data, the providers may label an identical text under two different entities. This distinction arises from the detection strategies the classifiers adopt. Data Discovery handles these conflicts by applying certain rules on these conflicting entities.

The rules for handling the conflicting entities are as follows:

No overlap: If the two entities do not conflict, retain the results in the original form.

For example, Jake Filbert lives in Connecticut. If only Jake Filbert is identified, the result will be labeled as [NAME] lives in Connecticut.

Sample Response Default

Tue, 20 Feb 2024 00:00:00 +0000

{ “transform”: { “text”: “[PERSON] lives at [LOCATION] [LOCATION], [LOCATION] [LOCATION], [LOCATION].” } }

The fields are described as follows:

Name	Example Response	Description
transform.text	[PERSON] lives at [LOCATION]..	The labed input text with classified entities listed by name in place of the original sensitive data

Sample Response with Detail

Tue, 20 Feb 2024 00:00:00 +0000

{
 "transform": {
 "text": "[PERSON] lives at [LOCATION] [LOCATION], [LOCATION] [LOCATION], [LOCATION]."
 },
 "providers": [
 {
 "name": "Pattern Classification Provider",
 "version": "1.1.0",
 "status": 200,
 "elapsed_time": 0.011328935623168945,
 "config_provider": {
 "name": "Pattern",
 "address": "http://pattern_provider_service:8051",
 "supported_content_types": []
 }
 },
 {
 "name": "Context Classification Provider",
 "version": "1.1.0",
 "status": 200,
 "elapsed_time": 0.03895401954650879,
 "config_provider": {
 "name": "Context",
 "address": "http://context_provider_service:8052",
 "supported_content_types": []
 }
 }
 ],
 "classifications": {
 "LOCATION": [
 {
 "score": 0.85,
 "location": {
 "start_index": 17,
 "end_index": 24
 },
 "classifiers": [
 {
 "provider_index": 0,
 "name": "SpacyRecognizer",
 "score": 0.85,
 "original_entity": "LOCATION",
 "details": {}
 }
 ]
 },
 {
 "score": 0.9240000128746033,
 "location": {
 "start_index": 26,
 "end_index": 32
 },
 "classifiers": [
 {
 "provider_index": 0,
 "name": "SpacyRecognizer",
 "score": 0.85,
 "original_entity": "LOCATION",
 "details": {}
 },
 {
 "provider_index": 1,
 "name": "context",
 "score": 0.9980000257492065,
 "original_entity": "CITY",
 "details": {}
 }
 ]
 },
 {
 "score": 0.9244499981403351,
 "location": {
 "start_index": 40,
 "end_index": 51
 },
 "classifiers": [
 {
 "provider_index": 0,
 "name": "SpacyRecognizer",
 "score": 0.85,
 "original_entity": "LOCATION",
 "details": {}
 },
 {
 "provider_index": 1,
 "name": "context",
 "score": 0.9988999962806702,
 "original_entity": "STATE",
 "details": {}
 }
 ]
 },
 {
 "score": 0.9958999752998352,
 "location": {
 "start_index": 14,
 "end_index": 16
 },
 "classifiers": [
 {
 "provider_index": 1,
 "name": "context",
 "score": 0.9958999752998352,
 "original_entity": "BUILDING",
 "details": {}
 }
 ]
 },
 {
 "score": 0.9983999729156494,
 "location": {
 "start_index": 33,
 "end_index": 38
 },
 "classifiers": [
 {
 "provider_index": 1,
 "name": "context",
 "score": 0.9983999729156494,
 "original_entity": "ZIPCODE",
 "details": {}
 }
 ]
 }
 ],
 "PERSON": [
 {
 "score": 0.8819000124931335,
 "location": {
 "start_index": 0,
 "end_index": 4
 },
 "classifiers": [
 {
 "provider_index": 1,
 "name": "context",
 "score": 0.8819000124931335,
 "original_entity": "NAME",
 "details": {}
 }
 ]
 }
 ]
 }
}

The fields for the transform section are described as follows:

Viewing Application Logs

Tue, 20 Feb 2024 00:00:00 +0000

The application logs can be viewed using the following commands:

 kubectl logs classification-deployment-{version} -n protegrity -f
 kubectl logs roberta-provider-deployment-{version} -n protegrity -f
 kubectl logs presidio-provider-deployment-{version} -n protegrity -f

Run the kubectl get pods -n <namespace-name> command to obtain the version of the images.

Setting the Log Level and other Logging Configuration

Set the log level and other valid Python Logging configuration.

Navigate to the helm/data-discovery-classification directory in your downloaded deployment package.

Harmonizing Provider Outputs

Tue, 20 Feb 2024 00:00:00 +0000

Based on the detection logic, the Pattern and Context classification providers might classify the same data in different labels. The classification service standardizes provider outputs into a unified response.

Consider the example, You can visit our office located in New York City.

Context provider might categorize New York City as CITY.
Pattern provider might categorize New York City as LOCATION.

This can cause an inconsistency in the outputs generated across the providers.

Performance and Accuracy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Performance and accuracy are critical metrics for data discovery tools. These ensure that large datasets can be processed swiftly and sensitive information is correctly identified. High performance minimizes latency and maximizes productivity, while accuracy reduces the risk of data breaches and ensures compliance with regulatory standards like GDPR and CCPA.

Together, these qualities are essential for maintaining data integrity and security in environments where unstructured data flows through various systems..

Input Validation

Mon, 01 Jan 0001 00:00:00 +0000

The Classification service in Data Discovery offers an input validation security feature that rejects invalid input data. Data that is malformed, non-normalized, containing homoglyphs, hieroglyphs, mixed Unicode variants, or control characters is considered as unsanitized or invalid data. These are rejected and will not be classified.

The following are few examples of data that will be rejected:

Ⅷ
𝓉𝑒𝓍𝓉
Ｐｅｐ

Before invoking the Classification endpoint, ensure that the input text is normalized. Replace invalid characters by their corresponding normalized plaintext characters. If the input text contains any invalid character, a status code of 422 and a message Untrusted input is returned.

Mon, 01 Jan 0001 00:00:00 +0000

When running commands from inside the Docker network (for example, from another container), use the service name directly. This leverages Docker’s internal DNS. e.g.,

curl -XPOST http://localhost:8050/pty/data-discovery/v1.1/classify --data 'You can reach Dave Elliot by phone 203-555-1286' -H "Content-Type: text/plain"

Mon, 01 Jan 0001 00:00:00 +0000

When running command from outside the docker network, e.g., from your host machine, use the published port mapping. e.g.,

curl -XPOST classification_service/pty/data-discovery/v1.1/classify --data 'You can reach Dave Elliot by phone 203-555-1286' -H "Content-Type: text/plain"

Mon, 01 Jan 0001 00:00:00 +0000

Response Code	Description
200	Successful Response.
206	Partial Content. Only some providers classifed data successfully.
400	Bad Request. Invalid input parameters or content.
413	Payload too large.
415	Unsupported media type.
422	Untrusted input. For more information, refer to Input Validation
502	Bad Gateway. All upstream providers failed; no successful data aggregation possible.
598	Unexpected internal server error. Check server logs.
599	Internal server error. Check server logs.

Mon, 01 Jan 0001 00:00:00 +0000

Name	Example Response	Description
providers	Array	Array of provider objects that participated in the request, including their respective success or failure codes.
providers[n].name	Pattern Classification Provider	Product name of the provider.
providers[n].version	1.0.0	Version of the provider.
providers[n].status	200	HTTP response code returned by the provider.
providers[n].elapsed_time	0.028	Time, in seconds, taken by the provider to process the request.
providers[n].config_provider	Object	Object containing configuration details for each provider.
providers[n].config_provider.name	Pattern	Internal name of the provider.
providers[n].config_provider.address	http://pattern_provider_service:8051	Network address or endpoint of the provider.
providers[n].config_provider.supported_content_types	[]	Array of supported content types. An empty array indicates support for all content types.

Mon, 01 Jan 0001 00:00:00 +0000

Navigate to the docker_compose directory.
Edit the docker-compose.yaml file.
Under the environment section of classification_service, append the security parameter as follows.

- SECURITY_SETTINGS={"ENABLE_ALL_SECURITY_CONTROLS":false}

Save the changes.
If the application is already running, stop the containers first:

docker compose down

Start the application with your configuration changes following the Docker Compose deployment guide:

docker compose up -d

Mon, 01 Jan 0001 00:00:00 +0000

Navigate to the /eks/helm/classification_app directory.
Create a values-override.yaml file with the required custom configuration.

securitySettings:
 ENABLE_ALL_SECURITY_CONTROLS: false

Save the changes.
If the application is already deployed, uninstall using the following command.

helm uninstall data-discovery-classification --namespace default --wait

Run the following installation command.

helm install data-discovery-classification . \
 --namespace default \
 --create-namespace \
 --wait \
 --wait-for-jobs \
 --timeout 900s \
 -f values-override.yaml

Mon, 01 Jan 0001 00:00:00 +0000

Name	Example Response	Description
transform.text	[PERSON] lives at [LOCATION]..	The labed input text with classified entities listed by name in place of the original sensitive data

Search Results

Mon, 01 Jan 0001 00:00:00 +0000