User Defined Functions and APIs on

MapReduce APIs

Mon, 01 Jan 0001 00:00:00 +0000

This section describes the MapReduce APIs available for protection and unprotection in the Big Data Protector to build secure Big Data applications.

Warning: The Protegrity MapReduce protector only supports bytes converted from the string data type.

If any other data type is directly converted to bytes and passed as input to the API that supports byte as input and provides byte as output, then data corruption might occur.

Caution: If you are using the Protect, or Unprotect, or Reprotect API which accepts byte as input and provides byte as output, then ensure that you pass the charset argument in APIs with the charset used to encode the string input data type.
For example, if the input String was encoded using the UTF-16LE charset, then ensure to pass the “UTF-16LE” charset argument in the ByteIn or ByteOut APIs.

Hive UDFs

Mon, 01 Jan 0001 00:00:00 +0000

Warning: If you are using Ranger or Sentry, then ensure that your policy provides create access permissions to the required UDFs.

This section lists the Hive UDFs available for protection and unprotection in the Big Data Protector.

ptyGetVersion()

This UDF returns the current version of the protector.

ptyGetVersion()

Parameters:

None

Result:

The UDF returns the current version of the protector.

Example:

create temporary function ptyGetVersion AS 'com.protegrity.hive.udf.ptyGetVersion';
select ptyGetVersion();

ptyGetVersionExtended()

This UDF returns the extended version information of the protector.

Pig UDFs

Mon, 01 Jan 0001 00:00:00 +0000

ptyGetVersion()

The function returns the current version of the protector.

Signature:

ptyGetVersion()

Parameters:

None

Result:

The function returns the version number in a chararray.

Example:

REGISTER </path/to/bdp/lib/>/peppig-<jar_version>.jar;
// register pep pig version
DEFINE ptyGetVersion com.protegrity.pig.udf.ptyGetVersion;
//define UDF
employees = LOAD ‘employee.csv’ using PigStorage(‘,’) AS (eid:chararray,name:chararray, ssn:chararray);
// load employee.csv from HDFS path
version = FOREACH employees GENERATE ptyGetVersion();
DUMP version;

ptyGetVersionExtended()

The function returns the extended version information of the protector.

Signature:

ptyGetVersionExtended()

Parameters:

None

Result:

The function returns a chararray in the following format:
```
BDP: <1>; JcoreLite: <2>; CORE: <3>;
```
where,
- 1. is the current version of the Protector
- 1. is the Jcorelite library version
- 1. is the Core library version

Example:

HBase Commands

Mon, 01 Jan 0001 00:00:00 +0000

HBase is a database, which provides random read and write access to tables, consisting of rows and columns, in real-time. HBase is designed to run on commodity servers, to automatically scale as more servers are added, and is fault tolerant as data is divided across servers in the cluster. HBase tables are partitioned into multiple regions. Each region stores a range of rows in the table. Regions contain a datastore in memory and a persistent datastore (HFile). The Name node assigns multiple regions to a region server. The Name node manages the cluster and the region servers store portions of the HBase tables and perform the work on the data.

Impala UDFs

Mon, 01 Jan 0001 00:00:00 +0000

This section explains the Impala protector, the UDFs provided, and the commands for protecting and unprotecting data in an Impala table.

Overview of the Impala Protector

Impala is an MPP SQL query engine for querying the data stored in a cluster. The Protegrity Impala protector extends the functionality of the Impala query engine and provides UDFs which protect or unprotect the data as it is stored or retrieved.

Impala Protector Usage

The Protegrity Impala protector provides UDFs for protecting data using encryption or tokenization, and unprotecting data by using decryption or detokenization.

Spark Java APIs

Mon, 01 Jan 0001 00:00:00 +0000

All the Spark Java APIs that are available for protection and unprotection in Big Data Protector to build secure Big Data applications are listed here.

Spark is an execution engine that carries out batch processing of jobs in-memory and handles a wider range of computational workloads. In addition to processing a batch of stored data, Spark is capable of manipulating data in real time.

Spark leverages the physical memory of the Hadoop system. It utilizes the Resilient Distributed Datasets (RDDs) to store the data in-memory and lowers latency, if the data fits in the memory size. The data is saved on the hard drive only if required. RDDs being the basic units of abstraction and computation in Spark, you can use the Spark protection and unprotection APIs to perform transformation operations on an RDD.

Spark SQL UDFs

Mon, 01 Jan 0001 00:00:00 +0000

All the Spark SQL UDFs that are available for protection and unprotection in Big Data Protector to build secure Big Data applications are listed here.

Introduction

The Spark SQL module provides relational data processing capabilities to Spark. The module allows you to run SQL queries with Spark programs. It contains DataFrames, which is an RDD with an associated schema, that provide support for processing structured data in Hive tables.

Spark SQL enables structured data processing and programming of RDDs providing relational and procedural processing through a DataFrame API that integrates with Spark.

PySpark - Scala Wrapper UDFs

Mon, 01 Jan 0001 00:00:00 +0000

All the Spark Scala Wrapper UDFs that are available for protection and unprotection in Big Data Protector to build secure Big Data applications are listed here.

For each of the Spark SQL UDF in Spark SQL UDFs, a Scala UDF wrapper class is created so that it can be registered in the PySpark and invoked using the spark.sql() method.

ptyGetVersionScalaWrapper()

The UDF returns the current version of the protector.

Signature:

ptyGetVersionScalaWrapper()

Parameters:

Unity Catalog Batch Python UDFs

Mon, 01 Jan 0001 00:00:00 +0000

The UDFs in this section is applicable only to install and configure the Big Data Protector using the Standard Compute in Databricks. The information presented in this section will not apply to the Dedicated Compute as well as SQL Warehouse.
This version of the build only supports Unity Catalog Batch Python UDFs that use the Cloud Protect APIs. The Hive and Spark UDFs and APIs that provide native protection within the cluster nodes are not packaged in this build. If you want to use those features, please use the 9.1.0.0 builds.