Trusted Appliances Cluster (TAC)
Network clustering is where a group of computers are organized so they function together, providing highly available resources. Clustering is highly desirable for disaster recovery. Failure of one system will not affect business continuity and the performance of resources is maintained.
A Trusted Appliances cluster (TAC) is a tool, where appliances, such as, ESA or DSG replicate and maintain information. In a TAC, multiple appliances are connected using SSH. A trusted channel is created to transfer data between the appliances in the cluster. You can also run remote commands, backup data, synchronize files and configurations across multiple sites, or import/export configurations between appliances that are directly connected to each other.
In a TAC, all the systems in the cluster are in an active state. The request for security operations are handled across the active appliances in the cluster. Thus, in case of a failure of an appliance, the requests are balanced across other appliances in the cluster.
1 - TAC Topology
The TAC is a connected graph with a fully connected cluster. In a fully connected cluster, every node directly communicates with other nodes in the cluster.
The following figure shows a connected graph with four nodes A, B, C, and D that are directly connected to each other.
In a TAC, each appliance is classified either as a client or a server.
- Client: A client is a stateless agent that requests information from a server.
- Server: A server maintains information about all the appliances in the cluster, performs regular health checks, and responds to queries from the clients.
A server can be further classified as a leader or a follower. The leader is responsible for maintaining the status of cluster and replicating cluster-related information among other servers in the cluster. The first appliance that is added the cluster is the leader. The other appliances added to the cluster are followers.
It is important to maintain the number of servers to keep the cluster available. For a cluster to be available, the number of servers available must be (N/2) + 1, where N is the number of servers in the cluster. Thus, it is recommended to have a minimum of three servers in your cluster for fault tolerance.
2 - Cluster Configuration Files
In a cluster, you can deploy an appliance as a server or a client by modifying the cluster configuration files. For deploying an appliance on a cluster, the following configuration files are available for an appliance.
agent.json
The agent.json file specifies the role of an appliance in the cluster. The file is available in the /opt/cluster-consul-integration/configure directory.
The following table describes the attributes that can be configured in the agent.json file.
| Attribute | Description | Values |
|---|
| type | The role of the appliance in the cluster. | - auto (default) – Role of the appliance is determined based
on state of the TAC and the parameters of the
agent_auto.json file. - client – Appliance is added to cluster as a client
- server– Appliance is added to cluster as a server
For more information about the deployment scenarios, refer
to section Deploying
Appliances in a cluster. |
agent_auto.json
This file is considered only if the type attribute in the agent.json file is set to auto. The agent_auto.json file specifies the maximum number of servers allowed in a cluster. Additionally, you can also specify which appliances can be added to the cluster as servers.
The agent_auto.json file is available in the /opt/cluster-consul-integration/configure directory.
The following table describes the attributes that can be configured in the agent_auto.json file.
| Attribute | Description | Values |
|---|
| maximum_servers | The maximum number of servers that can be deployed in a cluster. | 5 (default)Note- It is recommended to set the attribute value as 3 or 5.
- If the attribute value is 0, then all the appliances are added to the cluster as servers.
|
| PAP_eligible_servers | The list of appliances that can be deployed as servers. | - ESA (default) - ESA appliance
- CG – DSG appliance
|
config.json
This file contains the cluster-related information for an appliance, such as, data center, ports, Consul certificates, bind address, and so on. The config.json file is available in the /opt/consul/configure directory.
3 - Deploying Appliances in a Cluster
You can deploy the appliances in a cluster as a server or a client. The type attribute in the agent.json file and the PAP_eligible_servers and maximum_servers attributes in the agent_auto.json file determine how the appliance is deployed in the cluster.
The following flowchart illustrates how an appliance is deployed in a cluster.
Example process for deploying appliances in a cluster
Consider an ESA appliance, ESA001, on which you create a cluster. As this is the first appliance on the cluster, ESA001 is becomes the leader of the cluster. The following are the values of the default attributes of the agent.json and agent_auto.json files on ESA001.
- type: auto
- maximum_servers: 5
- PAP_eligible_servers: ESA
Now, you want to add another ESA appliance, ESA002, to this cluster as a server. In this case, you must ensure that the type attribute in the agent.json file of ESA002 is set as server.
If you want to add an ESA003 to the cluster as a client, you must ensure that the type attribute in the agent.json file of ESA003 is set as client.
The following figure illustrates the cluster comprising of nodes ESA001, ESA002, and ESA003.
Now, you add another ESA appliance, ESA004, to this cluster with the following attributes:
- type: auto
- maximum_servers: 5
- PAP_eligible_servers: ESA
In this case, the following checks are performed:
- Is the value of maximum_servers greater than zero? Yes.
- Is the number of servers in the cluster exceeding the maximum_servers? No
- Is the appliance code of ESA004 in the PAP_eligible_servers list? Yes.
The name or appliance code of appliances can be viewed in the Appliance_code file in the /etc directory.
As long as the limit of the number of servers on the cluster is not exceeded and the appliance is a part of the server list, ESA004 is added as a server as shown in the following figure.
Now add a DSG appliance named CG001 to this cluster with the following attributes:
- type: auto
- maximum_servers: 5
- PAP_eligible_servers: CG
In this case, the following checks are performed:
- Is the maximum_servers greater than zero? Yes.
- Is the number of servers in the cluster exceeding the maximum_servers? No.
- Is the appliance code of CG001 in the PAP_eligible_servers list? Yes.
Thus, DSG1 is added to the cluster as a server.
Now, consider a cluster with five servers, ESA001, ESA002, ESA003, ESA004, and ESA006 as shown in the following figure.
You now add another ESA appliance, ESA007 to this cluster, with the following attributes:
- type: auto
- maximum_servers: 5
- PAP_eligible_servers: ESA
In this case, the following checks are performed:
- Is the maximum_servers greater than zero? Yes.
- Is the number of servers in the cluster exceeding the maximum_servers? Yes
- Is the appliance code of ESA007 in the PAP_eligible_servers list? Yes
Thus, as the limit of the number of servers in a cluster is exceeded, ESA007 is added as a client.
4 - Cluster Security
This section describes about the Cluster Security.
Gossip Key
In the cluster, the appliances communicate using the Gossip protocol. The cluster supports encrypting the communication using the gossip key. This key is generated during the creation of the cluster. The gossip key is then shared across all the appliances in the cluster.
SSL Certificates
SSL certificates are used to authenticate the appliances on the cluster. Every appliance contains the following default cluster certificates in the certificate repository:
- Server certificate and key for Consul
- Certificate Authorities(CA) certificate and key for Consul
In a cluster, the server certificates of the appliances are validated by the CA certificate of the appliance that initiated the cluster. This CA certificate is shared across all the appliances on the cluster for SSL communication.
You can also upload your custom CA and server certificates to the appliances on the cluster. The CA.key file is not mandatory when you deploy custom certificates for an appliance.
Ensure that you apply a single CA certificate on all the appliances in the cluster.
If the CA.key is available, the appliances that are added to the cluster download the CA certificate and key. The new server certificate for the appliance are generated using the CA key file.
If the CA.key is not available, all the keys and certificates are shared among the appliances in the cluster.
Ensure that the custom certificates match the following requirements:
The CN attribute of the server certificate is set in the following format:
server.<datacenter name>.<domain>
The domain and datacenter name must be equal to the value mentioned in theconfig.json file. For example, server.ptydatacenter.protegrity.
The custom certificates contain the following entries:
localhost
- 127.0.0.1
- FQDN of the local servers in the clusterFor example, an SSL Certificate with SAN extension of servers ESA1, ESA2, and ESA3 in a cluster has the following entries:
localhost
- 127.0.0.1
- ESA1.protegrity.com
- ESA2.protegrity.com
- ESA3.protegrity.com
The following figure illustrates the certificates.
Ports
The following ports are used for enabling communication between appliances:
- TCP port of 8300 – Used by servers to handle incoming request
- TCP and UDP ports of 8301 – Used by appliances to gossip on LAN
- TCP and UDP ports of 8302 – Used by appliances to gossip on WAN
Appliance Key Rotation
If you are using cloned machines to join a cluster, it is necessary to rotate the keys on all cloned nodes before joining the cluster. If the cloned machines have proxy authentication, two factor authentication, or TAC enabled, it is recommended to use new machines. This avoids any limitations or conflicts, such as, inconsistent TAC, mismatched node statuses, conflicting nodes, and key rotation failures due to keys in use.
For more information about rotating the keys, refer
here.
5 - Reinstalling Cluster Services
If the configuration files for TAC are corrupted, you can reinstall the consul service.
Before you begin
Ensure that Cluster-Consul-Integration v1.0.0 service is uninstalled before reinstalling Consul v2.4.0 service.
To reinstall the Cluster-Consul-Integration service:
In the CLI Manager, navigate to Administration > Add/Remove Services.
Press ENTER.
Enter the root password and select OK.
Select Install applications.
Select only Consul v2.4.0 and select OK.
Select Yes.
The Consul product is reinstalled on your appliance.
Install the Cluster-Consul-Integration v1.0.0 service.
For more information about installing services, refer to the Protegrity Installation.
6 - Uninstalling Cluster Services
If there is cluster with a maximum of ten nodes and you do not want to continue with the integrated cluster services, then uninstall the cluster services.
To uninstall cluster services:
Remove the appliance from the TAC.
In the CLI Manager, navigate to Administration > Add/Remove Services.
Press ENTER.
Enter the root password and select OK.
Select Remove already installed applications.
Select Cluster-Consul-Integration v1.0.0 and select OK.
The integration service is uninstalled.
Select Consul v2.4.0 and select OK.
The Consul product is uninstalled from your appliance.
If the node contains scheduled tasks associated with it, then you cannot uninstall the cluster services on it. Ensure that you delete all the scheduled tasks before uninstalling the cluster services.
7 - FAQs on TAC
This section lists the FAQs on TAC.
| Question | Answer |
|---|
| Can I block communication between appliances? | No. Blocking communication between appliances is disabled from release v7.1.0 MR2. |
| What is the recommended minimum quorum of servers required in a cluster? | The recommended minimum quorum of servers required in a cluster is three. |
| How to determine which appliance is the leader of the cluster? | In the OS Console of an appliance, run the following command:
/usr/local/consul operator raft list-peers -http-addr https://localhost:9000 -ca-file /opt/consul/ssl/ca.pem -client-cert /opt/consul/ssl/cert.pem -client-key /opt/consul/ssl/cert.key |
| Can I change the certificates of an appliance that is added to a cluster? | Yes. Ensure that the certificates are valid. For more information about the validity of the certificates, refer here. |
| Can I remove the last server from the cluster? | No, you cannot remove the last server from the cluster. The clients depend on this server for cluster related information. If you remove this server, then you risk de-stabilizing the cluster. |
| How to determine the role of an appliance in a cluster? | In the Web UI, navigate to the Trusted Appliance Cluster. On the screen, the labels for the appliances appear. The label for the server is Consul Server and that of the client is Consul Client. |
| Can I add an appliance other than ESA as server? | Yes. Ensure that the value of the type attribute in the agent.json file under the /opt/cluster-consul-integration/configure directory is set as server. |
| Can I clone a machine and join it to the cluster? | Yes, you can clone a machine to join in the cluster.However, if you are using cloned machines to join a cluster, it is necessary to rotate the keys on all cloned nodes before joining the cluster.
If the cloned machines have proxy authentication, two factor authentication, or TAC enabled, it is recommended to use new machines. This avoids any limitations or conflicts, such as, inconsistent TAC, mismatched node statuses, conflicting nodes, and key rotation failures due to keys in use.
|
| For more information about rotating the keys, refer here. | |
8 - Creating a TAC using the Web UI
You can create a TAC, where you add an appliance to the cluster.
Before you begin
When setting up or adding appliances to your cluster, you may be required to request a license for new nodes from Protegrity.
For more information about licensing, refer to the Protegrity Data Security Platform Licensing and your license agreement with Protegrity.
Before creating a TAC, ensure that the SSH Authentication type is set to Password + PublicKey.
If you are using cloned machines to join a cluster, it is necessary to rotate the keys on all cloned nodes before joining the cluster.
If the cloned machines have proxy authentication, two factor authentication, or TAC enabled, it is recommended to use new machines. This avoids any limitations or conflicts, such as, inconsistent TAC, mismatched node statuses, conflicting nodes, and key rotation failures due to keys in use.
For more information about rotating the keys, refer here.
Creating a TAC
In the ESA Web UI, navigate to System > Trusted Appliances Cluster.
The Join Cluster screen appears.
Select Create a new cluster.
The following screen appears.
Select the preferred communication method.
Select Add New to add, edit, or delete a communication method.
For more information about managing communication methods, refer here.
Click Save.
A cluster is created.
9 - Joining an Existing Cluster using the Web UI
If your appliance is not a part of any trusted appliances cluster, then you can add it to an existing cluster. This section describes the steps to join a TAC using the Web UI.
Before you begin
If you are using cloned machines to join a cluster, it is necessary to rotate the keys on all cloned nodes before joining the cluster.
If the cloned machines have proxy authentication, two factor authentication, or TAC enabled, it is recommended to use new machines. This avoids any limitations or conflicts, such as, inconsistent TAC, mismatched node statuses, conflicting nodes, and key rotation failures due to keys in use.
For more information about rotating the keys, refer
here.
Important : When assigning a role to the user, ensure that the Can Create JWT Token permission is assigned to the role.If the Can Create JWT Token permission is unassigned to the role of the required user in the target node, then joining the cluster operation fails.To verify the Can Create JWT Token permission, from the ESA Web UI navigate to Settings > Users > Roles.
Adding to an existing cluster
On the ESA Web UI, navigate to System > Trusted Appliances Cluster.
The following screen appears.
Enter the IP address of the target node in the Node text box.
Enter the credentials of the user of the target node in the Username and Password text boxes.
Click Connect.
The Site drop-down list and the Communication Methods options appear.
If you need to add a new communication method, click Add New. Otherwise, continue on to the next step.
Select the site and the preferred communication method.
Click Join.
The node is added to the cluster and the following screen appears.
Handling Consul certificates after adding an appliance to the cluster
After joining an appliance to the cluster, during replication, the Consul certificates are copied from the source to the target appliance. In this case, it is recommended to delete the Consul certificates pertaining to the target node from the Certificate Management screen. Navigate to Settings > Network > Certificate Repository. Click the delete icon next to Server certificate and key for Consul.
10 - Managing Communication Methods for Local Node
Every node in a network is identified using a unique identifier. A communication method is a qualifier for the remote nodes in the network to communicate with the local node.
There are two standard methods by which a node is identified:
- Local IP Address of the system (ethMNG)
- Host name
The nodes joining a cluster use the communication method to communicate with each other. The communication between nodes in a cluster occur over one of the accessible communication methods.
Adding a Communication Method from the Web UI
This section describes the steps to add a communication method from the Web UI.
In the Web UI, you can add a communication method only before creating a cluster. Perform the following steps to add a communication method from the Web UI.
In the Web UI, navigate to System > Trusted Appliances Cluster.
The Join Cluster Screen appears.
Click Create a new Cluster.
Click Create.
Click Add New.
The Add Communication Method text box appears.
Type the communication method and select OK.
The communication method is added.
Editing a Communication Method from the Web UI
This section describes the steps to edit a communication method from the Web UI.
In the Web UI, you can edit a communication method only before you create a cluster. Perform the following steps to edit a communication method from the Web UI.
In the Web UI, navigate to System > Trusted Appliances Cluster.
The Join Cluster Screen appears.
Click Create a new Cluster.
The Create New Cluster screen appears.
Click Create.
Click the Edit 
icon corresponding to the communication method to be edited.
The Edit Communication Method text box appears.
Type the communication method and select OK.
The communication method is edited.
Deleting a Communication Method from the Web UI
This section describes the steps to delete a communication method from the Web UI.
To delete a communication method from the Web UI:
In the Web UI, navigate to System > Trusted Appliances Cluster.
The Join Cluster Screen appears.
In the Web UI, you can delete a communication method before you create a cluster.
Click Create New Cluster.
The Create New Cluster screen appears.
Click Create.
Click the Delete 
icon corresponding to the communication method to be deleted.
A message confirming the delete operation appears.
Select OK.
The communication method is deleted.
11 - Viewing Cluster Information
This section describes the how to view cluster information using the Web UI.
To execute commands using Web UI:
In the Web UI, navigate to System > Trusted Appliances Cluster .
The screen with the appliances connected to the cluster appears.
Select All in the drop-down list.
The following options appear:
- Node Summary
- Cluster Tasks
- DiskFree
- MemoryFree
- Network
- System Info
- Top 10 CPU
- Top 10 Memory
- All
Select the required option.
The selected information for the appliances appears in the right pane.
12 - Removing a Node from the Cluster using the Web UI
This section describes the steps to remove a node from a cluster using the Web UI.
Before you begin
If a node is associated with a cluster task that is based on the hostname or IP address, then the Leave Cluster operation will not remove node from the cluster. Ensure that you delete all such tasks before removing any node from the cluster.
Removing the node
On the Web UI of the node that you want to remove from the cluster, navigate to System > Trusted Appliances Cluster.
The screen displaying the cluster nodes appears.
Navigate to Management > Leave Cluster.
The following screen appears.
A confirmation message appears.
Select Ok.
The node is removed from the cluster.
Scheduled tasks and removed nodes
If the scheduled tasks are created between the nodes in a cluster, then ensure that after you remove a node from the cluster, all the scheduled tasks related to the node are disabled or deleted.