Observe and Restart Policy

Use this task as the platform operator during routine checks or incidents. These diagnostic checks are not installation steps. Read the error before restarting. Restarting does not correct policy fields or missing Policy Integration Manager (PIM) object mappings. The application programming interface (API) process also owns background scheduling.

Prerequisites

Prepare the following:

• Set CTX to the intended Kubernetes context.
• Obtain workload-read access for logs, events, and configuration references.
• Obtain policy:observability for metrics and policy:audit for restricted operation evidence.
• Obtain separate deployment authority only when a restart requires it.
• Obtain your platform hostname, trusted certificate authority (CA) bundle, and current access token.

Configure API Access

Complete the credential-safe request setup with your own user token in a dedicated Bash session. Then set the deployment endpoint and trusted CA file in that same shell:

export FQDN='your-platform-hostname'
export PPC_CA_FILE='your-platform-ca.pem'
export POLICY_BASE_URL="https://${FQDN}/pty/policy/v1.0"
POLICY_API="$POLICY_BASE_URL"
: "${POLICY_AUTH_HEADER:?Complete authentication setup}"

Use the configured API root if your platform overrides the default. All requests use the protected POLICY_AUTH_HEADER file from the shared setup. Do not include token values in logs or support records. Follow credential cleanup when you finish.

Neither audit access nor a support reference grants recovery mutation authority. Do not grant policy_admin merely to collect evidence. Use the permission reference.

Inspect Effective Scheduling

Complete the following steps:

1. Read the release’s nonsecret values.

   helm --kube-context "$CTX" --namespace pty-policy get values policy
   

   Keep credentials out of the evidence record. Review approved image identities alongside these values.

2. Read the generated recompilation interval.

   kubectl --context "$CTX" --namespace pty-policy get configmap policy-api-config \
     -o jsonpath='{.data.policy_recompile_interval_seconds}{"\n"}'
   

   The default chart produces 300. A nonpositive value disables scheduling. Use the configured name if your release changes apiService.containerName.

3. Verify the deployment binds POLICY_RECOMPILE_INTERVAL_SECONDS to that key. Inspect only the relevant nonsecret section of the workload specification. A value elsewhere in a file does not establish the running process’s configuration.

4. Read recent API logs.

   kubectl --context "$CTX" --namespace pty-policy logs \
     deployment/policy-api --container policy-api --since=30m
   

   Look for the startup interval message and subsequent cycle summaries. Retain timestamps, diagnostic phases, and superseded or deferred reasons. Redact sensitive details before sharing logs.

Source defaults and generated configuration can differ from a process that has not restarted after a change. Do not print the full process environment. For supported deployment settings, refer to configuration.

Inspect Health and Progress

Complete the following steps:

1. Inspect health through the authenticated gateway.

   curl --cacert "$PPC_CA_FILE" --fail-with-body --include \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/live"
   curl --cacert "$PPC_CA_FILE" --fail-with-body --include \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/ready"
   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/health"
   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/version"
   

   Liveness returns Hypertext Transfer Protocol (HTTP) 204 for a responding process. Readiness returns empty 204, or 503 when a startup prerequisite fails. Health and version return JavaScript Object Notation (JSON) responses. Structured health can report lifecycle degradation with HTTP 200. A 401 requires valid authentication. A 403 requires the appropriate permissions.

2. Inspect metrics with observability authority.

   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/metrics"
   

   Expect Prometheus text, not JavaScript Object Notation (JSON). The outbox is the durable queue of publication and withdrawal events. Terminal work has stopped automatic attempts. An operator must investigate before selecting recovery. The following metrics help distinguish stalled work:

   Metric	Interpretation
   pmx_policy_recompilation_total	Count cycle outcomes as complete, blocked, skipped, or error.
   pmx_policy_recompilation_last_pass_success	Report whether the last owned cycle completed.
   pmx_policy_recompilation_failed_policies	Count failed policies in the last owned cycle.
   pmx_policy_recompilation_last_success_timestamp_seconds	Record the current process’s last successful cycle time.
   pmx_policy_outbox_pending	Count retained pending publication or withdrawal work.
   pmx_policy_outbox_oldest_age_seconds	Measure age since the oldest pending event’s creation.
   pmx_policy_outbox_terminal	Count retained terminal work requiring investigation.

   Recompilation counters and last-success metrics reset with the process. Skipped cycles do not replace the last owned result. Repository-backed queue gauges refresh on scrape.

3. Inspect affected policies through permitted history and operation reads. Use the recovery API guide. Record revision, deployment generation, attempts, diagnostics, and replacement information.

4. Use the error message to select recompilation troubleshooting or publication recovery. Send connection failures to the named integration’s administrator and policy conflicts to the policy author and reviewer.

Do not infer protection from healthy services, an empty queue, or a successful cycle. Verify target-specific evidence and actual target behavior independently. Refer to automatic recompilation outcomes to distinguish unchanged, blocked, superseded, and deferred work.

Restart Under Change Control

Complete the following steps:

1. Capture current logs, metrics, deployment generations, and pending work. If a PIM request lost its response, ask the PIM administrator to check whether the change completed before restarting.

2. Confirm the restart uses the same approved image and database schema.

3. Restart the default API Deployment during the approved outage window.

   kubectl --context "$CTX" --namespace pty-policy rollout restart deployment/policy-api
   

   Expect service interruption with the single-instance deployment. This restarts all process-owned background services.

4. Wait for rollout completion.

   kubectl --context "$CTX" --namespace pty-policy rollout status \
     deployment/policy-api --timeout=5m
   

   If startup waits or fails, inspect schema checks, credentials, and events. Do not weaken readiness or runtime privileges to bypass the failure.

5. Repeat authenticated verification and inspect later worker cycles. Check that pending operations resume after existing worker leases expire. Use publication recovery for operations that stopped after failure.

Normal shutdown waits for active recompilation and publication work. The chart’s termination grace period is 30 seconds. Long external calls can exceed it. Shutdown does not guarantee every write drained. Preserve uncertain-write evidence and follow recovery.

For an approved extended stop, use your deployment-management pause procedure. Confirm no controller or Helm reconciliation will restart the API unexpectedly. Do not delete credentials, claims, or database rows to stop processing. On resumption, recheck external state and retained work before declaring recovery.

Related: Background processing, automatic recompilation, and backup and restore.