System Operations

• 1: Background Processing and Cadence
• 2: Automatic Recompilation of Approved Policies
• 3: Observe and Restart Policy

1 - Background Processing and Cadence

The application programming interface (API) process owns Policy’s background services. They are not separate Kubernetes Deployments. Each service performs a different task. One successful worker does not prove another worker completed its work. Native import and publication use Policy Integration Manager (PIM).

Services and Defaults

The following table summarizes each background service, its default cadence, and its purpose:

The configuration guide explains the distinction between chart values, process environment, and credentials. Record effective settings for the running release rather than relying on this table.

Periodic services sleep before work and run serial cycles within each task. Execution and dependency waits add time to the configured interval. These intervals are not activation guarantees or service-level commitments. The current chart requires one API replica. Do not scale it for faster processing.

Startup Import and Discovery Freshness

PIM startup import defaults to enabled independently of periodic import. A startup import failure does not itself block API startup. Disabling periodic import does not disable that startup attempt. Periodic import also requires a configured PIM base address.

Import discovers existing native policies. It does not automatically adopt them into governed publication. Resolve ownership and legacy adoption through authorized workflows.

Disabling periodic catalog refresh leaves lazy first-request loading. A failed ordinary refresh can leave the prior snapshot available. Automatic recompilation applies a stricter fresh-input gate: it cannot silently compile from that retained snapshot.

Publication and Protection

Approval queues publication rather than performing every target write synchronously. The publication worker processes bounded batches of retained events. These events form the outbox, a database-backed queue that survives a process restart. The worker can finish a request, schedule another attempt, or stop after failure. A processed-event count is not a successful-publication count.

Default retry delays are 5, 15, 30, 60, and 300 seconds. Deterministic refusals can stop earlier than transient connectivity failures. Worker ownership leases default to 300 seconds. A lease temporarily reserves work for one worker. Terminal means automatic attempts have stopped, not that publication succeeded or the policy disappeared. After process loss, reclaiming unfinished work can wait for lease expiry. Restarting does not retry work whose automatic attempts have stopped.

Protection reconciliation evaluates revision-scoped evidence independently. It compares observations with the current governed revision and deployment generation. That generation distinguishes delivery operations. It is not the policy’s authored revision number. For Data, missing, old, or incomplete observations keep Policy from reporting Enforced. For Federated Guardrails, a matching receipt records consumer retrieval, not runtime activation. A policy using both targets needs complete Data observations and the matching Guardrails receipt. Refer to lifecycle and delivery.

Disablement and Stopping

A nonpositive polling interval disables the corresponding periodic service. It does not erase durable work, withdraw native policies, or grant a manual trigger. Stopping the API stops scheduling all its background services and interrupts user access. It does not stop independent consumers or undo native writes.

Use the configuration workflow for supported cadence changes. Refer to monitoring and restarts for a controlled process change. Do not repurpose database restore mode as an ordinary pause mechanism.

2 - Automatic Recompilation of Approved Policies

Policy’s application programming interface (API) periodically recompiles approved policies against fresh discovery and live Policy Integration Manager (PIM) inventory. Source changes can alter generated output without changing an author’s intent. This background check does not replace human review or approval.

For example, fresh discovery can associate another location with an existing classification. The same approved intent can then produce different target artifacts. The worker validates the refreshed set before accepting compilation changes. It does not react immediately to each discovery change event.

Scheduling and Effective Configuration

The backend setting policy_recompile_interval_seconds defaults to 300 seconds. The environment variable POLICY_RECOMPILE_INTERVAL_SECONDS supplies that setting. The shipped chart also defaults to 300 through this value:

apiService:
  environment:
    POLICY_RECOMPILE_INTERVAL_SECONDS: "300"

Nonpositive values disable periodic recompilation. The worker sleeps before its first cycle and between cycles. Execution time adds to the interval. Five minutes is not a completion deadline. Disabling scheduling preserves stored revisions and queued publication work. It neither withdraws policies nor disables the other workers.

Check the actual release values, generated configuration, and startup message. Do not assume source defaults match an existing installation. Refer to observation and restarts for nonsecret checks.

There is no public run-now endpoint for automatic recompilation. Restarting the API is not a supported immediate-trigger mechanism. The next process also sleeps before its periodic cycle.

Fresh Inputs and Set-Wide Validation

An eligible cycle refreshes discovery, then reads live PIM inventory. It pins those fresh inputs for compilation and peer validation. Pinning means using the captured inputs throughout that cycle, rather than mixing later refreshes into the same result. It validates both individual policies and their combined generated output. Changed source context can reveal conflicts that earlier inputs did not expose.

Failed discovery refresh does not permit recompilation from cached discovery. Unavailable live inventory also prevents acceptance. The catalog’s separate ability to retain an old snapshot does not relax this rule. An apparently usable catalog can therefore coexist with blocked recompilation.

Blocking compilation, validation, or persistence failures preserve the previous governed set. Policy does not accept a partially refreshed subset. This preservation concerns local governed state and queued work. It does not certify target availability or undo earlier external changes. Here, fail-closed means refusing an unsafe replacement while retaining previous governed state. It does not mean automatically denying all target traffic.

What Stays Unchanged

Automatic recompilation preserves:

• The author’s saved intent and the approval that authorized it.
• The original approval identity and decision information.
• Any open replacement draft, including unfinished or in-review work.
• Earlier governed revisions and their historical evidence.

System compilation history remains distinct from a person’s approval event. The worker records its compilation action rather than impersonating an approver. An author still needs the ordinary submission and approval workflow to change intent. Refer to states and revisions.

Changed and Unchanged Output

Changed accepted output creates compilation minor versions under the existing governed major revision. It also queues asynchronous publication work. This is not a new authored major revision or a new human approval.

Unchanged output creates neither compilation minors nor publication work. A successful cycle with no new version can therefore be normal. Do not treat an unchanged version number as proof that the worker stopped.

Successful compilation establishes that Policy accepted refreshed artifacts. Queued publication does not establish delivery. Publication and consumer receipt do not independently establish actual target protection. Continue through the delivery and lifecycle guidance for target-specific verification.

Outcomes and Operator Interpretation

Use this table to interpret recompilation outcomes and choose an operator response:

Concurrent governance can supersede work during compilation or persistence. Deferred work can include active publication constraints or legacy adoption requirements. Legacy deployments do not gain automatic governed adoption or publication. Some safe source-preserving scope updates can proceed automatically. Do not infer authorization for every native ownership change.

Observation and Recovery

An operator should collect the following evidence before changing configuration:

• Effective interval, process start time, and recent cycle timestamps.
• Policy count, failed-policy count, completion flag, and diagnostic phase.
• Superseded or deferred indicators and any returned reason.
• Current governed revision, compilation minor, and relevant open draft.
• Dependency availability, request identifiers, and later deployment generation.

The API logs an Approved policy recompilation summary for owned cycles. Inspect structured log fields where your logging platform retains them. Metrics count cycle outcomes. API logs provide the phase and reason. A blocked metric can represent blocked, superseded, or deferred work. Read the logged reason before changing a setting or retrying publication.

For a transient dependency failure, restore access and observe a later scheduled cycle. Do not substitute cached inventory or reduce the interval to bypass validation. For missing policy inputs or conflicts, ask the policy author and reviewer to correct and approve the proposed change. For legacy adoption or missing PIM mappings, ask the Policy administrator to follow ownership recovery. Do not edit database records, erase checkpoints, or manufacture approval evidence.

Publication retry sends the configuration saved for that delivery request. It does not incorporate corrected drafts or fresh discovery. The reviewer’s validation request checks a saved draft. It does not start automatic recompilation.

Refer to monitoring and restarts, publication recovery, and API recovery and reconciliation.

3 - Observe and Restart Policy

Use this task as the platform operator during routine checks or incidents. These diagnostic checks are not installation steps. Read the error before restarting. Restarting does not correct policy fields or missing Policy Integration Manager (PIM) object mappings. The application programming interface (API) process also owns background scheduling.

Prerequisites

Prepare the following:

• Set CTX to the intended Kubernetes context.
• Obtain workload-read access for logs, events, and configuration references.
• Obtain policy:observability for metrics and policy:audit for restricted operation evidence.
• Obtain separate deployment authority only when a restart requires it.
• Obtain your platform hostname, trusted certificate authority (CA) bundle, and current access token.

Configure API Access

Complete the credential-safe request setup with your own user token in a dedicated Bash session. Then set the deployment endpoint and trusted CA file in that same shell:

export FQDN='your-platform-hostname'
export PPC_CA_FILE='your-platform-ca.pem'
export POLICY_BASE_URL="https://${FQDN}/pty/policy/v1.0"
POLICY_API="$POLICY_BASE_URL"
: "${POLICY_AUTH_HEADER:?Complete authentication setup}"

Use the configured API root if your platform overrides the default. All requests use the protected POLICY_AUTH_HEADER file from the shared setup. Do not include token values in logs or support records. Follow credential cleanup when you finish.

Neither audit access nor a support reference grants recovery mutation authority. Do not grant policy_admin merely to collect evidence. Use the permission reference.

Inspect Effective Scheduling

Complete the following steps:

1. Read the release’s nonsecret values.

   helm --kube-context "$CTX" --namespace pty-policy get values policy
   

   Keep credentials out of the evidence record. Review approved image identities alongside these values.

2. Read the generated recompilation interval.

   kubectl --context "$CTX" --namespace pty-policy get configmap policy-api-config \
     -o jsonpath='{.data.policy_recompile_interval_seconds}{"\n"}'
   

   The default chart produces 300. A nonpositive value disables scheduling. Use the configured name if your release changes apiService.containerName.

3. Verify the deployment binds POLICY_RECOMPILE_INTERVAL_SECONDS to that key. Inspect only the relevant nonsecret section of the workload specification. A value elsewhere in a file does not establish the running process’s configuration.

4. Read recent API logs.

   kubectl --context "$CTX" --namespace pty-policy logs \
     deployment/policy-api --container policy-api --since=30m
   

   Look for the startup interval message and subsequent cycle summaries. Retain timestamps, diagnostic phases, and superseded or deferred reasons. Redact sensitive details before sharing logs.

Source defaults and generated configuration can differ from a process that has not restarted after a change. Do not print the full process environment. For supported deployment settings, refer to configuration.

Inspect Health and Progress

Complete the following steps:

1. Inspect health through the authenticated gateway.

   curl --cacert "$PPC_CA_FILE" --fail-with-body --include \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/live"
   curl --cacert "$PPC_CA_FILE" --fail-with-body --include \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/ready"
   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/health"
   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/version"
   

   Liveness returns Hypertext Transfer Protocol (HTTP) 204 for a responding process. Readiness returns empty 204, or 503 when a startup prerequisite fails. Health and version return JavaScript Object Notation (JSON) responses. Structured health can report lifecycle degradation with HTTP 200. A 401 requires valid authentication. A 403 requires the appropriate permissions.

2. Inspect metrics with observability authority.

   curl --cacert "$PPC_CA_FILE" --fail-with-body \
     --header "@${POLICY_AUTH_HEADER:?Complete authentication setup}" "$POLICY_BASE_URL/metrics"
   

   Expect Prometheus text, not JavaScript Object Notation (JSON). The outbox is the durable queue of publication and withdrawal events. Terminal work has stopped automatic attempts. An operator must investigate before selecting recovery. The following metrics help distinguish stalled work:

   Metric	Interpretation
   pmx_policy_recompilation_total	Count cycle outcomes as complete, blocked, skipped, or error.
   pmx_policy_recompilation_last_pass_success	Report whether the last owned cycle completed.
   pmx_policy_recompilation_failed_policies	Count failed policies in the last owned cycle.
   pmx_policy_recompilation_last_success_timestamp_seconds	Record the current process’s last successful cycle time.
   pmx_policy_outbox_pending	Count retained pending publication or withdrawal work.
   pmx_policy_outbox_oldest_age_seconds	Measure age since the oldest pending event’s creation.
   pmx_policy_outbox_terminal	Count retained terminal work requiring investigation.

   Recompilation counters and last-success metrics reset with the process. Skipped cycles do not replace the last owned result. Repository-backed queue gauges refresh on scrape.

3. Inspect affected policies through permitted history and operation reads. Use the recovery API guide. Record revision, deployment generation, attempts, diagnostics, and replacement information.

4. Use the error message to select recompilation troubleshooting or publication recovery. Send connection failures to the named integration’s administrator and policy conflicts to the policy author and reviewer.

Do not infer protection from healthy services, an empty queue, or a successful cycle. Verify target-specific evidence and actual target behavior independently. Refer to automatic recompilation outcomes to distinguish unchanged, blocked, superseded, and deferred work.

Restart Under Change Control

Complete the following steps:

1. Capture current logs, metrics, deployment generations, and pending work. If a PIM request lost its response, ask the PIM administrator to check whether the change completed before restarting.

2. Confirm the restart uses the same approved image and database schema.

3. Restart the default API Deployment during the approved outage window.

   kubectl --context "$CTX" --namespace pty-policy rollout restart deployment/policy-api
   

   Expect service interruption with the single-instance deployment. This restarts all process-owned background services.

4. Wait for rollout completion.

   kubectl --context "$CTX" --namespace pty-policy rollout status \
     deployment/policy-api --timeout=5m
   

   If startup waits or fails, inspect schema checks, credentials, and events. Do not weaken readiness or runtime privileges to bypass the failure.

5. Repeat authenticated verification and inspect later worker cycles. Check that pending operations resume after existing worker leases expire. Use publication recovery for operations that stopped after failure.

Normal shutdown waits for active recompilation and publication work. The chart’s termination grace period is 30 seconds. Long external calls can exceed it. Shutdown does not guarantee every write drained. Preserve uncertain-write evidence and follow recovery.

For an approved extended stop, use your deployment-management pause procedure. Confirm no controller or Helm reconciliation will restart the API unexpectedly. Do not delete credentials, claims, or database rows to stop processing. On resumption, recheck external state and retained work before declaring recovery.

Related: Background processing, automatic recompilation, and backup and restore.