Configuration Management

Configuration management (CM) keeps software and machines in an intended, known state. It covers application settings, operating-system state, policy, and controlled differences between environments. The central problem is ownership and convergence, not which tool renders YAML.

Classify Configuration

Keep these concerns distinct:

• build-time inputs determine artifact contents;
• deploy-time configuration selects environment behavior;
• runtime controls change behavior without a new deployment;
• secrets carry authentication or confidential values;
• infrastructure definitions manage resource lifecycle;
• user or business data belongs in application storage.

Misclassification causes leaks and drift. A feature flag is not a secret store; an environment variable is not an audit trail; a configuration repository is not a business database.

Desired Properties

Configuration should be:

• explicit, typed, and validated before use;
• versioned when it is safe to store;
• minimal and owned;
• applied idempotently or through clear state transitions;
• attributable and auditable;
• scoped by environment and identity;
• observable without exposing sensitive values;
• recoverable to a known-good version.

Fail startup for invalid required configuration rather than silently choosing a dangerous fallback. Error messages should name the invalid key without printing its secret value.

Source and Precedence

Document one precedence order, for example:

safe defaults < versioned environment config < deployment overrides < emergency override

Fewer layers are easier to reason about. At runtime, expose which source supplied a non-sensitive setting and the effective configuration version. Ambiguous precedence creates incidents that source review cannot reproduce.

Application Configuration

Prefer a small schema with types, ranges, allowed values, cross-field rules, and deprecation warnings. Validate at build or deployment time when possible and again at the application boundary.

Changing configuration is a release:

1. review the proposed value and affected scope;
2. validate it against the target version;
3. roll it out progressively if risk warrants;
4. observe behavior;
5. roll back or continue by explicit criteria.

Hot reload avoids restart cost but introduces concurrent old/new state and failure handling. Use it only when the requirement justifies that complexity; otherwise restart safely with immutable configuration.

Secrets

Store secrets in a dedicated service or protected platform mechanism. Applications should receive the narrowest value or short-lived credential they need.

• do not commit, bake into images, or log secrets;
• avoid command-line arguments and broad environment dumps;
• encrypt in transit and at rest;
• restrict access by workload identity and purpose;
• rotate without coordinated downtime;
• audit reads and administrative changes;
• define behavior when retrieval or rotation fails.

Encryption does not solve authorization, leakage after decryption, or rotation.

Machine Configuration

Use a configuration-management tool when running machines must converge in place. A minimal idempotent playbook expresses state, not a sequence of hopeful shell commands:

- hosts: web
  become: true
  tasks:
    - name: Install web server
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Install reviewed configuration
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: "0644"
      notify: Reload nginx

  handlers:
    - name: Reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

Pin compatible collections and test against the supported operating systems. A task is idempotent only if its module and external side effects are idempotent.

Mutable vs Immutable Hosts

In-place convergence suits long-lived machines, edge devices, and software that cannot be replaced cheaply. Image replacement reduces accumulated drift and makes rollback clearer for replaceable compute.

Most systems combine both: a versioned base image, declarative infrastructure, and small runtime configuration. Choose one owner for each field and keep post-launch mutation minimal.

Drift

Drift is a difference between intended and observed state. Detect it continuously, then decide whether to:

• restore the declared state;
• accept the emergency change into source;
• transfer ownership to another controller;
• retire the obsolete declaration.

Do not auto-remediate every difference blindly. Reverting a legitimate incident action or provider-managed field can worsen an outage. Emergency changes need an owner, expiry, and follow-up reconciliation.

Environment Strategy

Keep structure common and data different. Copy-pasted environment directories drift; one giant template with conditional logic becomes unreadable. Use the smallest common base with explicit overlays only where environments genuinely differ.

Production data and credentials should not be required to test parsing, validation, or rendering. Test representative values and boundary cases in isolation.

Testing and Delivery

• lint and parse configuration;
• validate schemas and policy;
• render templates and inspect the result;
• apply twice to test convergence where safe;
• test supported OS and application versions;
• verify service behavior after application;
• test invalid inputs and rollback;
• canary high-risk changes.

Keep the same review, provenance, approval, and audit standards as code. A one-line configuration change can have a larger blast radius than a code change.

Tool Choice

Ansible, Puppet, Chef, Salt, and CFEngine differ in transport, agent model, language, and ecosystem. Prefer the tool already safely operated by the team. Introduce another only for a capability the current path cannot provide.

Use Infrastructure as Code for resource lifecycle, CI/CD for delivery, and a secret manager for sensitive values. Avoid making one tool own all three concerns.

Checklist

• Is every setting classified and owned?
• Is precedence short and documented?
• Are values typed and validated before impact?
• Are secrets separately stored, scoped, rotated, and audited?
• Can a change roll out progressively and roll back?
• Is drift detected without unsafe automatic correction?
• Can effective non-sensitive configuration be identified in production?
• Are temporary overrides owned and expiring?

Configuration management succeeds when the running state is explainable and intentional.