Ansible and Semaphore orchestrate all servers. Backups run nightly, integrity is checked weekly, and a probe restore runs monthly — all without touching a terminal.
All server configuration runs through Ansible, executed from a dedicated management server. Semaphore provides a web-based job runner and scheduler on top of Ansible, replacing manual terminal work with a trackable, repeatable interface.
The local AI has direct read and write access to the Ansible repository, with a second model reviewing every proposed change before it is written to disk. See the AI Ansible page for details.
All recurring tasks are scheduled through Semaphore. Jobs are staggered deliberately — the backup tool requires an exclusive lock on each repository, so no two backup jobs overlap. The schedule covers daily backups, weekly system updates, weekly integrity checks, and a monthly restore test.
Incremental encrypted backups of all servers to the local NAS. Database services get dedicated dump-then-snapshot jobs that run before the general backup window.
System package updates across all servers every week. Integrity check across all backup repositories. Web application container updates. All results reported by email.
A file is actually extracted from the most recent snapshot and verified. Confirms that data can be recovered, not just that the repository is structurally intact.
All servers are backed up daily using Restic — an incremental, encrypted, deduplicated backup tool. Backups are written to the local NAS. Restic's deduplication means that unchanged data is not stored again, keeping the backup size manageable even with daily snapshots.
Database services get their own backup jobs: the database is dumped to disk first, then the directory including the dump is snapshotted by Restic. The password manager and the job scheduler database are also backed up separately.
A second NAS acts as a backup target for disaster recovery — keeping a separate copy of all data in case of primary NAS failure.
An Ansible playbook runs a full integrity check across all backup repositories every week. The result is sent by email with a subject line that starts with [OK] or [FEHLER], making the outcome visible without opening the message.
A file is actually extracted from the most recent snapshot and verified. This confirms that not only is the repository structurally intact, but that data can actually be recovered from it. Both verification jobs skip the external mail relay servers, which have their own separate backup path.
Early in the project, a misconfigured schedule caused a DNS filter update job to run sixty times in an hour instead of once. The asterisk in the minute field of a cron expression means "every minute", not "at the start of the hour". The corrected expression runs once per week.
Reboots are never performed automatically by any playbook. When a package update requires a reboot, the playbook notes the fact and stops. Reboots happen manually at a chosen moment.
No two backup jobs run against the same repository at the same time. The backup tool holds an exclusive lock while writing, and a second job that finds the repository locked will fail rather than queue. The schedule was designed with this constraint in mind from the start.