Notes

Taming Ceph rebalance traffic

Swapping a failing disk kicked off a backfill that ate most of the cluster's replication bandwidth. The fix: throttle recovery so client I/O keeps priority.

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2
ceph config set osd osd_recovery_op_priority 1

After that a single OSD replacement rebalanced quietly overnight.

vmalert rules that actually page me

Alert fatigue is real. If I wouldn't get out of bed for it, it isn't an alert. Disk filling within 4h, a node down, cert expiry inside a week, replication lag past a threshold — those page. CPU spikes don't.

Ansible + Vault without leaking to logs

Tasks templating secrets print them in verbose diffs. no_log: true on the offending tasks fixed it.

- name: render app config
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    mode: "0640"
  no_log: true