Pre-release live HA test plan #11

New issue

Closed

opened 2026-10-18 17:29:27 -07:00 by eric · 1 comment

eric commented

2026-10-18 17:29:27 -07:00

Owner

Live HA-pair tests to run before tagging the next release. Add to this issue as further bugs are fixed and need their own tests.

Run on both nodes unless a step says otherwise. Follow the log with opnsense-log -f | grep wancarp.

Setup (after installing the new files on both nodes)

Script installed at /usr/local/etc/rc.syshook.d/carp/10-wancarp, mode 0700.
actions_wancarp.conf installed in /usr/local/opnsense/service/conf/actions.d/, then service configd restart.
configctl wancarp reconcile prints OK and logs nothing (state already correct).
System → Settings → Cron lists "Reconcile WAN with CARP role (10-wancarp)". Add it with minute * on both nodes (or let HA sync carry the cron job).
With the cron running and no failover, a few minutes go by with no wancarp lines in the log.

Node role (#4)

./10-wancarp --dryrun --reconcile on the MASTER: the role=MASTER line lists every VHID as MASTER, and WAN's VHID (if any) shows as skipped. Reports already up, skipping.
Same on the BACKUP: role=BACKUP, reports already down, skipping.
./10-wancarp --dryrun 1@<dev> MASTER on the BACKUP: what-if line says a real event would act as BACKUP, then shows WOULD enable.
Maintenance-mode failover in both directions. Each node's log shows one role= line per VHID event. WAN ends up live on exactly one node, with IP and default route.
(If you have more than one VHID) Mid-failover log: events that arrive while VHIDs disagree show role=BACKUP and do not bring WAN up. WAN only comes up once all counted VHIDs are MASTER.
Optional: set $carp_vhids = [<one vhid>] on both nodes, and confirm the role= line marks the others skipped (not in $carp_vhids). Revert afterwards.

Drift correction (#3)

On the BACKUP: configctl interface reconfigure wan. WAN comes back UP (expected, this is the bug). Within about a minute the cron logs START…END with disabling interface 'wan' plus the error-level reconcile: interface state had drifted ... line, and WAN is down again.
On the BACKUP: Save/Apply on the WAN interface page. Same result.
On the BACKUP: trigger a real LINK_UP (reboot the modem, re-seat the cable, or toggle the virtio link on the host). Same result.
On the MASTER: configctl interface reconfigure wan. WAN stays up and reconcile logs nothing.

CARP VIP on a toggled interface (#5)

./10-wancarp --dryrun --reconcile on each node with the real config: no DRYRUN: CARP VHID ... is on '<alias>' error. (If one appears, that interface carries a VIP and must come out of $iface_aliases before release testing goes further.)
On either node, copy the script somewhere other than the hook dir (cp /usr/local/etc/rc.syshook.d/carp/10-wancarp /root/wancarp-test), and in the copy add an alias that does carry a CARP VIP (e.g. 'lan') to $iface_aliases. Run /root/wancarp-test --dryrun --reconcile: an error-level DRYRUN: CARP VHID <n> is on 'lan' (<dev>) ... line appears for each VHID on it, and the role= line marks those VHIDs skipped (on a toggled interface). Nothing changes on the box (dry-run only). Delete the copy afterwards.

Default-route repoint, both directions (#6)

These need $remove_backup_route = true and $lan_vip set to the LAN CARP VIP in the installed hook on both nodes. If you don't run with it in production, set it back to false afterwards.

./10-wancarp --dryrun --reconcile on each node: preflight shows API system_routing_configure() present. and the current default route.
On the BACKUP: route -n get -inet default shows gateway $lan_vip. Repeated CARP events and reconciles log default route already points at '<vip>', skipping. or nothing.
Maintenance-mode failover in both directions. The newly promoted MASTER logs default route points at '<vip>', restoring it via system_routing_configure() and then default gateway is now <ISP gw>. If it says none (WAN gateway not ready yet), the WAN default must show up shortly after through newwanip. Either way, netstat -rn ends up with the default via the WAN gateway and external pings work. The newly demoted node repoints to $lan_vip.
(If you can set up a static-IP WAN) Repeat the failover with it. This is the case #6 couldn't confirm.
Already-down WAN: on the BACKUP, route delete -inet default. Within about a minute the cron logs repointing default route from none to '<vip>' plus the error-level drift line, and the default points at $lan_vip again.
Optional, costs the MASTER its Internet for up to a minute: on the MASTER, route change -inet default <lan_vip>. Within about a minute the reconcile restores OPNsense's default route and logs the drift line.

END banner reports failures (#7)

On the BACKUP, with $remove_backup_route = true set as above: cp /usr/local/etc/rc.syshook.d/carp/10-wancarp /root/wancarp-test, and in the copy set $lan_vip = '198.51.100.1' (an off-link address). Run /root/wancarp-test --reconcile (live, not dry-run). It deletes the current default route, then route add fails. The log shows 'route add -inet default 198.51.100.1' failed (rc=...) and the END banner says result=partial, not result=ok.
Then configctl wancarp reconcile (the real hook) puts the default back on the real $lan_vip, and its END banner says result=ok. Delete the copy.
Across this whole test run, every END line says result=ok except the deliberate failure above.

MASTER skips only a configured interface (#8)

On the BACKUP (WAN down): pgrep -F /var/run/dhclient.<wan dev>.pid prints nothing. A dhclient left running on the downed WAN would make the MASTER guard skip an unconfigured interface after promotion.
On the MASTER: ./10-wancarp --dryrun --reconcile still logs interface 'wan' (<dev>) already up, skipping. (no false reconfigure of a healthy WAN).
UP but unconfigured, costs the MASTER its WAN for up to a minute: on the MASTER, pkill -F /var/run/dhclient.<dev>.pid; ifconfig <dev> inet <wan ip> delete. Check that ifconfig <dev> still shows UP in flags but has no inet line and no global inet6 (if the WAN also has global IPv6, this test needs that removed too). Then run ./10-wancarp --dryrun --reconcile: it logs is UP but has no address or DHCP client; reconfiguring it. followed by WOULD enable. Within about a minute the cron reconcile does it for real: enabling interface 'wan', the drift line, and WAN has its IP, default route and Internet again.
Best effort (the lease often comes back too fast to catch): on the MASTER, configctl interface reconfigure wan, then straight away run ./10-wancarp --dryrun --reconcile while ifconfig shows no inet yet. It says already up, skipping because dhclient is running, not reconfiguring it.

Regression

Hard-reboot the MASTER: the BACKUP takes over and WAN moves with its IP and default route. When the old MASTER rejoins, it ends up with WAN in the correct state for its role.
Hard-reboot the BACKUP: no WAN on it after it rejoins, and the MASTER is unaffected. No reconcile activity in the log during boot.
Lock not leaked to children (feb906f): after any promotion, fstat /tmp/wancarp.lock shows no holders, and a later demotion of that same node (no reboot in between) takes WAN down without waiting on anything.
No START without a matching END anywhere in the log during the above.

Live HA-pair tests to run before tagging the next release. Add to this issue as further bugs are fixed and need their own tests. Run on both nodes unless a step says otherwise. Follow the log with `opnsense-log -f | grep wancarp`. ## Setup (after installing the new files on both nodes) - [x] Script installed at `/usr/local/etc/rc.syshook.d/carp/10-wancarp`, mode `0700`. - [x] `actions_wancarp.conf` installed in `/usr/local/opnsense/service/conf/actions.d/`, then `service configd restart`. - [x] `configctl wancarp reconcile` prints `OK` and logs **nothing** (state already correct). - [x] **System → Settings → Cron** lists "Reconcile WAN with CARP role (10-wancarp)". Add it with minute `*` on both nodes (or let HA sync carry the cron job). - [x] With the cron running and no failover, a few minutes go by with **no** `wancarp` lines in the log. ## Node role (#4) - [x] `./10-wancarp --dryrun --reconcile` on the MASTER: the `role=MASTER` line lists every VHID as `MASTER`, and WAN's VHID (if any) shows as skipped. Reports `already up, skipping`. - [x] Same on the BACKUP: `role=BACKUP`, reports `already down, skipping`. - [x] `./10-wancarp --dryrun 1@<dev> MASTER` on the BACKUP: what-if line says a real event would act as `BACKUP`, then shows `WOULD enable`. - [x] Maintenance-mode failover in both directions. Each node's log shows one `role=` line per VHID event. WAN ends up live on exactly one node, with IP and default route. - [x] (If you have more than one VHID) Mid-failover log: events that arrive while VHIDs disagree show `role=BACKUP` and do not bring WAN up. WAN only comes up once all counted VHIDs are MASTER. - [x] Optional: set `$carp_vhids = [<one vhid>]` on both nodes, and confirm the `role=` line marks the others `skipped (not in $carp_vhids)`. Revert afterwards. ## Drift correction (#3) - [x] On the BACKUP: `configctl interface reconfigure wan`. WAN comes back UP (expected, this is the bug). Within about a minute the cron logs START…END with `disabling interface 'wan'` plus the error-level `reconcile: interface state had drifted ...` line, and WAN is down again. - [x] On the BACKUP: Save/Apply on the WAN interface page. Same result. - [x] On the BACKUP: trigger a real LINK_UP (reboot the modem, re-seat the cable, or toggle the virtio link on the host). Same result. - [x] On the MASTER: `configctl interface reconfigure wan`. WAN stays up and reconcile logs nothing. ## CARP VIP on a toggled interface (#5) - [x] `./10-wancarp --dryrun --reconcile` on each node with the real config: **no** `DRYRUN: CARP VHID ... is on '<alias>'` error. (If one appears, that interface carries a VIP and must come out of `$iface_aliases` before release testing goes further.) - [x] On either node, copy the script somewhere other than the hook dir (`cp /usr/local/etc/rc.syshook.d/carp/10-wancarp /root/wancarp-test`), and in the copy add an alias that **does** carry a CARP VIP (e.g. `'lan'`) to `$iface_aliases`. Run `/root/wancarp-test --dryrun --reconcile`: an error-level `DRYRUN: CARP VHID <n> is on 'lan' (<dev>) ...` line appears for each VHID on it, and the `role=` line marks those VHIDs `skipped (on a toggled interface)`. Nothing changes on the box (dry-run only). Delete the copy afterwards. ## Default-route repoint, both directions (#6) These need `$remove_backup_route = true` and `$lan_vip` set to the LAN CARP VIP in the installed hook on **both** nodes. If you don't run with it in production, set it back to `false` afterwards. - [x] `./10-wancarp --dryrun --reconcile` on each node: preflight shows `API system_routing_configure() present.` and the current default route. - [x] On the BACKUP: `route -n get -inet default` shows gateway `$lan_vip`. Repeated CARP events and reconciles log `default route already points at '<vip>', skipping.` or nothing. - [x] Maintenance-mode failover in both directions. The newly promoted MASTER logs `default route points at '<vip>', restoring it via system_routing_configure()` and then `default gateway is now <ISP gw>`. If it says `none (WAN gateway not ready yet)`, the WAN default must show up shortly after through newwanip. Either way, `netstat -rn` ends up with the default via the WAN gateway and external pings work. The newly demoted node repoints to `$lan_vip`. - [ ] (If you can set up a static-IP WAN) Repeat the failover with it. This is the case #6 couldn't confirm. - [x] Already-down WAN: on the BACKUP, `route delete -inet default`. Within about a minute the cron logs `repointing default route from none to '<vip>'` plus the error-level drift line, and the default points at `$lan_vip` again. - [x] Optional, costs the MASTER its Internet for up to a minute: on the MASTER, `route change -inet default <lan_vip>`. Within about a minute the reconcile restores OPNsense's default route and logs the drift line. ## END banner reports failures (#7) - [x] On the BACKUP, with `$remove_backup_route = true` set as above: `cp /usr/local/etc/rc.syshook.d/carp/10-wancarp /root/wancarp-test`, and in the copy set `$lan_vip = '198.51.100.1'` (an off-link address). Run `/root/wancarp-test --reconcile` (live, not dry-run). It deletes the current default route, then `route add` fails. The log shows `'route add -inet default 198.51.100.1' failed (rc=...)` and the END banner says `result=partial`, **not** `result=ok`. - [x] Then `configctl wancarp reconcile` (the real hook) puts the default back on the real `$lan_vip`, and its END banner says `result=ok`. Delete the copy. - [x] Across this whole test run, every `END` line says `result=ok` except the deliberate failure above. ## MASTER skips only a configured interface (#8) - [x] On the BACKUP (WAN down): `pgrep -F /var/run/dhclient.<wan dev>.pid` prints nothing. A dhclient left running on the downed WAN would make the MASTER guard skip an unconfigured interface after promotion. - [x] On the MASTER: `./10-wancarp --dryrun --reconcile` still logs `interface 'wan' (<dev>) already up, skipping.` (no false reconfigure of a healthy WAN). - [x] UP but unconfigured, costs the MASTER its WAN for up to a minute: on the MASTER, `pkill -F /var/run/dhclient.<dev>.pid; ifconfig <dev> inet <wan ip> delete`. Check that `ifconfig <dev>` still shows `UP` in flags but has no `inet` line and no global `inet6` (if the WAN also has global IPv6, this test needs that removed too). Then run `./10-wancarp --dryrun --reconcile`: it logs `is UP but has no address or DHCP client; reconfiguring it.` followed by `WOULD enable`. Within about a minute the cron reconcile does it for real: `enabling interface 'wan'`, the drift line, and WAN has its IP, default route and Internet again. - [x] Best effort (the lease often comes back too fast to catch): on the MASTER, `configctl interface reconfigure wan`, then straight away run `./10-wancarp --dryrun --reconcile` while `ifconfig` shows no `inet` yet. It says `already up, skipping` because dhclient is running, **not** `reconfiguring it`. ## Regression - [x] Hard-reboot the MASTER: the BACKUP takes over and WAN moves with its IP and default route. When the old MASTER rejoins, it ends up with WAN in the correct state for its role. - [x] Hard-reboot the BACKUP: no WAN on it after it rejoins, and the MASTER is unaffected. No reconcile activity in the log during boot. - [x] Lock not leaked to children (feb906f): after any promotion, `fstat /tmp/wancarp.lock` shows no holders, and a later demotion of that same node (no reboot in between) takes WAN down without waiting on anything. - [x] No `START` without a matching `END` anywhere in the log during the above.

eric referenced this issue from a commit

2026-10-20 21:13:05 -07:00

fix: open the lock file close-on-exec

eric commented

2026-10-22 17:09:31 -07:00

Author

Owner