eric/10-wancarp
Watch
1
0
Fork
You've already forked 10-wancarp
 0
Code Issues 1 Pull requests Projects Releases 3 Packages Wiki Activity Actions

Release test plan: live checks before tagging the next release #21

New issue
Open
opened 2026-10-23 15:34:08 -07:00 by eric · 1 comment
eric commented 2026-10-23 15:34:08 -07:00
Owner

Purpose

Live tests to run on the HA pair before tagging the next release. The off-box harness (php tests/run.php) covers the logic; these cover what only a real box can confirm. Add a section here for each issue fixed before the release that needs a live check.

Install the main build on both nodes first and confirm the START banner reports version=<last release>+dev.

Watch the logs on both nodes throughout:

opnsense-log -f | grep wancarp

Pre-checks

  • On each node: /usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun --reconcile shows the expected role= line and no errors.
  • On each node, the same dry-run shows boot check product::booting() present; booting=no. That is the check the INIT rule relies on during boot (#15). If it reports class product MISSING instead, an INIT event during boot would take WAN down; reopen #15.

#15: all-INIT node takes its WAN down

A. Lost LAN link on the MASTER

  • Pull the MASTER's LAN cable, or disable its switch port or virtio link.
  • On the old MASTER, the log shows role=BACKUP counted=[1@<lan>=INIT] (MASTER for nothing, INIT), and WAN goes down (ifconfig <wan> has no UP flag).
  • The peer becomes MASTER and brings its WAN up, with Internet working.
  • Only one node has WAN up at any point after the event.
  • Wait at least one reconcile cycle (1 minute): the old MASTER's WAN stays down, and the reconcile logs no drift.
  • Restore the link. The node returns as BACKUP (or MASTER, if preempt takes it back) and WAN ends up live on exactly one node.

B. Temporarily Disable CARP

  • On the MASTER: Interfaces → Virtual IPs → Status → Temporarily Disable CARP.
  • Confirm the VHIDs show INIT in ifconfig (the issue flagged this as unconfirmed). If they show something else, note what and check the role line still takes WAN down.
  • WAN goes down on that node, and the peer takes over with its WAN up.
  • Re-enable CARP, and WAN ends up live on exactly one node.

Result (2026-10-24): the first run failed. Temporarily Disable CARP doesn't leave the VHIDs INIT: OPNsense removes them (no carp: lines in ifconfig) and sets net.inet.carp.allow=0. The hook logged role=undecided counted=[] and left WAN up, so both nodes had WAN live, and the internal network was down for about a minute. Fixed in 202d499: a node with net.inet.carp.allow=0 is BACKUP. On the re-run, the first INIT event logged role=BACKUP counted=[] (CARP disabled on this node, net.inet.carp.allow=0) and took WAN down. The peer took over, and after re-enabling, preempt moved WAN back. WAN was live on exactly one node throughout.

With $remove_backup_route = true, the returning MASTER restored its route while WAN was still waiting on DHCP, and OPNsense briefly set the default via an opt2 gateway (10.102.0.1) until the lease arrived. Afterwards it was via the WAN gateway on the MASTER and 192.168.0.1 on the BACKUP, as expected. The code comment and README now describe this (6188615); no code change needed.

C. Boot

  • Hard-reboot the MASTER. While booting, any INIT event logs (INIT while booting: left to the next event) and doesn't touch WAN (no disabling interface 'wan' line during boot). Before the #15 follow-up fix (222180a) the boot check always read "not booting" on a real box, so this is the check that matters. Once it rejoins, WAN ends up live on exactly one node.
  • Hard-reboot the BACKUP and check the same.
  • Boot a node with its LAN link down. After boot finishes, the next reconcile takes WAN down (role=BACKUP, had drifted).

Result (2026-10-24): MASTER and BACKUP hard resets (Proxmox Reset) both passed. Neither boot logged an INIT event. Each rejoined as BACKUP, and its BACKUP event took down the WAN that rc.bootup had brought up. The MASTER then took over again by preempt 6 s later, and WAN ended up live on exactly one node. With $remove_backup_route, OPNsense's boot routing set the default via 10.102.0.1 after the first CARP event. The next event, or the reconcile (had drifted), pointed it back at 192.168.0.1.

The LAN-down box is left unticked because it can't pass as written. This is OPNsense behaviour, not a script bug.

  • BACKUP rebooted with only vtnet1 down: VHID 2 reached BACKUP and took WAN down at once. A reconcile ran a minute later and logged had drifted (the route).
  • BACKUP rebooted with vtnet1 and vtnet2 both down: OPNsense's boot didn't finish (the last rc.bootup line was unbound_configure_do(1)). No CARP event fired, since the VHIDs were INIT from the start, and no reconcile ran because cron doesn't start until boot ends: none for 11+ minutes. The node kept rc.bootup's WAN up the whole time; no outage was noticed. Reconnecting the links fired BACKUP, which took WAN down within a second. In an earlier run, boot finished about 3 minutes after reconnecting.
  • Both boot-time gaps (WAN up until the first CARP event, and the all-links-down boot stall) are now in the README boot note (6058b86). Taking WAN down here would mean acting during boot, which the boot guard exists to prevent.

D. Dry-run

  • ./10-wancarp --dryrun 1@<lan> INIT on the MASTER prints the WOULD disable lines and changes nothing.

#17: BACKUP no longer calls interface_configure()

The removed call did nothing, so BACKUP should behave exactly as before. These checks confirm that on a real box.

  • Fail over the MASTER (maintenance mode is fine; it can be the same run as the regression check below). On the demoted node, the log shows disabling interface 'wan' and ends result=ok changed=yes, with no UNCAUGHT, partial or other error lines.
  • On that node, WAN is fully down: ifconfig <wan> has no UP flag and no inet address, and pgrep -fl 'dhclient.*<wan>' finds nothing.
  • If $remove_backup_route is set: netstat -rn shows default via $lan_vip, and the node can still reach the Internet through the peer (for example, pkg update works).
  • Wait one reconcile cycle (1 minute): WAN stays down and the reconcile logs no drift.
  • ./10-wancarp --dryrun 1@<lan> BACKUP on the MASTER prints would interface_reset('wan'), unset enable (in-memory), ifconfig <wan> down. with no interface_configure(), and changes nothing.

#18: lock in /var/run with a bounded wait

Run on the BACKUP, firing an event that matches its current role, so nothing changes even if the run went ahead. In one shell, hold the lock; in a second, fire the event:

lockf -k /var/run/wancarp.lock sleep 180
time /usr/local/etc/rc.syshook.d/carp/10-wancarp 1@<lan> BACKUP; echo "exit=$?"
  • The event run gives up after about 120 seconds with exit=1. The log shows timed out after 120s waiting for lock /var/run/wancarp.lock and END ... aborted=lock changed=no, and WAN is untouched.
  • With the lock still held, /usr/local/etc/rc.syshook.d/carp/10-wancarp --reconcile; echo "exit=$?" returns exit=0 at once and logs nothing.
  • After sleep ends, fire the same event again: it runs normally and ends result=ok changed=no.
  • ls -l /var/run/wancarp.lock shows -rw------- owned by root (not readable by other users, since flock() works on a read-only descriptor), and nothing new appears at /tmp/wancarp.lock (an old one left by 1.1.1 is harmless and can be deleted).
  • A non-root user can't take the lock: su -m nobody -c 'lockf -k -t 0 /var/run/wancarp.lock true'; echo "exit=$?" fails with Permission denied and a non-zero exit.

Regression: normal failover

  • Maintenance-mode failover (Enter Persistent CARP Maintenance Mode on the MASTER): WAN moves to the peer, then back when you leave maintenance mode.
  • Reconcile drift: on the BACKUP, run configctl interface reconfigure wan. Within a minute the reconcile takes WAN back down and logs had drifted.

Release

  • Every box above is ticked, or the reason it isn't is noted here.
  • Rename [Unreleased] in CHANGELOG.md, set VERSION, commit, and push the tag.
## Purpose Live tests to run on the HA pair before tagging the next release. The off-box harness (`php tests/run.php`) covers the logic; these cover what only a real box can confirm. Add a section here for each issue fixed before the release that needs a live check. Install the `main` build on **both** nodes first and confirm the START banner reports `version=<last release>+dev`. Watch the logs on both nodes throughout: ``` opnsense-log -f | grep wancarp ``` ## Pre-checks - [x] On each node: `/usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun --reconcile` shows the expected `role=` line and no errors. - [x] On each node, the same dry-run shows `boot check product::booting() present; booting=no.` That is the check the INIT rule relies on during boot (#15). If it reports `class product MISSING` instead, an INIT event during boot would take WAN down; reopen #15. ## #15: all-INIT node takes its WAN down ### A. Lost LAN link on the MASTER - [x] Pull the MASTER's LAN cable, or disable its switch port or virtio link. - [x] On the old MASTER, the log shows `role=BACKUP counted=[1@<lan>=INIT] (MASTER for nothing, INIT)`, and WAN goes down (`ifconfig <wan>` has no `UP` flag). - [x] The peer becomes MASTER and brings its WAN up, with Internet working. - [x] Only one node has WAN up at any point after the event. - [x] Wait at least one reconcile cycle (1 minute): the old MASTER's WAN stays down, and the reconcile logs no drift. - [x] Restore the link. The node returns as BACKUP (or MASTER, if preempt takes it back) and WAN ends up live on exactly one node. ### B. Temporarily Disable CARP - [x] On the MASTER: **Interfaces → Virtual IPs → Status → Temporarily Disable CARP**. - [x] Confirm the VHIDs show `INIT` in `ifconfig` (the issue flagged this as unconfirmed). If they show something else, note what and check the role line still takes WAN down. - [x] WAN goes down on that node, and the peer takes over with its WAN up. - [x] Re-enable CARP, and WAN ends up live on exactly one node. > **Result (2026-10-24):** the first run failed. Temporarily Disable CARP doesn't leave the VHIDs `INIT`: OPNsense removes them (no `carp:` lines in `ifconfig`) and sets `net.inet.carp.allow=0`. The hook logged `role=undecided counted=[]` and left WAN up, so both nodes had WAN live, and the internal network was down for about a minute. Fixed in 202d499: a node with `net.inet.carp.allow=0` is BACKUP. On the re-run, the first INIT event logged `role=BACKUP counted=[] (CARP disabled on this node, net.inet.carp.allow=0)` and took WAN down. The peer took over, and after re-enabling, preempt moved WAN back. WAN was live on exactly one node throughout. > > With `$remove_backup_route = true`, the returning MASTER restored its route while WAN was still waiting on DHCP, and OPNsense briefly set the default via an opt2 gateway (`10.102.0.1`) until the lease arrived. Afterwards it was via the WAN gateway on the MASTER and `192.168.0.1` on the BACKUP, as expected. The code comment and README now describe this (6188615); no code change needed. ### C. Boot - [x] Hard-reboot the MASTER. While booting, any INIT event logs `(INIT while booting: left to the next event)` and doesn't touch WAN (no `disabling interface 'wan'` line during boot). Before the #15 follow-up fix (222180a) the boot check always read "not booting" on a real box, so this is the check that matters. Once it rejoins, WAN ends up live on exactly one node. - [x] Hard-reboot the BACKUP and check the same. - [ ] Boot a node with its LAN link down. After boot finishes, the next reconcile takes WAN down (`role=BACKUP`, `had drifted`). > **Result (2026-10-24):** MASTER and BACKUP hard resets (Proxmox Reset) both passed. Neither boot logged an INIT event. Each rejoined as BACKUP, and its BACKUP event took down the WAN that `rc.bootup` had brought up. The MASTER then took over again by preempt 6 s later, and WAN ended up live on exactly one node. With `$remove_backup_route`, OPNsense's boot routing set the default via `10.102.0.1` after the first CARP event. The next event, or the reconcile (`had drifted`), pointed it back at `192.168.0.1`. > > **The LAN-down box is left unticked because it can't pass as written. This is OPNsense behaviour, not a script bug.** > - BACKUP rebooted with only `vtnet1` down: VHID 2 reached `BACKUP` and took WAN down at once. A reconcile ran a minute later and logged `had drifted` (the route). > - BACKUP rebooted with `vtnet1` and `vtnet2` both down: OPNsense's boot didn't finish (the last `rc.bootup` line was `unbound_configure_do(1)`). No CARP event fired, since the VHIDs were INIT from the start, and no reconcile ran because cron doesn't start until boot ends: none for 11+ minutes. The node kept `rc.bootup`'s WAN up the whole time; no outage was noticed. Reconnecting the links fired BACKUP, which took WAN down within a second. In an earlier run, boot finished about 3 minutes after reconnecting. > - Both boot-time gaps (WAN up until the first CARP event, and the all-links-down boot stall) are now in the README boot note (6058b86). Taking WAN down here would mean acting during boot, which the boot guard exists to prevent. ### D. Dry-run - [x] `./10-wancarp --dryrun 1@<lan> INIT` on the MASTER prints the `WOULD disable` lines and changes nothing. ## #17: BACKUP no longer calls interface_configure() The removed call did nothing, so BACKUP should behave exactly as before. These checks confirm that on a real box. - [ ] Fail over the MASTER (maintenance mode is fine; it can be the same run as the regression check below). On the demoted node, the log shows `disabling interface 'wan'` and ends `result=ok changed=yes`, with no `UNCAUGHT`, `partial` or other error lines. - [ ] On that node, WAN is fully down: `ifconfig <wan>` has no `UP` flag and no `inet` address, and `pgrep -fl 'dhclient.*<wan>'` finds nothing. - [ ] If `$remove_backup_route` is set: `netstat -rn` shows `default` via `$lan_vip`, and the node can still reach the Internet through the peer (for example, `pkg update` works). - [ ] Wait one reconcile cycle (1 minute): WAN stays down and the reconcile logs no drift. - [ ] `./10-wancarp --dryrun 1@<lan> BACKUP` on the MASTER prints `would interface_reset('wan'), unset enable (in-memory), ifconfig <wan> down.` with no `interface_configure()`, and changes nothing. ## #18: lock in /var/run with a bounded wait Run on the **BACKUP**, firing an event that matches its current role, so nothing changes even if the run went ahead. In one shell, hold the lock; in a second, fire the event: lockf -k /var/run/wancarp.lock sleep 180 time /usr/local/etc/rc.syshook.d/carp/10-wancarp 1@<lan> BACKUP; echo "exit=$?" - [ ] The event run gives up after about 120 seconds with `exit=1`. The log shows `timed out after 120s waiting for lock /var/run/wancarp.lock` and `END ... aborted=lock changed=no`, and WAN is untouched. - [ ] With the lock still held, `/usr/local/etc/rc.syshook.d/carp/10-wancarp --reconcile; echo "exit=$?"` returns `exit=0` at once and logs nothing. - [ ] After `sleep` ends, fire the same event again: it runs normally and ends `result=ok changed=no`. - [ ] `ls -l /var/run/wancarp.lock` shows `-rw-------` owned by root (not readable by other users, since `flock()` works on a read-only descriptor), and nothing new appears at `/tmp/wancarp.lock` (an old one left by 1.1.1 is harmless and can be deleted). - [ ] A non-root user can't take the lock: `su -m nobody -c 'lockf -k -t 0 /var/run/wancarp.lock true'; echo "exit=$?"` fails with `Permission denied` and a non-zero exit. ## Regression: normal failover - [ ] Maintenance-mode failover (**Enter Persistent CARP Maintenance Mode** on the MASTER): WAN moves to the peer, then back when you leave maintenance mode. - [ ] Reconcile drift: on the BACKUP, run `configctl interface reconfigure wan`. Within a minute the reconcile takes WAN back down and logs `had drifted`. ## Release - [ ] Every box above is ticked, or the reason it isn't is noted here. - [ ] Rename `[Unreleased]` in `CHANGELOG.md`, set `VERSION`, commit, and push the tag.
eric commented 2026-10-23 16:07:31 -07:00
Author
Owner

Updated after the follow-up audit of #15, #18 and #20 (222180a): replaced the method_exists() boot pre-check, which would have read bool(false) on a healthy box, with the new dry-run boot check line; flagged the boot tests in #15 C as the ones that confirm that fix; the #18 lock now expects -rw-------, plus a check that nobody can't take it.

Updated after the follow-up audit of #15, #18 and #20 (222180a): replaced the `method_exists()` boot pre-check, which would have read `bool(false)` on a healthy box, with the new dry-run `boot check` line; flagged the boot tests in #15 C as the ones that confirm that fix; the #18 lock now expects `-rw-------`, plus a check that `nobody` can't take it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
1.1.1
1.1.0
1.0.0
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
eric/10-wancarp#21
No description provided.