eric/10-wancarp
Watch
1
0
Fork
You've already forked 10-wancarp
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

CARP failback (SEC→PRIME) breaks a long-running LAN ping; BACKUP node sources from LAN CARP VIPs #2

New issue
Closed
opened 2026-09-05 21:16:19 -07:00 by eric · 2 comments
eric commented 2026-09-05 21:16:19 -07:00
Owner
Copy link

Status: open — root cause not confirmed, but the case has split in two (see §3).
Affects: GARLIC-01 (PRIME) / GARLIC-02 (SEC), OPNsense 26.7.x
Probably not a 10-wancarp defect — see §4.

Migrated from the ISSUE-carp-failback-lan-vip-arp.md handoff doc on docs/carp-failback-investigation, which was only ever a workaround for not having tea set up on the other machine. That branch can be deleted once this issue is up.

1. Symptom

Continuous ping from a Windows client on the LAN (192.168.0.x), across CARP failovers:

Direction Meaning Behaviour
PRIME → SEC SEC becomes MASTER Works. Loses exactly 1 packet, as expected.
SEC → PRIME PRIME becomes MASTER (failback) Breaks. Ping stops entirely. Requires Ctrl+C and retrying several times before it recovers.

Two load-bearing details, both confirmed:

  • Web browsing kept working while the ping was dead.
  • Recovery required restarting the ping, not merely waiting.
  • The ping target was an external host (8.8.8.8-class), not the gateway VIP.

Separately, for the record: while SEC is BACKUP it has no internet of its own. That is correct and by design — this topology has a single ISP address, so the BACKUP node has no uplink. Rebooting SEC appears to "restore" its internet only because the boot window briefly brings WAN up before CARP demotes it again. Do not reboot SEC to get it online; use CARP maintenance mode on PRIME so SEC legitimately becomes MASTER.

2. Environment

  • OPNsense 26.7.x (PHP 8.5, FreeBSD 15.1), upgraded from 26.1.x. The upgrade itself succeeded; failover/failback worked during the upgrade reboots.
  • Two nodes, both Proxmox VMs (MAC OUI bc:24:11), virtio NICs.
  • 10-wancarp installed at /usr/local/etc/rc.syshook.d/carp/10-wancarp on both. Survived the in-place upgrade.
Device Role CARP VIP vhid Virtual MAC
vtnet0 WAN (DHCP) — — —
vtnet1 LAN 192.168.0.1 1 00:00:5e:00:01:01
vtnet2 LAN/VLAN 10.102.0.1 2 00:00:5e:00:01:02

Script config: $iface_aliases = ['wan'], $lan_vip = '192.168.0.1', $remove_backup_route = false.

The ISP hands the same IPv4 address to both nodes (different MACs). This is why the script exists — only the MASTER may hold WAN up.

HA configuration (verified in GUI): PRIMARY has "Synchronize all states via" set to the Sync VLAN; SECONDARY has it Disabled. ⇒ pfsync state sync is currently inert. PRIMARY multicasts into the sync VLAN; SECONDARY neither sends nor listens.

3. Where the investigation actually stands

Two things are true at once and should not be conflated.

3a. There is a real defect on SEC — a BACKUP node sourcing from a CARP VIP

On PRIME, at exactly the two failbacks and nowhere else:

17:21:25  arp: 192.168.0.1 moved from 00:00:5e:00:01:01 to bc:24:11:2c:35:a7 on vtnet1
17:32:27  arp: 10.102.0.1  moved from 00:00:5e:00:01:02 to bc:24:11:4e:4f:88 on vtnet2

Both bc:24:11:… MACs are confirmed to belong to GARLIC-02 (SEC) — one per internal VLAN. So both LAN CARP VIPs moved from their correct CARP virtual MAC to the physical MAC of the node that was demoting to BACKUP.

FreeBSD emits arp: … moved from … to … when it receives an ARP packet claiming a cached IP from a different MAC. That includes ARP requests — the sender-IP field of a request is the sender's own address. So:

SEC, while going BACKUP, sourced traffic from the LAN CARP VIPs using its own physical MAC.

A BACKUP node must never emit anything sourced from a CARP VIP. This is wrong regardless of what it did or did not break here, and is worth fixing on its own merits. Correlation with the symptom: 2 of 2 on the broken direction, 0 on the working direction.

Likely culprits — anything that can source from a VIP: dpinger gateway monitoring, Unbound/dnsmasq bound to the VIP, a DHCP server, an mDNS/Avahi reflector, monit, or an NTP client.

3b. …but it is probably not what broke the ping

The ping target was external, so ping and browsing both left the client through the same gateway MAC. A poisoned neighbour entry would have killed both. Browsing survived. The ARP anomaly cannot explain the divergence.

What now carries the weight is that recovery was triggered by restarting the process, not by waiting. The network has no idea the ping was restarted — that is a per-flow, client-side signature:

  • A long-running ping -t holds an ICMP handle whose route/interface binding was resolved once, at start. Windows can wedge such a handle across a gateway transition and emit General failure. indefinitely until the process is restarted.
  • A browser opens new sockets constantly, each getting a fresh route lookup, and so is immune to exactly that wedge.

One mechanism, both observations — which the ARP story cannot do. This promotes the "Windows ping quirk" from unlikely (where the original triage had it) to the leading hypothesis. Not yet confirmed; §5 settles it.

A second, genuine but transient, factor is in the logs: the promoted node has no WAN address and no default route until DHCP completes:

ROUTING: refusing to set interface route on addressless wan(vtnet0)
The required WAN_DHCP IPv4 interface address could not be found, skipping.

That is a real outage window on every failback — but it is seconds long and hits browsing equally, so it explains the first few dropped packets, not a failure that persists until the ping is restarted.

4. Scope — this is probably not a 10-wancarp defect

10-wancarp only ever touches the wan alias (vtnet0). It never configures, resets or ARPs on vtnet1/vtnet2. The anomaly in §3a is on the LAN VIPs.

The logs show the script behaving correctly on every single event: right enable/disable decisions, idempotency skips firing properly on the duplicate per-VIP CARP events, every END banner result=ok.

§3a most likely belongs to the class already flagged in the README's "HA note: CARP-aware services" section — a service on SEC that is not CARP-aware. Tracked here because this repo is where the HA behaviour is documented.

5. The decisive test — client-side, costs nothing

At the next failback, while the original ping is still dead, open a second terminal on the Windows client and start a fresh ping to the same external target:

ping -t 8.8.8.8
Result Meaning
New ping works, old one stays dead Client-side per-flow wedge. The network recovered on time; §3a is a separate (real) bug that is not causing this. Done.
Both dead, both recover together The network really is black-holing. §3a or another factor is live; proceed to §6.

Record at the same moment, in the same window:

arp -a | findstr 192.168.0.1
ping 192.168.0.1
  • The exact failure text of the dead ping is itself diagnostic: General failure. ⇒ client-side handle wedge; Request timed out ⇒ packets are leaving and not coming back; Destination host unreachable ⇒ no usable route/ARP.
  • 00-00-5e-00-01-01 in the ARP output = correct. A bc-24-11-… MAC = the §3a poisoning did reach the client after all, which would overturn §3b.
  • If the gateway ping (192.168.0.1) succeeds while the external one fails, the LAN path is fine and the problem is upstream of PRIME — i.e. the DHCP/default-route window, not ARP.

Trigger the failback with CARP maintenance mode on PRIME (System → High Availability → Status), not a reboot — that isolates the CARP transition from boot-time noise.

6. Chasing §3a on its own

6a. Run on SEC (GARLIC-02) now — no failover required

Find anything bound to a VIP address rather than an interface address:

sockstat -46 | grep -E '192\.168\.0\.1|10\.102\.0\.1'
sockstat -46 -l    # broader sweep of what runs on the BACKUP node at all

Check the usual CARP-unaware suspects are actually CARP-aware or disabled on BACKUP:

  • Services → mDNS Repeater (Avahi) → "Enable CARP Failover" must be ticked.
    Start here — this pair has already been bitten by exactly this once (see
    the comment below): Avahi ran on both firewalls reflecting VLAN1↔VLAN102 and
    produced network-wide latency that was initially mistaken for a CARP storm.
    It was fixed then by ticking that box. The pair has since been upgraded in
    place to 26.7, so re-verify the setting survived. Two-minute GUI check, no
    failover needed.
  • Services → Unbound / Dnsmasq → interface bindings (should not be the VIP)
  • Services → Kea DHCP / Dnsmasq DHCP → HA / failover configuration
  • System → Gateways → whether dpinger is monitoring from the VIP

6b. Capture the next failback live

On PRIME, before triggering (leave running through the transition):

tcpdump -ni vtnet1 arp and host 192.168.0.1
tcpdump -ni vtnet2 arp and host 10.102.0.1     # second terminal

Looking for: an ARP from SEC's physical MAC with sender-IP = the VIP, and how long the LAN takes to relearn 00:00:5e:00:01:0x. Then identify the emitting process:

tcpdump -ni vtnet1 -e ether src <SEC-vtnet1-MAC> and host 192.168.0.1 -vv

Fix belongs in service configuration on SEC (make the offending service CARP-aware, or bind it to the interface address rather than the VIP) — not in 10-wancarp.

7. Log timeline (2026-09-05)

Reconstructed from grep -h wancarp /var/log/system/*.log on both nodes plus grep -hi "moved from" on PRIME.

Time PRIME (GARLIC-01) SEC (GARLIC-02) Direction Ping
17:17:29 rebooting MASTER, WAN up, changed=yes PRIME→SEC OK
17:21:25 MASTER, WAN up, changed=yes — ARP move logged BACKUP, WAN down, changed=yes SEC→PRIME broken
17:30:57 still MASTER MASTER, WAN up, changed=yes PRIME→SEC OK
17:31:20 BACKUP (1@vtnet1): "already down, skipping" —
17:31:21 BACKUP (2@vtnet2): "disabling", flags showed UP —
17:32:27 MASTER, WAN up, changed=yes — ARP move logged BACKUP, WAN down, changed=yes SEC→PRIME broken

Two secondary observations, both unexplained and both low priority:

  • 17:30:57 → 17:31:21 is a 24-second dual-WAN-up window (SEC brought WAN up before PRIME dropped it). Notably this is in the direction that works.
  • 17:31:20 vs 17:31:21 on PRIME contradict each other: the first BACKUP event found WAN already down, the second found it UP again one second later. Something outside the script raised WAN between the two events. Probably shutdown-time interface reconfiguration.

8. 26.7 compatibility — verified, no action needed

Checked against the stable/26.1 → stable/26.7 core diff and confirmed with --dryrun on an upgraded node. Recorded so nobody re-does this work. Landed in 5fb5793.

  • Invocation contract byte-identical. src/etc/devd/carp.conf unchanged (still matches [0-9]+@[0-9a-z_.]+ + MASTER|BACKUP|INIT), rc.syshook unchanged, actions_interface.conf [carp] action unchanged, rc.syshook.d/carp/ still present.
  • All five PHP APIs unchanged, except interface_configure() gained an optional 5th parameter ($batch = false) that defaults to 26.1 behaviour. The script's four-argument calls are unaffected.
  • "Interface assignments migrated to MVC/API" is not a storage change. The new NetworkInterface model is <mount>:memory:</mount> and reads/writes through to legacy config.xml <interfaces>. $config aliases and the enable flag are untouched.
  • LOWER_UP is new in the flag output (bit 24, IFF_NETLINK_1, ifconfig's netlink path). It reports carrier, not admin state. iface_is_up()'s strict in_array('UP', ..., true) handles it correctly. Do not loosen that to a substring test — see the comment on iface_is_up().
  • Minor: 26.7 marks VLAN devices 'reload' => false. Irrelevant for a physical or virtio WAN.

Also discovered along the way: interfaces is not an XMLRPC-syncable section, so the interface enable flag must be set manually on each node. The script header and README both claimed HA config sync carries it. Corrected in 70078c7.

9. Work items

  • Run §5. It is the cheapest test and it decides whether this issue is a firewall problem at all.
  • Run §6a on SEC regardless of the §5 outcome — §3a is a real defect either way.
  • Consider clearing "Synchronize all states via" on PRIMARY so both nodes match at disabled. In this topology each node NATs to its own WAN address, so a synced WAN state is wrong by construction on the peer. It is currently inert anyway (SECONDARY has it off). Do not "fix" this by enabling pfsync on SECONDARY — that would activate state sync and could introduce the cross-node NAT-state corruption this topology cannot support.
  • Delete the docs/carp-failback-investigation branch and its ISSUE-…md file once this issue is up. Done — branch deleted locally and on origin at b2e6219; its only unique content was the handoff doc, now superseded by this issue.

10. Hypotheses already eliminated

Recorded so they are not re-litigated.

Hypothesis Verdict Killed by
LAN ARP staleness needing a flush on the firewall Wrong as stated CARP shares a virtual MAC across both nodes, so the client's entry stays valid by design; and flushing the firewall's ARP table would not touch the client's cache. But the instinct was right about the LAN side — see §3a.
Stale/poisoned pfsync NAT state carried to PRIME Dead pfsync is inert — SECONDARY has state sync disabled, so no states are exchanged in either direction.
PRIME's own stale pf states surviving demotion Dead The failback was a reboot; PRIME starts with an empty state table.
Duplicate WAN IP during PRIME's boot window Dead No moved from on vtnet0; no "using my IP" on either node; the one real dual-WAN window is in the working direction.
Proxmox per-NIC firewall / MAC anti-spoof filtering the CARP vMAC Dead Firewall is disabled on all VM NICs in this environment.
ARP poisoning of the client's gateway entry explains the symptom Dead The ping target was external, so browsing would have died with it. See §3b.

Explicitly deferred — do not implement without new evidence

A proposed change to treat INIT as "not MASTER" (down the interfaces) plus a rc.syshook.d/start/ boot-time guard. This was designed for a duplicate-WAN-IP boot window hypothesis that the logs disproved (see the table above). The INIT no-op may still be a latent gap, but it is not this bug. Do not implement it as a fix for this issue.

**Status:** open — root cause not confirmed, but the case has split in two (see §3). **Affects:** GARLIC-01 (PRIME) / GARLIC-02 (SEC), OPNsense 26.7.x **Probably not a `10-wancarp` defect** — see §4. Migrated from the `ISSUE-carp-failback-lan-vip-arp.md` handoff doc on `docs/carp-failback-investigation`, which was only ever a workaround for not having `tea` set up on the other machine. That branch can be deleted once this issue is up. ## 1. Symptom Continuous ping from a Windows client on the LAN (`192.168.0.x`), across CARP failovers: | Direction | Meaning | Behaviour | |---|---|---| | PRIME → SEC | SEC becomes MASTER | **Works.** Loses exactly 1 packet, as expected. | | SEC → PRIME | PRIME becomes MASTER (failback) | **Breaks.** Ping stops entirely. Requires Ctrl+C and retrying several times before it recovers. | Two load-bearing details, both confirmed: - **Web browsing kept working while the ping was dead.** - **Recovery required restarting the ping**, not merely waiting. - The ping target was an **external host** (`8.8.8.8`-class), **not** the gateway VIP. Separately, for the record: while SEC is BACKUP it has no internet of its own. That is **correct and by design** — this topology has a single ISP address, so the BACKUP node has no uplink. Rebooting SEC appears to "restore" its internet only because the boot window briefly brings WAN up before CARP demotes it again. **Do not reboot SEC to get it online**; use CARP maintenance mode on PRIME so SEC legitimately becomes MASTER. ## 2. Environment - OPNsense **26.7.x** (PHP 8.5, FreeBSD 15.1), upgraded from 26.1.x. The upgrade itself succeeded; failover/failback worked during the upgrade reboots. - Two nodes, both **Proxmox VMs** (MAC OUI `bc:24:11`), virtio NICs. - `10-wancarp` installed at `/usr/local/etc/rc.syshook.d/carp/10-wancarp` on both. Survived the in-place upgrade. | Device | Role | CARP VIP | vhid | Virtual MAC | |---|---|---|---|---| | `vtnet0` | WAN (DHCP) | — | — | — | | `vtnet1` | LAN | `192.168.0.1` | 1 | `00:00:5e:00:01:01` | | `vtnet2` | LAN/VLAN | `10.102.0.1` | 2 | `00:00:5e:00:01:02` | Script config: `$iface_aliases = ['wan']`, `$lan_vip = '192.168.0.1'`, `$remove_backup_route = false`. **The ISP hands the *same* IPv4 address to both nodes** (different MACs). This is why the script exists — only the MASTER may hold WAN up. HA configuration (verified in GUI): PRIMARY has "Synchronize all states via" set to the Sync VLAN; SECONDARY has it **Disabled**. ⇒ **pfsync state sync is currently inert.** PRIMARY multicasts into the sync VLAN; SECONDARY neither sends nor listens. ## 3. Where the investigation actually stands Two things are true at once and should not be conflated. ### 3a. There is a real defect on SEC — a BACKUP node sourcing from a CARP VIP On PRIME, at exactly the two failbacks and nowhere else: ``` 17:21:25 arp: 192.168.0.1 moved from 00:00:5e:00:01:01 to bc:24:11:2c:35:a7 on vtnet1 17:32:27 arp: 10.102.0.1 moved from 00:00:5e:00:01:02 to bc:24:11:4e:4f:88 on vtnet2 ``` Both `bc:24:11:…` MACs are confirmed to belong to **GARLIC-02 (SEC)** — one per internal VLAN. So both LAN CARP VIPs moved from their correct CARP virtual MAC to the **physical MAC of the node that was demoting to BACKUP**. FreeBSD emits `arp: … moved from … to …` when it receives an ARP packet claiming a cached IP from a different MAC. That includes ARP **requests** — the sender-IP field of a request is the sender's own address. So: > **SEC, while going BACKUP, sourced traffic from the LAN CARP VIPs using its own physical MAC.** A BACKUP node must never emit anything sourced from a CARP VIP. This is wrong regardless of what it did or did not break here, and is worth fixing on its own merits. Correlation with the symptom: 2 of 2 on the broken direction, 0 on the working direction. Likely culprits — anything that can source from a VIP: dpinger gateway monitoring, Unbound/dnsmasq bound to the VIP, a DHCP server, an mDNS/Avahi reflector, monit, or an NTP client. ### 3b. …but it is probably *not* what broke the ping The ping target was external, so ping and browsing both left the client through the same gateway MAC. A poisoned neighbour entry would have killed both. Browsing survived. **The ARP anomaly cannot explain the divergence.** What now carries the weight is that **recovery was triggered by restarting the process, not by waiting**. The network has no idea the ping was restarted — that is a per-flow, client-side signature: - A long-running `ping -t` holds an ICMP handle whose route/interface binding was resolved once, at start. Windows can wedge such a handle across a gateway transition and emit `General failure.` indefinitely until the process is restarted. - A browser opens **new** sockets constantly, each getting a fresh route lookup, and so is immune to exactly that wedge. One mechanism, both observations — which the ARP story cannot do. This promotes the "Windows `ping` quirk" from *unlikely* (where the original triage had it) to the **leading hypothesis**. Not yet confirmed; §5 settles it. A second, genuine but **transient**, factor is in the logs: the promoted node has **no WAN address and no default route until DHCP completes**: ``` ROUTING: refusing to set interface route on addressless wan(vtnet0) The required WAN_DHCP IPv4 interface address could not be found, skipping. ``` That is a real outage window on every failback — but it is seconds long and hits browsing equally, so it explains the first few dropped packets, not a failure that persists until the ping is restarted. ## 4. Scope — this is probably not a `10-wancarp` defect `10-wancarp` only ever touches the `wan` alias (`vtnet0`). It never configures, resets or ARPs on `vtnet1`/`vtnet2`. The anomaly in §3a is on the **LAN** VIPs. The logs show the script behaving **correctly on every single event**: right enable/disable decisions, idempotency skips firing properly on the duplicate per-VIP CARP events, every `END` banner `result=ok`. §3a most likely belongs to the class already flagged in the README's "HA note: CARP-aware services" section — a service on SEC that is not CARP-aware. Tracked here because this repo is where the HA behaviour is documented. ## 5. The decisive test — client-side, costs nothing At the next failback, **while the original ping is still dead**, open a second terminal on the Windows client and start a *fresh* ping to the *same* external target: ``` ping -t 8.8.8.8 ``` | Result | Meaning | |---|---| | New ping **works**, old one stays dead | Client-side per-flow wedge. The network recovered on time; §3a is a separate (real) bug that is not causing this. Done. | | **Both** dead, both recover together | The network really is black-holing. §3a or another factor is live; proceed to §6. | Record at the same moment, in the same window: ``` arp -a | findstr 192.168.0.1 ping 192.168.0.1 ``` - The **exact** failure text of the dead ping is itself diagnostic: `General failure.` ⇒ client-side handle wedge; `Request timed out` ⇒ packets are leaving and not coming back; `Destination host unreachable` ⇒ no usable route/ARP. - `00-00-5e-00-01-01` in the ARP output = correct. A `bc-24-11-…` MAC = the §3a poisoning did reach the client after all, which would overturn §3b. - If the gateway ping (`192.168.0.1`) succeeds while the external one fails, the LAN path is fine and the problem is upstream of PRIME — i.e. the DHCP/default-route window, not ARP. Trigger the failback with **CARP maintenance mode** on PRIME (System → High Availability → Status), not a reboot — that isolates the CARP transition from boot-time noise. ## 6. Chasing §3a on its own ### 6a. Run on SEC (GARLIC-02) now — no failover required Find anything bound to a VIP address rather than an interface address: ```sh sockstat -46 | grep -E '192\.168\.0\.1|10\.102\.0\.1' sockstat -46 -l # broader sweep of what runs on the BACKUP node at all ``` Check the usual CARP-unaware suspects are actually CARP-aware or disabled on BACKUP: - **Services → mDNS Repeater (Avahi) → "Enable CARP Failover" must be ticked.** **Start here** — this pair has already been bitten by exactly this once (see the comment below): Avahi ran on both firewalls reflecting VLAN1↔VLAN102 and produced network-wide latency that was initially mistaken for a CARP storm. It was fixed then by ticking that box. The pair has since been upgraded in place to 26.7, so re-verify the setting survived. Two-minute GUI check, no failover needed. - Services → Unbound / Dnsmasq → interface bindings (should not be the VIP) - Services → Kea DHCP / Dnsmasq DHCP → HA / failover configuration - System → Gateways → whether dpinger is monitoring from the VIP ### 6b. Capture the next failback live On **PRIME**, before triggering (leave running through the transition): ```sh tcpdump -ni vtnet1 arp and host 192.168.0.1 tcpdump -ni vtnet2 arp and host 10.102.0.1 # second terminal ``` Looking for: an ARP from SEC's physical MAC with **sender-IP = the VIP**, and how long the LAN takes to relearn `00:00:5e:00:01:0x`. Then identify the emitting process: ```sh tcpdump -ni vtnet1 -e ether src <SEC-vtnet1-MAC> and host 192.168.0.1 -vv ``` Fix belongs in **service configuration on SEC** (make the offending service CARP-aware, or bind it to the interface address rather than the VIP) — not in `10-wancarp`. ## 7. Log timeline (2026-09-05) Reconstructed from `grep -h wancarp /var/log/system/*.log` on both nodes plus `grep -hi "moved from"` on PRIME. | Time | PRIME (GARLIC-01) | SEC (GARLIC-02) | Direction | Ping | |---|---|---|---|---| | 17:17:29 | rebooting | MASTER, WAN **up**, `changed=yes` | PRIME→SEC | OK | | 17:21:25 | MASTER, WAN **up**, `changed=yes` — **ARP move logged** | BACKUP, WAN **down**, `changed=yes` | SEC→PRIME | **broken** | | 17:30:57 | still MASTER | MASTER, WAN **up**, `changed=yes` | PRIME→SEC | OK | | 17:31:20 | BACKUP (1@vtnet1): "already down, skipping" | — | | | | 17:31:21 | BACKUP (2@vtnet2): "disabling", flags showed `UP` | — | | | | 17:32:27 | MASTER, WAN **up**, `changed=yes` — **ARP move logged** | BACKUP, WAN **down**, `changed=yes` | SEC→PRIME | **broken** | Two secondary observations, both unexplained and both low priority: - **17:30:57 → 17:31:21 is a 24-second dual-WAN-up window** (SEC brought WAN up before PRIME dropped it). Notably this is in the direction that *works*. - **17:31:20 vs 17:31:21 on PRIME contradict each other**: the first BACKUP event found WAN already down, the second found it `UP` again one second later. Something outside the script raised WAN between the two events. Probably shutdown-time interface reconfiguration. ## 8. 26.7 compatibility — verified, no action needed Checked against the `stable/26.1 → stable/26.7` core diff and confirmed with `--dryrun` on an upgraded node. Recorded so nobody re-does this work. Landed in `5fb5793`. - **Invocation contract byte-identical.** `src/etc/devd/carp.conf` unchanged (still matches `[0-9]+@[0-9a-z_.]+` + `MASTER|BACKUP|INIT`), `rc.syshook` unchanged, `actions_interface.conf` `[carp]` action unchanged, `rc.syshook.d/carp/` still present. - **All five PHP APIs unchanged**, except `interface_configure()` gained an optional 5th parameter (`$batch = false`) that defaults to 26.1 behaviour. The script's four-argument calls are unaffected. - **"Interface assignments migrated to MVC/API" is not a storage change.** The new `NetworkInterface` model is `<mount>:memory:</mount>` and reads/writes through to legacy `config.xml` `<interfaces>`. `$config` aliases and the `enable` flag are untouched. - **`LOWER_UP`** is new in the flag output (bit 24, `IFF_NETLINK_1`, ifconfig's netlink path). It reports *carrier*, not admin state. `iface_is_up()`'s strict `in_array('UP', ..., true)` handles it correctly. **Do not loosen that to a substring test** — see the comment on `iface_is_up()`. - Minor: 26.7 marks VLAN devices `'reload' => false`. Irrelevant for a physical or virtio WAN. Also discovered along the way: **`interfaces` is not an XMLRPC-syncable section**, so the interface `enable` flag must be set **manually on each node**. The script header and README both claimed HA config sync carries it. Corrected in `70078c7`. ## 9. Work items - [ ] Run §5. It is the cheapest test and it decides whether this issue is a firewall problem at all. - [ ] Run §6a on SEC regardless of the §5 outcome — §3a is a real defect either way. - [ ] **Consider clearing "Synchronize all states via" on PRIMARY** so both nodes match at disabled. In this topology each node NATs to its own WAN address, so a synced WAN state is wrong by construction on the peer. It is currently inert anyway (SECONDARY has it off). **Do not "fix" this by enabling pfsync on SECONDARY** — that would activate state sync and could introduce the cross-node NAT-state corruption this topology cannot support. - [x] Delete the `docs/carp-failback-investigation` branch and its `ISSUE-…md` file once this issue is up. Done — branch deleted locally and on origin at `b2e6219`; its only unique content was the handoff doc, now superseded by this issue. ## 10. Hypotheses already eliminated Recorded so they are not re-litigated. | Hypothesis | Verdict | Killed by | |---|---|---| | LAN ARP staleness needing a flush on the firewall | Wrong *as stated* | CARP shares a virtual MAC across both nodes, so the client's entry stays valid by design; and flushing the firewall's ARP table would not touch the client's cache. **But the instinct was right about the LAN side** — see §3a. | | Stale/poisoned pfsync NAT state carried to PRIME | **Dead** | pfsync is inert — SECONDARY has state sync disabled, so no states are exchanged in either direction. | | PRIME's own stale pf states surviving demotion | **Dead** | The failback was a reboot; PRIME starts with an empty state table. | | Duplicate WAN IP during PRIME's boot window | **Dead** | No `moved from` on `vtnet0`; no "using my IP" on either node; the one real dual-WAN window is in the *working* direction. | | Proxmox per-NIC firewall / MAC anti-spoof filtering the CARP vMAC | **Dead** | Firewall is disabled on all VM NICs in this environment. | | ARP poisoning of the client's gateway entry explains the symptom | **Dead** | The ping target was external, so browsing would have died with it. See §3b. | ### Explicitly deferred — do not implement without new evidence A proposed change to treat `INIT` as "not MASTER" (down the interfaces) plus a `rc.syshook.d/start/` boot-time guard. This was designed for a **duplicate-WAN-IP boot window** hypothesis that the logs **disproved** (see the table above). The `INIT` no-op may still be a latent gap, but it is not this bug. Do not implement it as a fix for this issue.
eric commented 2026-09-05 21:24:05 -07:00
Author
Owner
Copy link

Operational context carried over (machine-local notes → issue)

Claude Code's memory is per-machine — it lives in a local ~/.claude/… directory keyed by the repo's absolute path, and does not sync between computers. Work continues on a different PC tomorrow, so the durable parts are transcribed here. This comment is the carry-over; nothing below needs to be rediscovered.

Note: this repo and its issues are publicly readable without authentication (verified anonymously). WAN public addressing and the WAN interface MAC are therefore deliberately omitted from this comment. They aren't needed for anything in §5/§6. If a future step needs them, pull them off the box directly.

Node identities

Node Role advskew
GARLIC-01 PRIMARY (PRIME) 1
GARLIC-02 SECONDARY (SEC) 101

WAN is vtnet0, virtio, DHCP, with a software-set shared MAC on both boxes — set in OPNsense, not duplicated on the Proxmox VM NIC hardware. That detail matters if WAN addressing is ever re-examined: the ISP sees one MAC, which is how both nodes get handed the same IPv4 address.

Triggering failover safely — read before running §5

  • Force a transition with Persistent CARP Maintenance Mode (System → High Availability → Status): Enter it on the box you want demoted, then Leave it on that same box. Not a reboot — that keeps the CARP transition clean of boot-time noise, which is what §5 needs.

  • Persistent maintenance is sysctl net.inet.carp.demotion = 240.

  • The sysctl is additive on write. So if the GUI ever loses its Leave button and a node is stuck demoted, clear it with:

    sysctl net.inet.carp.demotion=-240

    This has been needed before. Worth knowing before deliberately demoting a node on a firewall you are not physically next to.

  • 10-wancarp --dryrun is safe on a live firewall and makes no changes:

    /usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun 1@vtnet1 MASTER
/usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun 1@vtnet1 BACKUP

  • CARP passes its arguments as <vhid>@<interface> (e.g. 1@vtnet1) — not the <address>@<vhid> that older documentation claimed.

Why Avahi is the prime suspect for §3a, not just one of five

This pair has already had one CARP-unaware-service incident, and it was the mDNS Repeater. Avahi ran on both firewalls reflecting VLAN1↔VLAN102, forming an mDNS (224.0.0.251) reflection loop that caused network-wide latency and packet loss — initially misdiagnosed as a CARP storm. The fix was the repeater's "Enable CARP Failover" checkbox, which restricts it to the MASTER.

That is exactly the shape of §3a: a service on the BACKUP node emitting traffic it has no business emitting on a LAN VIP. The pair has since been upgraded in place 26.1 → 26.7, so the setting is worth re-verifying before anything more elaborate. §6a has been updated to put this first.

General rule established for this pair: any multicast reflector, or any service that must not run on both nodes simultaneously, has to be either CARP-aware or pinned to a single node.

Corroborating an item already in the issue

Boot behaviour is resolved, not open: hard reboots of both the MASTER and the BACKUP were tested and each hands off and reclaims cleanly, because CARP fires MASTER/BACKUP events at boot and the runtime-only hook follows them. This independently supports the "Explicitly deferred" note at the bottom of the issue — the INIT default-deny change is not needed, and should not be implemented as a fix for this bug.

Repo state as of this comment

  • main is at 70078c7, pushed. It carries the config-sync correction (interfaces is not an XMLRPC-syncable section, so the per-interface enable flag must be set manually on each node).
  • The docs/carp-failback-investigation branch and its ISSUE-…md handoff file have been deleted — this issue supersedes them. Do not go looking for that file tomorrow.
  • Nothing is uncommitted.
## Operational context carried over (machine-local notes → issue) Claude Code's memory is **per-machine** — it lives in a local `~/.claude/…` directory keyed by the repo's absolute path, and does not sync between computers. Work continues on a different PC tomorrow, so the durable parts are transcribed here. This comment is the carry-over; nothing below needs to be rediscovered. > **Note:** this repo and its issues are **publicly readable without authentication** (verified anonymously). WAN public addressing and the WAN interface MAC are therefore deliberately **omitted** from this comment. They aren't needed for anything in §5/§6. If a future step needs them, pull them off the box directly. ### Node identities | Node | Role | advskew | |---|---|---| | GARLIC-01 | PRIMARY (PRIME) | 1 | | GARLIC-02 | SECONDARY (SEC) | 101 | WAN is `vtnet0`, virtio, DHCP, with a **software-set shared MAC on both boxes** — set in OPNsense, *not* duplicated on the Proxmox VM NIC hardware. That detail matters if WAN addressing is ever re-examined: the ISP sees one MAC, which is how both nodes get handed the same IPv4 address. ### Triggering failover safely — read before running §5 - **Force a transition with Persistent CARP Maintenance Mode** (System → High Availability → Status): *Enter* it on the box you want **demoted**, then *Leave* it on that **same** box. Not a reboot — that keeps the CARP transition clean of boot-time noise, which is what §5 needs. - **Persistent maintenance is `sysctl net.inet.carp.demotion` = 240.** - **The sysctl is *additive* on write.** So if the GUI ever loses its *Leave* button and a node is stuck demoted, clear it with: ```sh sysctl net.inet.carp.demotion=-240 ``` This has been needed before. Worth knowing before deliberately demoting a node on a firewall you are not physically next to. - **`10-wancarp --dryrun` is safe on a live firewall** and makes no changes: ```sh /usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun 1@vtnet1 MASTER /usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun 1@vtnet1 BACKUP ``` - CARP passes its arguments as `<vhid>@<interface>` (e.g. `1@vtnet1`) — **not** the `<address>@<vhid>` that older documentation claimed. ### Why Avahi is the prime suspect for §3a, not just one of five This pair has already had one CARP-unaware-service incident, and it was the mDNS Repeater. Avahi ran on **both** firewalls reflecting VLAN1↔VLAN102, forming an mDNS (`224.0.0.251`) reflection loop that caused network-wide latency and packet loss — initially misdiagnosed as a CARP storm. The fix was the repeater's **"Enable CARP Failover"** checkbox, which restricts it to the MASTER. That is exactly the shape of §3a: a service on the BACKUP node emitting traffic it has no business emitting on a LAN VIP. The pair has since been upgraded in place 26.1 → 26.7, so the setting is worth re-verifying before anything more elaborate. §6a has been updated to put this first. **General rule established for this pair:** any multicast reflector, or any service that must not run on both nodes simultaneously, has to be either CARP-aware or pinned to a single node. ### Corroborating an item already in the issue Boot behaviour is **resolved, not open**: hard reboots of both the MASTER and the BACKUP were tested and each hands off and reclaims cleanly, because CARP fires `MASTER`/`BACKUP` events at boot and the runtime-only hook follows them. This independently supports the "Explicitly deferred" note at the bottom of the issue — the `INIT` default-deny change is not needed, and should not be implemented as a fix for this bug. ### Repo state as of this comment - `main` is at `70078c7`, pushed. It carries the config-sync correction (`interfaces` is not an XMLRPC-syncable section, so the per-interface `enable` flag must be set manually on each node). - The `docs/carp-failback-investigation` branch and its `ISSUE-…md` handoff file have been **deleted** — this issue supersedes them. Do not go looking for that file tomorrow. - Nothing is uncommitted.
eric commented 2026-09-06 12:28:11 -07:00
Author
Owner
Copy link

Closing — not a bug in this script. Root cause was a misconfiguration on my side.

System → High Availability → Settings → "Synchronize all states via" was configured on the PRIMARY but left Disabled on the SECONDARY. That setting is per-node and is not carried by XMLRPC config sync, so pfsync state synchronisation was inert in both directions even though the primary looked correctly configured.

Setting it to the dedicated CARP sync VLAN on the secondary and re-running the reboot test gives a clean result in both directions: 1 packet lost on failover, 1 on failback, with no other disruption.

The LAN CARP VIP ARP moves noted during the investigation were transition noise, not the cause. 10-wancarp behaved correctly throughout — every CARP event was handled with the right enable/disable decision, and the idempotency guards worked as intended.

26.7 / PHP 8.5 / FreeBSD 15.1 compatibility was verified separately and is documented in the README.

The gap is in the official OPNsense HA documentation, not in this script. No code change required.

Closing — not a bug in this script. Root cause was a misconfiguration on my side. **System → High Availability → Settings → "Synchronize all states via"** was configured on the PRIMARY but left **Disabled** on the SECONDARY. That setting is per-node and is *not* carried by XMLRPC config sync, so pfsync state synchronisation was inert in both directions even though the primary looked correctly configured. Setting it to the dedicated CARP sync VLAN on the secondary and re-running the reboot test gives a clean result in both directions: **1 packet lost on failover, 1 on failback**, with no other disruption. The LAN CARP VIP ARP moves noted during the investigation were transition noise, not the cause. `10-wancarp` behaved correctly throughout — every CARP event was handled with the right enable/disable decision, and the idempotency guards worked as intended. 26.7 / PHP 8.5 / FreeBSD 15.1 compatibility was verified separately and is documented in the README. The gap is in the official OPNsense HA documentation, not in this script. No code change required.
eric 2026-09-06 12:28:11 -07:00
  • closed this issue
  • added
    invalid
         and removed
    bug
         labels
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
1.0.0
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
eric/10-wancarp#2
No description provided.