10-wancarp — OPNsense CARP Helper

When a CARP event occurs on an OPNsense firewall in an HA (High Availability) pair, this script runs and enables or disables interfaces based on the node's overall CARP role (worked out from all of its VHIDs, not just the one that fired the event):

• MASTER firewall → enables the configured interface(s).
• BACKUP firewall → disables the configured interface(s).

This keeps the uplink UP on exactly one node at a time. Typical use is a WAN link that must not be active on both firewalls simultaneously (e.g. a single ISP handoff, a modem in bridge mode, or any upstream that gets confused by two devices claiming the same MAC/IP).

You can manage one or many interfaces. The common case is WAN only.

On its own, this script only runs on transitions. After you first set up HA, all interfaces are UP. To force a quick event for testing, go to Interfaces → Virtual IPs → Status → Enter Persistent CARP Maintenance Mode on the MASTER (then leave maintenance mode to flip back). The reconcile cron job is part of Installation for this reason: without it, OPNsense can bring the BACKUP's WAN back up between transitions.

AI Disclosure

• I used Claude Code (Opus 4.8) to write this script in its entirety including the README.
• I have reviewed the README for accuracy based on my experience using the script.
• I have fully tested the script in my Homelab to verify it works as intended.
• I have only read the script header and not performed a full source review
• I had Claude Code use Context7 while developing this script so the latest relevant documentation would be used

How it works

OPNsense executes every file in /usr/local/etc/rc.syshook.d/carp/ on a CARP transition and passes two arguments:

Arg	Meaning	Example
$argv[1]	subsystem	1@vtnet1 (<vhid>@<interface>)
$argv[2]	type	MASTER, BACKUP, or INIT

So the system effectively runs:

/usr/local/etc/rc.syshook.d/carp/10-wancarp 1@vtnet1 MASTER

The event only wakes the script up. Each VHID fires its own event, so with several VIPs the "last event wins" approach would leave WAN wherever the last one put it, even if the VHIDs disagree. Instead, the script reads every VHID's live state from ifconfig and acts on the node's role:

Counted VHIDs	Role	Action
any one is BACKUP	BACKUP	take interfaces down
all are MASTER	MASTER	bring interfaces up
none is MASTER, at least one INIT (e.g. all INIT)	BACKUP	take interfaces down
any, with CARP disabled on the node (net.inet.carp.allow=0)	BACKUP	take interfaces down
anything else (MASTER+INIT mix, none)	undecided	no change

BACKUP wins a disagreement on purpose: the peer is then MASTER for something, and two live WANs is exactly what this script exists to prevent.

INIT means CARP isn't running on that VIP, typically because its link is down. The peer then promotes itself and brings its WAN up, so a node that is MASTER for nothing takes its WAN down.

Interfaces → Virtual IPs → Status → Temporarily Disable CARP doesn't leave the VHIDs in INIT: it sets net.inet.carp.allow=0 and removes them, so ifconfig shows no carp: lines at all. The script reads that sysctl and treats a node with CARP disabled the same way, as BACKUP. Two consequences:

• If CARP is INIT on both nodes at once, both WANs go down. That fails safe, and with the LAN dead the WAN is no use anyway.
• A MASTER+INIT mix stays undecided: the node is still MASTER for something.

While the node is booting, every VHID is briefly INIT, so the INIT and CARP-disabled rules are off then (see Boot behaviour below).

"Counted" means every VHID on the node, or only those listed in $carp_vhids (see Configuration). An interface this script toggles must not carry a CARP VIP at all (see below); as a safeguard, any VHID found on one is never counted, since taking that interface down on the BACKUP would push it to INIT.

On MASTER: brings the real device link up and reconfigures it. On BACKUP: resets each interface, forces the link down, and (optionally) re-points the default route.

Runs are serialized with a root-only (0600) lock (/var/run/wancarp.lock), so concurrent per-VHID events can't interleave: the last run to finish always acted on the latest state. An event run waits up to 120 seconds for the lock (enough for a MASTER run that brings up two PPP WANs); if it's still held, the run logs timed out waiting for lock and ends aborted=lock, and the next reconcile re-applies the role.

Runtime-only — does not write config

The script makes interface changes at runtime only; it does not call write_config() and never modifies config.xml. This is deliberate:

• Persisting the per-node enable flag during a CARP event raced OPNsense's own config writes and could clobber the persistent CARP maintenance-mode flag, stranding a node demoted with no GUI button to recover.
• The enable flag is a static prerequisite (enabled on both nodes, set manually on each — interfaces is not an XMLRPC-syncable section), not a per-event value. Persisting enable=0 on a demoted node would fight that prerequisite and leave it booting with the gateway subsystem skipping the interface.

Because nothing is persisted, the idempotency guard checks live state, not the config enable flag (which by prerequisite reads 1 on both nodes regardless of who is MASTER). BACKUP skips an interface whose UP flag is already clear. MASTER skips one only if it is UP and configured, meaning it has an IPv4 or non-link-local IPv6 address or a running dhclient; an interface that is UP with neither is reconfigured. Note status: is not used — on virtio (vtnet) NICs it stays active even when the interface is down; only the UP flag is honest.

A PPP-type interface (PPPoE, PPTP, L2TP; a <ppps> entry for its device) is judged by its mpd5 daemon (/var/run/<type>_<alias>.pid), since mpd5 creates the device (e.g. pppoe0), raises it only once the session is up, and destroys it on exit. MASTER skips the interface while mpd5 runs, connected or still dialing, so duplicate events and reconciles don't restart it mid-negotiation. BACKUP takes it down while either the device is UP or mpd5 runs. After interface_reset() has stopped mpd5 the device is gone, so the final ifconfig down is skipped. The parent NIC is left up: it may carry other VLANs, and nothing dials on it without mpd5. PPPoE has only been tested off-box (see Tested on).

Boot behaviour: since state isn't persisted, a rebooting node reaches the correct interface state from the CARP MASTER/BACKUP event fired as it rejoins. While OPNsense is still booting, an all-INIT role is left undecided so the script doesn't fight rc.bootup's own interface setup; if the node is still INIT once boot finishes (say its LAN link is down), the next reconcile takes WAN down. Confirmed by hard-rebooting both the MASTER and the BACKUP: each hands off and reclaims cleanly, WAN ending up live on exactly one node.

Two gaps the script can't close, since it only runs on a CARP event or from cron:

• rc.bootup brings WAN up (it's enabled in config) before CARP fires its first event, so a rebooting node has WAN up for a few seconds alongside the peer's until that event takes it down.
• If every CARP interface has no link at boot, OPNsense's boot doesn't finish until the links return: on the test pair it stalled past unbound_configure_do() for over 11 minutes, and completed a few minutes after reconnecting. The VHIDs sit in INIT from the start, so no CARP event fires, and cron (and with it the reconcile) doesn't run until boot ends. The node keeps rc.bootup's WAN up the whole time, alongside the peer's. When a link returns, the first CARP event takes it down. If only some CARP links are down, boot finishes normally and a VHID that reaches BACKUP takes WAN down at once.

Prerequisite — interfaces must be enabled in config on BOTH firewalls. Because the script is runtime-only and never persists the enable flag, every alias in $iface_aliases must be enabled in config on both nodes (Interfaces → [WAN] → Enable interface → Save/Apply). Do this separately on each node — interfaces is not an XMLRPC-syncable section, so HA config sync does not carry this flag to the peer. If an interface is disabled in config, OPNsense's gateway/routing subsystem skips it and won't install its default route even when the script brings the link up — the promoted MASTER then gets a DHCP IP but no default route and no Internet (symptom: WAN can ping its own gateway, but netstat -rn shows no default and external pings say "No route to host").

Keeping state from drifting (reconcile)

Runtime-only has a cost. The BACKUP's WAN is down in memory only, while config.xml says it's enabled (the prerequisite above requires that). OPNsense has its own path that re-reads config and brings interfaces up:

• devd turns an ethernet LINK_UP into configctl interface linkup start <dev>.
• rc.linkup checks only the on-disk enable flag, never CARP, and calls interface_configure(). That brings WAN up, starts DHCP and restores the default route.

So on the BACKUP, any of these bring WAN back up with no CARP event to undo it, and both nodes now have the WAN:

• the modem or ISP equipment reboots, or a cable is re-seated,
• on virtio, the hypervisor toggles the link (vtnet still reports carrier while administratively down, so it sends LINK_UP),
• someone clicks Save/Apply on the WAN interface page,
• configctl interface reconfigure wan is run.

(rc.linkup skips all of this during boot, which is why the reboot tests pass.)

The fix is to re-apply the role on a schedule. 10-wancarp --reconcile works out the role exactly like an event would and corrects any interface that's in the wrong state. It is quiet when there's nothing to fix: its lines only reach the system log if it changed something or hit an error. A correction is logged at error level as reconcile: interface/route state had drifted from CARP role ... so you can see that drift happened. It does nothing during boot, and backs off if a CARP event run currently holds the lock.

Cron runs it through a configd action, actions_wancarp.conf, once a minute, so a drifted WAN is taken back down within about a minute. Steps 3, 6 and 7 of Installation set this up.

Configuration

Edit the USER CONFIGURATION block near the top of 10-wancarp:

$iface_aliases       = ['wan'];        // OPNsense interface ALIASES
$carp_vhids          = [];             // VHIDs that decide the role ([] = all)
$lan_vip             = '192.168.0.1';  // default route target on BACKUP (optional)
$remove_backup_route = false;          // set true to repoint default route on BACKUP

• $iface_aliases — use OPNsense aliases (the internal names like wan, lan, opt1, opt2 — the keys under <interfaces> in config.xml). Do not use device names (igc0, em0) or descriptions; the script resolves the real device automatically via get_real_interface().

  • Single uplink: ['wan']
  • Multiple: ['wan', 'opt2']

  No interface in $iface_aliases may carry a CARP VIP. List only uplinks with no VHID on them (typically a WAN with a DHCP/PPPoE address). Taking such an interface down on the BACKUP puts its VHID into INIT, so the node can never become MASTER through it, and raises the node's CARP demotion counter (net.inet.carp.ifdown_demotion_factor), which skews the election on every other VHID too. That breaks failover, including maintenance-mode failover. The dry-run preflight reports any VHID it finds on a listed interface.

• $carp_vhids: which CARP VHIDs decide the node's role (see How it works). Leave it empty to count every VHID, which is right when all your VIPs fail over together (preempt on, the OPNsense default). List specific VHIDs, e.g. [1], to tie the WAN to just those VIPs. A VHID on an interface in $iface_aliases (a misconfiguration, see above) is never counted either way.

• $remove_backup_route / $lan_vip — leave false for almost everyone. See "Giving the BACKUP node Internet access" below for what this is for and why a failover gateway is the better answer.

Giving the BACKUP node Internet access

When a firewall becomes BACKUP, this script disables its WAN — so that box now has no path to the Internet of its own. That's usually a problem: a firewall still wants outbound access while it's the backup (package updates, DNS, NTP, gateway monitoring, etc.). It needs an alternate default route — typically out through the other firewall (the current MASTER) via the LAN.

There are two ways to provide that, and they solve the same problem:

• A failover gateway (recommended). Configure an OPNsense gateway group with WAN as Tier 1 and a LAN-side gateway (pointing at the peer/MASTER) as Tier 2, monitored by dpinger. When WAN goes down, OPNsense automatically and dynamically shifts the default route to the LAN gateway, and shifts it back when WAN returns. This is native, monitored, and self-healing. If you have this, leave $remove_backup_route = false and ignore $lan_vip — the script does not need to touch routing at all.

• $remove_backup_route (the blunt fallback). If you have no failover gateway, set $remove_backup_route = true and $lan_vip to a LAN-side gateway (often a shared LAN CARP VIP). Whenever the node is BACKUP and its default route doesn't point at $lan_vip, the script deletes it and adds one that does. Whenever the node is MASTER and its default route still points at $lan_vip, the script deletes it and has OPNsense reinstall its own (system_routing_configure()). If WAN is still waiting on DHCP at that point, OPNsense briefly uses whatever other gateway it has (or none) until the lease arrives and it switches the default back to WAN. Both checks run on every CARP event and every --reconcile, so drift is corrected within a minute — but there's no gateway monitoring, which is why a failover gateway is preferred.

Do not use both. If you configure a failover gateway and set $remove_backup_route = true, the two will fight over the default route on the BACKUP node. Pick one — the gateway group if you can.

Installation

Do steps 1–7 on each firewall (step 7 can come from HA sync instead, see there), then step 8 once. Run the commands as root in a shell (SSH, or console menu option 8).

1. Enable the interface in config. For every alias in $iface_aliases (just wan by default): Interfaces → [WAN], tick Enable interface, Save, Apply. HA config sync does not carry this flag, so set it on each node. See the prerequisite for why.

2. Download the release and verify it. Each release on the releases page publishes the script, its configd action file and a 10-wancarp.sha256 checksum covering both. Replace 1.1.1 with the release you want:

   cd /tmp
   curl -fsSLO https://git.pickysysadmin.ca/eric/10-wancarp/releases/download/1.1.1/10-wancarp
   curl -fsSLO https://git.pickysysadmin.ca/eric/10-wancarp/releases/download/1.1.1/actions_wancarp.conf
   curl -fsSLO https://git.pickysysadmin.ca/eric/10-wancarp/releases/download/1.1.1/10-wancarp.sha256
   sha256sum -c 10-wancarp.sha256
   

   You want OK on both lines. If anything prints FAILED, stop and do not install. (Releases before 1.1.0 have no actions_wancarp.conf and no reconcile; don't use them.)

3. Install both files and reload configd:

   mv 10-wancarp /usr/local/etc/rc.syshook.d/carp/10-wancarp
   chmod 0700 /usr/local/etc/rc.syshook.d/carp/10-wancarp
   mv actions_wancarp.conf /usr/local/opnsense/service/conf/actions.d/actions_wancarp.conf
   service configd restart
   

4. Configure (only if the defaults don't fit). The defaults toggle wan, count every VHID and leave routing alone, which suits most setups. To change that, edit the USER CONFIGURATION block in /usr/local/etc/rc.syshook.d/carp/10-wancarp (see Configuration). Use the same settings on both nodes. Upgrading overwrites the file, so re-apply your edits after each upgrade.

5. Dry-run it. This changes nothing:

   /usr/local/etc/rc.syshook.d/carp/10-wancarp --dryrun --reconcile
   

   Check that no line says MISSING, NOT present or resolves to NO real device, that each alias shows config=enabled, and that the role= line matches this node (MASTER or BACKUP). See Dry-run.

6. Check the configd action:

   configctl wancarp reconcile
   

   It should print OK. On the BACKUP this also takes WAN down if it is currently up, which is the intended state.

7. Schedule the reconcile. System → Settings → Cron → +: Minutes *, Hours *, all other fields *, Command Reconcile WAN with CARP role (10-wancarp), Save, Apply. If your HA sync includes Cron, add it on the primary and sync; otherwise add it on each node. The command only appears in the list on a node where step 3 has been done.

8. Test a failover. On the MASTER, Interfaces → Virtual IPs → Status → Enter Persistent CARP Maintenance Mode. WAN should go down there and come up on the other node; leave maintenance mode to flip back. Follow it with opnsense-log -f | grep wancarp (see Verifying).

Installing from main instead (no checksum)

main moves with every commit, so there is no published hash for it. If you want the newest changes and accept that, replace steps 2 and 3 with:

curl -fsSL https://git.pickysysadmin.ca/eric/10-wancarp/raw/branch/main/10-wancarp -o /usr/local/etc/rc.syshook.d/carp/10-wancarp
chmod 0700 /usr/local/etc/rc.syshook.d/carp/10-wancarp
curl -fsSL https://git.pickysysadmin.ca/eric/10-wancarp/raw/branch/main/actions_wancarp.conf -o /usr/local/opnsense/service/conf/actions.d/actions_wancarp.conf
service configd restart

Dry-run (test before going live)

Pass --dryrun (or -n) to run the script from the CLI without changing anything — no interface_configure/interface_reset, no ifconfig/route. Output is echoed to the terminal as well as the system log.

./10-wancarp --dryrun 1@vtnet1 MASTER
./10-wancarp --dryrun 1@vtnet1 BACKUP

(The flag may go in any position; the <vhid>@<interface> source and the role are still required. A malformed source string only warns in dry-run instead of aborting.)

With a role argument, dry-run is a what-if: it acts on the role you pass, so you can exercise both branches on either node (INIT acts as BACKUP, as on a node that is MASTER for nothing). It also prints the live role a real event would act on right now:

wancarp: role=BACKUP counted=[1@vtnet0=BACKUP, 2@vtnet2=BACKUP]
wancarp: DRYRUN: acting on the role you passed ('MASTER') as a what-if; a real event now would act as BACKUP.

To see what a reconcile would do right now (live role, no what-if), use:

./10-wancarp --dryrun --reconcile

Before printing what each role would do, dry-run runs a preflight that verifies the things most likely to make a live failover silently fail:

• the legacy global $config is actually populated (if not, a live run resolves nothing and no-ops),
• every OPNsense PHP API the script calls exists on this build — including interface_reset(), which has varied between releases, and, with --reconcile, the exit_on_bootup() boot guard (a missing one doesn't fatal; a live reconcile falls back to product::booting()),
• the product::booting() boot check the INIT rule relies on, and whether the node is booting right now,
• each configured alias resolves to a real device, with its current enable flag and live link state (flags/status),
• the current default route (when $remove_backup_route is set),
• every CARP VHID on the node, which ones were counted or skipped (and why), and the resulting role,
• that no CARP VHID sits on an interface in $iface_aliases (reported as an error naming the VHID and alias).

It is safe to run on a live firewall; it makes no changes.

Verifying / troubleshooting

Every log line is prefixed with wancarp, so one grep pulls the whole story of any event:

opnsense-log -f | grep wancarp

opnsense-log replaced clog in 21.x; -f follows, and the log type defaults to system. The logs are plain text now, so this works too:

grep wancarp /var/log/system/*.log

(GUI: System → Log Files → General, filter on wancarp.)

For each event you get:

• a START banner with the script version and the raw arguments (proves the hook fired, which version is installed, and with what),
• a role= line listing each VHID's state, which were counted or skipped, and the role the script acted on,
• one line per interface action — or a skip line with the reason,
• explicit errors (return code + captured output) for any failed shell command,
• explicit errors for an unknown/typo'd interface alias,
• any PHP exception: one thrown while handling an interface is logged as '<alias>' (<device>) failed: ... and the run carries on with the next alias; anything else is caught by the top-level handler,
• an END banner: a normal run reports role=MASTER|BACKUP|none and result=ok|partial|exception with changed=yes|no (dry-run uses would-change=yes|no). partial means at least one action failed (a shell command such as ifconfig down or route add, or an exception on one interface, logged as an error above it), so the node may be only partly switched; a rejected event reports aborted=invalid-type|bad-source|lock changed=no.

A cron reconcile logs nothing when state is already correct. If it does log, it changed something or hit an error, and you get the full START…END story for that run.

If you see a START banner but no END banner, the script died mid-run on an uncaught fatal — check opnsense-log php (/var/lib/php/tmp/PHP_errors.log), if PHP error logging is on.

Manually simulate an event (testing only — prefer --dryrun for a no-op run):

/usr/local/etc/rc.syshook.d/carp/10-wancarp 1@vtnet1 BACKUP

Off-box tests

tests/ runs the real script on any machine with PHP 8 (no OPNsense needed), against stub OPNsense functions and fake ifconfig/route/pgrep that share one simulated firewall state. CI runs it on every push.

php tests/run.php            # every case
php tests/run.php backup     # only cases whose name contains "backup"
php tests/run.php -v         # show calls and log for anything not passing

Each case in tests/run.php sets a starting state (interface UP/addresses, each VHID's CARP state, default route, running dhclient), optional USER CONFIGURATION values and the script arguments, then checks the final state, the OPNsense calls made, the log lines and the exit code. Results:

• PASS / FAIL: as you'd expect; any FAIL fails the run.
• XFAIL: a case marked 'xfail' => '#N' for a known open bug, still broken.
• XPASS: an xfail case now passes. The run fails so you remember to delete its 'xfail' line.

To reproduce a bug, add a case that describes the correct behaviour and mark it xfail with the issue number; the fix then removes the marker. The stubs model current opnsense/core closely enough for the script's logic (for example, interface_configure() returns at once when enable is unset, and interface_reset() doesn't down the link), but they are not OPNsense: a change to how the script talks to OPNsense still needs a --dryrun and a failover on a real pair.

Tested on

OPNsense 26.1.x (PHP 8, FreeBSD 14.x), validated on a live two-node HA pair (virtio NICs, DHCP WAN, config sync enabled): maintenance-mode and hard-reboot failover in both directions, WAN moving with its IP + default route, idempotent across the duplicate per-VIP CARP events. Uses only stable OPNsense PHP APIs (get_real_interface, legacy_interface_flags, interface_configure, interface_reset, log_msg / log_error). --reconcile also calls exit_on_bootup() (falling back to product::booting() if it's missing), the role rule asks product::getInstance()->booting() (treated as not booting if the class is missing), and $remove_backup_route calls system_routing_configure().

OPNsense 26.7.x (PHP 8.5, FreeBSD 15.1) — reviewed against the 26.1 -> 26.7 core diff and confirmed with --dryrun on an upgraded node. Nothing the script depends on changed:

• The invocation contract is byte-identical: devd/carp.conf still matches <vhid>@<device> + MASTER|BACKUP|INIT, rc.syshook is unchanged, and rc.syshook.d/carp/ still exists. An in-place upgrade leaves the script installed and executable.
• All five PHP APIs above are unchanged, except that interface_configure() gained an optional 5th parameter ($batch = false). It defaults to the 26.1 behaviour, so the four-argument calls here are unaffected.
• "Interface assignments migrated to MVC/API" is not a storage change. The new NetworkInterface model is <mount>:memory:</mount> and reads/writes straight through to legacy config.xml <interfaces>, so $config aliases (wan, opt1, ...) and the enable flag are untouched. The prerequisite above still applies exactly as written.

One behavioural note, benign here: 26.7 marks VLAN devices 'reload' => false, so a VLAN parent is no longer re-created by the $reload = true path in interface_configure(). The VLAN already exists at runtime, and this does not apply at all to a plain physical/virtio WAN.

Not yet validated on a live pair: the node-role logic ($carp_vhids, role from all VHIDs) and --reconcile / the configd action. They have been tested only off-box, in the off-box tests. Before relying on them, run --dryrun --reconcile on each node and check the role= line. Then, on the BACKUP, run configctl interface reconfigure wan to force the drift and confirm the next cron run takes WAN back down.

PPPoE (and other mpd5 WANs) is untested on a live pair. The live validation above used a DHCP WAN. PPP handling follows current interfaces.inc and is covered off-box only. If you run it on PPPoE, check that the --dryrun alias line shows ppp daemon=running on the MASTER, that a failover ends with result=ok on both nodes, and that a reconcile while the MASTER is redialing logs nothing. Please report how it goes.

ifconfig LOWER_UP (FreeBSD 15.1)

On 26.7 a downed WAN reports:

flags=[BROADCAST,SIMPLEX,MULTICAST,LOWER_UP] status=active

LOWER_UP is a synthetic token (bit 24, IFF_NETLINK_1) emitted by ifconfig's netlink output path — it reports carrier state, not administrative state, and is absent from sys/net/if.h. So on a BACKUP node three separate signals (LOWER_UP, status: active, and virtio's always-live link) all claim the interface is up while it is administratively DOWN.

iface_is_up() therefore matches the exact token UP with a strict in_array(..., true) against the comma-split flag list. Do not loosen this to strpos() / str_contains() — a substring test matches LOWER_UP, the MASTER branch concludes the link is already up and skips it, and the promoted node comes up with no WAN. Both dry-run directions were verified on 26.7: MASTER reports WOULD enable, BACKUP reports already down, skipping.

HA note: CARP-aware services

This script handles the WAN, but it's worth auditing anything else that runs on both nodes. A multicast reflector running on both firewalls at once is a classic trap — e.g. the mDNS Repeater (Avahi) with two nodes reflecting between the same VLANs forms a reflection loop (224.0.0.251 flood) that looks like a CARP/L2 storm but isn't. Enable its "Enable CARP Failover" option (or otherwise make it run on the MASTER only). Rule of thumb: any service that should be active on just one node must be CARP-aware or single-node.

References

Original script concept — kronenpj on GitHub