<![CDATA[BEESLEY.TECH

Here is a recent design that was implemented for a customer due to the need for spanning their layer 2 network across different sites that are physically separated.
This post will describe the specifics of the configuration of this design and why things are configured the way they are.This configuration is actually a lab that I have running, in order to test the configuration.

Firstly the sites are physically separate, therefore it is not possible (without significant cost) to run a cable or use an ISP service for this. Therefore IPsec was chosen due to this traffic traversing the Internet.

IPsec Configuration
We need to configure the IPsec tunnel first, in order to facilitate the VXLAN. I had attempted to configure the tunnel in the GUI, then make the edits to allow for VXLAN afterwards, however this did not work as I was presented with an error.

The config for the IPsec tunnel on the left side FW is:
config vpn ipsec phase1-interface
edit "VPN"
set interface "port1"
set peertype any
set net-device disable
set proposal des-md5 des-sha1
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 172.16.1.1
set encap-remote-gw4 172.16.1.2
set remote-gw 172.16.1.2
set psksecret ENC

So a couple of things that I found with this piece of the configuration:

IKEv1 & IKEv2 both work
This is an eval/lab firewall, that is why des-md5 is used
The set encapsulation vxlan defines the encapsulation method as VXLAN, other possible options are GRE and IKE. This means that when traffic enters the tunnel a new header will be added with that protocol (in this case VXLAN).
The encap-local-gw4 and encap-remote-gw4 define the IP address at each end of the tunnel.
remote-gw is the IP address of the remote end of the gatway (in this case it is the same), there may be a situation where this is necessary, however I have not run into that. Possibly if you are terminating the VXLAN on a device that is not the firewall.

Switch Configuration
The switch configuration is fairly straight forward, we will need to add both the local port and IPsec interface into the virtual switch.
config system switch-interface
edit "VXLAN"
set vdom "root"
set member "VPN" "port10"
set intra-switch-policy explicit
next
end
We need to specifically define the VDOM, it most cases root, if you do not it will throw an error and abort your config. We then set the members with the set member command, and finally we use the command set intra-switch-policy explicit to tell the firewall that policy needs to be defined in order for the VXLAN to operate. I have seen some places say that this is optional, however I could not get it to work without this command. Also if you go back to the switch configuration after initially creating the switch then it will not allow you to change this option, which is not a big deal, because you can just recreate the software switch without much issue.

VXLAN Encapsulation
In the end the result is that you can ping across the tunnel to a device on the same LAN.
Here I am using the VPCs in GNS3 to demonstrate that:

Note that this device does not have a default gateway, I did that intentionally to show that the PC does not reach out to a router.

Ultimately, this configuration is not too complex, once the fundamentals are understood. This configuration will add on an extra 88 bytes to the header (50 for VXLAN, 38 for IPsec), so if you have a reliable core network that will support larger packet headers then it may be a good idea to enable them.

UPDATE
After this configuration was put in place for the customer issues started to arise when access different subnets across the VXLAN (e.g. subnet 1 to subnet 2, but not subnet 1 to subnet 1). After some investigation it was found that the firewall was dropping these packets, this was due to the fact that the packet was coming in the firewall, going out the virtual switch interface, then attempting to come in again, the kernel of the firewall saw this as duplicate packets and dropped them:

id=20085 trace_id=127 func=resolve_ip_tuple_fast line=5609 msg="Find an existing session, id-006341bc, reply direction"
id=20085 trace_id=127 func=ip_session_core_in line=6260 msg="outgoing dev changed:24->18 dir=reply, drop"

The traffic flow was as follows:
Tunnel in -> virtual switch out -> physical interface in

Ultimately the solution to this is to enable auxiliary sessions, there are a couple of caveats to this though:

This setting is only available on hardware that has a CP9 ASIC or newer.
This setting is only available on version 6.2.3 or newer.

This means that the only option here is to upgrade to a newer model unit if you find yourself in this position.

You can see the maximum firmware version for your hardware unit by going to support.fortinet.com and using the upgrade path tool. You can also see the ASIC in use by using the command get hardware status.