Static Route with Ubuntu 18.04 and Netplan

Mindwatering Incorporated

Author: Tripp W Black

Created: 01/26/2020 at 02:35 PM

 

Category:
Ubuntu
Configuration Files

Issue:
Have two connections to another network in separate buildings on same campus.
The default connection goes through firewall/gateway and through the VPN tunnel, and the firewall redirects on the second ping to go through the faster "pipe" that is direct.

Main Gateway/Firewall: 10.0.12.1 --> 10.0.2.1
Faster Connection: 10.0.12.3 --> 10.0.2.3

Yes, it would be nice is the first packet goes through the "direct" way, but Ubuntu and the Cisco gateway we use don't seem to do static routes the same as our older slower one. We haven't found a direct solution. The routing technically works, but NFS targets are failing between the Ubuntu NFS servers on the 10.0.12.x and the ESXi servers on the 10.0.2.x networks.

The ESXi hosts a static route, which they honor on the first packet.
However, the Ubuntu and sometimes Mac clients on the 12.x network ask the 10.0.12.1 firewall and it sends the first packet goes through the gateway, and then gives the hint on the second packet to go through the "direct" 10.0.12.3 connection. This causes the NFS stores to go inaccessible for the ESXi servers whose first packet comes back the "long" way. We can reconnect manually, but then a certain number of hours later when the Ubuntu NFS server's routing tables go stale, the problem shows up again.

Solution:
Since the Ubuntu 18.04 NFS serves are using NetPlan, we had to use the new YAML config file format.
$ route -v
< confirms the direct route has dropped out>
$ cd /etc/netplan/
$ sudo vi /50-cloud-init.yaml
Being VERY careful to use spaces and not tabs, and keep the indenting perfect, add the following bolded section below, and save:
<a>
network:
version: 2
renderer: networkd
ethernets:
ens160:
dhcp4: no
addresses: [10.0.12.176/24]
gateway4: 10.0.12.1
nameservers:
addresses: [10.0.12.1,8.8.8.8,8.8.4.4]
routes:
- to: 10.0.12.0/24
via: 10.0.12.3
metric: 10
<esc>:wp

Restarting the network doesn't touch netplan, and we wanted to know that it would stick for a reboot.
$ sudo reboot

Alternately, to make the network active w/o a reboot:
$ sudo netplan apply
$ ip addr show dev ens160


Note:
In Ubuntu 22, the gateway4 is deprecated. Instead, remove the gateway4 line above. Add the following line before the other routes:
...
routes:
- to: default
via: 10.0.12.1
...


After the reboot, we performed another route test, and confirmed the route "stuck".
$ sudo route -v
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 0 0 0 eth1
10.0.12.0 10.0.12.3 255.255.255.0 UG 10 0 0 eth1
10.0.12.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

Returning to the ESXi hosts, we reconnect the NFS stores. List the NFS stores to confirm they aren't already "back".
[root@mwvhost1:/] esxcfg-nas -l
< shows two of the NFS servers still disconnected>

Reconnected with:
[root@mwvhost1:/] esxcfg-nas -r
< nothing output>

Check again with:
[root@mwvhost1:/] esxcfg-nas -l
< shows all the NFS server shares all connected properly.


Note:
If you deleted the NFS stores, you'll have to re-add them. In addition, the VCSA will remember the "inaccessible" ones old names and the new ones with the same name will have a (1) added. Before you read the NFS stores, do a rename on the inaccessible NFS stores add add a suffix "dead", or something. They stick until all the hosts have been rebooted.





ESXi Helpful Commands:
List static routes on ESXi servers:
esxcfg-route -l

Add the NFS store back to a ESXi host:
[root@mwvhost1:/etc] esxcfg-nas -a -o 10.0.12.176 -s /local2/ MW12NFS03

Review recent log:
[root@mwvhost1:/etc] tail /var/log/vobd.log

Test a NAS Port:
[root@mwvhost1:/etc] nc -z 10.0.12.176 2049

Test that Jumbo Frames are making it through using Jumbo Ping:
[root@mwvhost1:/etc] vmkping -s 9000 10.0.12.186


Static Routes on ESXi 5.5 +
[root@mwvhost1:~] esxcli network ip route ip4 add --gateway 192.168.2.1 --network 192.168.22.0/24

[root@mwvhost1:~] esxcli network ip route ipv4 list



Warning:
Ubuntu 20 Changed the NetPlan Naming. See sample below.
In addition, static routes entered network GUI settings, did not get populated to the 01-network-manager-all.yaml file. The result was that ESXi hosts and the vCenter time-out connecting to NFS shares as the packages were going down both paths.

Sample:
$ cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
version: 2
renderer: NetworkManager
ethernets:
enp1s0:
dhcp4: false
addresses: [10.1.22.176/24]
gateway4: 10.1.22.1
nameservers:
addresses: [123.125.1.10,125.125.5.10]
routes:
- to: 10.1.20.0/24
via: 10.1.22.3
metric: 4
- to: 10.1.20.0/24
via: 10.1.22.1
metric: 100




Mac OSX to ESXi Test:
System --> Network --> Ethernet --> Details --> Hardware --> Manual (slider) --> Change MTU to 9000 from 1500.
$ ping -D -s 8164 10.1.22.176


Win Test:
Network and Sharing --> Local Area Connection --> Properties (button) --> Configure (button) --> Advanced (tab) --> Jumbo Packet --> Change to 9014.
(Win requires the Jumbo packet to include the 14 ethernet header bytes overhead. Otherwise, the PC/server will not handle 9000 MTU jumbo frames.)
$ ping -l 8164 10.1.22.176
<verify ping successful>
$ ping -l 9000 10.1.22.176
<verify ping successful>

Static Route on MS Windows:
(to network 10.1.22.0 via 10.1.20.3)
route add 10.1.22.0 MASK 255.255.255.0 10.1.20.3



previous page