RHEL Network Profile Notes
I'm normally an Ubuntu user, but I recently configured a redundant pair of servers running RHEL5, and I wanted to have two pools of IPs (one for production and one for testing) that I could manually move between the boxes quickly and easily. Network configuration is very different in RHEL5 than in Ubuntu, but I figured that RedHat's network profiles were probably the easiest way to do this. However, I quickly discovered that the configuration and use of network profiles in RedHat is poorly documented and doesn't behave as I would expect, particularly when using Aliases. So, I'm posting my notes here in the hope they will save others some of the trouble I went through.
First, a quick summary of network configuration in RHEL5:
The system's hostname and default gateway are configured in /etc/sysconfig/network .
Each interface's IP and Subnet is configured in /etc/sysconfig/network-scripts/ifcfg-<device> . Multiple versions of the config file may be created as ifcfg-<device>-<clone> or ifcfg-<device>_<clone> (ifcfg-<device>-<clone> inherits configuration directives from ifcfg-<device>, while ifcfg-<device>_<clone> does not). Aliases (multiple IPs on a single interface) may be configured using ifcfg-<device>:<alias> files (use DEVICE=<device>:<alias> in the file as well).
Static routes are configured in /etc/sysconfig/network-scripts/route-<device> using either ADDRESS0, NETMASK0, and GATEWAY0 directives or <Target Network>/<Mask Length> via <Gateway> syntax.
ifup <device> , ifup <device>-<clone> , or ifup <device>_<clone> will read the above files and configure the specified device. ifdown <device> , ifdown <device>-<clone> , or ifdown <device>_<clone> will de-configure the specified device. By default, Aliases are automatically configured/de-configured with the parent device (you can add ONPARENT=no to the Alias file to disable this, but note that ONBOOT=yes is ignored so the Alias will not be configured at boot time).
And some notes on ARP caching:
Packets traveling between subnets are routed using their destination IP address. Packets traveling within a subnet are routed using their destination MAC address. So, when a remote machine sends a local machine a packet, the local gateway will receive the packet, and before it can forward the packet to the local machine, it will need to determine the MAC address of the local machine. It does this by broadcasting an ARP request on the local subnet, asking for the MAC address that is associated with the packet's destination IP address. The local machine configured with that IP address will respond with it's MAC address, and the router will forward the packet to that MAC address.
Of course, the local gateway will cache the IP and MAC association so it doesn't have to send an ARP request every time it receives a packet. This causes problems when we move an IP between servers, as the old server will continue to receive the packets destined for that IP until the gateway's ARP cache entry expires (which could be several minutes). If the new server happens to send a packet to the gateway with its MAC and the IP in the packet's source fields, the gateway's ARP cache will be updated immediately, but most servers will not send any packets unless they have received a packet.
To correct this problem, ifup automatically runs arping after configuring a device, which broadcasts an ARP update to the local subnet, causing the ARP cache on all local routers and hosts to be updated. arping is only run for the parent device's IP, and is not run for Aliases unless ifup <device>:<alias> is run manually for each Alias.
Now on to network profiles:
All ifcfg-* files used by all profiles should be placed in /etc/sysconfig/networking/devices/
files are supported, but ifcfg-<device>-<clone>
files are not.
A directory should be created for each profile in /etc/networking/profiles/
. A copy of /etc/hosts
, and /etc/sysconfig/network
must be provided for each profile (the same files may be hard-linked into all of the profile directories). The relevant ifcfg-* files from /etc/sysconfig/networking/devices/
should be hard-linked into each profile's directory. route-<device>
files do not appear to be supported (one copy may be used for all profiles, but different profiles cannot use different copies of these files, so for all practical purposes they may as well not be supported).
will dump the parsed network configuration (and may automatically edit the above files to correct certain errors).
system-config-network-cmd --profile <profile> --activate
will switch to the specified profile.
on the kernel command line will boot the machine using the specified profile (though this may have a bug - see https://bugzilla.redhat.com/show_bug.cgi?id=544105
When switching profiles, the following occurs:
ifdown is run for all files in /etc/sysconfig/networking/devices/ that are not in the new profile but are associated with a device that is currently assigned an IP. (ifdown will fail on any files that are not currently in /etc/sysconfig/network-scripts/, but those failures will be ignored).
Any new /etc/sysconfig/network-scripts/ifcfg-* files are added to devices/ and profiles/<old profile>/.
The old profile's config files are removed and replaced with hard-links to files in profiles/<new profile>/. If the old profile used an ifcfg-<device> file and the new profile uses an ifcfg-<device>_<clone> file for the same device, the ifcfg-<device> file does not appear to be removed (bug?).
ifup is run for all files in the new profile that contain ONBOOT=yes and are associated with a device that is not currently up.
When switching profiles that use Aliases:
ifdown is run for all files (including Aliases) in /etc/sysconfig/networking/devices/ that are not in the new profile but are associated with a base device that is currently assigned an IP. If the base device is assigned an IP and the old and new profiles use different config files for the base device, then all Aliases will be brought down. If the base device is assigned an IP and the old and new profiles use the same config file for the base device, then any Aliases that are no longer used in the new profile will be brought down. If the base device is not currently assigned an IP (if all IPs are assigned to the device as Aliases), then no Aliases will be brought down.
Since Alias config files do not normally include ONBOOT=yes, ifup is not normally run for any Alias files. If the old and new profiles use different config files for the base device, then all Aliases will be brought up when the base device is brought up (in addition to the old profile's Aliases if the base device was not previously assigned an IP). If the old and new profiles use the same config file for the base device, then no new Aliases will be brought up.
In other words, if the old and new profiles use a different config file for the base device, the final Alias configuration is as expected, but all Aliases must be brought down and up in the process, even if some Aliases are used in both profiles. If the old and new profiles use the same config file for the base device, or if the base device is not assigned an IP, system-config-network-cmd will do strange things.
Making profiles work better with Aliases:
In my case, I wanted to be able to configure each of my servers to use no IPs (offline), production IPs only, test IPs only, or both production and test IPs simultaneously, and I wanted to be able to change between the production<->both and test<->both profiles without bringing down the IPs common to both profiles. In addition, I wanted arping
to be automatically run for each new Alias, to ensure that incoming traffic would begin hitting the server immediately.
In order to change between profiles without bringing down the common IPs, all of the profiles must share the same config file for the base device (since using different config files will cause the base device to be brought down, and the kernel will automatically bring down all Aliases when the base device is brought down). So, I've posted a patch at https://bugzilla.redhat.com/show_bug.cgi?id=624850
, which causes system-config-network-cmd
to work properly even if the old and new profiles use the same config file for the base device and/or the base device is not assigned an IP.
As it turns out, this patch naturally causes arping
to be run for each new Alias, as long as the same config file is used for the base device in each profile and the base device is assigned an IP (since Aliases are brought up using explicit ifup <device>:<alias>
commands instead of being brought up automatically with the base device). However, if the base device is not assigned an IP, ifup
will still be run for the base device, so Aliases will be brought up automatically and not explicitly, and arping
will not be run for Aliases. If there are no IPs common to all of the profiles (as in my case, where the production and test IPs are orthogonal), but you still want arping
to be run for each new Alias, you can either assign the base device a bogus IP, or (as a hack) set ONBOOT=yes
in each Alias config file (which will cause each Alias to be brought up twice, once automatically and once explicitly, though this shouldn't hurt anything).