PRIMECLUSTER Global Link Services Configuration and Administration Guide: Redundant Line Control Function 4.2 (for Linux)

3.2.2.1 Setup common to modes

(1) Physical interface settings

Set up physical interfaces. The physical interface settings vary depending on redundant network methods and operating system.

See the following table 3.6.

[Table 3.6 Physical interface settings]

Redundant network methods		Operating system
		RHEL3	RHEL4		SLES
			Tagged VLAN disabled	Tagged VLAN enabled	Tagged VLAN disabled	Tagged VLAN enabled (SLES9 only)
Fast switching mode (IPv4)		Setup 1	Setup 3	Setup 5	Setup 7	Setup 9
Fast switching mode (IPv6)		Setup 2	Setup 4	Setup 6	Setup 8	Setup 10
Fast switching mode (Dual)		Setup 1	Setup 3	Setup 5	Setup 7	Setup 9
NIC switching mode (IPv4)	Primary interface	Setup 1	Setup 3	Setup 5	Setup 7	Setup 9
	Secondary interface	Setup 2	Setup 4	Setup 6	Setup 8	Setup 10
NIC switching mode (IPv6)	Primary interface	Setup 2	Setup 4	Setup 6	Setup 8	Setup 10
	Secondary interface	Setup 2	Setup 4	Setup 6	Setup 8	Setup 10
NIC switching mode (Dual)	Primary interface	Setup 1	Setup 3	Setup 5	Setup 7	Setup 9
	Secondary interface	Setup 2	Setup 4	Setup 6	Setup 8	Setup 10

When a operating system is RHEL, to specify "HWADDR=XX:XX:XX:XX:XX:XX" for the GLS physical interface settings (the "/etc/sysconfig/network-scripts/ifcfg-ethX" file), add "HOTPLUG=no" to the settings.

If "HOTPLUG=no" is set when the operating system is RHEL3 or RHEL4, it will not disable the PCI Hot Plug. Active maintenance of NIC (PCI cards) can be performed for the physical interface where "HOTPLUG=no" is set.

Setup 1

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
BROADCAST=XXX.XXX.XXX.XXX
IPADDR=XXX.XXX.XXX.XXX
NETMASK=XXX.XXX.XXX.XXX
NETWORK=XXX.XXX.XXX.XXX
ONBOOT=yes
TYPE=Ethernet

Setup 2

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

Setup 3

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
HWADDR=XX:XX:XX:XX:XX:XX
HOTPLUG=no
BROADCAST=XXX.XXX.XXX.XXX
IPADDR=XXX.XXX.XXX.XXX
NETMASK=XXX.XXX.XXX.XXX
NETWORK=XXX.XXX.XXX.XXX
ONBOOT=yes
TYPE=Ethernet

Setup 4

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
HWADDR=XX:XX:XX:XX:XX:XX
HOTPLUG=no
ONBOOT=yes
TYPE=Ethernet

Setup 5

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
HWADDR=XX:XX:XX:XX:XX:XX
HOTPLUG=no
ONBOOT=yes
TYPE=Ethernet

/etc/sysconfig/network-scripts/ifcfg-ethX.Y

DEVICE=ethX.Y
BOOTPROTO=static
BROADCAST=XXX.XXX.XXX.XXX
IPADDR=XXX.XXX.XXX.XXX
NETMASK=XXX.XXX.XXX.XXX
NETWORK=XXX.XXX.XXX.XXX
ONBOOT=yes

Setup 6

/etc/sysconfig/network-scripts/ifcfg-ethX

DEVICE=ethX
BOOTPROTO=static
HWADDR=XX:XX:XX:XX:XX:XX
HOTPLUG=no
ONBOOT=yes
TYPE=Ethernet

/etc/sysconfig/network-scripts/ifcfg-ethX.Y

DEVICE=ethX.Y
BOOTPROTO=static
ONBOOT=yes

Setup 7

/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX

BOOTPROTO='static'
BROADCAST='XXX.XXX.XXX.XXX'
IPADDR='XXX.XXX.XXX.XXX'
NETMASK='XXX.XXX.XXX.XXX'
NETWORK='XXX.XXX.XXX.XXX'
STARTMODE='onboot'
PERSISTENT_NAME='ethX'
UNIQUE='XXXXXXXXXXXXXXXX'
_nm_name='bus-pci-XXXX:XX:XX.X'

• Edit the setup file (/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX) that is corresponding to the MAC address "XX:XX:XX:XX:XX:XX" of physical interfaces for GLS. Do not create the configuration file of /etc/sysconfig/network/ifcfg-ethX.
• In certain machine environment, it happens that the interface names of your network devices switch from one boot to another. If you want to give your interfaces persistent names, add "PERSISTENT_NAME=ethX" in /etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX. Set ethX to the number of more than the implementable network interfaces (more than 300 is recommended). See also ifup(8)'s online manual(man ifup).
• The "UNIQUE=" and "_nm_name=" values are automatically set in the setup file. Users are not supposed to change the values.

Setup 8

/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX

BOOTPROTO='static'
STARTMODE='onboot'
PERSISTENT_NAME='ethX'
UNIQUE='XXXXXXXXXXXXXXXX'
_nm_name='bus-pci-XXXX:XX:XX.X'

• Edit the setup file (/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX) that is corresponding to the MAC address "XX:XX:XX:XX:XX:XX" of physical interfaces for GLS. Do not create the configuration file of /etc/sysconfig/network/ifcfg-ethX.
• In certain machine environment, it happens that the interface names of your network devices switch from one boot to another. If you want to give your interfaces persistent names, add "PERSISTENT_NAME=ethX" in /etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX. Set ethX to the number of more than the implementable network interfaces (more than 300 is recommended). See also ifup(8)'s online manual(man ifup).
• The "UNIQUE=" and "_nm_name=" values are automatically set in the setup file. Users are not supposed to change the values.

Setup 9

/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX

BOOTPROTO='static'
STARTMODE='onboot'
PERSISTENT_NAME='ethX'
UNIQUE='XXXXXXXXXXXXXXXX'
_nm_name='bus-pci-XXXX:XX:XX.X'

• Edit the setup file (/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX) that is corresponding to the MAC address "XX:XX:XX:XX:XX:XX" of physical interfaces for GLS. Do not create the configuration file of /etc/sysconfig/network/ifcfg-ethX and /etc/sysconfig/network/ifcfg-ethX.Y.
• In certain machine environment, it happens that the interface names of your network devices switch from one boot to another. If you want to give your interfaces persistent names, add "PERSISTENT_NAME=ethX" in /etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX. Set ethX to the number of more than the implementable network interfaces (more than 300 is recommended). See also ifup(8)'s online manual(man ifup).
• The "UNIQUE=" and "_nm_name=" values are automatically set in the setup file. Users are not supposed to change the values.

/etc/opt/FJSVhanet/script/conf/vlan_if.conf

DEVICE<n>=ethX
VLANID<n>=Y
IPADDR<n>=XXX.XXX.XXX.XXX
NETMASK<n>=XXX.XXX.XXX.XXX
TYPE<n>=vlan

• The above setup file does not exist after GLS installation. If you use tagged VLANs on SLES9, create the setup file.
• In the setup file, a string from sharp "#" to line feed indicates a comment.
• Add alphanumerical characters and/or periods "." after equal "=" for each parameter setup value. If you specify values other than alphanumerical characters and/or periods ".", the setting will be disabled. For example, such characters are set in the second line, the second line and after will be disabled.
• "<n>" of DEVICE<n>=" and "VLANID<n>=" must begin with "0" in the setup file. If you set multiple tagged VLAN interfaces, specify a sequential range of numbers beginning with "0" (0, 1, 2, 3, 4, ...).

# Tagged VLAN interface eth1.2
DEVICE0=eth1
VLANID0=2
IPADDR0=192.168.20.10
NETMASK0=255.255.255.0
TYPE0=vlan

# Tagged VLAN interface eth2.5
DEVICE1=eth2
VLANID1=5
IPADDR1=192.168.50.10
NETMASK1=255.255.255.0
TYPE1=vlan
.....

Setup 10

/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX

BOOTPROTO='static'
STARTMODE='onboot'
PERSISTENT_NAME='ethX'
UNIQUE='XXXXXXXXXXXXXXXX'
_nm_name='bus-pci-XXXX:XX:XX.X'

• Edit the setup file (/etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX) that is corresponding to the MAC address "XX:XX:XX:XX:XX:XX" of physical interfaces for GLS. Do not create the configuration file of /etc/sysconfig/network/ifcfg-ethX and /etc/sysconfig/network/ifcfg-ethX.Y.
• In certain machine environment, it happens that the interface names of your network devices switch from one boot to another. If you want to give your interfaces persistent names, add "PERSISTENT_NAME=ethX" in /etc/sysconfig/network/ifcfg-eth-id-XX:XX:XX:XX:XX:XX. Set ethX to the number of more than the implementable network interfaces (more than 300 is recommended). See also ifup(8)'s online manual(man ifup).
• The "UNIQUE=" and "_nm_name=" values are automatically set in the setup file. Users are not supposed to change the values.

/etc/opt/FJSVhanet/script/conf/vlan_if.conf

DEVICE<n>=ethX
VLANID<n>=Y
IPADDR<n>=
NETMASK<n>=
TYPE<n>=vlan

For the secondary interface, no value must be set for "IPADDR<n>=" and "NETMASK<n>=".

Example: IPADDR0=(LF)

• The above setup file does not exist after GLS installation. If you use tagged VLANs on SLES9, create the setup file.
• In the setup file, a string from sharp "#" to line feed indicates a comment.
• Add alphanumerical characters and/or periods "." after equal "=" for each parameter setup value. If you specify values other than alphanumerical characters and/or periods ".", the setting will be disabled. For example, such characters are set in the second line, the second line and after will be disabled.
• "<n>" of DEVICE<n>=" and "VLANID<n>=" must begin with "0" in the setup file. If you set multiple tagged VLAN interfaces, specify a sequential range of numbers beginning with "0" (0, 1, 2, 3, 4, ...).

# Tagged VLAN interface eth1.2
DEVICE0=eth1
VLANID0=2
IPADDR0=192.168.20.10
NETMASK0=255.255.255.0
TYPE0=vlan

# Tagged VLAN interface eth2.5
DEVICE1=eth2
VLANID1=5
IPADDR1=192.168.50.10
NETMASK1=255.255.255.0
TYPE1=vlan

# Tagged VLAN interface eth3.2
DEVICE2=eth3
VLANID2=2
IPADDR2=
NETMASK2=
TYPE2=vlan

# Tagged VLAN interface eth4.5
DEVICE3=eth4
VLANID3=5
IPADDR3=
NETMASK3=
TYPE3=vlan
.....

(2) Verification of the physical interface

Verify if the physical interface is inserted into the system using ifconfig command.
Also, if the physical interface is UP, check whether it is marked as "RUNNING". If "RUNNING" is not displayed, the links might be down on the interface. Check the cable switch and HUB speed settings. Use the "ethtool" command to check the link state.

# ifconfig -a
eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:192.168.70.2  Bcast:192.168.70.255  Mask:255.255.255.0
          inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2140 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2140 errors:0 dropped:0 overruns:0 carrier:0
          collisions:22 txqueuelen:1000
          RX bytes:278285 (271.7 KiB)  TX bytes:273656 (267.2 KiB)
          Base address:0xec80 Memory:d2fc0000-d2fe0000
eth1      Link encap:Ethernet  HWaddr 0xx:xx:xx:xx:xx:xx
          inet addr:192.168.71.2  Bcast:192.168.71.255  Mask:255.255.255.0
          inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2118 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:278492 (271.9 KiB)  TX bytes:273666 (267.2 KiB)
          Base address:0xecc0 Memory:d2fe0000-d3000000

In the above example, it is possible to use eth0 and eth1. For details regarding ifconfig command, refer to the Linux manual.

When using Tagged VLAN, ensure that the NIC supports tagged VLAN functionality (IEEE 802.1Q). In addition, in a Redundant Line Control function, the effective range of VLAN-ID which can be specified is from 1 to 4094.

(3) Checking the name service

When using name services such as DNS or NIS, define keywords such as hosts in /etc/nsswitch.conf file to first refer to the local file. This allows to solve the address even if the DNS, NIS or LDAP sever is unreachable. The following is an example of /etc/nsswitch.conf.

#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Legal entries are:
#
#       nisplus or nis+         Use NIS+ (NIS version 3)
#       nis or yp               Use NIS (NIS version 2), also called YP
#       dns                     Use DNS (Domain Name Service)
#       files                   Use the local files
#       db                      Use the local database (.db) files
#       compat                  Use NIS on compat mode
#       hesiod                  Use Hesiod for user lookups
#       [NOTFOUND=return]       Stop searching if not found so far
#

# To use db, put the "db" in front of "files" for entries you want to be
# looked up first in the databases
#
# Example:
#passwd:    db files nisplus nis
#shadow:    db files nisplus nis
#group:     db files nisplus nis

passwd:     files
shadow:     files
group:      files

#hosts:     db files nisplus nis dns
hosts:      files dns
.....

(4) Configuring IPv4 routing daemon

GLS allows you to select zebra(ripd) for IPv4 routing daemon. If you are using the system as an IPv4 router, you must configure a routing daemon. Configuration procedure is as follows.

zebra(ripd) configuration

1) When defining static route information, first startup zebra and login to zebra using telnet then proceed with the configuration. For details on zebra configuration, refer to zebra(8) manual.

# telnet localhost zebra

2) Configure zebra to startup during system startup (when run level is 2, 3, or 5).

RHEL:
# chkconfig --level 235 zebra on

SLES:
# chkconfig --set zebra 235

3) Verify zebra is configured to startup on run level 2, 3, 5.

RHEL:
# chkconfig --list zebra
zebra          0:off   1:off   2:on   3:on    4:off    5:on    6:off

SLES:
# chkconfig --list zebra
zebra                     0:off  1:off  2:on   3:on   4:off  5:on   6:off

(5) IPv6 RA daemon configuration

GLS supports radvd(router advertisement daemon for IPv6) for RA (router advertisement) daemon. To use IPv6(dual) on Fast switching mode, you must start RA daemon on the host running GLS in order to transmit RA from virtual interfaces. Other than this purpose, RA daemon is not necessary. The following describes configuration procedure.

radvd configuration

1) Define the configuration in /etc/radvd.conf

When transmitting network information (network fec0:1::, prefix length 64) from sha0 with RA(router advertisement), define the configuration description as shown below.
In addition, the difference in the version of radvd needs to define the following kernel parameters.
For details on radvd configuration, refer to radvd manual.

RHEL (/etc/sysctl.conf):

net.ipv6.conf.all.forwarding=1

SLES (/etc/sysconfig/sysctl):

IPV6_FORWARD=yes

/etc/radvd.conf

interface sha0
{
        AdvSendAdvert on;        # Sending router advertisements
        MinRtrAdvInterval 3;
        MaxRtrAdvInterval 10;       
        prefix fec0:1::0/64      # Sending Prefix fec0:1::0/64 from sha0
        {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
};

2) Configure radvd to startup during system startup (when run level is 2, 3, or 5).

RHEL:
# chkconfig --level 235 radvd on

SLES:
# chkconfig --set radvd 235

3) Verify radvd is configured to startup on run level 2, 3, 5.

RHEL:
# chkconfig --list radvd
radvd            0:off   1:off   2:on   3:on    4:off    5:on    6:off

SLES:
# chkconfig --list radvd
radvd                     0:off  1:off  2:on   3:on   4:off  5:on   6:off

(6) route configuration

Static route configuration on the environment not using IPv4 routing daemon is described below.

Default gateway configuration

Define the default gateway address (GATEWAY). Note that default gateway device (GATEWAYDEV) can not be configured to a physical interface NIC switching mode bundles.

RHEL (/etc/sysconfig/network-scripts/ifcfg-ethX or /etc/sysconfig/network):

GATEWAY=192.168.1.254

SLES (/etc/sysconfig/network/routes):

default 192.168.1.254 - -

Static route configuration

To configure static route on a routing table, define the configuration on /etc/sysconfig/network-scripts/route-ethX.

RHEL (/etc/sysconfig/network-scripts/route-ethX):

GATEWAY0=192.168.40.10
NETMASK0=255.255.255.0
ADDRESS0=192.168.100.0

SLES (/etc/sysconfig/network/routes):

192.168.100.0 192.168.40.10 255.255.255.0 ethX

(7) IPv6 module configuration

Supporting IPv6(dual) on GLS Fast switching mode or NIC switching mode, it is required to load IPv6 module on Linux. See the following for configuring IPv6 module.

Loading ipv6 module (RHEL only)

1) Configure /etc/sysconfig/network to use IPv6. Ignore the tunneling configuration, since GLS does not support tunneling feature.

NETWORKING_IPV6=yes
IPV6TO4INIT=no

2) Reboot the system.

# /sbin/shutdown -r now

3) Verify IPv6 module is loaded.

# lsmod | grep ipv6
ipv6                  662756  10

(8) Tagged VLAN configuration

To use tagged VLAN interfaces in Fast switching mode or NIC switching mode, set up the tagged VLANs as follows:

When the system is RHEL (RHEL4 only):

1) Configure /etc/sysconfig/network to use tagged VLAN.

VLAN=yes

2) Reboot the system.

# /sbin/shutdown -r now

3) Verify tagged VLAN module is loaded.

# lsmod | grep 8021q
8021q                  18760   1

When the system is SLES (SLES9 only):

1) Copy the script file.

# cd /etc/opt/FJSVhanet/script/system
# cp -p rc_event.slesvlan.sam rc_event

2) Reboot the system.

# /sbin/shutdown -r now

The "-p" option of the "cp" command is required to take over permission of the script file. To copy the script file, be sure to use the "-p" option. You do not need to edit the "/etc/opt/FJSVhanet/script/system/rc_event" script file copied.

Also, to cancel the tagged VLAN interfaces, delete the script file using the following steps:

1) Delete the script file.

# cd /etc/opt/FJSVhanet/script/system
# rm rc_event

2) Reboot the system.

# /sbin/shutdown -r now

If you are using SLES9 for the operating system, it is necessary to create the "/etc/opt/FJSVhanet/script/conf/vlan_if.conf" setup file to use tagged VLAN interfaces. If you have deleted the script file, it is not necessary to delete the "/etc/opt/FJSVhanet/script/conf/vlan_if.conf" setup file. Deleting the above script file will disable the setup file.

The "/etc/opt/FJSVhanet/script/conf/vlan_if.conf" setup file and the "/etc/opt/FJSVhanet/script/system/rc_event" script file are not backed up or restored. They are different than the GLS configuration file that can be backed up and restored with the "hanetbackup" and "hanetrestore" commands. You need to back up or restore the setup and script files if necessary.

(9) Network hotplug configuration (SLES9 only)

To invalidate the network hotplug function of the interface (sha*,eth*.*), edit /etc/hotplug/net.agent file.

. . . . . . . .

case $ACTION in

    add|register)
        case $INTERFACE in

            # interfaces that are registered after being "up" (?)
            #ppp*|ippp*|isdn*|plip*|lo*|irda*|dummy*|ipsec*|tun*|tap*|bond*|vlan*)
            ppp*|ippp*|isdn*|plip*|lo*|irda*|dummy*|ipsec*|tun*|tap*|bond*|vlan*|sha*|eth*.*)
                debug_mesg assuming $INTERFACE is already up
                exit 0
                ;;

. . . . . . . .

    remove|unregister)
        case $INTERFACE in

            # interfaces that are registered after being "up" (?)
            #ppp*|ippp*|isdn*|plip*|lo*|irda*|dummy*|ipsec*|tun*|tap*|bond*|vlan*)
            ppp*|ippp*|isdn*|plip*|lo*|irda*|dummy*|ipsec*|tun*|tap*|bond*|vlan*|sha*|eth*.*)
                debug_mesg assuming $INTERFACE is already up
                exit 0
                ;;

. . . . . . . .

• Communication become impossible when this setting is not configured. Make sure to configure it when you use GLS.
• Configure the above-mentioned setting again if the OS's patch or hotplug's patch rewrites /etc/hotplug/net.agent file.
• Restore /etc/hotplug/net.agent file, if you uninstall GLS to stop the operation.