I was involved for a project for one of our customers that involved configuring remote replication between two Sun 6180 storage boxes, one located in Cairo and the other located in Port Said (which is 200km away).
We had two brocade 7800 SAN switches. The two sites are connected via a WAN link.
The WAN link (if you are not familiar with it), is usually a long distance link between two routers. These routers are connected via a medium that supports the required distance; for example an E1 link, or a bundle of E1’s to increase bandwidth. The result is two network subnets connected via the routers. The diagram below illustrates how network clients on both sides of the routers can reach each other via regular network routing.
FC over IP = FCIP
SAN replication uses the FC protocol. To allow the FC protocol to travel over the WAN link we will use the FCIP protocol. FCIP will encapsulate the FC packets into IP packets which can then be routed over the long distance network between the two sites.
Sending large amounts of data over a WAN link usually includes sizing. The WAN connection must have enough bandwidth to accommodate the traffic. Proper sizing has to be performed such that the link can complete the data transfer complete in a timely fashion according to the environment requirements.
Connections & Configuration
Connecting the 6180’s to the SAN switch is pretty straight forward, just connect the replication port of the 6180 to one of the FC ports on the SAN switch.
Configuring the FCIP tunnel is where all the work is. The entire FCIP configuration has to be performed on both SAN switches via command line. The configuration should match on both sides, otherwise it will not work.
First we need to configure the network ports on the SAN switch, this will just define the IP addresses of the FC network ports. Don’t get this mixed up with the management port and IP, this is a different port and different configuration.
Once the network ports have been configured we then need to define the network routes. This way the packets know which router to use to reach the remote subnet.
Finally, we configure the tunnel. The tunnel defines the source and target IPs in addition to any other options required for the connection, such as IPSEC, fastwrite, compression, etc… On the brocade 7800 switch, the FCIP tunnel is designated a virtual port number. This port is not a physical port, it is a virtual port used to identify and configure the tunnel. The first FCIP port on the 7800 is port 16, which we will be using.
Connection Diagram
The following diagram illustrates the target configuration;
Command Syntax
portcfg ipif <network_interface> create <IP_Address> <Subnetmask> <MTU> portcfg iproute <network_interface> create <host_subnet> <Subnetmask> <Gateway> portcfg fciptunnel 16 create <destination_IP> <source_IP> <speed_in_bytes>
Step-by-Step Configuration
To configure the tunnel as shown in the configuration diagram, log on to the Main Switch and run the following commands;
This following command configures the physical ge0 port with;
IP: 192.168.1.100
subnet: 255.255.255.0
MTU: 1500
SAN-MAIN:admin> portcfg ipif ge0 create 192.168.1.100 255.255.255.0 1500
Now configure port ge0 such that;
To reach 192.168.2.0/24 (the remote SAN switch FC network) use the router 192.168.1.1
SAN-MAIN:admin> portcfg iproute ge0 create 192.168.2.0 255.255.255.0 192.168.1.1
Finally, create the FCIP tunnel 16 with target 192.168.2.100 and source 192.168.1.100
SAN-MAIN:admin> portcfg fciptunnel 16 create 192.168.2.100 192.168.1.100 10000
By running the above three commands, the configuration on the MAIN SAN switch is done. We now need to configure the other side of the tunnel on the DR switch;
SAN-DR:admin> portcfg ipif ge0 create 192.168.2.100 255.255.255.0 1500 SAN-DR:admin> portcfg iproute ge0 create 192.168.1.0 255.255.255.0 192.168.2.1 SAN-DR:admin> portcfg fciptunnel 16 create 192.168.1.100 192.168.2.100 10000
The configuration is now complete, the two 6180’s should be able to communicate as if they were connected to the same SAN switch.
Finally, additional configuration such as fast write, compression and IPSEC security can be configured by modifying the tunnel configuration. Note that this will disrupt the FCIP traffic for a few seconds while the reconfiguration completes. Again, this has to be done on both SAN switches otherwise the tunnel will not come up.
Syntax for the options we used is as follows;
portcfg fciptunnel <fcip_port> modify -f 1 -c 1 -i 1 -K <32-byte key>
To apply this to our example, the 32-byte key is in hexadecimal, and can be anything as long as it is the same on both sides.
portcfg fciptunnel 16 modify -f 1 -c 1 -i 1 -K AABBCCDDEEFF112233445566778899AA
You can also limit the bandwidth used by the tunnel. In this example I am limiting the tunnel to use 10Mbps.
SAN-MAIN:admin> portCfg fciptunnel 16 modify -b 10000 -B 10000 !!!! WARNING !!!! Modify operation can disrupt the traffic on the fciptunnel specified for a brief period of time. This operation will bring the existing tunnel down (if tunnel is up) before applying new configuration. Continue with Modification (Y,y,N,n): [ n] y Circuit 16.0 modify: Operation Succeeded
Viewing Your Configuration
The following output show the commands and example output of a configured tunnel;
Viewing the ge interface configuration
SAN-DR:admin> portshow ipif ge0 Port: ge0 Interface IPv4 Address NetMask Effective MTU Flags -------------------------------------------------------------- 0 10.3.202.241 255.255.255.0 1500 U R M Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp SAN-DR:admin> portshow iproute ge0 Port: ge0 IP Address Mask Gateway Metric Flags ------------------------------------------------------------- 10.2.3.0 255.255.255.0 10.3.202.150 0 U G S 10.3.202.0 255.255.255.0 * 0 U C 10.3.202.150 255.255.255.255 * 0 U H L Flags: U=Usable G=Gateway H=Host C=Created(Interface) S=Static L=LinkLayer(Arp)
Viewing the FCIP tunnel configuration
SAN-MAIN:admin> portshow fciptunnel all ------------------------------------------------------------------------------- Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met ------------------------------------------------------------------------------- 16 - Up cf--- 11m13s 0.00 0.00 4 - - ------------------------------------------------------------------------------- Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf circuit: s=sack SAN-MAIN:admin> portshow fciptunnel all -c ------------------------------------------------------------------------------- Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met ------------------------------------------------------------------------------- 16 - Up cf--- 11m27s 0.00 0.00 4 - - 16 0 ge0 Up ----s 11m27s 0.00 0.00 4 1000/1000 0 ------------------------------------------------------------------------------- Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf circuit: s=sack SAN-DR:admin> portshow fciptunnel 16 ------------------------------------------- Tunnel ID: 16 Tunnel Description: Admin Status: Enabled Oper Status: Up Compression: On (Standard) Fastwrite: On Tape Acceleration: Off TPerf Option: Off IPSec: Enabled Remote WWN: Not Configured Local WWN: 10:00:00:05:33:d1:ac:c2 Peer WWN: 10:00:00:05:33:a3:73:fa Circuit Count: 1 Flags: 0x00000000 FICON: Off SAN-DR:admin> portshow fciptunnel 16 -c ------------------------------------------- Tunnel ID: 16 Tunnel Description: Admin Status: Enabled Oper Status: Up Compression: On (Standard) Fastwrite: On Tape Acceleration: Off TPerf Option: Off IPSec: Enabled Remote WWN: Not Configured Local WWN: 10:00:00:05:33:d1:ac:c2 Peer WWN: 10:00:00:05:33:a3:73:fa Circuit Count: 1 Flags: 0x00000000 FICON: Off ------------------------------------------- Circuit ID: 16.0 Circuit Num: 0 Admin Status: Enabled Oper Status: Up Remote IP: 10.2.3.241 Local IP: 10.3.202.241 Metric: 0 Min Comm Rt: 10000 Max Comm Rt: 10000 SACK: On Min Retrans Time: 100 Max Retransmits: 8 Keepalive Timeout: 10000 Path MTU Disc: 0 VLAN ID: (Not Configured) L2CoS: (VLAN Not Configured) DSCP: F: 0 H: 0 M: 0 L: 0 Flags: 0x00000000
Deleting FCIP tunnel
If you want to change the IP addresses of the ge interfaces you will need to delete the entire configuration and start over.
The following sequence of commands will delete the FCPIP tunnel, the route and IP address assigned to the FC network interface. The tunnel is FC IP is dependent on the route, and the route is dependent on the tunnel. The commands are arranged by dependency, so if you want to delete a components, you must also delete all the components above it.
portcfg fciptunnel <fcip_port_number> delete portcfg iproute <fc_network_interface> delete <host_subnet> <subnet_mask> portcfg ipif <fc_network_interface> delete <fc_ip> <subnetmas>
Waseem Fayed's blog 
Hello everyone, it’s my first visit at this site,
and article is really fruitful designed for me, keep up posting these content.
LikeLike
Thanks for the great post. I have a question: How do I make the two switches act as one switch? For example, how do I manage the ports on one SAN switch using the other switch after creating an FCIP tunnel between the two?
LikeLike
I did want to configure zoning between the two switches in this solution but was not able to. I don’t remember if the fabric actually joined or not, I’ll update again soon with notes on this though.
But your question made me curious to look into it again so quick search and I found this blog post. The author says that the fabric should merge, and that if different zoning exists then zone conflicts will occur. Check it out and let me know how it works out for you.
LikeLike
Hello,
Could you show us which would be the trace from 192.168.1.100 to 192.168.1.200 ??
portcmd –tracerout ge0 -s ipsource -d ipdestination
LikeLike
I am so sorry, I never saw this comment, I know this is ancient, but I thought I should answer anyways.
I don’t have access to this environment anymore, nor did I still have it back when the comment was made, so I am really sorry I cannot provide the traceroute output.
LikeLike