LANE Design Recommendations                   Printable Pdf

Document ID: 10459

Introduction

This document provides practical LAN emulation (LANE) network design guidelines. These guidelines will assist you in the design of high performance, scalable, and high availability LANE networks. While this document focuses on Cisco equipment, the same concepts can be applied when integrating third party products.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

Readers of this document should be knowledgeable of basic operations and configurations of LANE networks.

Components Used

This document focuses on Ethernet LANE configurations. The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Understanding the Server Requirements

The various LANE servers and their requirements are presented below. The LAN Emulation Configuration Server (LECS)
When the LECs join the emulated LAN (ELAN), they each contact the LECS individually. However, when the LANE network undergoes a disaster (for example, when the primary LECS fails), all clients go down.
Note: The exception to this is when Fast Simple Server Redundancy Protocol (FSSRP) is used.
Since all LECs go down at the same time, they will all contact the backup LECS at the same time. Therefore, for hosting the LECS, you need a device that:

• can handle a sudden burst of traffic directed at the process level.
• can accept almost all incoming call setups from the LECs simultaneously.
• is known for its stability. If the LECS goes down, the whole network goes down (again, with the exception of FSSRP). Therefore, putting the LECS on a device running an experimental software version is not recommended.

The LAN Emulation Server (LES)

You cannot implement a simple hardware cell replication from control direct to control distribute since some frames (such as the join requests) have to be analysed by the LES process. Therefore, a device that can act as a good LES is a device that:

• has a powerful CPU and can accept a lot of call setups in a short amount of time. This is necessary when many clients join the ELAN at the same time, but less vital than for the LECS, since only the LECs in the ELAN have to join.
• has strong segmentation and reassembly (SAR) hardware support. As all incoming cells have to be reassembled into frames, if a lot of join requests arrive at the same time, they will have to be reassembled very quickly.

Remember that in Cisco's implementation, LES and Broadcast and Unknown Server (BUS) processes are combined (that is, you can't put the LES for ELAN-1 on one device, and the BUS for ELAN-1 on another device).
Another thing to keep in mind is possible pre-emptive behaviour. If pre-emption is enabled, the LES/BUS with the highest priority (according to the LANE database) will always take over primary LES/BUS duty. In other words, if the primary LES/BUS fails, all LECs of the ELAN will go down and reconnect to the backup LES/BUS. If pre-emptivity is configured, should the primary LES/BUS go up again, all LECs will go down once more and reconnect to the LES/BUS with the highest priority. In LANE Module Software release 3.2.8 and later, and Cisco IOS® Software release 11.3(4) and later, the pre-emptivity feature can be turned on and off. The pre-emptivity feature can be configured as described in Configuring LAN Emulation documentation. The Broadcast and Unknown Server

• has hardware support for the frame copy from the incoming multicast sent to the outgoing multicast forward. If you have "smart" hardware, this copy operation can be done before reassembly. This means that cells incoming on the multicast send are forwarded on the multicast forward. This saves one segmentation and reassembly per frame.
• requires a strong CPU if there is no hardware support for the BUS.
• must be able to process a lot of call setups at the same time, but with a lower limit than the LECS.

Catalyst 6K LANE/MPOA Module (OC-12)

600

Catalyst 5K LANE/MPOA Module (OC-12)

600

Catalyst 5K LANE/MPOA Module (OC-3)

166

Catalyst 5K LANE Module (OC-3)

122

RSP4 - VIP-2-50+PA-A1

92

RSP4 - VIP-2-500+PA-A3

84

RSP4 - VIP-2-40+PA-A3

78

RSP4 - VIP-2-40+PA-A1

77

4700

40

LS1010

30

Understanding Cisco Device Capabilities

This section covers the capabilities of the most common Cisco devices used to run LEC, LECS, LES, and BUS. These devices are the Cisco LANE modules, the Lightstream 1010, the Catalyst 8510MSR and 8540MSR, and the 7500/RSP. Their capabilities are compared against the requirements listed above. LANE Modules

Segmentation and reassembly is performed by the hardware. The SAR chip is somewhat intelligent, and can directly forward the reassembled frames to the frame bus of the catalyst (the catalyst backplane). For control frames, the SAR chip can forward the frames to the CPU of the LANE module. A control frame is any frame that has to be analysed (for example, Interim Local Management Interface (ILMI), signalling, and frames destined to the LES), coming to the LANE module via a specified VC.

The SAR chip can also re−direct cells coming on the multicast send to the multicast forward (that is, multicast, broadcast, and unknown cells coming from the LECs). The cells are not reassembled into frames. Its ease of implementation results in very good BUS performance.

Once a "data direct" and an entry in the content−addressable memory (CAM) table have been created, the reassembled frames are sent directly to the frame BUS and tagged with the correct virtual LAN (VLAN) ID. A LANE module makes a very good LEC because once the "data direct" has been established, the CPU is no longer involved.

LightStream 1010 and Catalyst 8510MSR8540MSR

• The BUS performance is around 50Kpps for 64 byte packets, far below any LANE module. This is because there is no hardware acceleration for the BUS.
• If the 8540MSR is used with ATM and Ethernet cards, the CPU may be used primarily to talk with Ethernet line cards. In this case, the 8540MSR's CPU should not be used as an LES.

Sample Designs

LANE is mainly used in large and critical networks. As such, redundancy is mandatory. Simple Server Redundancy Protocol (SSRP) is the most widely used redundancy protocol. If the software is recent, FSSRP is the preferred protocol (refer to Guideline #11).

Let's assume we have a fairly large network, for example 100 VLANs/ELANs and 100 catalysts, each with a dual uplink LANE module. This means that on each LANE module, we might need one LEC per ELAN, in this case 10,000 LECs. In addition, we assume that IP is used, and that the design includes a safe Class C (254 IP host addresses, 254 MAC addresses) per VLAN. Design 1: Simple, but to be avoided...

When creating the LECs on the LANE module, all LECs go up as soon as they are configured. During operation, the LES process might become overloaded, and the LANE module will run out of memory. Design 2 below solves both these problems.

The main issue in this network is when a major problem occurs. Assume that the LANE module hosting the LECS, LES, or BUS becomes unreachable. This might happen, for example, if the LANE module of catalyst 1 becomes faulty. You can see redundancy taking place, but the redundancy time (that is, the time between the primary LECS, LES, or BUS failure, and the last LEC becoming operational again) might last up to two hours! A good design might bring this number down to some dozens of seconds, or a few minutes in large networks.

The problem lies in the signalling involved with the LECs joining the ELAN. If the LECS must be contacted by every LEC, it will receive 10,000 call setups (100 LANE modules with 100 LECs on each) almost simultaneously. The LANE module is designed to bridge efficiently between the frame bus and the cell link, but not to handle a lot of call setups per second. The CPU of the LANE module is not powerful enough to handle this volume of call setups. The following output illustrates redundancy time in a LANE network with appproximately 1600 LECs (only part of the show processes cpu command is displayed):

ATM#show processes cpu CPU utilization for five seconds: 99%/0%; one minute: 98%; five minutes: 69% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process <snip> 7 13396 207 64714 16.55% 10.85% 3.77% 0 ATM ILMI Input 8 13600 188 72340 13.45% 10.54% 3.72% 0 ILMI Process <snip> 35 107892 553 195103 68.94% 55.34% 26.72% 0 ATMSIG Input 36 34408 1125 30584 12.29% 9.45% 6.63% 0 ATMSIG Output <snip>

As you can see, the LANE module is over−utilized due to the incoming signalling activity. What accounts for the two hour redundancy time? The answer lies in the notion of time−out. The signalling specifications clearly mention that if the device does not get a "connect" message back after a specified amount of time when a "call setup" is sent, it must start over. LANE specifications require that the LEC must go back to its initial state, and start all over again. This means that if a LEC is able to contact the LECS and get connected to it, its call setup to the LES might time−out, and it goes back to its initial state of trying to contact the LECS! This can also happen with connections from the LES, and from/to the BUS.

Based on the explanations above, here are some basic design recommendations:

• Try to spread the LES/BUS for the different ELANs on the various devices that can implement it efficiently. Ideally, one primary LES/BUS on each LANE module, with the next ones backing up the first. In practice, this would generate a very long LECS database. Experience shows that 10 LES/BUS servers per LANE module seem to be a safe number.
• Try not to put the LECS in the same location as other important LES/BUS servers. Also try to put the LECS on a device with sufficient CPU power so it can handle the signalling information efficiently. The LECS should be on a router (the Cisco 7200 or 7500 is recommended, ideally without LES/BUS), or on an ATM switch.
• Assuming IP and one Class C range is used for each VLAN, approximately 250 MAC addresses are a good number for the LES duty. For 10 LES on a LANE module, this means the CPU of one LANE module for a maximum of 2500 MAC addresses. Bear in mind that there are no fixed and official numbers, but this is a safe and conservative estimate. On the other hand, 200 LES/BUS on a LANE module, with each ELAN containing 1000 end stations, are safe as long as the station remains practically idle (refer to Guideline #3 for more details).

Design 2: More complex, but safer and more efficient...

Guidelines

The guidelines presented below are practical recommendations only, based on the deployment of production LANE networks. While examples of successful networks that exceed the recommendations do exist, a thorough understanding of how they will affect the network is required before exceeding these guidelines.

Guideline #1

If you plan to use Hot Standby Router Protocol (HSRP) over LANE, make sure that you upgrade to a recent release and have read Implementing HSRP over LANE.

Guideline #2

The LANE BUS is responsible for forwarding all broadcast, multicast, and unknown destination unicast frames received from members of an ELAN, to all members of the ELAN. Since LANE uses ATM adaptation layer 5 (AAL5) which does not allow the interleaving of cells from different protocol data units (PDUs), the BUS must serialize the frames prior to forwarding. This requires the BUS to reassemble the received frames, segment each frame one by one, and forward the cells. The requirement to reassemble and segment each frame significantly limits the forwarding throughput of the BUS, which considerably influences the scalability of an ELAN. The proliferation of IP multicast applications further intensifies this task. Remember that only LANE modules can receive the cells on multicast send and forward them on the multicast forward. This is done without reassembly.

Guideline #3

We stated above that 10 LES/BUS with each ELAN corresponding to a Class C IP network (approximately 250 users) are safe and conservative; however, successful LANE networks with 10−60 LES/BUS pairs per module do exist. This depends slightly on the module, but checking the design will always involve checking the CPU utilization (using the show processes cpu command), and the lowest available memory (using the show memory command). This should, of course, be carried out during peak network utilization, as the LES' overall CPU utilization is directly related to the LE_ARP process.

In a LANE environment, it is common to see the LES/BUS pairs located on a single device supporting the entire LANE network. Not only does this represent a single point of failure, but it limits the BUS performance within each ELAN.

Distributing LANE services across multiple platforms provides greater scalability in multi−ELAN environments, as well as higher system availability and increased aggregate BUS performance (for example, the aggregate BUS performance in the network increases as more devices and interfaces are configured for BUS support). For maximum BUS capacity from a design perspective, the Catalyst 5000 and 6000 ATM modules can be dedicated to LES and BUS services.

Knowing the capacity of the BUS, and estimating the amount of broadcast or multicast traffic expected in each ELAN, you can calculate the number of LES/BUS pairs that can be applied to a given interface. You can also measure the capacity of the BUS.

Estimating the amount of broadcast or multicast traffic for each ELAN is, however, more challenging. One method for estimating the amount of broadcast or multicast traffic for each ELAN is to measure this traffic on the existing network. A network analyzer or Remote Monitoring (RMON) probe device can be inserted into the existing LAN to measure the amount of broadcast and multicast traffic. Another way is to query the "ifOutMulticastPkts" and "ifOutBroadcastPkts" mib objects. Check first whether or not they are supported on your IOS/platform.

Alternatively, you can, to some extent, calculate the amount of broadcast or multicast traffic by calculating the bandwidth consumed by routing protocol broadcasts, for example. For Internetwork Packet Exchange (IPX), Routing Information Protocol (RIP), and Service Advertising Protocol (SAP), bandwidth consumption can be accurately determined if the number of IPX routes and SAPs are known. The same is true for IP and the particular routing protocol being used.

Additional BUS capacity headroom should be reserved for:

• Supporting unicast traffic while a data direct VC is being established and until a flush packet is acknowledged on the receiving LEC.
• On−demand IP multicast applications that are utilized at various times of the day (these should be considered in the overall multicast volume).
• Additional routing traffic when a protocol is running and in a convergence state (that is, link−state advertisements (LSAs) exchanged during an Open Shortest Path First (OSPF) topology change).
• High volumes of Address Resolution Protocol (ARP) requests, specifically in the morning when workstations first log into the LAN and network servers.

Using whatever method is available, the goal is to have an accurate depiction of the amount of broadcast and multicast traffic that will exist on each ELAN. Unfortunately, this information is seldom available to the network designer for various reasons. When faced with this situation, some general conservative guidelines can be used. As a recommendation, a typical network with 250 users per ELAN, running the more common applications, should be allocated at least 10 Kpps of BUS capacity. Table 1 illustrates the maximum recommended number of LES/BUS pairs per interface.

These numbers should be used in conjunction with Guideline #4, which limits to 250 the number of LECs serviced by all the LES/BUS pairs configured on an interface. Also, these numbers should be adjusted according to the actual number of users in each ELAN, while paying particular attention to any broadcast or multicast applications that will be run on the ELAN.

Guideline #4

Limit the total number of LECs serviced by the LES/BUS pair to a maximum of 250. During initialization, and following a network failure, in order for the LANE clients to join their ELAN, they must establish multiple connections and make requests to their LANE service components. Since the devices supporting the LANE services possess a finite rate at which they can process the connections and requests, it is recommended that the LES/BUS pairs configured on an interface service a maximum of 250 LANE clients. For example, an interface may be configured with 10 LES/BUS pairs, each servicing 25 LECs for a total of 250 LECs being serviced by the interface. This will ensure timely initialization and failure recovery.

Guideline #5

Put the LES/BUS for a given ELAN in close proximity to any major broadcast or multicast traffic source.

In a LANE environment, specifically where multicast applications are in use (that is, IP/TV), it is good design practice to place the BUS as close to the known multicast source as possible. Since multicast traffic must first be sent to the BUS, which in turn forwards the traffic to all clients, situating the BUS in close proximity to the multicast source saves the traffic from crossing the ATM backbone twice.

This allows the LANE network to scale to a greater magnitude. In addition, the BUS should not be located on the same interface as the LEC supporting the multicast source, since the multicast traffic would cross the transmit link twice.

Exercise caution if you are considering LANE as the networking technology to support a multicast environment. While LANE does support multicast traffic, it does so rather inefficiently. LANE simply floods the multicast traffic to all clients in the ELAN regardless of whether or not they are part of the multicast group. Excess multicast traffic can significantly degrade the performance of workstations (as discussed in Guideline #6), while the flooding behavior wastes backbone bandwidth.

Guideline #6

Limit the number of end systems in a given ELAN to 500 or less, should the network carry only IP packets. The table 2 below gives some basic recommendation based on the quantity of broadcast generated by the protocol. Again, in case you are not fully sure what protocols will be needed, keep in mind the 250 end station recommendation we gave in the past.

By definition, an ELAN represents a broadcast domain. Therefore, within an ELAN, all broadcast and multicast packets are flooded to all members of the ELAN. Workstations must process each received broadcast and multicast packet to determine if it is of interest. The processing of "uninteresting" broadcast packets wastes workstation CPU cycles. When the level of broadcast activity becomes high (relative to the processing capacity of the workstations), they can be seriously impacted and prevented from performing their intended tasks.

The number of end systems, applications, and the protocol(s) in use determine the level of broadcasting within an ELAN. Tests have illustrated that, in the absence of broadcast intensive applications, the number of end systems that can safely be configured in a single ELAN ranges from 200 to 500 depending upon the protocol mix.

Table 2: Maximum Number of Recommended End Systems per ELAN based on Protocol Mix

IP

500

IPX

300

AppleTalk

200

Mixed

200

Guideline #7

Calculate the network VC usage to ensure it is within the capacity of the ATM devices.

VC Usage

ATM switches and edge devices support a limited number of VCs. When designing ATM networks, it is important to ensure that the VC capacity of the equipment is not exceeded. This is particularly important in LANE networks, since LANE is not noted for its VC efficiency. During the network design phase, you should calculate the anticipated VC usage for the backbone, as well as for each individual edge device. The VC usage of the backbone corresponds to the total number of VCs expected in the network. This quantity should be compared to the number of VCs supported by the ATM switches.

Since not all VCs cross a given switch, this number serves as an upper bound. The actual topology of the backbone and traffic patterns must be considered, in relation to the total number of VCs, to determine if the VC capacity of the ATM switches will be exceeded.

Similarly, the VC usage for each edge device should be calculated. This relates to the number of VCs that will terminate on a given interface of an edge device. This number must then be compared to the VC capacity of the interface.

The following formulas can be used in calculating the network's VC usage. These formulas assume the use of Cisco LANE services and clients, and apply to SSRP and FSSRP. When present, differences in VC usage between the two protocols are indicated.

Backbone VC Usage   a. LEC-LANE Service VCs:

    SSRP: 4 (#LEC_per_ELAN)(#ELAN)

    FSSRP: 4 (#LEC_per_ELAN)(#LES/BUS_per_ELAN)(#ELAN)

  b. LECS-LES Control VCs:

    (#LES/BUS_per_ELAN)(#ELAN)

  c. LECS-LECS Control VCs:

    (#LECS)(#LECS - 1) / 2

  d. LEC-LEC Data Direct VCs:

    If mesh_factor < 1.0:

    (#LEC_per_ELAN) [(#LEC_per_ELAN)(mesh_factor)/2](#ELAN)

    If mesh_factor = 1.0: (recommended in most designs)

    (#LEC_per_ELAN) [((#LEC_per_ELAN) - 1)/2](#ELAN)

    where:

Edge Device Interface VC Usage

  a. LEC-LANE Service VCs:

    SSRP: (#active_LES/BUS_on_interface) (2 * #LEC_per_ELAN + 2)

    FSSRP: (#LES/BUS_on_interface) (2 * #LEC_per_ELAN + 2)

  b. LECS-LES Control VC's:

    (#LES/BUS_on_interface)

  c. LECS-LECS Control VCs

    (#LECS - 1)

  d. LEC-LEC Data Direct VCs:

    (#LEC)[(#LEC_per_ELAN)(#LEC_per_ELAN)(mesh_factor)/2]

    Interface VC usage = a + b + c + d

Once you have calculated the VC usage, compare the results to the VC capacity of the relevant devices using Table 3.

Table 3: Inter-ELAN Routing - VC Capacity for Various Cisco Devices

Catalyst 8540 MSR

256k

Catalyst 8510 MSR/LS1010

16 MB dynamic random-access memory (DRAM) = 4k

32 MB DRAM = 16k

64 MB DRAM = 32k

Cisco 7500/7200 ATM Deluxe

4k

Cisco 7500/7200 ATM Lite

2k

Catalyst 6K - LANE/MPOA OC-12

4k

Catalyst 5K - LANE/MPOA OC-12

4k

Catalyst 5K - LANE/MPOA OC-3

4k

Catalyst 5K - LANE OC-3

4k

Catalyst 2900 XL - LANE OC-3

1k

Guideline #8

If you want to link different campus ATM networks with permanent virtual paths (PVPs), always "route" between the sites rather than allow native ELANs to span different campus ATM networks.

Guideline #9

Assess the router capacity needed by estimating the amount of inter-ELAN routing required.

The amount of routing capacity required in a given LANE network varies widely. Therefore, the amount of routing capacity must be estimated during the network design process. After determining the capacity required, the number of routers and router interfaces needed can be determined using the following forwarding throughput table:

Table 4: Inter-ELAN Routing Capacity for Various Cisco Devices

RSP4/VIP2-50 ATM PA-A3

118

101

RSP4/VIP2-50 ATM PA-A1

91

91

RSP4/VIP2-40 ATM PA-A3

83

60

RSP4/VIP2-40 ATM PA-A1

66

66

While the "one-armed" router configuration is popular in LANE designs, typically this does not provide adequate routing capacity. Instead, multiple interfaces and/or multiple routers are required. The CEF forwarding rates listed in the table above assume a one-armed router configuration. To reach these rates, the router's central processor is pushed to nearly 100% utilization. By contrast, the distributed forwarding rates are achieved using the processor residing on the Versatile Interface Processor (VIP), with essentially no impact on the centralized router processor. As a result, multiple ATM interfaces can be installed in the router leading to a much higher aggregate throughput.

Guideline #10

Provide dual-home ATM edge devices to at least two different ATM switches for redundancy.

In a LANE network, the ATM switch supporting the edge devices can be a single point of failure for connectivity to the backbone. The Catalysts 6K and 5K provide OC-12/OC-3 dual-physical sublayer (PHY) uplink modules for redundant connectivity to downstream ATM switches. The dual-homed LANE modules provide a "Fiber Distributed Data Interface (FDDI)-like" dual-PHY feature. This dual-PHY uplink module provides a primary and secondary ATM interface. If the primary interface loses link connectivity to the ATM switch, the module will automatically switch the connection over to the secondary interface.

It is highly recommended that the network designer take advantage of the dual-PHY interfaces on the LANE modules and provide dual-homed uplinks to two different ATM switches in the core. This will protect the edge devices from the failure of a single ATM switch.

Guideline #11

Use FSSRP unless the VC budget has constraints.

Since the various LANE service components are single points of failure in a LANE network, production networks should be designed with redundancy. Cisco supports two redundancy schemes for LANE services: Simple Server Redundancy Protocol (SSRP) and Fast SSRP (FSSRP).

FSSRP is the recommended redundancy scheme in most cases. It provides almost immediate failover with no loss of data, even in large networks. On the other hand, SSRP will result in loss during a failover, and recovery times may be substantial (sometimes minutes) in large networks.

There exists one situation where SSRP is recommended over FSSRP: when the network is VC-constrained. In contrast to SSRP, FSSRP LECs maintain backup connections to the redundant LES/BUS pairs. Up to three backup LES/BUS pairs can be configured compared to a total of four per ELAN. The VC usage increase that the network will experience under FSSRP can be calculated using the following formula:

Therefore, if the network reaches its VC capacity, SSRP is recommended over FSSRP. If you use FSSRP, then you should reduce the number of redundant LES/BUS components. In most circumstances, a total of two LES/BUS pairs per ELAN offers an acceptable balance between VC usage and eliminating single points of failure.

Related Information

• Implementing HSRP Over LANE
• Technical Support - Cisco Systems
• ATM LANE
• ATM General Information

All contents are Copyright © 1992-2005 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Updated: May 18, 2005 Document ID: 10459