Version: 1.0
Status: DRAFT: 8/16/99
Contact: Dr. John Philbrick

PURPOSE

To provide a general description of the process used for designing a campus infrastructure.

SCOPE

This guideline describes the general requirements for the campus hardware used to support the campus infrastructure and the campus connection to Net.Work.Virginia.

This guideline is applicable to all campuses within the Virginia Community College System.

ATM (Asynchronous Transfer Mode) - The transmission protocol used by Net.Work.Virginia.

Class C Address - An address space assigned by the Internet that supports a maximum of 254 IP hosts.

IP Telephone - An IP host running a voice application. These devices are connected to an IP subnet and require an IP address. They do not have direct connectivity to the PSTN, which must be provided by a PBX attached to the LAN and the PSTN.

Intranet - An enterprise wide network that may serve more than one location and that uses the Internet Protocol (IP) as its transmission protocol.

LANE (LAN Emulation) - An older protocol that provides IP support in an ATM environment (see MPOA).

LEC (Local Exchange Carrier) - The local telephone company that provides communications services to the site. Each NWV campus connection physically terminates on a LEC ATM switch.

Level 3 Switching - A router feature that allows the router to "route" the first few packets in a data stream, and then switch the remaining packets. Thus data is "switched" based on the destination address in the packet.

MPOA (MultiProtocol Over ATM) - The newest protocol that provides IP support in an ATM environment (see LANE).

NAT (Network Address Translation) - An Internet protocol that allows individual sites to support more IP hosts than the number of IP addresses assigned to it. This is done using special Internet addresses that have been reserved for this purpose. These special addresses are invalid in the Internet itself. The hosts using these addresses may communicate among themselves, but they cannot access the Internet directly. NAT allows them to access the Internet by using one of the valid Internet addresses assigned to the site. The number of concurrent Internet sessions is limited to the number of valid Internet addresses assigned to the site.

Non-Time Dependent Application - An application, such as file transfer that does not require the use of an end-to-end timing mechanism between the sending and receiving applications for proper operation. These applications typically use an error correction mechanism to ensure correct and complete transmission of the data. These mechanisms detect data that is in error and have it retransmitted until it is received without error.

NWV (Net.Work.Virginia) - The network used by VCCS and other Agencies of the Commonwealth to provide inter site communications services. Each campus has a connection that terminates on the campus ATM switch.

PBX (Private Branch Exchange) - A privately owned telephone network switch located on the customer premises.

Policy Based Management - A centralized mechanism that allows network managers to assign network privileges, including traffic priorities by user, a group of users, and/or an application. Network devices are automatically configured by the system to enforce the privileges.

PSTN (Public Switched Telephone Network) - The worldwide system of telephones.

RIP (Routing Information Protocol) - An Internet protocol used by routers to determine the best path between themselves and the destination network.

Spanning Tree Protocol - A mechanism for providing redundant physical connections between devices when their native transmission protocols don't support them. The protocol allows devices to negotiate routes between them so that there is one, and only one active connection between them at any given time. Since the devices are aware of the other routes, they can automatically switch to them if the primary route fails.

Time Dependent Application - An application, such as voice that requires the use of an end-to-end timing mechanism between the sending and receiving applications for proper operation. Since the retransmission of data would disrupt the timing relationship between the two applications, faulty data is typically ignored by these applications. This will create situations where the receiver may misinterpret the meaning of the data. In a voice application, if "may not" is sent and the packet containing the "not" is ignored, the receiver will hear only the "may".

Variable Length Subnet Masking - An IP protocol that allows any IP address space to be subdivided into multiple, smaller address spaces.

VC (Virtual Circuit) - The ATM mechanism that provides connectivity between two points. They may be Permanent Virtual Circuits (PVCs) or Switched Virtual Circuits (SVCs). PVCs are manually configured, requiring human intervention. SVCs are created dynamically, at the request of a network device.

Virtual IP Subnet - A virtual IP address space that can be created in a switched infrastructure that assigns IP hosts to a common IP address space, without requiring them to be on the same physical LAN segment.

VLAN (Virtual LAN) - The mechanism used in switched environments to implement virtual LAN segments (broadcast domains).

WAN (Wide Area Network) - A generic term referring to the communications network that provides inter site connectivity (for VCCS, Net.Work.Virginia).

GUIDELINE

This guideline serves as a reference for colleges planning a switched campus infrastructure. These guidelines define a set of minimum expectations for the campus infrastructure. They will be review and updated, as necessary, to reflect changes in technology and customer requirements.

Design Goal:

The end-to-end management of traffic flows such that application performance meets customer expectations.

Considerations and Constraints:

Campus traffic flows between end devices on a single campus. Thus, bandwidth management strategies are limited to the tools available in the campus equipment and the amount of bandwidth available in the campus infrastructure.

Off-campus traffic flows between a device on one VCCS campus and a device(s) on a different VCCS campus or a non-VCCS location. In this case, bandwidth management strategies are limited to the tools available in ALL of the networking devices, both campus and WAN that the traffic must pass through.

Environments Requiring Management:

The VCCS Intranet consists of three environments, all of which require effective bandwidth management if end-to-end control is to be achieved. As the source and/or the destination for all traffic, college infrastructures play a pivotal role in determining end-to-end application performance. The devices selected to implement the infrastructure determine which tools are available to manage the campus bandwidth. But, the campus infrastructure designer determines the amount of bandwidth actually available.

All off campus traffic must flow through the campus Net.Work.Virginia (NWV) connection. Given the bandwidth available in the campus connections, it is hard to view them as finite resources, but they are. The Local Exchange Carrier (LEC) switch serving the campus and the campus ATM switch used for WAN access are the devices that control traffic through the campus WAN connection.

Net.Work.Virginia ATM transport services support all off campus traffic flows through the LEC ATM switches, and possibly through the NWV backbone ATM switches. In the worst-case scenario, intercampus VCCS traffic passes through two LEC and two NWV backbone switches to get from one campus to another.

Bandwidth management tools available in each environment:

Each environment contains a set of "tools" that can be used to manage how bandwidth is allocated within the environment.

Net.Work.Virginia ATM transport services:

The selection of ATM Virtual Circuit (VC) type and ATM class of service parameters are the two primary mechanisms available to control the allocation of bandwidth across the ATM infrastructure. End-to-end control of these mechanisms may or may not be available to the campus infrastructure designer. This is because some of them require manual configuration of the WAN ATM switches. When Switched Virtual Circuits (SVCs) are used these mechanisms are invoked by the network hardware. When SVCs are used network designers can control these mechanisms, as long as the devices that generate the SVC requests allow the designer to configure them.

The ATM transport services provide a platform for the VCCS IP Intranet. The IP tool kit includes something called "level 3 switching". It is a router feature that allows the router to "route" the first few packets in a data stream and then "switches" the remaining packets. This minimizes the processing delay in the router. It is most effective on long data streams such as a file transfer that contain a large number of packets. It is least effective on short data streams such as voice that contain few packets. For example: The word "no" might be the voice reply to a question. If the digitized form of "no" fits into one packet, the packet will be routed not switched.

The effectiveness of "policy management" depends on the manager controlling all of the devices used in the end-to-end communications path. Since no college manager has control of the NWV resources or the resources of another college, the use of policy management has limited applicability in the ATM environment.

NWV makes no claim about being a single vendor environment, and is known to have ATM switches from multiple vendors. Campus infrastructure designers should not depend on NWV to support or ignore proprietary features in campus ATM switches. What is ignored today may be used by NWV for some other purpose tomorrow, without user notification. NWV cannot be relied upon to provide anything but standards based native ATM level support.

Campus Net.Work.Virginia WAN connections (DS-3):

There are no tools available to manage the campus WAN connection. It must be managed manually. The campus infrastructure designer is limited to the bandwidth available in the connection, and must understand how each application utilizes the bandwidth in order to ensure that it is not exceeded.

College infrastructures:

The tools available in the campus infrastructure depend on the hardware used in the infrastructure. The hardware selected should include the features (tools) noted in this document. All of the tools mentioned here are based on existing or proposed standards.

Campus infrastructures may also benefit from the use of level 3 switching. The same considerations apply on the WAN and the campus. The primary benefit will be to applications that generate long data streams.

The effectiveness of "policy management" depends on the manager controlling all of the devices used in the end-to-end communications path. Since the college manager has control of the infrastructure resources on his/her campus, the use of policy management has great potential in the campus environment.

At the IP network level (VCCNet) colleges may agree to do anything they wish, with the understanding that the campus WAN access routers implement these features across the VCCNet WAN. However, if a "session" is to be managed end-to-end both campuses must be under the control of a single management system, or the management systems on both campuses must be configured the same. In the later case, both systems must also use the same protocol to implement the controls (implying that it is a standard protocol or the same vendors management systems).

If there is one system, there could be multiple managers with the authority to make changes. But the changes would apply to all campuses, and would be implemented without the knowledge of the manager(s) who did not make the change.

If there is a separate system on each campus, they either have to be from the same vendor or they must use the same protocol. The managers could then agree on a service level and each manager implement it on his/her own campus.

The use of a single vendor solution at the campus level may provide the college with additional proprietary services. However, as noted above, the campus infrastructure designer cannot rely on NWV to faithfully reproduce ATM based feature(s) across the WAN. This is especially true for multi campus colleges who may already be using IP or other protocols that do span multi campus environments. Make sure that the features you want to use are based in the higher-level protocols, NOT at the ATM level.

Campus Infrastructure Guidelines:

In order to meet the bandwidth needs of existing and future applications, all campus infrastructure designs should incorporate a switched architecture. The architecture should include the use of the following features and functions:

• Redundancy in inter switch connections can be achieved if the campus switches support the spanning tree protocol.

• More efficient use of Class C IP address spaces may be achieved using variable length subnet masking or with IP virtual subnets. Network Address Translation (NAT) with private IP addresses is not a recommended solution since it may cause problems for some of the new video applications that require workstations to register with an authentication server prior to running the application. However, since IP telephones do not require access to the Internet, the use of private IP addresses may prove to be a valuable tool for conserving IP addresses when using IP telephones.

• Any switched architecture may be used (Token-Ring, Ethernet, or ATM).

• Any backbone technology that provides sufficient bandwidth may be used (ATM, Gigabit Ethernet, or Fast Ethernet).

• Infrastructures must be able to support a worst-case scenario for all application types (Non-time dependent, Time dependent, and mission critical).

• Advanced network services such as policy based management, and global directory services should be supported.

• Given the use of IP and ATM, support for LANE or MPOA should be considered.

• There are no plans to change the backbone routing protocol (RIP) any time in the near future. However, there is no requirement that campus infrastructures must use RIP. Colleges are free to use the most appropriate routing protocol for their environment. The only requirement is that the selected routing protocol broadcast all of the subnet Ids that support users who require Internet access into the network, so that the users will have WAN connectivity.

Shared to switched infrastructure migration:

It is not necessary to convert the entire campus infrastructure to a switched environment at the same time. The migration plan should allow the switched backbone to be put in place first, and then move groups of users in a way that is most efficient for the college.

Bandwidth calculations:

The bandwidth calculation process starts by identifying the bandwidth needs of each end device. The bandwidth requirements for the end devices are then aggregated to determine the upstream bandwidth requirements for each of the lines connecting the work group switches to the infrastructure switches above them. The aggregation process is repeated through the infrastructure switches to the top of the infrastructure (the single hub switch). Using a worst-case analysis for each element in the infrastructure is the only way to ensure that each end device will be provided with enough bandwidth to meet its needs.

The design of the physical infrastructure must be developed first. This includes where the switches will be located, what devices will be on each switch, and how devices will be divided into virtual IP subnets and VLANs (if used).

Once the design is completed, total the bandwidth of each end device NIC on its work group switch. Since no device can send or receive data faster than its NIC can process, this represents the most bandwidth that the switch must support.

Note that even if the devices do not send enough data to fully utilize the bandwidth of their NICs all of the time, any data that is sent is sent at the NIC rate. Therefore, any time all of the devices on a particular work group switch are sending data at the same time, the aggregate upstream bandwidth that the switch must handle is the total of the NICs in all of the devices on the switch.

Designers may consider reducing the cost of the switches by rationalizing that all of the devices on the switch will not be sending data at the same time. The designer is, then, assuming that the applications on the subordinate devices are known, known to be of less than full NIC bandwidth, and not going to change in the near future. Though this may be true, whatever data is sent is sent at the rate of the NIC. This is the same logic that results in an effective bandwidth of 4 Mbps for a 10 Mbps shared Ethernet, because devices do send at the same time. It must be understood that this method will lead to ongoing performance problems when the devices do send at the same time. Since they are caused by transient conditions they will be extremely difficult to troubleshoot, and the only way to fix them will be to replace the switches.

Size the work group switches and the upstream lines from them so that they provide the total bandwidth needed by the devices on each switch. Continue the process of aggregating the upstream bandwidth from the work group switches toward the top of the tree (the hub switch). Ensure that each upstream switch and line can support all of the bandwidth needed by the switches beneath them. Following this process will identify the bandwidth requirements for each switch and line, and result in an infrastructure that can concurrently support all of the end devices operating at full capacity.

Do not include any redundant upstream lines in the bandwidth calculations. Since only one line will carry data at a time, ALL of the lines must be able support the load. Size the redundant lines to match the lines that they replace in the event of a line failure.

These calculations may disclose instances where the original design cannot be supported by the physical infrastructure. In this case there are two choices:

1. The design can be modified to meet your needs.
2. The possibility of congestion can be accepted, knowing that performance will be impacted if it occurs.

Bandwidth allocation strategies:

Following the above design process will provide a campus infrastructure that can support all of the intra campus bandwidth requirements.

In order to provide inter campus bandwidth management, the campus WAN connection must be managed. As noted above, this is a manual process that relies on the designers understanding of the types and amounts of inter campus traffic that the users generate. The WAN connection has considerably less bandwidth than the campus infrastructure (only 45 Mbps). It is, therefore, almost certain to be the primary bottleneck for inter campus traffic. The WAN connection bandwidth may be over allocated based on the fact that not every VC will be operating at its maximum capacity, all of the time. If this is done it must be understood that if the situation does occur, congestion will result and performance will be impacted. All that can be done is to prioritize the traffic so that mission critical traffic will experience minimum delay. This means that when the WAN line is congested, non-mission critical traffic will be delayed.

Equipment options:

The criteria for infrastructure switches are that they be sized appropriately so that they do not become a bottleneck and that they support the spanning tree protocol (if redundant lines are being used).