There are occasions where you need a good load balancer but don’t have the budget. Microsoft offers Network Load Balancing services (NLB) as part of their Windows server operating systems, but although we’re looking for a cheap solution we try to avoid problems. This is where the Citrix NetScaler comes in. Gartner positioned the NetScaler ADC in the leaders quadrant of the Magic Quadrant for Application Desktop Controllers (for the 7th consecutive year in 2013), proving it to be good solution and a reliable partner. Using Direct Server Return (DSR) mode we can offer a “poor man’s solution”.

In this article I’ll explain why you might (not) want to implement Direct Server Return (DSR), how it works and of course how to configure it!

Costs

As said, sometimes we need a “cheap” solution because the budget isn’t sufficient. The costs of a Citrix NetScaler are determined by a number of factors:

• Platform – Ranges from VPX (virtual appliance) to MPX (hardware appliance to SDX (hardware appliance with advanced virtualization to consolidate up to 40 independently managed NetScalers)
• Edition – Three editions (Standard / Enterprise / Platinum) are available. Each edition offers a (sub)set of features
• Model – A variety of models exist to suit the most demanding IT and business needs. The model range offers an increasing throughput and other performance factors. Once you hit your thoughput limit packets are queued, not dropped. If the same NetScaler is used for other purposes –for instance Access Gateway – this could affect those services.

Depending on your needs there’s a NetScaler that fits your need. One of the criteria – looking at load balancing – is throughput. All platforms have different models which are limited on the throughput they can deliver. The more throughput you need the more expensive model you need. To give you a rough overview on the cost of a NetScaler VPX –  the virtual appliance – this are the list prices of the Standard edition (which offers load balancing) .

Source: Citrix Store – NetScaler VPX – January 10th 2014

NetScaler VPX Express

There is a Citrix NetScaler VPX Express available for Free!  The Express edition is limited to 5 Mbps throughput but offers full NetScaler Standard functionality. It even includes five free Access Gateway Enterprise edition Concurrent licenses!  The Express edition does not entitle you to file a tech support case, you need a retail NetScaler VPX license. Therefor it is not recommended for production use 😉

More information can be found in the NetScaler VPX Express FAQ and Citrix Blogs.

Throughput

As you can see throughput equals money, so with less throughput we can use a less expensive model (or platform). One way of lowering the throughput is by using Direct Server Return (DSR) mode.

In a normal scenario where a client communicates with a server via a load balancer the following steps are involved.

1. The client requests a file from the load balancer
2. The load balancer forwards the request to the server
3. The server sends the file to the load balancer
4. The load balancer sends the file to the client

In a Direct Server Return (DSR) mode the server doesn’t answer the load balancer but returns the file to the client directly, resulting in the following steps:

1. The client requests a file from the load balancer
2. The load balancer forwards the request to the server
3. The server sends the file to the client

As a result the file returned by the server no longer travels through the NetScaler and less throughput is required. This – of course – only works if the returned data is the big data since all requests are made at the load balancer. In case you’re uploading large files from the client to a server the Direct Server Return (DSR) mode won’t help you here.

Applications

So when could Direct Server Return (DSR) mode be beneficial for you?  If you need to load balancing client – server traffic where the response causes the majority of the throughput and you need a good load balancer that monitors the health of the servers, the Citrix NetScaler is a good option. If you don’t want to let all traffic flow back through the load balancer, due to throughput limitations DSR could be a solution.

Examples are:

• Trivial File Tranfer Protocol (TFP) – For instance for Citrix Provisioning Services (PVS)
• Microsoft App-V streaming  via HTTP 
• Web server offering large downloads (like bigfile.iso in the example above)

PS: Microsoft App V.4x streaming via RTSP and DSR mode does not seem to work according to Barry Schiffer. Apparently this has to do with the way Microsoft implemented the RTSP protocol.

Considerations

Before you start reading how you can configure your environment to use Direct Server Return (DSR) mode there are some things to consider. To achieve the aforementioned functionality a number of tricks are applied that might not work or break existing functionality. More information about features and limitations can be found in the Citrix eDocs.

Changes on load balancer AND server

implementing Direct Server Return (DSR) mode requires you not only change the load balancer but also the server. In other words, you required administrative access to the load balanced servers and need to make system changes. Not just the load balancer.

Packet Processing Flow

A Citrix NetScaler processes packets in a pre-defined order. When traffic flows through a NetScaler it evaluates its feature sets, logging matching policy actions.

As can be seen in the diagram a packet is evaluated when it travels from the client to the server and again when it travels back from the client to the server. When Direct Server Return (DSR) mode is used the packet never travels back through the NetScaler, as a result the a number of actions are never applied: Using Direct Server Return (DSR) mode the NetScaler can offer less functionality.

• CF + CMP + CKA
• Response Rewrite
• Apply Rewrite
• NS Body Transformer
• Caching – Read more about Integrated Caching here

Firewall and routing

Instead of changing the destination IP the destination MAC is changed (see paragraph “How it works” for details) the SNIP must reside in the same routing subnet as the VIP of the virtual server. If for example the VIP and SNIP are in 10.0.0.x and the server in 192.16.0.x then the packet is never routed . The SNIP sent outs a package from the 10.0.0.x subnet and therefore should not be routed, as a result no server picks up the packet (the NIC with the provided MAC listens on the 192.168.0.x subnet).

Incomplete SYN

From eDocs: Because the appliance does not proxy TCP connections (that is it does not send SYN-ACK to the client), it does not completely shut out SYN attacks. By using the SYN packet rate filter, you can control the rate of SYNs to the server. To control the rate of SYNs, set a threshold for the rate of SYNs. To get protection from SYN attacks, you must configure the appliance to proxy TCP connections. However, that requires the reverse traffic to flow through the appliance.

Because there’s an incomplete SYN Intrusion Detection / Protection Systems (IDS / IPS) could mark the traffic as malicious and therefore break it.

Monitors

Not all monitors can be used to monitor the health of services. This is is due to the fact that the NetScaler forwards packets using the server MAC address instead of the destination server IP. The following monitors are affected: Citrix-WI-Extended  – FTP  –  LDAP – MySQL – NNTP – POP3 – Radius – SMTP – SNMP – USER (Custom Perl Script). More information (and a solution) can be read in CTX138969.

How it works

In short Direct Server Return (DSR) mode works by replacing the MAC addresses of the sender to the MAC address of the load balancer server (MAC based forwarding) and by providing the source IP of the client instead of the NetScaler (Use Source IP – USIP).

Let’s see how the the packets are changed in an example. Let’s assume the following fictitious IP and MAC addresses are used:

If we look at the previous example, in normal mode four packets are sent (simplified):

When using Direct Server Return (DSR) mode only three packets are sent (simplified):

As you can see the “magic happens” in the second step when the NetScaler requests the file at the server. Mac Based Forwarding: Instead of changing the destination IP to the IP of the server (10.0.0.100) the VIP is used (10.0.10), to ensure the packet is delivered at the correct server the MAC of the destination server is in the packet.
Use Source IP (USIP): Now the server needs to answer the client instead of the SNIP of the NetScaler. Instead of providing the SNIP as the source IP and MAC the IP and MAC of the client are provided in the packet.

Now the server receives a packet with an IP it doesn’t own (it receives a packet with IP 10.0.0.10 while it only owns 10.0.0.100). To prevent that the packet is dropped a loopback interface is created on the server with the IP of the VIP (10.0.0.10).  To avoid problems with the ARP table the loopback interface is configured as a non-arping interface (for an example see below).

How to configure Direct Server Return (DSR) mode

Finally! Here’s the part where I describe the steps that are needed to get Direct Server Return (DSR) mode working . Configuring Direct Server Return (DSR) mode requires you to configure both the NetScaler and the server.

NetScaler

Modes

Since we use MAC based forwarding this mode needs to be enabled, by default it’s disabled. In the Configuration tab go to System> Settings and click on Configure modes. Select the MAC based forwarding mode and click OK.

Or via CLI

enable ns mode mbf

Basic features

Of course the load balancing feature needs to be enabled, this is disabled by default. In the Configuration tab go to System> Settings and click on Configure modes. Select the MAC based forwarding mode and click OK.

Or via CLI

enable ns feature lb

Server

For each load balanced server a Server-object needs to be created. Nothing special here, just add a normal server. In the Configuration tab go to Traffic Management > Load Balancing > Servers and click on Add.

Or via CLI

add server Server1 10.0.0.100

Service

Each service offers one or more services (like HTTP, DNS, MySQL, etc.). A NetScaler load balances traffic across services, not across servers. We need to create a service with the protocol ANY, a basic monitor (as said earlier, not all monitors work with DSR– CTX138969) and Use Source IP (USIP) needs to be enabled. Of course the service needs to be bound to a server on a specific port, in the example port 80 (HTTP). In the Configuration tab go to Traffic Management > Load Balancing > Services and click on Add.

Or via CLI

add service service_Server1_ANY server1 ANY 80 -usip Yes

Virtual Server

Last we need a virtual server that load balances traffic to one or more virtual service. What’s important is that protocol is ANY (just like the service), the load balancing method is Source IP Hash and the redirection mode is MAC based (aka MAC based forwarding). Since no return traffic passes the NetScaler is makes no sense to keep track of sessions, therefor it is recommended to make the virtual server Sessionless. In the Configuration tab go to Traffic Management > Load Balancing > Virtual Servers and click on Add.

Or vla CLI

add lb vserver vserver_DSR ANY 10.0.0.11 80 -lbmethod SOURCEIPHASH -m MAC -sessionless ENABLED
bind lb vserver vserver_DSR service_Server1_ANY

PS: For certain services (such as FTP) you need to enable connection failover: stateless

Server

Loopback interface

One each load balanced server a loopback interface is created with the IP of the virtual server VIP. This ensures that the server doesn’t drop the packet when it enters the IP stack.

Windows

In Windows you can add a Loopback interface using the Add Hardware Wizard (hdwwiz.exe).  In Windows Server 2012 the loopback interface is renamed to Microsoft KM-TEST Loopback Adapter.

Rename the new loopback network to “Loopback” in Control Panel > Network and Internet > Network Connections.

Open the properties of the Loopback adapter and disable all services except Internet Protocl Version 4 (TCP/IPv4) and specify the IP address of the virtual server VIP (10.0.0.100 in the example from above). The same subnet should be 255.255.255.255 (limitiing it to just this IP) , do not specify a gateway!  Important as well is to disable DNS registration and NetBIOS over TCP/IP in the Advanced tab.

Linux

In Linux you can add a loopback interface via CLI

ifconfig dummy0 up
ifconfig dummy0:0 inet 10.0.0.10 netmask 255.255.255.255 up

Non-arping interface

To avoid problems with the ARP table the loopback interface is configured as a non-arping interface.

Windows

Open a command  prompt and enable weak host receiving and sending on the loopback interface. Also enable weak host receiving on the production interface  (bound to 10.0.0.100 in the example).

netsh int ipv4 set int "Loopback" weakhostreceive=enabled weakhostsend=enabled 
netsh int ipv4 set int "Ethernet" weakhostreceive=enabled 

It very well could be that your machine already cached some arp data. You could wait until the cache is invalidated or clear the cache by issuing the command

arp -d *

More information about strong and weak host models can be found at TechNet Magazine – The Cable Guy.

Linux

In Linux the loopback interface can be confiured as non-arping by issueing the following commands:

echo 1 > /proc/sys/net/ipv4/conf/dummy0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/dummy0/arp_announce

More information about arp annound / arp ignore to disable ARP can be read at the WIKI of Linux Virtual Server. For more information see Configuring LINUX Servers in DSR Mode on Citrix eDocs

.