Network Issues with Windows Server 2008 RDP and VS/Hyper-V on Dell Servers

Last month I had a client buy and install several Dell PowerEdge 2950 servers. These servers had 2 Quad processors, 16GB or RAM, and 2 300GB 15K RAPM SAS hard disks, making them ideal for medium-range departmental servers. When the machines arrived, I installed Windows Server 2008 Standard Edition 64-bit on them, and proceeded to configure them based upon the client’s needs.

One of the machines was installed as a Domain Controller. Another was configured to host a SQL database, and 3 more were to act as Virtual Server machines. At that time. Microsoft has not yet released Hyper-V as RTM, and I decided against deploying it at an RC1 phase on that client’s network. So instead, I installed Microsoft Virtual Server 2005 R2 SP1 (with the KB948515 update). All machines were configured to allow remote administration through RDC (Remote Desktop Connection).
At first all seemed well. However, after some time, I began seeing strange network issues. Sometime the servers would just lose connectivity. Their NICs where in a “Connected” status, they had static IP addresses assigned, but they would sometimes lose Internet connectivity. The Virtual Machines hosted on these servers would act even more erratically. They sometimes lost connectivity with the host server, sometimes responding to PINGs, sometimes losing it. Furthermore, each VM was accessed through RDC, and whenever I tried to use the RDC client (mstsc.exe) to connect to the VM, every second attempt would fail. I would press “Connect” and it would just sit there doing nothing. Pressing “Connect” again would sometimes work, and sometimes fail.
The servers themselves worked flawlessly, but connecting to the host OS with RDC would sometimes freeze, and in most cases it would disconnect me with this error message:
Remote Desktop Disconnected Because of an error in data encryption, this session will end.  Please try connecting to the remote computer again.
The nasty part was that this error only appeared when connecting to the servers from my Vista SP1 laptop. All other computers on the network had no issues when connecting to the remote servers.
I have changed from Remote tab for System Properties on both the 2008 Server and Vista system to have a more secure and less secure session and it doesn’t seem to matter.
After some looking around I discovered that uninstalling VS2005 on one of the servers solved the issue of network disconnections for that server, but I needed it so I looked further. Luckily for me, at that same time, Microsoft released Hyper-V RTM, so I decided it was a good opportunity to use it instead of Virtual Server 2005. Furthermore, some posts on various forums and newsgroups on the Internet said it had nothing to do with VS2005. So I needed to investigate further.
After some more digging it turns out that the blame is on the server’s NIC – a Broadcom BCM5708C NetXtreme II GigE.
In order to fix the RDC disconnection issues you need to disable the IPv4 Large Send Offload setting in the NIC’s properties.
From the Broadcom Gigabit Ethernet Teaming Services: Broadcom NetXtreme II™ Network Adapter User Guide:
“Large Send Offload (LSO) is a feature provided by Broadcom network adapters that prevents an upper level protocol such as TCP from breaking a large data packet into a series of smaller packets with headers appended to them. The protocol stack need only generate a single header for a data packet as large as 64 KB, and the adapter hardware breaks the data buffer into appropriately-sized Ethernet frames with the correctly sequenced header (based on the single header originally provided).”


http://manuals.fujitsu-siemens.com/serverbooks/content/manuals/html/broadcom/netxtreme_ii/teamsvcs.htm#wp398933
Follow these steps:

    1. Right-click Networks on the desktop (if you have it present. I do) and select Properties. In the Network and Sharing Center, click on the Manage Network Connections link on the left-hand side.

Note: You can get to the same spot by typing ncpa.cpl in the Run menu.

    1. Right-click on each NIC icon and select Properties.

 

    1. In the NIC Properties window, click on the Configure button.

 

    1. In the Broadcom BCM5708C NetXtreme II GigE Properties window, click on the Advanced tab.
    2. In the Advanced tab, find the IPv4 Large Send Offload setting, and select Disabled from the drop-down list.

 

    1. Click Ok. Your network connection might be dropped for a few seconds.

This should take care of things. However, one reader also suggested changing the Speed and Duplex setting of the NIC to 100 MB Full from the default Auto.
Another reader wrote: “I would not recommend anyone change the Link Speed and Duplex setting as suggested without having a good understanding of how autonegotiation works. In a nutshell, the link speed and duplex setting must be the same at each end of the connection, i.e. on the network adapter and on the switch, or a duplex mismatch can occur. With modern equipment, Link Speed and Duplex should always be set to Auto at each end. In fact, the setting MUST be set to Auto to achieve gigabit connection speeds on capable equipemnt.”
If you have additional feedback, please drop me a note by using the Feedback page.
Another important note is the usage of 2 NICs on the server whenever running the Hyper-V role on it. Make sure you always use both NICs on the Hyper-V server, one for the Virtual Switch network, and the other for regular management tasks of the host machine.
Links
Enumeration Keywords
http://msdn.microsoft.com/en-us/library/aa503756.aspx
Using Registry Values to Enable and Disable Task Offloading
http://msdn.microsoft.com/en-us/library/aa938424.aspx
“The RDP Protocol Component “DATA ENCRYPTION” Detected an Error…” error message – 323497
http://support.microsoft.com/?kbid=323497

Recent Windows Server 2008 Forum threads

Got a question? Post it on our Windows Server 2008 forums!