When new sessions are started, either via Microsoft RDP of Citrix ICA, they are disconnected within seconds. This applies to normal users and users with administrative privileges. This problem is caused by a chain of events. One components crash leads to an ungraceful shutdown of other components leaving a garbage configuration, preventing new connections.

Symptoms

Users connect to a Citrix XenApp server and are immediately disconnected. No sessions are visible on the server (no disconnected sessions either) and the same symptoms apply for sessions via Microsoft RDP. There are no events logged in the event log that indicate a potential problem.In some occasions the first RDP/ICA session is successful and subsequent sessions fail.

Situation

Consider the following scenario:

• Remote Desktop Session Host role (RDSH) is installed
• Require secure RPC communication is enabled for RDSH
• Set client connection encryption level is configured for RDSH
• There are active (or disconnected) sessions either via RDP or ICA

Chain of events

The following events occur causing a chain of events leading to the described symptoms.

Group policy refresh

Every 90 minutes group policies are automatically refreshed (or manually via gpupdate /force). When this happens the Remote Desktop service reloads to load the GPO changes.

In the Application log we can see an event is raised by SceCli (Security Configuration Editor Client for Windows) with ID 1704 informing us that a new security policy is applied successfully.

Remote Desktop services crash

When the Remote Desktop service restarts it unloads and reloads the module winsta.dll (the module that handles the WinStations), but reloads it incorrectly. As a result the Remote Desktop services will crash.

First an event is raised by Application Error with ID 1000 informing us that the application svchost.exe_TermService (the process of the Remote Desktop service) has an error. The faulting module is WINSTA.dll_unloaded.

90 seconds after the Remote Desktop Service stops working the winlogon notification subscriber notices the <TermSrv> is taking a long to handle notification events, as can be seen in an event from Winlogon with ID 6005.

Around 5 minutes later the Citrix Health monitors notice something’s wrong, both the Terminal Services test and the Ticketing test fail. Two error events are raised by CitrixHealthMon with ID 2005.

Since the Remote Desktop Service is no longer working the Citrix Health monitors can’t perform a recovery action (disabling logon). A warning is logged from CitrixHealthMon with ID 1001.

After 1 hour and 1 minute <TermSrv>  handles the notification event sent by the winlogon notification subscriber.

Sure enough, not much later the Citrix Health monitor was able to successfully run and pass the Terminal Services test and the state and failure threshold has been reset. Am information event is logged from CitrixHealthMon with ID 2006.

Remote Desktop services starts

The Service Control Manager detects when services fail and perform recovery actions. All services are configured by default to restart the service after 1 minute.

Although the service seemed to start working correctly again (the notification events are received and the health checks are passed) the Service Control Manager detects that the Remote Desktop Service is terminated unexpectedly and it will perform a correction action (Restart the service) in 60000 milliseconds, or 1 minute. Notice this happens 1 hour after the initial problem started.

Sure enough an event is raised by Service Control Manager with ID 7036 informing us that the Remote Desktop Services service has entered the running state.

At the same time an event is raised by DistributedCOM with ID 10010 informing us that the server {F9A874B6-F8A8-4D73-B5A8-AB610816828B} did not register in time. The server referred to is the “Terminal Services Connection Manager Class”, which makes sense if the Remote Desktop Services service is restarted.

Sessions (RDP/ICA) are reset

Since the Remote Desktop Service is restarted the active (and disconnected) sessions are reset. In a normal situation an event is sent to all depending services such as Citrix IMA (used in Citrix XenApp 6.5) to inform when sessions are ended. When Citrix receives receives an event it can do some housekeeping such as cleaning up the Sessions key in the registry.
HKLM\SOFTWARE\Citrix\Ica\Session

Since this event is never sent (the Remote Desktop Services service is in an inconsistent state and restarting) no housekeeping takes places and garbage remains in the registry. In this example three sessions where active. ID 3, 4 and 14.

Citrix XenApp reloads

Citrix XenApp detects that Remote Desktop Services is restarted and will act accordingly. For instance it will contact the Citrix license server to verify if it is available. An informational event is raised by MetaFrame with ID 9019.

What Citrix will not do is housekeeping, not even when a server restarts. As a result the garbage of the previous sessions (ID 3,4 and 14) remain in the registry causing problems for new sessions.

New sessions

So what happens if a new session is initiated? Since Citrix has hooked into Remote Desktop Service it is informed immediately when new Winstations are created (which happens for both RDP and ICA). Citrix is kind enough to keep track of all sessions in the HKLM\SOFTWARE\Citrix\Ica\Sessions key, regardless of the protocol used. In other words, RDP sessions are stored in the Ica sessions key as well.

As a result the following scenario could occur (example):

• A  user initiates a new session via Microsoft RDP
• A new Winstation is created with ID 3
• Citrix hooks into the new WinStation creation and tries to register the session
• It detects that ID 3 already exists and this is an ICA session
• Citrix and Remote Desktop Services are confused, what happened? Must be a glitch in the network or somehome hacking into the system. Better safe then sorry ,  let’s end this session right now!

Solutions

For this problem there are two solutions. One is already in place, the other needs to be implemented.

• The described problem with the Remote Desktop Services host crashing after refreshing group policies (with increased security) is a know bug that’s fixed with a hotfix. It is described in KB2479710.
• Citrix needs to add a housekeeping task in Citrix (XenApp) and clean all known sessions on a restart. This needs to be implemented in a future version (or hotfix).

Both Microsoft and Citrix need to increase the events generated in the scenario described above. You can’t disconnect a session without raising an event in the event log describing why.

.