{"id":5409,"date":"2013-08-07T09:29:47","date_gmt":"2013-08-07T07:29:47","guid":{"rendered":"https:\/\/ingmarverheij.com\/?p=5409"},"modified":"2013-08-07T10:43:59","modified_gmt":"2013-08-07T08:43:59","slug":"citrix-xenapp-sessions-disconnected-right-after-connection","status":"publish","type":"post","link":"https:\/\/ingmarverheij.com\/en\/citrix-xenapp-sessions-disconnected-right-after-connection\/","title":{"rendered":"Citrix XenApp: Sessions disconnected right after connection"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" title=\"\" style=\"border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; float: right; padding-top: 0px; padding-left: 0px; margin: 0px 0px 0px 5px; display: inline; padding-right: 0px; border-top-width: 0px\" border=\"0\" alt=\"\" align=\"right\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Disconnect.jpg\" width=\"119\" height=\"69\" \/>When new sessions are started, either via Microsoft RDP of Citrix <a href=\"https:\/\/ingmarverheij.com\/en\/the-citrix-ica-file-explained-and-demystified\/\" target=\"_blank\">ICA<\/a>, they are disconnected within seconds. This applies to normal users and users with administrative privileges. This problem is caused by a chain of events. One components crash leads to an ungraceful shutdown of other components leaving a garbage configuration, preventing new connections.<\/p>\n<p><!--more--><\/p>\n<h1>Symptoms<\/h1>\n<p>Users connect to a Citrix XenApp server and are immediately disconnected. No sessions are visible on the server (no disconnected sessions either) and the same symptoms apply for sessions via Microsoft RDP. There are no events logged in the event log that indicate a potential problem.<em>In some occasions the first RDP\/ICA session is successful and subsequent sessions fail.<\/em><\/p>\n<p><em><\/em><\/p>\n<p>&#160;<\/p>\n<h1>Situation<\/h1>\n<p>Consider the following scenario:<\/p>\n<ul>\n<li><strong>Remote Desktop Session Host role<\/strong> (RDSH) is installed <\/li>\n<li><strong>Require secure RPC communication <\/strong>is enabled for RDSH <\/li>\n<li><strong>Set client connection encryption level <\/strong>is configured for RDSH <\/li>\n<li>There are <strong>active <\/strong>(or <strong>disconnected<\/strong>) sessions either via RDP or ICA <\/li>\n<\/ul>\n<p>&#160;<\/p>\n<h1>Chain of events<\/h1>\n<p>The following events occur causing a chain of events leading to the described symptoms.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Events.png\"><img loading=\"lazy\" decoding=\"async\" title=\"\" style=\"border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; float: none; padding-top: 0px; padding-left: 0px; margin: 0px auto; display: block; padding-right: 0px; border-top-width: 0px\" border=\"0\" alt=\"\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Events_thumb.png\" width=\"400\" height=\"233\" \/><\/a><\/p>\n<h6>Group policy refresh<\/h6>\n<p>Every 90 minutes group policies are automatically refreshed (or manually via gpupdate \/force). When this happens the Remote Desktop service reloads to load the GPO changes. <\/p>\n<p>In the Application log we can see an event is raised by <strong>SceCli<\/strong> (<u>S<\/u>ecurity <u>C<\/u>onfiguration <u>E<\/u>ditor <u>Cli<\/u>ent for Windows) with ID 1704 informing us that a new security policy is applied successfully.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1704-SceCli1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 1704, SceCli\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 1704, SceCli\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1704-SceCli_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>&#160;<\/p>\n<h6>Remote Desktop services crash<\/h6>\n<p>When the Remote Desktop service restarts it unloads and reloads the module winsta.dll (the module that handles the WinStations), but reloads it incorrectly. As a result the Remote Desktop services will <strong>crash<\/strong>. <\/p>\n<p>First an event is raised by <strong>Application Error<\/strong> with ID <strong>1000<\/strong> informing us that the application svchost.exe_TermService (the process of the Remote Desktop service) has an error. The faulting module is WINSTA.dll_unloaded.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1000-Application-Error1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 1000, Application Error\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 1000, Application Error\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1000-Application-Error_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>90 seconds after the Remote Desktop Service stops working the winlogon notification subscriber notices the &lt;TermSrv&gt; is taking a long to handle notification events, as can be seen in an event from <strong>Winlogon <\/strong>with ID <strong>6005<\/strong>.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-6005-Winlogon1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 6005, Winlogon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 6005, Winlogon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-6005-Winlogon_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>Around 5 minutes later the Citrix Health monitors notice something&#8217;s wrong, both the Terminal Services test and the Ticketing test fail. Two error events are raised by <strong>CitrixHealthMon<\/strong> with ID <strong>2005<\/strong>.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2005-CitrixHealthMon-11.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 2005, CitrixHealthMon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 2005, CitrixHealthMon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2005-CitrixHealthMon-1_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2005-CitrixHealthMon-21.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 2005, CitrixHealthMon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 2005, CitrixHealthMon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2005-CitrixHealthMon-2_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>Since the Remote Desktop Service is no longer working the Citrix Health monitors can\u2019t perform a recovery action (disabling logon). A warning is logged from <strong>CitrixHealthMon<\/strong> with ID <strong>1001<\/strong>.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1001-CitrixHealthMon1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 1001, CitrixHealthMon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 1001, CitrixHealthMon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-1001-CitrixHealthMon_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>After 1 hour and 1 minute &lt;TermSrv&gt;&#160; handles the notification event sent by the winlogon notification subscriber.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-6006-Winlogon1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 6006, Winlogon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 6006, Winlogon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-6006-Winlogon_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>Sure enough, not much later the Citrix Health monitor was able to successfully run and pass the Terminal Services test and the state and failure threshold has been reset. Am information event is logged from <strong>CitrixHealthMon<\/strong> with ID <strong>2006<\/strong>.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2006-CitrixHealthMon1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 2006, CitrixHealthMon\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 2006, CitrixHealthMon\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-2006-CitrixHealthMon_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>&#160;<\/p>\n<h6>Remote Desktop services starts<\/h6>\n<p>The Service Control Manager detects when services fail and perform recovery actions. All services are configured by default to restart the service after 1 minute.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Remote-Desktop-Services-Recovery1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Remote Desktop Services - Recovery\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Remote Desktop Services - Recovery\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Remote-Desktop-Services-Recovery_thumb1.png\" width=\"154\" height=\"173\" \/><\/a><\/p>\n<p>Although the service seemed to start working correctly again (the notification events are received and the health checks are passed) the Service Control Manager detects that the Remote Desktop Service is terminated unexpectedly and it will perform a correction action (Restart the service) in 60000 milliseconds, or 1 minute. Notice this happens <strong>1 hour <\/strong>after the initial problem started.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-7031-Service-Control-Manager1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 7031, Service Control Manager\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 7031, Service Control Manager\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-7031-Service-Control-Manager_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>Sure enough an event is raised by <strong>Service Control Manager <\/strong>with ID <strong>7036<\/strong> informing us that the Remote Desktop Services service has entered the running state.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-7036-Service-Control-Manager1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 7036, Service Control Manager\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 7036, Service Control Manager\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-7036-Service-Control-Manager_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>At the same time an event is raised by <strong>DistributedCOM<\/strong> with ID <strong>10010<\/strong> informing us that the server {F9A874B6-F8A8-4D73-B5A8-AB610816828B} did not register in time. The server referred to is the \u201cTerminal Services Connection Manager Class\u201d, which makes sense if the Remote Desktop Services service is restarted.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-10010-DistributedCOM12.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 10010, DistributedCOM\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; float: left; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 10010, DistributedCOM\" align=\"left\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-10010-DistributedCOM12_thumb.png\" width=\"154\" height=\"108\" \/><\/a><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/F9A874B6-F8A8-4D73-B5A8-AB610816828B1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"{F9A874B6-F8A8-4D73-B5A8-AB610816828B}\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; float: left; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"{F9A874B6-F8A8-4D73-B5A8-AB610816828B}\" align=\"left\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/F9A874B6-F8A8-4D73-B5A8-AB610816828B1_thumb.png\" width=\"411\" height=\"89\" \/><\/a><\/p>\n<p>&#160;<\/p>\n<p>&#160;<\/p>\n<p>&#160;<\/p>\n<p>&#160;<\/p>\n<h6>Sessions (RDP\/ICA) are reset<\/h6>\n<p>Since the Remote Desktop Service is restarted the active (and disconnected) sessions are reset. In a normal situation an event is sent to all depending services such as Citrix IMA (used in Citrix XenApp 6.5) to inform when sessions are ended. When Citrix receives receives an event it can do some housekeeping such as cleaning up the Sessions key in the registry.    <br clear=\"all\" \/><em>HKLM\\SOFTWARE\\Citrix\\Ica\\Session<\/em><\/p>\n<p>Since this event is never sent (the Remote Desktop Services service is in an inconsistent state and restarting) no housekeeping takes places and garbage remains in the registry. In this example three sessions where active. ID 3, 4 and 14. <\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Active-sessions3.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Active sessions\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Active sessions\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Active-sessions3_thumb.png\" width=\"204\" height=\"178\" \/><\/a><\/p>\n<p>&#160;<\/p>\n<h6>Citrix XenApp reloads<\/h6>\n<p>Citrix XenApp detects that Remote Desktop Services is restarted and will act accordingly. For instance it will contact the Citrix license server to verify if it is available. An informational event is raised by <strong>MetaFrame <\/strong>with ID <strong>9019<\/strong>.<\/p>\n<p><a href=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-9019-MetaFrame1.png\"><img loading=\"lazy\" decoding=\"async\" title=\"Event 9019, MetaFrame\" style=\"border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px 5px 0px 0px; border-left: 0px; display: inline; padding-right: 0px\" border=\"0\" alt=\"Event 9019, MetaFrame\" src=\"https:\/\/ingmarverheij.com\/wp-content\/uploads\/2013\/08\/Event-9019-MetaFrame_thumb1.png\" width=\"154\" height=\"108\" \/><\/a><\/p>\n<p>What Citrix will not do is housekeeping, not even when a server restarts. As a result the garbage of the previous sessions (ID 3,4 and 14) remain in the registry causing problems for new sessions.<\/p>\n<p>&#160;<\/p>\n<h1>New sessions<\/h1>\n<p>So what happens if a new session is initiated? Since Citrix has hooked into Remote Desktop Service it is informed immediately when new Winstations are created (which happens for both RDP and ICA). Citrix is kind enough to keep track of all sessions in the <strong>HKLM\\SOFTWARE\\Citrix\\Ica\\Sessions<\/strong> key, regardless of the protocol used. In other words, RDP sessions are stored in the Ica sessions key as well.<\/p>\n<p>As a result the following scenario could occur (example):<\/p>\n<ul>\n<li>A&#160; user initiates a new session via Microsoft RDP <\/li>\n<li>A new Winstation is created with ID 3 <\/li>\n<li>Citrix hooks into the new WinStation creation and tries to register the session <\/li>\n<li>It detects that ID 3 already exists and this is an ICA session <\/li>\n<li>Citrix and Remote Desktop Services are confused, what happened? Must be a glitch in the network or somehome hacking into the system. Better safe then sorry ,&#160; let\u2019s end this session right now! <\/li>\n<\/ul>\n<p>&#160;<\/p>\n<h1>Solutions<\/h1>\n<p>For this problem there are two solutions. One is already in place, the other needs to be implemented.<\/p>\n<ul>\n<li>The described problem with the Remote Desktop Services host crashing after refreshing group policies (with increased security) is a know bug that\u2019s fixed with a hotfix. It is described in <a href=\"https:\/\/support.microsoft.com\/kb\/2479710\/en-us\" target=\"_blank\">KB2479710<\/a>. <\/li>\n<li>Citrix needs to add a housekeeping task in Citrix (XenApp) and clean all known sessions on a restart. This needs to be implemented in a future version (or hotfix). <\/li>\n<\/ul>\n<p>Both Microsoft and Citrix need to increase the events generated in the scenario described above. You can\u2019t disconnect a session without raising an event in the event log describing why.<\/p>\n<p>&#160;<\/p>\n<p>&#160;<\/p>\n<p>.<\/p>","protected":false},"excerpt":{"rendered":"<p>When new sessions are started, either via Microsoft RDP of Citrix ICA, they are disconnected within seconds. This applies to normal users and users with administrative privileges. This problem is caused by a chain of events. One components crash leads to an ungraceful shutdown of other components leaving a garbage configuration, preventing new connections.<\/p>","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-container-style":"default","site-container-layout":"default","site-sidebar-layout":"default","disable-article-header":"default","disable-site-header":"default","disable-site-footer":"default","disable-content-area-spacing":"default","footnotes":""},"categories":[295,305],"tags":[667,584,583,585],"class_list":["post-5409","post","type-post","status-publish","format-standard","hentry","category-remote-desktop-terminal-server","category-xenapp-presentation-server","tag-citrix","tag-citrix-xenapp","tag-remote-desktop-services","tag-winstation"],"_links":{"self":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/5409","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/comments?post=5409"}],"version-history":[{"count":5,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/5409\/revisions"}],"predecessor-version":[{"id":5444,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/5409\/revisions\/5444"}],"wp:attachment":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/media?parent=5409"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/categories?post=5409"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/tags?post=5409"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}