{"id":2815,"date":"2011-06-09T17:53:20","date_gmt":"2011-06-09T15:53:20","guid":{"rendered":"https:\/\/ingmarverheij.com\/2011\/06\/true-client-side-testing-best-practices\/"},"modified":"2011-06-09T17:53:20","modified_gmt":"2011-06-09T15:53:20","slug":"true-client-side-testing-best-practices","status":"publish","type":"post","link":"https:\/\/ingmarverheij.com\/en\/true-client-side-testing-best-practices\/","title":{"rendered":""true" client side testing best practices"},"content":{"rendered":"

\"iRobotWhen performing a LoadTest user actions are simulated. This implicates that mouse or keyboard actions are executed based on a script, based on a scenario, and that the script waits for a response on the screen.<\/p>\n

The response on the screen can be determined using API’s giving information about windows present, or the controls on the windows. For instance: the script waits until a window is active with the caption "Microsoft Word".<\/p>\n

Another way of determing if a response is given is by comparing the content of the screen with a bitmap. For instance: the script waits until an empty document is displayed in Microsoft Word.<\/p>\n

The difference between the two techniques is that a window caption is present right when the application is launched (even if the application is still loading) while the content on the screen is more simular to the way users interact in a session. So looking at a screen region is more accurate, it prevents assumptions (best practice #9 in loadtesting best practices<\/a>) like "how much time should we wait between lauching an application and clicking on a menu?\u201d.<\/p>\n

In this article I will be discussing some best about practices about \u201ctrue\u201d client side testing (best practice #12 in loadtesting best practices<\/a>).<\/p>\n

<\/p>\n

What is \u201ctrue\u201d client side testing?<\/h3>\n

First of all, let me try to explain what I mean with \u201ctrue\u201d client side testing. <\/p>\n

During a LoadTest a users actions are simulated, or in other words whey\u2019re simulating a user<\/strong>. A user is controlling a workstation via input devices like a keyboard or a mouse, and receiving the results on a display<\/strong>. The workstation is connected, via a network, to servers \/ virtual desktops. For the record: they don\u2019t care what\u2019s \u201cbehind\u201d the curtain (workstation). \"User<\/a><\/p>\n

 <\/h6>\n
Server side testing<\/h6>\n

Traditional loadtesting applications use a process on the server to simulate user actions. This process can be a script<\/strong> based on batch, VBscript, AutoIT, you name it or a proprietary<\/strong> application. Executing the process on the server side enables the script to \u201csee\u201d and \u201ccontrol\u201d the session, enabling the script to simulate user actions. This is what I call \u201cserver side testing\u201d\"Server<\/a>Downside<\/strong> of this method is that the process doesn\u2019t care <\/strong>about what happens between the server and the user (the left side from the servers). Or, in other words, the effects <\/strong>of the network <\/strong>components\/connection are not visible for the loadtest. Another downside is that the script uses the system clock of the server (or virtual desktop). If the server is virtualized, a hypervisor (VMM) under heavy load might cause a clock drift <\/strong>affecting the accuracy of the measurement (best practive #13 in loadtesting best practices<\/a>).<\/p>\n

 <\/p>\n

Client side testing<\/h6>\n

Like a person in real life<\/strong>, the content of the screen on the client device (a workstation<\/strong>) can be used to determine if a action is completed <\/strong>(for instance launching an application). When a loadtest application is able to \u201clook\u201d <\/strong>at the content of the screen from the client, including network components\/connection, this would be result in a more accurate <\/strong>measurement. \"Client<\/a><\/p>\n

I frequently use DeNamiK LoadGen<\/a> to simulate user actions and perform LoadTests. With DeNamiK LoadGen I\u2019m able to simulate a user, by controlling the keyboard and mouse and waiting for results <\/strong>by looking at the content of the screen<\/strong>. In other words: I tell the DeNamiK User Action Framework (DUAF<\/strong>) to wait until a certain bitmap (hash)<\/strong> is visible on the screen of the client, and then I\u2019ll continue with next action.<\/p>\n

 <\/h6>\n
Summary<\/h6>\n

So, that\u2019s what I mean with \u201ctrue\u201d client side testing. The content of the screen on the workstation, including the network components\/connection like a WAN, and remoting protocol effects. If a large image is displayed via a slow WAN connection, resulting in blocks building slowly on the screen, the script wait until it is displayed (like a user) instead of continuing because the caption exists.<\/p>\n

 <\/h3>\n

 <\/p>\n

\u201ctrue\u201d client side best practices<\/h3>\n

Now I\u2019ve explained what I mean with \u201ctrue\u201d client side testing, here are some best practices. Like the loadtesting best practices<\/a>, they are based on my experience.<\/p>\n

\"Dilbert<\/a><\/p>\n

source: Dilbert comic strips<\/a><\/p>\n

 <\/p>\n

<\/h6>\n
1. Use screen regions, unless\u2026<\/h6>\n

Try to use the content of the screen as a condition in your script as much as possible, unless this is impossible or requires assumptions. Since the content of the screen, a bitmap, is the most accurate condition this is the preferred method. Especially because it prevents assumptions, assumptions like \u201cwait x-milliseconds before the executing a command\u201d or \u201cthe document is loaded\u201d.<\/p>\n

Use other (API based) functions when this is more accurate or necessary to determine the location of a window.<\/p>\n

 <\/h6>\n
2. Position of windows <\/h6>\n

Determine the position of windows using API functions and continue using screen regions. An API is capable of retrieving properties of windows, like the window metrics, which is more efficient than looking for a bitmap on each pixel on the screen.<\/p>\n

 <\/p>\n

3. Use small regions<\/h6>\n

The region on the screen, which will be used to compare with a bitmap(hash), should be as small as possible. This has two benefits. First of all performance, the smaller the region the easier it is to calculate the hash \/ compare the bitmaps. Second, the smaller the chance that the content of the region is changed due to another process (see the next best practice).<\/p>\n

 <\/p>\n

4. Use static content<\/h6>\n

The content of the region, which will be used to compare with a bitmap(hash), should be static. When the content of the region on the screen is changed the bitmap(hash) won\u2019t match, and the script won\u2019t continue. An example is the name of a document in the region, a username or a systemtime. Although these are valid items to check, in most conditions they\u2019re irrelevant.<\/p>\n

 <\/p>\n

5. Re-use regions and bitmap(hash)<\/h6>\n

When you\u2019re testing a certain region on the screen multiple times, for instance when you checking if the content of a document is displayed, try to re-use the same region and matchin bitmap(hash). This way, when the environment changes, the script can be updated easily. And, it makes troubleshooting a lot easier. A good example is the progressbar in a browser like Internet Explorer. I frequently check the progressbar to see if the page is loaded, if the progressbar isn\u2019t visible (in IE) the page is loaded. By re-using the region and matchin bitmap(hash), maybe in a function, I\u2019m able to quickly find and solve problems.<\/p>\n

 <\/p>\n

6. Mouse events<\/h6>\n

Mouse events, like mousebutton down or up events, should be executed when the content of the screen matches a certain condition. For instance, when the Start button (I keep on calling that) is pressed, you\u2019ll want to wait before clicking on \u2018All Programs\u2019. As long as the menu isn\u2019t visible, there\u2019s no need to click (in fact, if you do the script will fail). This is an ideal situation where compare the content on the screen with a small bitmap (10×10) helps preveing assumptions.<\/p>\n

 <\/p>\n

7. Color depth<\/h6>\n

The hash of a bitmap, representing a certain region on the screen, is calculated with a formula that uses the color of each pixel and returns a hash. If one of the pixels changes the hash changes. This means that if the color depth (16bpp, 24bpp, etc.) changes, the hash will change. This sounds logical, and it is, but this has a big impact on the script. If the script is created with a different color depth then when it is exectuted, all hashes are invalid and need to be recreated (loads of work!)<\/p>\n

Also keep in mind that not only the color depth of the session is important, the color depth of the client is important aswell. The hash of a bitmap changes when the color depth of the client is changed (for instance from 24bpp to 16bpp), even when the color depth of the session remains equal (for instance 16bpp).<\/p>\n

 <\/p>\n

8. Speedscreen \/ font smoothing (etc)<\/h6>\n

Remoting protocol techniques like speedscreen, progressive display or other techniques that uses image compression change the pixels of a bitmap. Or in other words : the content of the screen is displayed in a different way each time you display it (to be honest, there are usually around 3 to 4 hashes). This means that the hash of the bitmap changes when a different compression (or other technique..) is used, some of these techniques kick in under certain conditions (like a slow WAN or heavy load) which makes it difficult to get all hashes at the first time. <\/p>\n

In the script you should be able to enter multiple hashes for the same bitmap. In the DeNamiK LoadGen this is done by giving an array of bitmaps : <\/p>\n

New String(){\u201chash1\u201d, \u201chash2\u201d, \u201chash3\u201d}<\/pre>\n
Font smoothing, which is enabled by default since Microsoft Window Server 2008, smooths the edges of fonts. This makes the content look better, and better readable for users, but makes it difficult to get a static bitmap(hash).<\/p>\n
Best practice is either avoid using regions that are influenced by the optimizations techniques OR to disable the technique. In fact, none of the two is ideal but (for now) necessary. I\u2019ve heard rumours that a new algorithm is developed by DeNamiK to prevent this \u2018workaround\u2019. But as long as I haven\u2019t seen it, this is the way to go!<\/p>\n
 <\/p>\n
Any comments, additions or questions? Please leave a comment, send me an e-mail or a tweet. <\/p>\n
 <\/p>\n
Ingmar Verheij<\/p>","protected":false},"excerpt":{"rendered":"
When performing a LoadTest user actions are simulated. This implicates that mouse or keyboard actions are executed based on a script, based on a scenario, and that the script waits for a response on the screen. The response on the screen can be determined using API’s giving information about windows present, or the controls on […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-container-style":"default","site-container-layout":"default","site-sidebar-layout":"default","disable-article-header":"default","disable-site-header":"default","disable-site-footer":"default","disable-content-area-spacing":"default","footnotes":""},"categories":[302],"tags":[318,338,100,316],"class_list":["post-2815","post","type-post","status-publish","format-standard","hentry","category-performance-testing","tag-best-practice","tag-client-side","tag-denamik-loadgen","tag-loadtest"],"_links":{"self":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/2815","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/comments?post=2815"}],"version-history":[{"count":0,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/2815\/revisions"}],"wp:attachment":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/media?parent=2815"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/categories?post=2815"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/tags?post=2815"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}