{"id":2626,"date":"2011-05-25T15:00:00","date_gmt":"2011-05-25T13:00:00","guid":{"rendered":"https:\/\/ingmarverheij.com\/2011\/05\/loadtesting-best-practices-part-2\/"},"modified":"2011-05-25T09:53:01","modified_gmt":"2011-05-25T07:53:01","slug":"loadtesting-best-practices-part-2","status":"publish","type":"post","link":"https:\/\/ingmarverheij.com\/en\/loadtesting-best-practices-part-2\/","title":{"rendered":"Loadtesting best practices – Part 2"},"content":{"rendered":"

\"\"This is the second part in a series two about loadtesting best practices.<\/p>\n

The first part focused on the \u201cbasics\u201d of loadtesting, most of them where about preparation. You can find the first part here<\/a>.<\/p>\n

In this second part I\u2019ll focus on some more advanced topics which are usefull in a later stage of the process.<\/p>\n

 <\/p>\n

<\/p>\n

8 \u2013 First time setup<\/h5>\n

\"\"The first time a user logs in a Windows environment or the first time an application is used, a first time setup <\/strong>might be displayed. Since this is a first time setup, it will happen only once for each user.<\/p>\n

When you\u2019re automating a test with simulated user actions you usually don\u2019t want to test the first time setup, you\u2019ll probably want to test the application itself. Therefore it is recommended to prevent the first time setup, usually this is done by setting a some registry keys.<\/p>\n

Common first time setup\u2019s are the initials<\/strong> in Microsoft Office, or the preference wizard <\/strong>in Internet Explorer (or Firefox).<\/p>\n

Before you do a full blown test, test the complete script again with a new user. During the creation of the script you\u2019ve probably configure the first time setup (or even changed some settings\u2026).<\/p>\n

 <\/p>\n

9 \u2013 Assumptions<\/h5>\n

\"\"It is hard not to make any assumption, but each assumption<\/strong> you make can result in an unexpected result. Try to think about what you expect to happen and then what you expect not<\/em> to happen. Are you really sure?<\/p>\n

Before giving some examples of assumptions, here are two quotes to remember:<\/p>\n

\u201cAssume makes an ass out of u and me (ass-u-me)\u201d <\/em><\/div>\n
\u201cAssumption is the mother of all fuck-ups\u201d<\/em><\/div>\n

So in short, don\u2019t be an ass and don\u2019t fuck-up due to assumptions.<\/p>\n

The most common assumptions are about timing, the time between actions like launchen an application and using the application. Another common assumption is that each step, in a series of actions, is correct and that you don\u2019t have to check everything, you should.<\/p>\n

 <\/p>\n

10 \u2013 Load changes everything<\/h5>\n

\"\"<\/a>Applications tend to react in the same manner each time you use them, under heavy load <\/strong>this changes. Since the application relies on a available resources, these might get scarce due to resource contention. This results in unexpected behaviour, usually due to timeouts and assumptions in the application.<\/p>\n

There is no need to accomodate <\/strong>for weird error messages, unless that\u2019s what you\u2019re testing. You should however accomodate for slow responsetimes, or unresponsive applications.<\/p>\n

 <\/p>\n

11 \u2013 No changes<\/h5>\n

\"\"<\/p>\n

An automated test simulating user actions is dumb script, it executes the commands you tell him. This means that if you change the environment, and don\u2019t change the script, it stops working.<\/p>\n

For instance if you start a script by waiting for the desktop, which means the user is logged in, by lookin for an icon on the desktop. If this icon is removed (or moved) the script fails. A real user would ignore the missing icon and start working, a script won\u2019t (unless you accomodate for it).<\/p>\n

A best practice is to agree <\/strong>a freeze period <\/strong>where no changes are allowed. During the creation of the script and the execution of the test, usually during a non-working day, are usually a few days.<\/p>\n

Even though these agreements are made, some sysadmins still change the environment because they think it has no (or hardly any) impact on the environment. So another best practice is to test the script<\/strong> the (working) day before the test<\/strong>, this enables you to change the script or undo the changes made.<\/p>\n

 <\/p>\n

12 \u2013 Client side testing<\/h5>\n

\"\"<\/p>\n

User actions are simulated using either a server or a client component<\/strong>.<\/p>\n

A server side component is a script, utility or some other process that is launched in the session of a user. This enables the script to execute commands and detect windows and objects inside a session. Downside of server side components is that they give an additional load, it\u2019s overhead. The server side component has (almost) no knowledge about the client or the connection quality. Since the process is active on the server, the content on the screen are displayed with local speed. A server side component also relies on the local clock to time results, see best practice 13 about clock drift.<\/p>\n

A client side component is a process executed on the client instead of on the server. This enabled the process to look at the end-result<\/strong>, including the effects of the quality of the connection. In other words: if a large bitmap is displayed on a slow connection, the client side components waits until the content is displayed (unlike the server component).<\/p>\n

Since the client side component executed on the client, no server side component is required. This prevents overhead on the server and enable the use of a reliable clock, even when you\u2019re testing a virtualized server under heavy load.<\/p>\n

 <\/p>\n

13 \u2013 Clock drift<\/h5>\n

\"\"Each system is equipped with a system timer <\/strong>which enabled the clock to work. On a set interval a pulse if given, this is called a clock tick.<\/p>\n

Timers (used in scripts to measure the time an action required to complete) and performance metrics rely on the system clock. If the system clock is inaccurate, the results are inaccurate<\/strong>.<\/p>\n

On a virtualized system the hypervisor (VMM) is responsible for distributing clock ticks generated by the hardware. Each guest OS (VM) should get the same amount of clock ticks with the same interval, this way the guest OS can run is own clock.<\/p>\n

A hypervisor (VMM) under heavy load <\/strong>isn\u2019t able to distribute the clock ticks to guest os\u2019s (VM). Tick may get lost by the VM unwittingly. Davit Ott (Intel) wrote a short article about virtualization and performance, read it here<\/a>.<\/p>\n

Try to avoid the use of a virtualized clock when the system is under heavy load. Using a virtualized system for timing is possible, of course, but the results might be influenced when the clock ticks gets lost. Using an external clock source (like a SQL server) helps, using a client side component can prevent the problem.<\/p>\n

 <\/p>\n

14 \u2013 Monitor<\/h5>\n

\"\"<\/strong>During a test you should monitor the results, like performance metrics, and the end-result. This way you can experience first hand <\/strong>what the impact of the test is. This not only helps in validating the result, it also might help you finding the bottleneck and solving the problem, if that is an objective.<\/p>\n

Especially in a virtual desktop environment, like SBC or VDI, you should launch a dedicated session for monitoring. Use that session to browse, click around and experience the impact.<\/p>\n

During the test write down all the properties and parameters of the test.\u00a0 Write down the events that occur, for instance spikes in performance metrics, slow response times, sessions failing. Keep the notes with the test results , they are very usefull when analysing the results.<\/p>\n

I once had a user connecting his laptop to the network during an extensive loadtest, his laptop was configured to synchronise his homefolder and mailbox. Since we where testing on a WAN connection, this had an impact on the test. Since I wrote this down in my notes, I could explain the impact on the perceived performance.<\/p>\n

 <\/p>\n

15 \u2013 Validate results<\/h5>\n

\"\"After a test is completed and the results are collected, try to validate the results. Are the results what you expected<\/strong>? Do they represent the results you noticed during the test?<\/p>\n

Data collected don\u2019t lie<\/strong>, they are collected by a dumb process. But if you collected the wrong data, interprete it in a wrong way or if the load generated don\u2019t match the scenario described (during preparation, see best practice 1 and 2) then the results might be invalid.<\/p>\n

 <\/p>\n

16 \u2013 Lifecycle management<\/h5>\n

\"\"<\/p>\n

Performaing a loadtest to scaling a system once is good, repeating<\/strong> the process is better. Once you\u2019ve set a baseline<\/strong> you can measure the impact of changes<\/strong>.<\/p>\n

(Almost) each environment is subject to change. Applications are added, upgraded or replaced, users are added and the workload changes.<\/p>\n

 <\/p>\n

Loadtesting should be a part of lifecycle management <\/strong>resulting in a better overall quality<\/strong>, as described in the Deming cycle.<\/p>\n

Plan <\/strong>– Create a plan with the results you want to achieve.<\/p>\n

For instance adding a software package or increasing load.<\/em><\/div>\n

Do<\/strong> \u2013 Implement the plan<\/p>\n

In a test environment according to DTAP <\/em><\/div>\n

Check<\/strong> \u2013 Compare the results with the baseline<\/p>\n

Are the results achieved as expected?<\/em><\/div>\n

Act<\/strong> \u2013 Resolve issues and store results<\/p>\n

If the results are not achieve, solve the problem. Finally store all results for archiving.<\/em><\/div>\n

 <\/p>\n

 <\/p>\n

Ingmar Verheij<\/p>","protected":false},"excerpt":{"rendered":"

This is the second part in a series two about loadtesting best practices. The first part focused on the \u201cbasics\u201d of loadtesting, most of them where about preparation. You can find the first part here. In this second part I\u2019ll focus on some more advanced topics which are usefull in a later stage of the […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-container-style":"default","site-container-layout":"default","site-sidebar-layout":"default","disable-article-header":"default","disable-site-header":"default","disable-site-footer":"default","disable-content-area-spacing":"default","footnotes":""},"categories":[302],"tags":[318,316,317],"class_list":["post-2626","post","type-post","status-publish","format-standard","hentry","category-performance-testing","tag-best-practice","tag-loadtest","tag-stresstest"],"_links":{"self":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/2626","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/comments?post=2626"}],"version-history":[{"count":8,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/2626\/revisions"}],"predecessor-version":[{"id":2627,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/posts\/2626\/revisions\/2627"}],"wp:attachment":[{"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/media?parent=2626"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/categories?post=2626"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ingmarverheij.com\/en\/wp-json\/wp\/v2\/tags?post=2626"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}