Kickstart your Performance Testing


My notes from a performance testing workshop I followed at AADays 2018 in Poland. It was my very first encounter with performance testing.


Spartez, Atlassian partner

Senior Quality Engineer

If you are a cloud company: “Monitor performance on production

Workshop Manifesto

Focus on this workshop:

  1. learn doing over theoretical study
  2. test feature under current development over test regression of key features
  3. test application over environment configuration

That is, while things you learn will be helpful in items on the right, we concentrate on the items on the left more this time.

Meet our SuT (System under Test)

JIRA: http://18.184.154.65:8080

Performance checks during exploratory testing

Trick: Charles proxy

  • web debugging proxy
  • can repeat requests even simultaneously

Google Chrome <-> Charles <-> Jira Software remove cache

What is good performance?

  • between 10 and 300ms: you can be proud
  • between 300ms and 1000ms: you know something is wrong, but it still feels natural to the user
  • above 1000ms: users loose focus and you can start loosing money

source: google performance testing

Typical results:

  • All response codes 2**. Duration +/- the same when increasing the number of users.
  • All response codes 2**. Long durations of 1-2s.
  • All response codes 2**. Timeout errors.
  • Not all response codes 2**. Errors occur: 500.

What metric describes application performance?

Response Code and Duration.

in real world you will have 1000 of these numbers

=> Median vs Average

median is primarily used for skewed distributions

99th percentile: the value below which 99% of the observations may be found

{1, 2, 2, 2, ,3, 14}
  • mean: 2
  • average (arithmetic mean): 4
  • 99th percentile: 14

Apdex - application performance index

  • 0 = no users satisfied
  • 1 = all users satisfied

apdext = (satisfiedcount + tolerating count/2)/totalsamples

=> apdex3 = (60 + 30/2)/100 = .75

tolerable time = 4 x satisfied time

sources: newrelic and wikipedia

Load Generation

JMeter:

  • load test functional behaviour
  • measure performance

design your test plan in GUI

Thread Group: configures the number of users (number of threads)

HTTPRequest: request to send

To get Reports: add listeners

Exercise

JMeter simple request

find out how many concurrent requests our application can handle so that response time stays below <1s

Load Shaping

How many concurrent users were there really?

use the JMeter Listener plugin: jp@gc - Active users over time

there were only 2 concurrent users instead of 10 => ramp up time

not very intuitive => thread group: jp@gc - Ultimate Thread Group

Different types of performance tests

what we have done up until now is: Load Tests

so many users, for that amount of time

   /--------------\
  /                \
 /                  \
/                    \

Stress Tests:looking for a break point - there are so many users, the application breaks

increasing the number of users over time

we are putting more load than the requirements of the application defined

what if the response time is already in the frustrating range: that is already a break point

          |----
     |----|
|----|
|

Spike Tests: big amount of users starting to use the application at a specific time

how good can our application handle spikes

there is no behaviour in our system that creates instability in the system

     |     |
_____|_____|_____

Soak Tests or Endurance Tests: testing for a long period of time (like 10 users for 8 hours)

your application works fine for 1h, but all of a sudden it dies after 1h due to a memory leak

|-------------------------|
|                         |

What should be the load test duration?

most people would say: around 1h

there is no book that says how long the load test should be

use the real time performance monitoring to identify what is an acceptable load test duration

Listener: jp@gc - Response Time over Time

Is this thread a real user?

No!

don’t worry, we’ll fix it (after the break)

people don’t throw the same request immediately after the other

people have thinking time, send different requests: open a page, create an issue, go to another issue, …

=> Blazemeter for Chrome: records interactions with the browser and download as JMeter test plan (needs creation of a user at Blazemeter)

  • JMeter
  • Selenium
  • Taurus JMeter

Exercise:

refresh JIRA Kanban board with Blazemeter => 100% error rate in JMeter: problems with authentication => record JIRA login and add that step to the JMeter test plan

cookie based authentication using JSESSIONID cookie

it works in JMeter because we have this HTTP Cookie Manager in the Test Plan (most difficult in JMeter is cross site authentication)

Our measurement now measures: send request, network latency, server response time, network latency

Doesn’t measure: JS execution, DOM tree, rendering, painting in browser

can you measure the latter part: yes, JMeter can work with web drivers

What are we measuring (test data)?

we are using the same user 10 times

opening the same kanban board 10 times

=> Data diversity

use a CSV file with the data that needs to dynamically change in the test plan

JMeter test plan and Git and CI:

can be stored in version control, but files are so big, they are not that repo friendly

can be run from CI using the command line tool

Assertions

the kanban board should have at least 10 issues

How did we stress JIRA during the workshop?

New Relic: performance monitoring in production

(less) Typical Results: fast responses => functional errors

Testing gotchas we’ve seen

  • discard Response Time measurements with errors
  • measure all relevant requests
  • use representative test data
  • assertions to make sure you get what you expect
  • jmeter should be close in the network to application. Good bandwidth.

real measurement: don’t run in GUI mode, disable listeners => collect the raw results and analyse in JMeter

JMeter drawback

  • it is not a browser: it does not execute JS
  • single instance of JMeter can spawn couple of hundreds of threads (users)
  • => more users: use distributed JMeter
  • hard to track .jmx files in version control: Taurus helps here a bit

Signal to Noise

repeat 3 times and see if you have the same results

Store performance testing data

JMeter has the option to write data to InfluxDB

use Grafana to read from InfluxDB