Context-Driven Testing: Why it Matters More Than Ever


Testing is a cognitive profession, not a tool-first profession. Using your brain is the basis for everything, for some reason people keep forgetting that.

Let’s go back to the basics of what testing is supposed to be doing. “Oh, how convenient, a machine can now do testing for me, let me send my brain cells on a well-earned holiday”.

If that is you, please, don’t be like that. As a tester, your brain is your most important asset, so it pains me to see that so many people in the testing community seem eager to offload important cognitive testing tasks to a machine that doesn’t care, feel, or can take responsibility.

Using LLM’s without question or care is highly irresponsible. This is ironic for us testers because we are supposed to be the critical realists in any project. We should aim to find problems, not blindly let a machine do…whatever.

As an antidote, let’s go back to the basics of what testing is supposed to be doing. Let me show you how and why you can use the Context-Driven Testing approach in your work. Using your brain isn’t optional for this, it’s the basis for everything.

I’ll keep repeating this: testing is a cognitive profession, and tools play a smaller role in doing it well than you think.


Context Driven Testing (CDT)

The context-driven testing approach is about doing the best we can with what we get.

Circumstances differ, so the testing approaches will differ

– James Bach, Brian Marick, Bret Pettichord, Cem Kaner

based on the book: Taking Testing Seriously, James Bach and Michael Bolton

testing cannot ever be complete, catching all the bugs is not possible => risk-based testing

Comparison

Agile Testing vs CDT

Agile Testing Context-driven Testing
follows principles of agile software development agnostic of software methodology
a practice tied to agile methodology a testing philosophy, not tied to a certain practice methodology
a set of principles a model for deciding what to test and why
can be prescriptive: advocates unit testing, automated testing, testing in the sprint (cfr. scrum) flexible, depends on context what the tester does
team collab “people working together are the most important part”

agile testing puts a lot of focus on delivery

agile testing: team collab, CDT shares the same idea

Does this matter? No

CDT is about learning, does not assume a context

Agile Testing, assumes the agile context, risks to narrow the testing to the automation side

in practice, agile testing and CDT philosophy can complement each other

what matters more: the degree of freedom you get to shape the testing process

Remember: we can be a CDT tester in any type of software development methodology, when we get a large degree of freedom to shape the testing

CDT principles

  1. The value of any practice depends on its context.
  2. There are good practices in context, but there are no best practices.
  3. People, working together, are the most important part of any project’s context.
  4. Projects unfold over time in ways that are often not predictable.
  5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
  6. Good software testing is a challenging intellectual process.
  7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

https://context-driven-testing.com/

2. there are good practices in context, but there are no best practices

how to recognise a CDT tester: ask them about testing: “It depends”

3. people are central, working together are the most important part of any context

a machine cannot test, no emotions, it cannot evaluate, it cannot learn

a person creates a story about the product learning, intelligence are marketing terms to fool us that machines can learn however, they can’t and have no accountability => The Accountability Gap

people are responsible for testing

which people? it depends; not only our team, it is also leadership (NARRATOR: see EU NIS2)

enormous risk: code generated by LLMs, test code generated by LLMs, test cases are generated by LLMs; who cares about the risks? who get the blame when something goes wrong in production

5. the product is a solution. If the problem isn’t solved, the product doesn’t work

6. good software testing is a challenging intellectual process

principle 5 and 6 are in contention, because of LLMs

=> an eight principle

8. none of the 7 principle matters if the story of testing is being ignored

=> Fake Testing ties back to the accountability displacement

in this AI era truth and evidence have never been so popular as now if testers do not find bugs, managers cannot be blamed for bugs hence the popularity of Fake Testing

assumption: where you work, testing is taken seriously and findings are handled

Core Skill: How to tell a compelling story

It’s about credibility towards management

we are:

  • a diplomat
  • an influencer
  • a story-teller

Core Skill: Work on your story teller

a few of our important weapons in the AI era

distill a story about why this matters right now for the product, why the business should care

  • gather evidence
  • keep your audience in mind
  • curate the story

better not differ to an LLM, this is the critical thinking

Risks

remember how Fake Testing matters to management => need to be fast in politics

  • beware of fake testing and who benefits from that

  • do NOT put yourself in the position of the business

    => creates an accountability displacement

    decision to release is a business decision not a tester’s decision

three-level testing story

  • the product story: quality status of the product
  • the process story: how did we test
  • the value story: explain the risks

Remember: build credibility that we are testing well & know our audience to support the business

Core Skill: How to create a Test Strategy and why you should care

it is easier to build a story when there is a Test Strategy

  • high level: use heuristics to get started
  • living document: must be updated on what we learn
  • a model: not complete, but an attempt to make our testing priorities explicit

Heuristics

  • See it as a way to build a mental model of what matters (open to change!)
  • SFDIPOT
  • test, take notes, question
  • risk-assessment
  • Heuristic Test Strategy Model

spend some time using the application, and take notes, feel the questions and write that down

talk to the people who know the product well and let them explain the product to build a mental model

the Test Strategy can be read by the team, they can amend it, do proposals

strategies can change!

Core questions

  • why are we here?
  • we got the required skills?
  • what is hindering good testing?

    -> always try to improve good testing or at least improve what is hindering good testing

  • what is already being done

    do not trust existing testing artefacts, they can be wrong

  • what is expected from us

Avoid this: a test strategy is not an utopia; do not promise what we cannot deliver. Remember: credibility!

Mitigate: explain the time constraint vs risks we perceive: this is an important story!

Juniors

Start by learning more about risks and risk assessment. Avoid doing “math” with risk.

I see people doing maths with risks, risk mitigation is mostly based on consensus

Mediors/Seniors

should have a strong mental model about our testing approach in any context

Remember: a test strategy is based on heuristics it is exploratory, constantly evolves and different by each execution

Core Skill: Automation in Testing: how a Context-driven tester sees it

there is this weird idea that CDT-ers are against automated testing; are we skeptical? Yes!

Core Skill: automation should support good testing

testing is a cognitive process, it is not a mechanical, repetitive process

Mainstream Paradigm

  • “automate the user” -> automate user intend
  • “automate testing”

it is very reductionist

CDT

  • tools cannot replace humans
  • assess the trade-offs

Irony

take into account the real cost of the test automation, the time and energy put in it

to understand whether automation creates value, we first need to be proficient in the basics of testing -> ZERO to do with automation

risk based testing, decision making and recognition has nothing to do with test automation

Remember: we are looking for trouble

Is it worth it?

trade-offs should be painfully obvious: the more time we spend automating the less time we have to spend on our core mission

Core Mission

look for new trouble

find NEW information

(instead of being in the verification business 🙄)

What I do

  • ruthless pragmatism: we have to have a testing goal in mind
  • can I speed it up?
  • code quality varies greatly

Remember: Risk of automation is that people think or pretend that this is all that testing is. be wary of reductionism

Remember: a CDT tester is pro-tool but not a tool maxxer. they are aware of the tradeoffs

be skeptical of tool vendors that make grandiose claims

This was an incomplete look at CDT.

Please read the book!

key take aways

  • study: mull over the CDT principles
  • do/try: apply CDT
  • tdo/try: story-telling
  • remember: the human is at the center

Questions

I am the only QA in a team of nine (six engineers). I face constant time pressure. AI speeds up my testing, but I feel I’m loosing deeper understanding of new projects. How can I balance efficiency with gaining solid system knowledge?

the speed fallacy

you should not do all the testing alone 💪

take it up with leadership, they should fix it (NARRATOR: see Deming’s systemic problems are created by leadership) ✔️

Regarding the identificato of test cases for automation, do you rely on specific metrics and objective criteria, to evaluate test cases and features; or is the selection primarily driven by experience and engineering judgment?

CDT testers do not need test cases, they have test goals, so it is primarily based on experience

it requires knowledge, when knowledge lacks we have to talk with stakeholders and the team

Is there a way to exploit Fake Testing?

whenever I encountered people that don’t know what testing is I tried to show them what it really is -> let’s try exploratory