Testing is a cognitive profession, not a tool-first profession. Using your brain is the basis for everything, for some reason people keep forgetting that.
Let’s go back to the basics of what testing is supposed to be doing. “Oh, how convenient, a machine can now do testing for me, let me send my brain cells on a well-earned holiday”.
If that is you, please, don’t be like that. As a tester, your brain is your most important asset, so it pains me to see that so many people in the testing community seem eager to offload important cognitive testing tasks to a machine that doesn’t care, feel, or can take responsibility.
Using LLM’s without question or care is highly irresponsible. This is ironic for us testers because we are supposed to be the critical realists in any project. We should aim to find problems, not blindly let a machine do…whatever.
As an antidote, let’s go back to the basics of what testing is supposed to be doing. Let me show you how and why you can use the Context-Driven Testing approach in your work. Using your brain isn’t optional for this, it’s the basis for everything.
I’ll keep repeating this: testing is a cognitive profession, and tools play a smaller role in doing it well than you think.
Context Driven Testing (CDT)
The context-driven testing approach is about doing the best we can with what we get.
Circumstances differ, so the testing approaches will differ
– James Bach, Brian Marick, Bret Pettichord, Cem Kaner
based on the book: Taking Testing Seriously, James Bach and Michael Bolton
testing cannot ever be complete, catching all the bugs is not possible => risk-based testing
Comparison
Agile Testing vs CDT
| Agile Testing | Context-driven Testing |
|---|---|
| follows principles of agile software development | agnostic of software methodology |
| a practice tied to agile methodology | a testing philosophy, not tied to a certain practice methodology |
| a set of principles | a model for deciding what to test and why |
| can be prescriptive: advocates unit testing, automated testing, testing in the sprint (cfr. scrum) | flexible, depends on context what the tester does |
| team collab | “people working together are the most important part” |
agile testing puts a lot of focus on delivery
agile testing: team collab, CDT shares the same idea
Does this matter? No
CDT is about learning, does not assume a context
Agile Testing, assumes the agile context, risks to narrow the testing to the automation side
in practice, agile testing and CDT philosophy can complement each other
what matters more: the degree of freedom you get to shape the testing process
Remember: we can be a CDT tester in any type of software development methodology, when we get a large degree of freedom to shape the testing
CDT principles
- The value of any practice depends on its context.
- There are good practices in context, but there are no best practices.
- People, working together, are the most important part of any project’s context.
- Projects unfold over time in ways that are often not predictable.
- The product is a solution. If the problem isn’t solved, the product doesn’t work.
- Good software testing is a challenging intellectual process.
- Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
https://context-driven-testing.com/
2. there are good practices in context, but there are no best practices
how to recognise a CDT tester: ask them about testing: “It depends”
3. people are central, working together are the most important part of any context
a machine cannot test, no emotions, it cannot evaluate, it cannot learn
a person creates a story about the product learning, intelligence are marketing terms to fool us that machines can learn however, they can’t and have no accountability => The Accountability Gap
people are responsible for testing
which people? it depends; not only our team, it is also leadership (NARRATOR: see EU NIS2)
enormous risk: code generated by LLMs, test code generated by LLMs, test cases are generated by LLMs; who cares about the risks? who get the blame when something goes wrong in production
5. the product is a solution. If the problem isn’t solved, the product doesn’t work
6. good software testing is a challenging intellectual process
principle 5 and 6 are in contention, because of LLMs
=> an eight principle
8. none of the 7 principle matters if the story of testing is being ignored
=> Fake Testing ties back to the accountability displacement
in this AI era truth and evidence have never been so popular as now if testers do not find bugs, managers cannot be blamed for bugs hence the popularity of Fake Testing
assumption: where you work, testing is taken seriously and findings are handled
Core Skill: How to tell a compelling story
It’s about credibility towards management
we are:
- a diplomat
- an influencer
- a story-teller
Core Skill: Work on your story teller
a few of our important weapons in the AI era
distill a story about why this matters right now for the product, why the business should care
- gather evidence
- keep your audience in mind
- curate the story
better not differ to an LLM, this is the critical thinking
Risks
remember how Fake Testing matters to management => need to be fast in politics
-
beware of fake testing and who benefits from that
-
do NOT put yourself in the position of the business
=> creates an accountability displacement
decision to release is a business decision not a tester’s decision
three-level testing story
- the product story: quality status of the product
- the process story: how did we test
- the value story: explain the risks
Remember: build credibility that we are testing well & know our audience to support the business
Core Skill: How to create a Test Strategy and why you should care
it is easier to build a story when there is a Test Strategy
- high level: use heuristics to get started
- living document: must be updated on what we learn
- a model: not complete, but an attempt to make our testing priorities explicit
Heuristics
- See it as a way to build a mental model of what matters (open to change!)
- SFDIPOT
- test, take notes, question
- risk-assessment
- Heuristic Test Strategy Model
spend some time using the application, and take notes, feel the questions and write that down
talk to the people who know the product well and let them explain the product to build a mental model
the Test Strategy can be read by the team, they can amend it, do proposals
strategies can change!
Core questions
- why are we here?
- we got the required skills?
-
what is hindering good testing?
-> always try to improve good testing or at least improve what is hindering good testing
-
what is already being done
do not trust existing testing artefacts, they can be wrong
- what is expected from us
Avoid this: a test strategy is not an utopia; do not promise what we cannot deliver. Remember: credibility!
Mitigate: explain the time constraint vs risks we perceive: this is an important story!
Juniors
Start by learning more about risks and risk assessment. Avoid doing “math” with risk.
I see people doing maths with risks, risk mitigation is mostly based on consensus
Mediors/Seniors
should have a strong mental model about our testing approach in any context
Remember: a test strategy is based on heuristics it is exploratory, constantly evolves and different by each execution
Core Skill: Automation in Testing: how a Context-driven tester sees it
there is this weird idea that CDT-ers are against automated testing; are we skeptical? Yes!
Core Skill: automation should support good testing
testing is a cognitive process, it is not a mechanical, repetitive process
Mainstream Paradigm
- “automate the user” -> automate user intend
- “automate testing”
it is very reductionist
CDT
- tools cannot replace humans
- assess the trade-offs
Irony
take into account the real cost of the test automation, the time and energy put in it
to understand whether automation creates value, we first need to be proficient in the basics of testing -> ZERO to do with automation
risk based testing, decision making and recognition has nothing to do with test automation
Remember: we are looking for trouble
Is it worth it?
trade-offs should be painfully obvious: the more time we spend automating the less time we have to spend on our core mission
Core Mission
look for new trouble
find NEW information
(instead of being in the verification business 🙄)
What I do
- ruthless pragmatism: we have to have a testing goal in mind
- can I speed it up?
- code quality varies greatly
Remember: Risk of automation is that people think or pretend that this is all that testing is. be wary of reductionism
Remember: a CDT tester is pro-tool but not a tool maxxer. they are aware of the tradeoffs
be skeptical of tool vendors that make grandiose claims
This was an incomplete look at CDT.
Please read the book!
key take aways
- study: mull over the CDT principles
- do/try: apply CDT
- tdo/try: story-telling
- remember: the human is at the center
Questions
I am the only QA in a team of nine (six engineers). I face constant time pressure. AI speeds up my testing, but I feel I’m loosing deeper understanding of new projects. How can I balance efficiency with gaining solid system knowledge?
the speed fallacy
you should not do all the testing alone 💪
take it up with leadership, they should fix it (NARRATOR: see Deming’s systemic problems are created by leadership) ✔️
Regarding the identificato of test cases for automation, do you rely on specific metrics and objective criteria, to evaluate test cases and features; or is the selection primarily driven by experience and engineering judgment?
CDT testers do not need test cases, they have test goals, so it is primarily based on experience
it requires knowledge, when knowledge lacks we have to talk with stakeholders and the team
Is there a way to exploit Fake Testing?
whenever I encountered people that don’t know what testing is I tried to show them what it really is -> let’s try exploratory