Software (SaMD) mobile application verification testing: objective evidence

Icculus · Aug 27, 2021

Hello - I'm working on performing verification testing for medical device software applications. There's a debate on what qualifies as objective evidence that the software complies with the design/specs/requirements. In the past I've used screenshots as objective evidence. However, some believe that attestations of the test result (e.g. basically a signoff that the test passes and the expected result is met) can be used as objective evidence. The concern with screenshots is that it's incredibly time consuming and inefficient, so we're wondering if auditors value screenshots or if they are actually required.

Does anybody have experience performing software application validation testing and methods of collecting objective evidence?
Thanks.

Ninja · Aug 27, 2021

Moved to Medical software section for higher and more applicable visibility.

yodon · Aug 27, 2021

The goal behind objective evidence should be to allow a reviewer (with sufficient knowledge) to draw the same conclusion made by the tester. Anything less than that is less than ideal... which, frankly, is a lot of what I've seen.

Kind of a side note here: one thing I've seen with screenshots is that people tend to verify the implementation rather than the requirements. See a button, take a screenshot (but is that really the requirement)? My experience has been that if you (re)focus on the requirements the number of screenshots is reduced (and you can use attestation along the way to get there; e.g., click the button).

It's been my experience, most times in (regulatory) reviews, that the reviewers tend to "weigh" the results (is the paper stack high enough). I kind of get that since such results would be quite tedious to go through (exacerbated if they have to jump to separate screenshots).

The bottom line is "who knows" what a reviewer/auditor will care about. Strive for ideal but balance with practicality. But don't let practicality completely win.

v9991 · Feb 14, 2023

yodon said:
The goal behind objective evidence should be to allow a reviewer (with sufficient knowledge) to draw the same conclusion made by the tester. Anything less than that is less than ideal... which, frankly, is a lot of what I've seen.

This is very simple - straight forward definition; I agree; :agree:

there is opportunity to further optimize the need of 'objective evidence' ( screenshots), through use of the application activity log and audit trails. ( i.,e., pre-requisite that they are non-editable - secured - traceable tec )

yodon said:
The bottom line is "who knows" what a reviewer/auditor will care about. Strive for ideal but balance with practicality. But don't let practicality completely win.

this happens because, broadly, there are people who approach this through procedural controls vs systematic controls vs process based controls., i.e.,
* 1. did the test script define how to test / challenge = voluminous documentation.
*2 did the testing itself happened in a way to simulate/sequence/flow of the process = once again volumnous documentation/.
* 3. did the test script define the expected/required outcome = right fit of screen shots ( as the flow/sequence/pre requisites are defined either in user manuals or sops or urs or config document etc

by the way, can you please help me direct/point out to any guidance/standard pertaining to the objective evidence. this is one thing which atleast gives atleast an neutral platform to engage an open discussion with stake holders.

Icculus · Feb 14, 2023

yodon said:
"...if you (re)focus on the requirements the number of screenshots is reduced...Strive for ideal but balance with practicality. But don't let practicality completely win.

This is great advice and makes perfect sense. The practicality also becomes a function of how complicated the software actually is. When verifying requirements, having 100's or 1000's of requirements also yields an impractical - or even impossible - level of manual testing in terms of collecting screenshots. At this point it becomes a practice in audit/compliance risk.

v9991 · Feb 15, 2023

v9991 said:
there is opportunity to further optimize the need of 'objective evidence' ( screenshots), through use of the application activity log and audit trails. ( i.,e., pre-requisite that they are non-editable - secured - traceable tec )

by the way, can you please help me direct/point out to any guidance/standard pertaining to the objective evidence.
this is one thing which atleast gives an neutral platform to engage an open discussion with stake holders.[/QUOTE]

v9991 · Feb 16, 2023

v9991 said:
This is very simple - straight forward definition; I agree;
there is opportunity to further optimize the need of 'objective evidence' ( screenshots), through use of the application activity log and audit trails. ( i.,e., pre-requisite that they are non-editable - secured - traceable tec )
.

there is one more much simpler tool to capture the both the objective evidence and the sequence of steps followed to test the script.
"steps recorder " an standard feature of the windows. ( from Win 7 onwards ) . ( enclosed is an example for it , where it captures each click and screen for same )

"C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Accessories\Steps Recorder.lnk"

what are your thoughts on using this. ( provided we make the saved document non-editable and traceable ; which can be complimented by application/system log and audit trails )

Software (SaMD) mobile application verification testing: objective evidence

Icculus

Starting to get Involved

Ninja

Looking for Reality

yodon

v9991

Icculus

Starting to get Involved

v9991

v9991

Attachments

Similar threads