2009-Jul-17 - Tips to Test Engineers

Early involvement during the project initial phase provides ample time for testers to gain good understanding on the scope and can also contribute in refining good reqmts.

Proactively initiate and involve in reqmts & design discussions & reviews. It helps a lot in gaining good understanding of the system.

Ensure test basis is ready before start of any test activity (Eg: SRS, Project overview is the basis for Test Strategy.)

Adopt structured approach in arriving at an appropriate test design for deriving effective test cases.

Ensure atleast 2 test techniques are applied for high risk reqmts.

Take every opportunity to apply test design at any test levels to derive good test coverage.

Adopt risk based test approach and classify tests based on risks & priority (H,M,L) during execution.

Document testcases with clear & concise steps, covering positive, negative, invalid & alternate flows with pre & post conditions.

Ensure the testcase documents are reviewed by internal and external reviewers.

Know the scope & risks/impact before starting the test cycle. It helps in better planning of tests.

Prepare test execution plan with estimated effort before start of test cycle.

Plan test execution such that high risk test cases are executed first so that critical defects are found early.

System testing must cover tests addressing the quality characteristics (FURPS).

Adopt out of box testing methods in test cycle apart from the normal tests such as “Error guessing / Exploratory” methods etc.

Test engineers must maintain personal bug dairy to record most important defects/scenarios that they come across during testing various applications during their test career so that it becomes handy for reference and also useful as pattern to reproduce in future testing of applications.

Lastly, make a habit of continuous learning and enhancing testing skills. Read atleast one article on testing daily.

2009-Mar-28 - Building Competency in the Test team

Many Software Organizations pose a major challenge in delivering the projects/products within committed timelines and meeting customer expectations. In making a project success in meeting the KSF of the project, the team contribution in technical, domain and process areas play a significant role. The team must be competent enough in all the 3 areas in order to deliver efficiently and effectively.

A project will pose a major risk in competency if the team has fairly fresh inexperienced staff, new technology intro, lack of required skilled staff etc., which will hamper the project deliveries.

Deployment of the competency FW/development program helps the project & test teams to build competency in defined areas and gain confidence in deliveries and customer expectations.

Test team competency plays a vital role in the projects. The competency of the team has its impact on the quality of deliveries based on the expertise knowledge & skills present in the team members. Higher the competency in the team, the higher the efficiency and effectiveness in the deliveries.

The need for competency development arises due to the following reasons: 

-High expectations from customer/management on project goals (KSF)

-lack of sufficient knowledge & skill in the team

-new & inexperienced members in the team

-customer feedback on the quality of deliverables

-new technology/advancement introduced in the project

-lack of sufficient domain knowledge in team

-functional knowledge is average

-process needs to be improved.

-post release defects are high

-Testers involvement late in projects

Basis for competency development:

2009-Mar-8 - Test design: Importance in Testing

Test design: Importance in Testing

Testing has always been complex and the most challenging part in the project life cycle. Though the timelines for various stages of project are defined during the kickoff phase, in the real time scenario testing cycle has always been crunched and shortened. Testing has to be well planned, prepared, designed, executed and tracked taking into consideration on the risks and timelines. 

Test design forms the crucial part of the testing phase, which uncovers defects well before the logical/physical test cases are derived. Test design provides a user based conceptual model in evolving the big picture of the system/subsystem in use. Based on TMap model, test design is derived from the test basis and test strategy.

Test design is a structured approach in selection and implementation of techniques to derive, from a specific test basis, test cases (logical and physical) that realize a specific coverage. 

Need for test design:

Most of the time, the test cases are derived based on testers knowledge/experience on the application or based on requirements. If the same application were given to another tester, he would derive some additional set of test cases. There is no consistency in the standard derivation of test cases. As the application grows with functionalities, maintenance of test cases becomes difficult, lot of redundant test cases, chances of important test cases being missed out, results in exhaustive test cases and coverage may not be adequate. In order to avoid this situation, structured approach in the derivation of test cases through test design helps in obtaining good set of test cases, better maintenance of test cases and results in good coverage.

Basis for test design:

Test basis forms the basis for determining the test strategy. Test strategy details the classification and application of appropriate test technique for the requirements based on risks and quality characteristics.

For the detailed test design, requirements & design forms the starting points in formulating the test design based on the selection of technique.

Characteristics of Test design techniques:

Test design technique	Quality Characteristics	Application area	Test Objective
Equivalence Partitioning	Functionality, Interoperability	All types of systems	To derive a set of test cases in a structured way, covering each partition of input class(es)
Boundary Value Analysis	Functionality	All types of systems, e.g. list displays All boundary requirements	Verification of boundary values
State Transition Test	Functionality	Screen transitions, GUI, Workflow applications, Technical automation and embedded software	Verification of state transitions
Decision table Test	Functionality	Complex processing	Processing logic: decision coverage in combination with path coverage / condition coverage.
Control Flow Test	Functionality, Usability, Interoperability	GUI, Workflow applications, Interactive parts everywhere within the system,	Verify the all-possible path testing.
Data Flow Test	Functionality	Integration between functions and data. Patient data handling Workflow testing Data driven appln Database applications	CRUD: on the basis of the lifecycle of data
Cause Effect - Graphing	Functionality	License combinations, Critical functionalities	Verify processing
Elementary Comparison Test	Functionality	Critical functionalities, Complex processing	Verification of functional paths and conditions.
Orthogonal Array Test	Functionality, Interoperability	GUI, Configuration parameters Installation, Interoperability	Interaction between various parameters combination, coverage effectiveness.
Process Cycle Test	Usability, Security, Suitability, Functionality	Workflow, User-manual, Service manual, Hospital procedures	Verify system behavior within organizational procedures
Test Use Cases	Functionality, Usability, Interoperability	Interactive parts everywhere within the system, Especially for interactive parts	Verification of user actions on system
Syntax Test Reporting	Functionality, Interoperability, Reliability, Usability	Reporting, GUI, Error messages Interfaces conformance test (interoperability)	Interaction between system and user
Real-life test	Reliability (Availability, Maintainability)	All products and components, simulation of practical use	Verification of the system in a real-life situation
Error Guessing	All	Widely applicable	Investigate areas in the system where defects are expected.
Exploratory testing	All, but mainly Reliability, Availability, Maintainability, Functionality, Usability	End-user tests Work in progress Non-functional	To look for defects additional to or instead of formal testing.

Test case derivation flow diagram:    

2008-Apr-16 - Challenges Faced while Testing in Healthcare domain.

Testing has always been challenging in the software development life cycle of a product irrespective of any area of domain. Testing Healthcare products requires more of domain knowledge and testing in general is very critical in the life cycle stage of any software development activity.

Healthcare products pose much more challenges for its complexity in design, development and testing,  patient data, diagnosis results, safety characteristics than compared to other domain products. On the other hand, Healthcare products has to comply for Safety and Regulatory standards and implement i.e, DICOM, HL7, IHE, HIPAA standards which makes good business sense to be in market competing with the other vendors. This in turn makes the test engineers face with multiple challenges while testing the  software and delivering to the customer with high quality.

Some of the challenges faced while testing in the Healthcare domain are listed below.

2008-Apr-13 - Robustness Test Framework (HAST)

Testing is now an accepted and vital part of any software development project. In fact, accurate testing is essential in order for the software to be implemented successfully.

Automated testing is recognized as a cost-efficient way to increase application reliability, while reducing the time and cost of software quality programs. Automated tests can be designed to *******te any user/application activity in a high-speed and easily *******ted environment.

However, not all tests should, or can be, automated. A software tool cannot automate certain processes that require human intervention, such as loading special paper into a printer, image quality, etc. Manual tests are those that require human intervention to perform a test procedure.

Healthcare product testing requires more of domain knowledge, and testing in general is critical in the life cycle stage of any software development activity. In today's situation, healthcare software is gaining more importance and is becoming more complex with the incorporation of new features and design based on customer feedback and market needs.

During the ongoing product development phases, it is also necessary to focus on the CTQ (Critical To Quality) of the system, i.e. performance and reliability parameters that define the system availability to the end user. Testing for robustness, performance and reliability are the critical focus areas and are very much necessary when developing healthcare software.

In the new product development lifecycle, testing for robustness, performance and reliability takes a backseat in the early stages of the product development and is rigorous during the later and final stages of the testing cycle. To avoid repeatedly crashing and system non-availability issues during later phases, you should catch these issues during early phases of development. This can be achieved using an automated test framework called HAST -- Highly Accelerated Stress Testing.

HAST is an automated framework of running cycling scenarios (derived from system-level use cases) of an Application Under Test (AUT) on an independent test machine to derive performance and reliability growth during the product development cycle. HAST is an essential part of any product development cycle that aims to deliver a reliable product at the end. It is run continuously over a longer period of time to determine the robustness of the system.

HAST is meant solely for robustness and reliability testing. The whole idea is to have the system perform use case operations repeatedly until it breaks down. It is more significant in hardware systems than purely software systems. In healthcare scanner systems, HAST scans and generates images repeatedly to ensure the hardware/software/firmware interaction can withstand the continuous scanning.

The infrastructure for building HAST is simply the ability to execute the functionality repeatedly without manual intervention.

• For UI based systems, it is achieved by GUI tools such as Quick Test Pro without any code change.
• For non-UI subsystems, it is driven by script providing hooks (interface) within the code to achieve automation of the use cases.

HAST helps identify reliability issues early in the development phase. Later, when the application is stable, helps in defining the Mean Time Between Crashes (MTBC) for the systems.

Needs for HAST

• To beat the system by simulating simultaneous/concurrent operations on the system.
• To detect critical system issues such memory issues, response time, application hangs and crashes, recovery issues, unknown exceptions, deadlocks, and performance issues early in the product development phase, which are impossible to find in manual testing.
• To simulate workflow scenarios, increase load and simulate concurrency on the system.
• To derive performance and reliability growth of the system under load conditions.

Building HAST
Identify a simple use case/workflow scenario for the AUT and develop automated use case for the same using a test script/test tool. Start the application and run the use case for "N" number of iterations and parse data from log files.

Infrastructure for HAST setup

• Hardware and software -- Test System and AUT
• Create HAST scripts -- QTP test scripting or any other Scripting technique.
• Parser scripts-- Parse log files to extract perf & reli data and publish on dashboard.
• Application logs -- Availability of structured logging mechanism in recording transactions, delays, time outs, etc.

Benefits of HAST

• It helps determine the stability of the application.
• It covers complete functional use case coverage.
• It helps find critical issues during the early stages of product development.
• Any user familiar with the operation can operate HAST.
• It helps determine the performance of the system.
• It helps derive the performance and reliability growth of the system.
• As a framework, it can be deployed on any of the domain applications (platforms, Web-based, etc.)

Simple use case of healthcare application for HAST operation

1. Start AUT
2. Create New Examination for a patient
3. Prepare for scanning
4. Start the scan procedure
5. Select the scanned exam and perform import and export operation to network & archive hosts (DVD, PACS).
6. Open viewer and perform viewing operations (change layout, zoom, pan, window leveling)
7. Perform post-processing operations
8. Print the images
9. Clean up the job queue for finished jobs
10. Format media
11. Delete the patient from database
12. Repeat steps 2-12

2008-Mar-24 - Testers involvement in the Requirements Review Process

In this increasingly complex software development era, it is extremely important to include Testing as early in the project as possible. In the predictable old Waterfall lifecycle world, testing was typically included in the coding phase right before they were needed.  In today’s iterative world, testing needs to be included at the project’s inception. 

Even today the Requirements Phase of a project generates most of the defects.  Over 50% of the defects are injected during the requirements phase.  This is important since this phase forms the foundation for the product that more of the later work will build on.  So, if the requirements are not correct there may be fundamental issues that are not surfaced until the later phases, which may require complete redesign of major portions of the product.  The amount of rework to fix assumptions or undiscovered issues can have a major impact on a project.  

Let’s first look at some of the effects it has on testing with the testers involvement during the later phases: 

• Testers understanding on the reqmts will be limited and tests designed may not be robust.
• Reqmts defects get carried on to next phase due to the insufficient review process.
• Testers need to learn the application on the fly from the reqmts.
• Due to late involvement, the risk analysis on the reqmts are skipped, the tests designed may not be of adequate coverage.
• Reqmts based testing gets skipped and most of non-clarity, measurable, consistency and ambiguity reqmts are not corrected.
• Traceability, relationships, correlation, and dependency in reqmts are not traced.
• Rapidly accelerated project cycles result in many requirement gray areas that force testers to determine expected results.  

The above issues convey the need for testing to be active participants in the Requirements Review process.  Testers should not be passive recipients of requirements in the form of a Use Case or a requirements document.  By involving testing in the requirements phase (such as Requirement Workshops, Review Sessions, Interviews, etc), Testing will benefit by the following:

·        The Test Plan will include more than a regurgitation of the Project Features.  By understanding the requirements as they are being created, the test plan will include a better plan of attack for testing each of the areas of functionality. 

·        As the project progresses, Testing will typically be asked to refine their resource estimates.  Understanding the requirements as they are being created will help refine resource estimates with greater confidence.  Even if test cases are not yet developed, you can get a sense of the number of requirements and estimate the necessary number of test cases and multiply by your time factor to create and execute test cases. 

·        The result of Testers attending requirements review sessions will result in clearer, concise requirements.  For example, is the requirement testable?   Another example is clarifying the gray area requirements that frequently get overlooked, for example, Error Messages, Usability issues, etc. 

·        Participation in requirements review allows Testers to capture the feelings about certain requirements that Testers cannot get from an after the fact Requirements review.  Feelings are not captured as requirement attributes.  Testers can also hear about how development feels about certain requirements.  This dichotomy is essential for risk based testing.  If you don’t have enough time to test the entire system, test efforts may focus on:   

o   Requirements that are important to the business (high priority) 

And 

o   Requirements that make development uneasy (high risk).

·        While the System Spe******t's role is to elicit requirements, Testers may better understand the impact of the requirement on other parts of the system.  Participation from Testing in requirements elicitation sessions may result in fewer changing requirements throughout the project, and fewer design problems.    

·        Understanding the requirements can result in a jump-start on Test Case creation.  You may be able to create several tests with steps from basic flows of a Use Case.  This preparation may allow time later on to automate between iteration releases. 

·        Understanding the requirements (and the interdependencies between requirements) will make it easier for Testing to set test case to business requirement traceability. 

·        Part of the role of Testing is to ensure that projects adhere to the software development process that the organization has in place.  Requirements review is a distinct activity and testing should ensure that it is performed and that it is performed correctly.

If testing passively waits until requirements are published, it may already be too late.  The time needed to then learn, question and understand the requirements eats into precious time necessary for Test Plan and Manual / Automated Test Case development. 

If your organization already includes testers in Requirements phase, keep up the good work. If not, help the leaders understand the importance of early involvement and the value add that Testing gives this activity.  The benefits far outweigh the costs.