Original post by Ahamad and software by Elliott Back

Work with management early on to understand how requirements might change, so that alternate test plans and strategies can be worked out in advance. It is helpful if the application’s initial design allows for some adaptability, so that later changes do not require redoing the application from scratch. Additionally, try to…

- In the project’s initial schedule, allow for some extra time to

A formal test plan is a document that provides and records important information about a test project, for example:

1. Project assumptions
2. Project background information
3. Available resources
4. Project Schedule
5. Entry and exit criteria
6. Test milestones
7. Use cases and/or test cases

For a range of reasons — both good and bad — many software and web development projects don’t budget enough time for complete and comprehensive testing. A quality test team must be able to test a product or system quickly and constructively in order to provide some value to the project. This essay describes how to test a web site or application in the absence of a detailed test plan and facing short or unreasonable deadlines.


Identify High-Level Functions First
High-level functions are those functions that are most important to the central purpose(s) of the site or application. A test plan would typically provide a breakdown of an application’s functional groups as defined by the developers; for example, the functional groups of a commerce web site might be defined as shopping cart application, address book, registration/user information, order submission, search, and online customer service chat. If this site’s purpose is to sell goods online, then you have a quick-and-dirty prioritization of:

1. Shopping cart - credit card validation and security.
2. Registration/user information
3. Taking Orders
4. Search the site
5. Online customer service like chat, email etc

I’ve prioritized these functions according to their significance to a user’s ability to complete a transaction. I’ve ignored some of the lower-level functions for now, such as the modify shopping cart quantity and edit saved address functions because they are a little less important than the higher-level functions from a test point-of-view at the beginning of testing.
Your opinion of the prioritization may disagree with mine, but the point here is that time is critical and in the absence of defined priorities in a test plan, you must test something now. You will make mistakes, and you will find yourself making changes once testing has started, but you need to determine your test direction as soon as possible.

Test Functions Before Display
Any web site should be tested for cross-browser and cross-platform compatibility — this is a primary rule of web site quality assurance. However, wait on the compatibility testing until after the site can be verified to just plain work. Test the site’s functionality using a browser/OS/platform that is expected to work correctly — use what the designers and coders use to review their work.

Concentrate on Ideal User Actions First
Ideal User Actions are those actions and steps most likely to be performed by users. For example, on a typical commerce site, a user is likely to

1. identify an item of interest
2. add that item to the shopping cart
3. buy it online with a credit card
4. ship it to himself/herself

Now, this describes what the user would want to do, but many sites require a few more functions, so the user must go through some more steps, for example:

1. login to an existing registration account (if one exists)
2. register as a user if no account exists
3. provide billing & bill-to address information
4. provide ship-to address information
5. provide shipping & shipping method information
6. provide payment information
7. agree or disagree to receiving site emails and newsletters

Most sites offer (or force) an even wider range of actions on the user:

1. change product quantity in the shopping cart
2. remove product from shopping cart
3. edit user information (or ship-to information or bill-to information)
4. save default information (like default shipping preferences or credit card information)

All of these actions and steps may be important to some users some of the time (and some developers and marketers all of the time), but the majority of users will not use every function every time. Focus on the ideal path and identify those factors most likely to be used in a majority of user interactions.

Concentrate on Intrinsic Factors First
Intrinsic factors are those factors or characteristics that are part of the system or product being tested. An intrinsic factor is an internal factor. So, for a typical commerce site, the HTML page code that the browser uses to display the shopping cart pages is intrinsic to the site: change the page code and the site itself is changed. The code logic called by a submit button is intrinsic to the site.
Extrinsic factors are external to the site or application. Your crappy computer with only 8 megs of RAM is extrinsic to the site, so your home computer can crash without affecting the commerce site, and adding more memory to your computer doesn’t mean a whit to the commerce site or its functioning.
Given a severe shortage of test time, focus first on factors intrinsic to the site:

1. does the site work?
2. do the functions work? (again with the functionality, because it is so basic)
3. do the links work?
4. are the files present and accounted for?
5. are the graphics MIME types correct? (I used to think that this couldn’t be screwed up)

Once the intrinsic factors are squared away, then start on the extrinsic points:

1. cross-browser and cross-platform compatibility
2. clients with cookies disabled
3. clients with javascript disabled
4. monitor resolution
5. browser sizing
6. connection speed differences

The point here is that with myriad possible client configurations and user-defined environmental factors to think about, think first about those that relate to the product or application itself. When you run out of time, better to know that the system works rather than that all monitor resolutions safely render the main pages.

Boundary Test From Reasonable to Extreme
You can’t just verify that an application works correctly if all input and all actions have been correct. People do make mistakes, so you must test error handling and error states. The systematic testing of error handling is called boundary testing (actually, boundary testing describes much more, but this is enough for this discussion).
During your pedal-to-the-floor, no-test-plan testing project, boundary testing refers to the testing of forms and data inputs, starting from known good values, and progressing through reasonable but invalid inputs all the way to known extreme and invalid values.

Good Values
Enter in data formatted as the interface requires. Include all required fields. Use valid and current information (what “valid and current” means will depend on the test system, so some systems will have a set of data points that are valid for the context of that test system). Do not try to cause errors.

Expected Bad Values
Some invalid data entries are intrinsic to the interface and concept domain. For example, any credit card information form will expect expired credit card dates — and should trap for them. Every form that specifies some fields as required should trap for those fields being left blank. Every form that has drop-down menus that default to an instruction (”select one”, etc.) should trap for that instruction. What about punctuation in name fields?

Reasonable and Predictable Mistakes
People will make some mistakes based on the design of the form, the implementation of the interface, or the interface’s interpretation of the relevant concept domain(s). For example, people will inadvertently enter in trailing or leading spaces into form fields. People might enter a first and middle name into a first name form field (”Mary Jane”).
Not a mistake, per se, but how does the form field handle case? Is the information case-sensitive? Or does the address form handle a PO address? Does the address form handle a business name?

Compatibility Test From Good to Bad
Once you get to cross-browser and cross-platform compatibility testing, follow the same philosophy of starting with the most important (as defined by prevalence among expected user base) or most common based on prior experience and working towards the less common and less important.
Do not make the assumption that because a site was designed for a previous version of a browser, OS, or platform it will also work on newer releases. Instead, make a list of the browsers and operating systems in order of popularity on the Internet in general, and then move those that are of special importance to your site (or your marketers and/or executives) to the top of the list.


The Drawbacks of This Testing Approach
Many projects are not mature and are not rational (at least from the point-of-view of the quality assurance team), and so the test team must scramble to test as effectively as possibly within a very short time frame. I’ve spelled out how to test quickly without a structured test plan, and this method is much better than chaos and somewhat better than letting the developers tell you what and how to test.
This approach has definite quality implications:

1. Incomplete functional coverage – this is no way to exercise all of the software’s functions comprehensively.
2. No risk management – this is no way to measure overall risk issues regarding code coverage and quality metrics. Effective quality assurance measures quality over time and starting from a known base of evaluation.
3. Too little emphasis on user tasks – because testers will focus on ideal paths instead of real paths. With no time to prepare, ideal paths are defined according to best guesses or developer feedback rather than by careful consideration of how users will understand the system or how users understand real-world analogues to the application tasks. With no time to prepare, testers will be using a very restricted set input data, rather than using real data (from user activity logs, from logical scenarios, from careful consideration of the concept domain).
4. Difficulty reproducing – because testers are making up the tests as they go along, reproducing the specific errors found can be difficult, but also reproducing the tests performed will be tough. This will cause problems when trying to measure quality over successive code cycles.
5. Project management may believe that this approach to testing is good enough – because you can do some good testing by following this process, management may assume that full and structured testing, along with careful test preparation and test results analysis, isn’t necessary. That misapprehension is a very bad sign for the continued quality of any product or web site.
6. Inefficient over the long term – quality assurance involves a range of tasks and foci. Effective quality assurance programs expand their base of documentation on the product and on the testing process over time, increasing the coverage and granularity of tests over time. Great testing requires good test setup and preparation, but success with the kind testplan-less approach described in this essay may reinforce bad project and test methodologies. A continued pattern of quick-and-dirty testing like this is a sign that the product or application is unsustainable in the long run.

Original post by Ahamad and software by Elliott Back

Verification ensures the product is designed to deliver all functionality to the customer; it typically involves reviews and meetings to evaluate documents, plans, code, requirements and specifications; this can be done with checklists, issues lists, walkthroughs and inspection meetings. You CAN learn to do verification, with little or no outside help.

Validation ensures that functionality, as

Alpha testing is final testing before the software is released to the general public. First, (and this is called the first phase of alpha testing), the software is tested by in-house developers. They use either debugger software, or hardware-assisted debuggers. The goal is to catch bugs quickly. Then, (and this is called second stage of alpha testing), the software is handed over to us, the

Execution of tests is completed by following the test documents in a methodical manner. As each test procedure is performed, an entry is recorded in a test execution log to note the execution of the procedure and whether or not the test procedure uncovered any defects. Checkpoint meetings are held throughout the execution phase. Checkpoint meetings are held daily, if required, to address and

Why is a nightly called a nightly?

A nightly is a test case that must be run every time there is a new build. Generally, a new build is published every night, and these tests are run nightly, hence the name, nightly tests. Other names are Acceptance or Self-Host, but the concept remains the same.

What should a nightly cover?

A good nightly test case verifies the following:

1. Is the feature usable?
2. Can the user perform basic end-to-end scenarios?
3. Most importantly, if a certain feature/scenario is broken, does the dev need to drop everything and immediately fix the issue?

Think of nightlies as regression tests. The nightlies cover the most important scenarios, so if the test case fails, a regression has occurred that the dev must investigate immediately. (If you’re wondering why don’t devs just run tests before checking in, they do – which is a topic for a later time. Or if you’re wondering why don’t devs just have the testers test the build before checking in, we do – called buddy tests – which is a topic for a later time.)

Consider creating a new text file. Putting the above into practice, a create text file nightly may verify the following:

1. Is a Text File template available on the New File Dialog? Is the Text File created upon ok’ing the New File Dialog with the text file selected?
2. Can the user insert text into the Text File?
3. Can the user perform basic editing scenarios, like cut, copy, paste, and selection?
4. Can the user save the Text File? Does the File – Save As dialog appear upon save?
5. Does the text file appear dirty upon editing it?
6. Can the user close the file?
7. Does the file appear in the Most Recently Used list?

Why doesn’t a nightly cover more?

One might think, “that isn’t very much testing for a text file,” but consider how many different languages we support, how many different OS we support, the different SKUs, and the Express Skus. After a while, the testing matrix really starts to get big, especially given that we’re not only running these tests with every new build, but we’re also analyzing the results for each new build. And, there are more than 5 builds a week, believe me, when you add in all the different flavors of runs I’ve mentioned above.

To reiterate, the point of a nightly is to ensure that the feature is “testable”, also called “Self-Test”, and to find any regressions. If nightlies are passing at 100%, the feature is said to be “Self-Host” or testable for other scenarios. Once QA signs off on a feature / build, the feature / build is said to be shippable. Once nightlies are finished running, we are able to say, “This build or feature is Self-Host.” If the build fails (obviously implying no features are testable), the build is considered Self-Toast, a term used by the build lab, which is a great play-on-words.

Original post by Ahamad and software by Elliott Back