The AQuA Testing Criteria and Tool

What is the AQuA Testing Criteria?

The Testing Criteria is an evolving set of platform-specific tests, built up from input from developers, test houses and AQuA members. It provides tests that compliment the functional testing carried out during development, and includes a range of non-functional, usability, installation, performance, compatibility and functional sanity testing.

However, it intentionally does not look at the usefulness or function of the application, and isn 'tintended to capture everything. It is simply a guide to the level of acceptable usability AQuA members feel an app should have before it is commercialised.

All apps detailed within the Quality App Directory need to have been tested against the AQuA Testing Criteria for Android Apps, or the AQuA Testing Criteria for iOS Apps.

Those displaying the quality level of 'Test House Verified' need to have been successfully tested by one of the QAD-approved Test Houses, with the test report uploaded into the Directory by the test house.

Carry on reading to find out about ...

And then ...

For Android:
Download the AQuA Testing Criteria for Android Apps updated with v1.6 November 2014 (opens in a new window)

For Apple iOS:
Download the AQuA Testing Criteria for iOS Apps updated with v1.1 November 2014 2014 (opens in a new window)

The test areas and scoring system

Areas covered by the tests

The tests within the AQuA Testing Criteria look at the usability of the app in the following areas:

Install and Launch Functionality
Memory Use Keys
Connectivity Device and Extra Hardware Specific Tests
Event Handling Stability
Messaging & calls Data Handling
External Influence Security
User Interface Multi Player
Language Meta Data
Performance Privacy and User Permissions
Media Platform Compliance in App Commerce
Menu Performance: Network and Battery Usage

Types of Apps

With the help of the AQuA members, test houses and developers, we identified 4 types of mobile apps for both the Android and iOS platforms:

  • A Simple App
  • A Framework App
  • Content from a Framework App
  • A Complex App

As these apps all have different attributes and functionality levels, they need different levels of testing, with not all types of app needing to be tested against all of the tests in the Criteria.

An example of a Simple App is a sports news and scores reporting app.

In Android, a Simple App ...

  • does not send SMS/MMS
  • does not write data to standard data files e.g. contacts, calendar
  • does not write data to external services e.g. social networks
  • can access but does not change the state of networking services e.g. 3G/Wifi/Bluetooth
  • can access external sites to retrieve information
  • can access location information
  • can read standard data files
  • can read SMS/MMS
  • can access screen, sound, camera, keyboard
  • can write its own data e.g. store pictures, create documents

In Apple iOS, a Simple App ...

  • has no Multiuser or Multiplayer features
  • has no in-app purchasing
  • has no in-app advertising

In any network, a Framework App is where the same application framework is used repeatedly to create new applications. This is especially prevalent for dictionaries, travel guides, other books and magazine applications but may occur in any application type.

Then there is the Content from a Framework App. For such content, it is clearly excessive to fully test the existing framework and the new content as it is to a large degree an existing application with new resource files (content).

Most Framework Apps will be simple apps so the relevant 'simple' tests should be used in the first instance to test the Framework with the first piece of content, and then a smaller set of tests should be performed for subsequent content (resource files) within that same framework.

An example of a Complex App is an integrated business mileage tracker.

Critical tests, warning tests and types of warnings

Input from developers has made us recognize that many of the tests that are performed can be subjective, leaving the interpretation to the tester. It is unfair therefore to fail an application for one minor error that may be down to a tester's opinion.

To account for this, the individual tests in the criteria are each graded as either Critical or Warning.

Critical Tests

As the name suggests, a Critical Test must be passed.

If an application fails a Critical test then the application has an overall fail.

Warning Tests

For a test which is considered a Warning Test, we have allowed for four different results; pass, annoying, difficult and impossible.

The warning levels are described as follows;

Pass This means the app has passed that specifc test. There are no issues
Annoying This means that a minor error has occurred with the app - e.g. one or two typos that would make the application not perfect but still very useable
Difficult This means that a more serious issue has occurred with the app e.g. making the application difficult to use but not impossible
Impossible This means a very serious issue has occurred with the app - the errors are so bad as to make the application unusable.

Points are awarded as follows, depending on the test type (Warning or Critical) and the severity of the error found.

Severity of error Warning test Critical test
No error 0 points 0 points
Annoying error 1 point
Difficult error 2 points
Impossible error 4 points
Fail test 5 points

If an app gains 4 points or more, it will have failed the Testing Criteria.

(As an example, an application could have 3 annoying results from Warning tests, or 1 difficult and 1 annoying result and still pass. An application with just 1 Critical test failure will fail).

The Online Testing Criteria Tool

What is the OnlineTesting Criteria Tool?

New for February 2013, it’s a tool available from your dashboard to help you run through the tests within the AQuA Testing Criteria for Android Apps and iOS Apps and assess the result.

  • Input a few details of your app, and the tool selects the relevant tests. Then run through the tests, scoring the app as you go. (You can pause and then go back to the testing whenever you like.)
  • Once you've finished the test suite for an app, you'll see the result - pass or fail - and can create a report.

All your existing and past tests are clearly shown in the system, for you to go back to at any time.

The tool is available to anyone to use free of charge.  Simply create an account and use it.