FIWARE is rapidly moving from experimental to production environments in which the platform must scale in reliable and real workload conditions. This fact implies that all FIWARE GEris must work at an adequate quality, reliability and at performance level appropriate for these conditions. A dedicated activity has been launched in the framework of the initiative to analyze and assess the level of quality of each GE, providing diverse kind of reports and an assessment dashboard.

The quality is evaluated from different points of view:

• Curation of GEs documentation (documentation testing), both inspecting the code and the accompanying documentation (installation manuals, user guidelines, and similar). The goal of this assessment is to support FIWARE users with high-quality support for installation, configuration and operation of FIWARE technology, thereby improving the FIWARE user experience in general.
• Verification of the GE specification (functional testing), developing the appropriate test cases to assess if the GEs implementation corresponds to what is defined in the specification.
• Assessment of performance, stability and scalability of GEs in operational environments, like under excessive workload (stress testing). Test scenarios are defined and executed such that limits of a GE under test are identified, and can be compared with reference levels. The goal of this assessment is to favor the applicability of FIWARE in purely commercial scenarios.

The testing of the documentation and verification has been done for all GE not deprecated in FIWARE Catalogue (28 in total). Three phases are required to complete the QA functional test process. The first phase verifies for each GE the completeness of documentation, the consistency of artefacts and the soundness of information.  The usability of documentation, by example, in case of installation manual is checked installing step by step the GE. In the second phase specific method calls verify the single APIs and the response correctness of each GEs. The last phase consists of functional verifications based on reference architectures integrating some GEs. As result a live dashboard collects and maintains the assessment information and GE owners are punctually requested to correct the encountered deficiencies. The 90% of the high priority GEs has passed successfully the documentation and verification tests. The medium and low priority GEs are above 70% of success but they are working on solving the issues

On the other hand, the stress testing has been performed only for those GEs most critical in terms of performance in the overall architecture. An iterative process and operative methodology have been put in place, obtaining after each iteration, a complete report with the measures obtained after stress test and analysis of the data. The reports are sent to the GE owners for considering improvements about performance and stability for next release. Three iterations will be achieved before end of September this year: one took place in February testing 9 GEs (Orion, Proton, Aeron, IDAS, Kurento, Wilma, KeyRock, Cepheus, Bosun); the second one in May testing new versions of these GEs; and final one due by September testing again a new updated version of these GEs plus two more identified (AuthZForce and Cygnus) and more frequent combination of GEs.

Once the first iteration of stress testing was conducted, a quality assurance expert was consulted for carrying out an independent assessment of the followed process and executed tests to produce an assessment of the achieved work. The main conclusions of his assessment were:

• Important performance borders were identified
• Robustness of use within bounds was shown
• Documentation needs to be improved

According to this assessment, FIWARE GEs are fit for being released in a commercial operational environment with some adjustments. A new external independent assessment is currently being requested after second iteration.

As part of the overall testing process and based on the obtained results in the three aspects (documentation, verification and stress) above mentioned, an overall label of quality is granted to each GE. This global label represents the degree of quality of the GE by adopting the energy labelling system[1] used by EU for devices. Specific labels for each analyzed category (usability, reliability, efficiency, scalability, performance and stability) are also granted. Thus, in the Catalogue each GE will be labelled with a global label expanded by clicking of detailed labels map.

Now, after two phases in the process some overall conclusions can be stated. There exists a significant heterogeneity in the GEs quality, having more mature GEs and ready for market than others. There is still room for improvement in documentation and support for most of the GEs, which is currently in progress thanks to this activity. It can be also stated significant improvements in performance from first iteration to the second one, due to the following of recommendations in first iteration testing report by the GE responsible, which is also a demonstration of the value this activity can bring to FIWARE.

In near future, the main focus will be to enlarge number and type of tests and to automate the tests as much as possible, but in the meantime a set of guidelines have been created in order to be able to replicate all the conducted tests by anyone. All the tests and code are already public in FIWARE software repository and all the reports available through the FIWARE wiki.

For further information:

• GitHub repository containing all docs (guidelines) and scripts (to run the tests) about non functional testing task (stress tests)
   
• All the reports, up to date, in Docman, under Quality Assurance folder.
   
• The FIWARE Quality Assurance chapter in our wiki.