GStreamer QA system  TODO



Milestone 1
-----------
Goal : Compatible behaviour as gst-media-test, but using the new API.

* testrun
 * Add possibility to modify the environment

* Monitor support
 * gdb/backtrace monitor

* Reporting support



Milestone 2
-----------
Goal : Database support

* DB storage support
 * Store monitors
* Remote DB storage support
 * Might have to be done with Milestone 3+4

* Important speed issue for search
 * Add a 'similarity' table that contains tests that have:
  * similar type and
  * similar arguments
 * This can be done either on the first run of compare.py or
   when importing one database in another.

Milestone 3
-----------
Goal : Centralized scheduling/control system

Using buildbot



Milestone 4
-----------
Goal : Search/Query/Reporting centralized system




Non-milestone-specific / Bonus features
---------------------------------------
* Speed optimizations
 * DBusRemoteTest : Add features so that, for some tests, we don't spawn a new
   remote process for every instance, but instead reuse existing ones (and
   re-spawning new instances if that one crashes).

* Smarter introspection
 * Make __test_arguments__ have more information :
  * default value
  * full documentation on that argument
 * Make __test_checklist__ have more information :
  * full documentation , especially explaining what could have caused
    that step to not be validated, to make debugging/fixing easier.

* Ordered list of tuples instead of dictionnaries of tuples
 * Maybe this could also apply to other properties

* Database
 * Make sure we use cPickle only for generic (non-gstqa-specific) objects
 * Add convenience functions for merging two DB together (or one in another).
 * Add methods for updating old-versioned DB

* Tests
 * Add some form of versioning so we can track changes.
  * Maybe upgrade version based on variations of arguments/checklist/extrainfo/behaviour

* DBUS
 * Fix issue in a 'asv' when one of the values is a list, we currently avoid this
   by removing all values of type 'list'

* Regression testing
 * Create an easy script to test regressions



DBUS ISSUES
-----------

* the type detection is extremely dumb for integers.
  Ex : python integers have a maximum range of 2**63-1 on 64bit
    -> this will be considered by dbus-python as an int
      -> dbus-python will convert that to an Int32
       -> BOOM
  We currently solve this by pre-processing some values to make sure
   they have the correct type.