feat(mdtest auto): introduce app-device coverage concept
Introduce app-device coverage concept. We first cluster device specs
based on the app path, all specs with the same app path is clustered
together. We also cluster devices based on model name, devices with
same model name is clustered together. The clustering rule can be
easily changed. In the current version, only the above clustering rule
is supported. Some app-device coverage definition:
* Coverage Definition:
If an app in an app cluster A is installed and run on a device in a
device cluster D, then we say app cluster A covers device cluster D
and the app-device path A-D is covered.
* Spec-Device Mapping Suite Definition:
A spec-device mapping suite is defined as the union of all spec to
device mappings for the same test script. For example, a single
application can be run on different types of devices. In such case,
the test script and spec never change but the spec-device mapping is
changed so an application can be run on different devices.
* Full Coverage Definition:
If a test script is repeatedly executed according to a spec-device
mapping suite S, in all such runs, if every app cluster covers
every device cluster, then we say the spec-device mapping suite S
gives full app-device coverage.
* Coverage Score Definition:
Suppose there are N app clusters, M device clusters, and a
spec-device mapping suite S covers X app-device paths, then the
coverage score is computed by C(S) = X / (N * M)
An algorithm is implemented to find the minimum set of spec-device
matches that gives the maximum app-device coverage. A new command
"mdtest auto ..." is implemented to invoke the algorithm.
Change-Id: Idcc7e33daba4754d26e409d8701f8e3b857935de
13 files changed
