docs(mdtest readme): Update readme file to reflect the latest changes in
mdtest
Update reademe file and add more details and explanations for the latest
mdtest version.
Change-Id: I0a096f75ae5ba2030e92dc51b4f3eeb7ffd33f34
diff --git a/mdtest/README.md b/mdtest/README.md
index f5637cc..ecf005e 100644
--- a/mdtest/README.md
+++ b/mdtest/README.md
@@ -3,9 +3,9 @@
`mdtest` is a command line tool built on top of [flutter](https://flutter.io/)
for integration testing. The tool wraps several flutter commands and implements
algorithms to deliver a robust end to end testing experience for multi-device
-applications. `mdtest` is targeted at flutter apps and provides a public
-wrapper of flutter driver API and allows testers to write portable test scripts
-across platforms.
+applications. `mdtest` targets at multi-device flutter apps and provides a
+public API that wraps flutter driver API and allows testers to write portable
+test scripts across platforms.
# Requirements:
@@ -16,11 +16,17 @@
* Tools
- [Dart](https://www.dartlang.org/): must be installed and accessible from
`PATH`
- - PUB: comes with Dart
+ - PUB: comes with Dart and must be accessible from `PATH`.
- [Flutter](https://flutter.io/): must be installed and accessible from `PATH`.
`flutter doctor` should report no error
- [ADB](http://developer.android.com/tools/help/adb.html): must be installed
and accessible from `PATH`
+ - LCOV: `sudo apt-get install lcov` on Linux, `brew install lcov` on Mac OS.
+ Must be installed and accessible from `PATH`.
+ - [Homebrew](http://brew.sh/): must be installed on Mac OS and accessible from
+ `PATH`.
+ - [mobiledevice](https://github.com/imkira/mobiledevice): must be installed on
+ Mac OS and accessible from `PATH`.
# Installing mdtest
@@ -42,7 +48,7 @@
```
with
```
- dlog:
+ dlog: 0.0.5
```
replace
```
@@ -56,7 +62,9 @@
```
The first time you run the `mdtest` command, it will build the tool ifself. If
-you see Build Success, then `mdtest` is ready to go.
+you see Build Success, then `mdtest` is ready to go. You can run `mdtest
+doctor` to check if all dependent tools are installed before you run any test
+script.
# Quick Start
@@ -68,28 +76,27 @@
is the way to tell `mdtest` what kind of devices you want your applications to
run on. The spec file gives you the flexibility to choose your app device
either uniquely by specifying the device id, or roughly by specifying some
-properties of the devices. The device nickname refers to a device that
-satisfies your specification. You can use the nickname to create a flutter
-driver in your test script. The ability to roughly specify device requirements
-and refer a device by its nickname makes your test script portable to any
+properties of the devices. The device nickname refers to a flutter driver
+instance that will be used to automate the application and device that satisfiy
+the test spec. You can use the nickname to create a flutter driver instance in
+your test script. The ability to roughly specify device requirements and refer
+a driver instance by its nickname makes your test script portable to any
platform anywhere in the world, as long as sufficient available devices are
detected by `mdtest`. The test scripts specified in the test spec should
contain flutter driver tests for integration testing.
-`mdtest` uses a test spec to initialize the test runs. The test spec is in JSON
-format and should follow the style below:
+`mdtest` uses a test spec to find devices that each application can run on based
+on the given device properties and initiate the test runs. The test spec is in
+JSON format and should follow the style below:
+
```json
{
- "test-paths": [
- "${path/to/test_script1.dart}",
- "${path/to/test_script2.dart}",
- "${path/to/*_test.dart}"
- ...
- ],
"devices": {
"${device_nickname1}": {
+ "platform": "${platform}",
"device-id": "${device_id}",
"model-name": "${device_model_name}",
+ "os-version": "${os_version}",
"screen-size": "${screen_size}",
"app-root": "${path/to/flutter/app}",
"app-path": "${path/to/instrumented/app.dart}"
@@ -101,111 +108,220 @@
}
```
-### `test-paths` (optional)
-
-All paths in the test spec should either be absolute paths or paths relative to
-the test spec file. You can also specify test paths using glob patterns. The
-"test-paths" attribute is optional if you specify the test script path(s) from
-the command line when invoking `mdtest`. But you should at least specify a test
-path from either the test spec or the command line, otherwise `mdtest` will
-complain.
-
### `devices` (required)
"devices" attribute is required in the test spec. You can list a number of
device specs inside "devices" attribute. Each device spec has a unique
"$device_nickname" mapping to several device/application properties.
+ * `platform` (optional): The "platform" property is optional and should be
+ either 'android' or 'ios'.
+
* `device-id` (optional): The "device-id" property is optional and should be
the device id obtained by `flutter devices` if set.
- * `model-name` (optional): The"model-name" property is optional and should be
- the device model name (e.g. Nexus 5) if set.
+ * `model-name` (optional): The "model-name" property is optional and should be
+ the device model name (e.g. Nexus 5, iPhone 6S Plus, etc.) if set.
+
+ * `os-version` (optional): The "os-version" property is optional and should
+ follow rules in [semantic versioning](http://semver.org/). This property is
+ platform specific and thus "platform" must be set to use this property.
* `screen-size` (optional): The "screen-size" property is optional and values
can only be one of
- ["small"(<3.5), "normal"(>=3.5 && <5), "large"(>=5 && <8), "xlarge"(>=8)]
+ ["small"(<3.6), "normal"(>=3.6 && <5), "large"(>=5 && <8), "xlarge"(>=8)]
where the size is measured by screen diagonal in inches. The screen size
generally follows
[Android Screen Support](https://developer.android.com/guide/practices/screens_support.html)
- with overlapping screen ranges resolved.
+ and [iOS Device List](https://en.wikipedia.org/wiki/List_of_iOS_devices) with
+ overlapping screen ranges resolved.
* `app-root` (required): The "app-root" attribute specifies the path to the
- flutter app which you want to run on the device.
+ flutter app which you want to run on the device. The path should point to
+ the app root directory. If a relative path is used, it must be relative to
+ the directory that contains the test spec file.
* `app-path` (required): The "app-path" attribute points to the instrumented
- flutter app that uses flutter driver plugin. For more information, please
- refer to
+ flutter app that uses flutter driver plugin. The path should point to a dart
+ file that contains an instrumented main function which invokes the actual app
+ main function. If a relative path is used, it must be relative to the path
+ in the `app-root`. For more information, please refer to
[flutter integration testing](https://flutter.io/testing/#integration-testing).
You can add arbitraty number of device specs by repeatedly adding attributes
-following the rules above.
+following the rules above. `mdtest create` can be used to create a test spec
+template.
## Commands
-Currently, `mdtest` supports two commands: `run` and `auto`. You can run
-`mdtest run args...` or `mdtest auto args...` to invoke the commands. Run
-`mdtest -h` to list the supported commands and `mdtest command -h` for more
-information for that command. `mdtest` has a global verbose flag `--verbose`,
-which will report more information to the user during execution if set to true.
+Currently, `mdtest` supports 5 commands: `doctor`, `create`, `run`, `auto` and
+`generate`. Run `mdtest -h` to list the supported commands and `mdtest command
+-h` for more information for that command. `mdtest` has a global verbose flag
+`--verbose`, which reports trace information to the user during execution if set
+to true.
+
+### Doctor
+
+`mdtest doctor` command checks if all required dependent tools are installed.
+You can follow the installation instruction if any tool is not installed. If
+you see 'mdtest is ready to go.' after running this command, then you can start
+to run test scripts. This command does not provide any options except the
+default help flag.
+
+### Create
+
+`mdtest create` command helps create test spec template or test script template.
+
+* Arguments
+ - `--spec-template` points to the target path that will be used to generate a
+ test spec template.
+ - `--test-template` points to the target path that will be used to generate a
+ test script template.
### Run
-`mdtest run` command is used to run test suite(s) on devices that `mdtest` finds
+`mdtest run` command is used to run test suites on devices that `mdtest` finds
according to the test spec. The tool computes an app-device mapping based on
the device requirements in the test spec. The app-device mapping is bijective
and is used to launch each application on the mapped device `mdtest` finds. In
-this mode, `mdtest` will use the first app-device mapping it finds, and install
-and start the applications on devices to execute test scripts.
+this mode, `mdtest` will use the first app-device mapping it finds, then install
+and start the applications on devices to execute test scripts. After test
+execution finishes, mdtest will uninstall testing applications on all used
+devices.
* Arguments
- - `--spec` points to the path of the spec file
- - `--coverage` collects code coverage and stores the coverage info for each
- application under ${application_folder/coverage/code_coverage} if set
- - `--format` report test output in TAP format if set to tap, default is none
- which uses the default dart-lang test output format
+ - `--brief` disable logging and only print out test execution results if set.
+ - `--spec` points to the path of the test spec file. Must be specified.
+ - `--coverage` collects code coverage and stores the coverage info in LCOV
+ format for each application under ${application_root_folder/coverage/*.lcov}
+ if set.
+ - `--format` reports test output in TAP format if set to tap, default is none
+ which uses the default dart-lang test output format.
+ - `--save-report-data` points to the path to save test execution results. The
+ results will be saved in JSON format and can be used to generate test
+ report.
- The rest of the arguments would be either paths or glob patterns for the
- test scripts
+ test scripts.
### Auto
-`mdtest auto` command is used to run test suite(s) in a small number of times to
-cover as many device settings for each unique application as possible. More
+`mdtest auto` command is used to run test suites in a small number of times to
+cover all possible device settings for each unique application. More
specifically, `mdtest` groups user specified applications based on the
-uniqueness of the application root path, and groups available devices based on
-model name (will support more grouping rules later). Then, `mdtest` computes
-the maximum number of possible app group to device group mappings. Finally,
-`mdtest` will try to compute the smallest number of test runs to cover those
-maximum possible mappings. The heuristic here is to make sure at least one app
-in each application group runs on at least one device in each device group
-possibly according to the test spec in some test run. However, since the
-problem is a set cover problem and is NP-complete, `mdtest` uses a approximation
-algorithm that has a complexity of O(log(n)).
+uniqueness of the application app-path, and groups available devices based on
+any of device-id (default), platform, model-name, os-version and screen-size.
+Then, `mdtest` computes all possible app group to device group mappings based on
+the test spec and available devices. Finally, `mdtest` will try to compute a
+small number of test runs to cover all possible app-device mappings. The
+heuristic here is to make sure at least one app in each application group runs
+on at least one device in each device group with minimum test runs. However,
+since the problem is a set cover problem, it is NP-complete, `mdtest` uses a
+[greedy approximation algorithm](https://en.wikipedia.org/wiki/Set_cover_problem#Greedy_algorithm)
+that tries to compute a small number of test runs that achieve all app-device
+paths. The algorithm has a complexity of O(log(n)).
* Arguments
- - `--spec` points to the path of the spec file
- - `--coverage` collects code coverage and stores the coverage info for each
- application under ${application_folder/coverage/code_coverage} if set
- - `--format` report test output in TAP format if set to tap, default is none
- which uses the default dart-lang test output format
+ - `--brief` disable logging and only print out test execution results if set.
+ - `--spec` points to the path of the test spec file. Must be specified.
+ - `--coverage` collects code coverage and stores the coverage info in LCOV
+ format for each application under ${application_root_folder/coverage/*.lcov}
+ if set.
+ - `--format` reports test output in TAP format if set to tap, default is none
+ which uses the default dart-lang test output format.
+ - `--save-report-data` points to the path to save test execution results. The
+ results will be saved in JSON format and can be used to generate test
+ report.
+ - `--groupby` is the device property that will be used to group all available
+ devices. The value can only be one of
+ ['device-id'(default), 'platform', 'model-name', 'os-version', 'screen-size'].
- The rest of the arguments would be either paths or glob patterns for the
- test scripts
+ test scripts.
+
+### Generate
+
+`mdtest generate` command is used to load report data and generate either code
+coverage or test execution output report. The code coverage report only
+contains statement coverage info and it uses genhtml internally, which comes
+with lcov, to generate the web report. The test execution output report
+contains the app-device hitmap info as well as the status of test entities like
+test round, test suite, test group and test method. The HTML format report will
+be stored under the given output directory users specify.
+
+* Arguments
+ - `--report-type` can be one of 'test' or 'coverage'. If set to 'coverage',
+ `mdtest` tries to generate a code coverage report. If set to 'test',
+ `mdtest` tries to generate a test execution output report. Must be set.
+ - `--load-report-data` points to the path to the report data that is used to
+ generate HTML report. If you want to generate a coverage report, you must
+ provide a coverage data file in LCOV format. If you want to generate a test
+ report, you must provide a test data file in JSON format. Must be set.
+ - `--lib` points to the path to the flutter application lib folder that your
+ code coverage data refers to. This option is only used if you want to
+ generate a code coverage report.
+ - `--output` points to the path of the directory where the HTML report will be
+ stored. Must be set.
## Writing Tests
-`mdtest` provides a wrapper of flutter driver API and allows users to create a
-driver instance by a device nickname specified in the test spec. To use this
-wrapper, you should add the following import statement in your test scripts:
+`mdtest` provides a DriverMap class which maps every nickname to a flutter
+driver instance. You can retrieve the corresponding flutter driver instance by
+providing the nickname, which is specified in the test spec. DriverMap class
+will lazy initialize the flutter driver the first time you retrieve it. Once
+you get the flutter driver instance, you can invoke any public methods that
+the FlutterDriver class provides. To use this wrapper, you should add the following
+import statement in your test scripts:
```
import 'package:flutter_driver/flutter_driver.dart';
import 'package:mdtest/driver_util.dart';
```
-Then you can create a flutter driver instance like this:
-```
-FlutterDriver driver = await DriverUtil.connectByName('${device_nickname}');
+Here is a full example test suite using DriverMap class:
+```dart
+import 'dart:async';
+
+import 'package:flutter_driver/flutter_driver.dart';
+import 'package:test/test.dart';
+import 'package:mdtest/driver_util.dart';
+
+void main() {
+ group('Multi-device application tests', () {
+ DriverMap driverMap;
+
+ setUpAll(() {
+ driverMap = new DriverMap();
+ });
+
+ tearDownAll(() {
+ if (driverMap != null) {
+ driverMap.closeAll();
+ }
+ });
+
+ test('Test 1', () async {
+ // Lazy initialize driver
+ FlutterDriver driver = await driverMap['nickname'];
+ // Write normal flutter driver tests
+ ...
+ });
+
+ test('Test 2', () async {
+ // Get all flutter driver instances
+ List<FlutterDriver> drivers = await Future.wait(driverMap.values);
+ // Invoke tap() on all flutter drivers
+ await Future.wait(
+ drivers.map((driver) => driver.tap(...))
+ );
+ // Invoke getText() on all flutter drivers and check values
+ await Future.forEach(drivers, (driver) async {
+ String result = await driver.getText(...);
+ expect(result, equals(...));
+ });
+ });
+ });
+}
```
-The way to write integration tests for flutter apps follows
+`mdtest create` can be used to create a sample test script for you. The way to
+write integration tests for flutter apps follows
[flutter integration testing](https://flutter.io/testing/#integration-testing).
diff --git a/mdtest/lib/src/commands/doctor.dart b/mdtest/lib/src/commands/doctor.dart
index 961255d..151c914 100644
--- a/mdtest/lib/src/commands/doctor.dart
+++ b/mdtest/lib/src/commands/doctor.dart
@@ -80,6 +80,9 @@
);
return 1;
}
+ printInfo(
+ 'All required tools are installed correctly. mdtest is ready to go.'
+ );
return 0;
}
}
