From 2f94d12b57cf01668126bcd2ab7009e614dd4f2a Mon Sep 17 00:00:00 2001 From: Marc Bonnici Date: Mon, 25 Jun 2018 09:30:14 +0100 Subject: doc: Restructure developer information Re-organise and add additional information to developer guide. --- doc/source/developer_information.rst | 8 + .../developer_information/developer_guide.rst | 12 + .../developer_guide/writing_plugins.rst | 863 +++++++++++++ .../developer_information/developer_reference.rst | 9 +- .../developer_reference/plugins.rst | 376 ++++++ .../developer_reference/writing_plugins.rst | 1315 -------------------- .../how_tos/adding_plugins.rst | 4 +- doc/source/developer_reference.rst | 29 - doc/source/migration_guide.rst | 7 +- doc/source/user_information/how_tos/agenda.rst | 2 +- 10 files changed, 1271 insertions(+), 1354 deletions(-) create mode 100644 doc/source/developer_information/developer_guide.rst create mode 100644 doc/source/developer_information/developer_guide/writing_plugins.rst create mode 100644 doc/source/developer_information/developer_reference/plugins.rst delete mode 100644 doc/source/developer_information/developer_reference/writing_plugins.rst delete mode 100644 doc/source/developer_reference.rst (limited to 'doc') diff --git a/doc/source/developer_information.rst b/doc/source/developer_information.rst index ff7cb99b..e2caa45f 100644 --- a/doc/source/developer_information.rst +++ b/doc/source/developer_information.rst @@ -6,6 +6,14 @@ Developer Information :depth: 4 :local: +------------------ .. include:: developer_information/how_to.rst + +------------------ + +.. include:: developer_information/developer_guide.rst + +------------------ + .. include:: developer_information/developer_reference.rst diff --git a/doc/source/developer_information/developer_guide.rst b/doc/source/developer_information/developer_guide.rst new file mode 100644 index 00000000..44565732 --- /dev/null +++ b/doc/source/developer_information/developer_guide.rst @@ -0,0 +1,12 @@ +.. _developer_guide: + +******************** +Developer Guide +******************** + +.. contents:: + :depth: 3 + :local: + +.. include:: developer_information/developer_guide/writing_plugins.rst + diff --git a/doc/source/developer_information/developer_guide/writing_plugins.rst b/doc/source/developer_information/developer_guide/writing_plugins.rst new file mode 100644 index 00000000..f44f84db --- /dev/null +++ b/doc/source/developer_information/developer_guide/writing_plugins.rst @@ -0,0 +1,863 @@ +.. _writing-plugins: + + +Writing Plugins +================ + +Workload Automation offers several plugin points (or plugin types). The most +interesting of these are + +:workloads: These are the tasks that get executed and measured on the device. These + can be benchmarks, high-level use cases, or pretty much anything else. +:targets: These are interfaces to the physical devices (development boards or end-user + devices, such as smartphones) that use cases run on. Typically each model of a + physical device would require its own interface class (though some functionality + may be reused by subclassing from an existing base). +:instruments: Instruments allow collecting additional data from workload execution (e.g. + system traces). Instruments are not specific to a particular workload. Instruments + can hook into any stage of workload execution. +:output processors: These are used to format the results of workload execution once they have been + collected. Depending on the callback used, these will run either after each + iteration and/or at the end of the run, after all of the results have been + collected. + +You can create a plugin by subclassing the appropriate base class, defining +appropriate methods and attributes, and putting the .py file containing the +class into the "plugins" subdirectory under ``~/.workload_automation`` (or +equivalent) where it will be automatically picked up by WA. + + +Plugin Basics +-------------- + +This sub-section covers things common to implementing plugins of all types. It +is recommended you familiarize yourself with the information here before +proceeding onto guidance for specific plugin types. + +.. _context: + +The Context +^^^^^^^^^^^ + +The majority of methods in plugins accept a context argument. This is an +instance of :class:`wa.framework.execution.ExecutionContext`. It contains +information about the current state of execution of WA and keeps track of things +like which workload is currently running. + +Notable methods of the context are: + +:context.get_resource(resource, strict=True): + This method should be used to retrieve a resource using the resource getters rather than using the ResourceResolver directly as this method additionally record any found resources hash in the output metadata. + +:context.add_artifact(name, host_file_path, kind, description=None, classifier=None): + Plugins can add :ref:`artifacts ` of various kinds to the run + output directory for WA and associate them with a description and/or + :ref:`classifier `. + +:context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None): + This method should be used to add :ref:`metrics ` that have been + generated from a workload, this will allow WA to process the results + accordingly depending on which output processors are enabled. + +Notable attributes of the context are: + +:context.workload: + :class:`wa.framework.workload` object that is currently being executed. + +:context.tm: + This is the target manager that can be used to access various information + about the target including initialization parameters. + +:context.current_job: + This is an instance of :class:`wa.framework.job.Job` and contains all + the information relevant to the workload job currently being executed. + +:context.current_job.spec: + The current workload specification being executed. This is an + instance of :class:`wa.framework.configuration.core.JobSpec` + and defines the workload and the parameters under which it is + being executed. + +:context.current_job.current_iteration: + The current iteration of the spec that is being executed. Note that this + is the iteration for that spec, i.e. the number of times that spec has + been run, *not* the total number of all iterations have been executed so + far. + +:context.job_output: + This is the output object for the current iteration which + is an instance of :class:`wa.framework.output.JobOutput`. It contains + the status of the iteration as well as the metrics and artifacts + generated by the workload. + + +In addition to these, context also defines a few useful paths (see below). + + +Paths +^^^^^ + +You should avoid using hard-coded absolute paths in your plugins whenever +possible, as they make your code too dependent on a particular environment and +may mean having to make adjustments when moving to new (host and/or device) +platforms. To help avoid hard-coded absolute paths, WA defines a number of +standard locations. You should strive to define your paths relative +to one of these. + +On the host +~~~~~~~~~~~ + +Host paths are available through the context object, which is passed to most +plugin methods. + +context.run_output_directory + This is the top-level output directory for all WA results (by default, + this will be "wa_output" in the directory in which WA was invoked. + +context.output_directory + This is the output directory for the current iteration. This will an + iteration-specific subdirectory under the main results location. If + there is no current iteration (e.g. when processing overall run results) + this will point to the same location as ``root_output_directory``. + + +Additionally, the global ``wa.settings`` object exposes on other location: + +settings.dependency_directory + this is the root directory for all plugin dependencies (e.g. media + files, assets etc) that are not included within the plugin itself. + +As per Python best practice, it is recommended that methods and values in +``os.path`` standard library module are used for host path manipulation. + +On the target +~~~~~~~~~~~~~ + +Workloads and instruments have a ``target`` attribute, which is an interface to +the target used by WA. It defines the following location: + +target.working_directory + This is the directory for all WA-related files on the target. All files + deployed to the target should be pushed to somewhere under this location + (the only exception being executables installed with ``target.install`` + method). + +Since there could be a mismatch between path notation used by the host and the +target, the ``os.path`` modules should *not* be used for on-target path +manipulation. Instead target has an equipment module exposed through +``target.path`` attribute. This has all the same attributes and behaves the +same way as ``os.path``, but is guaranteed to produce valid paths for the target, +irrespective of the host's path notation. For example: + +.. code:: python + + result_file = self.target.path.join(self.target.working_directory, "result.txt") + self.command = "{} -a -b -c {}".format(target_binary, result_file) + +.. note:: Output processors, unlike workloads and instruments, do not have their + own target attribute as they are designed to be able to be ran offline. + +.. _resource-resolution: + +Dynamic Resource Resolution +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The idea is to decouple resource identification from resource discovery. +Workloads/instruments/devices/etc state *what* resources they need, and not +*where* to look for them -- this instead is left to the resource resolver that +is part of the execution context. The actual discovery of resources is +performed by resource getters that are registered with the resolver. + +A resource type is defined by a subclass of +:class:`wa.framework.resource.Resource`. An instance of this class describes a +resource that is to be obtained. At minimum, a ``Resource`` instance has an +owner (which is typically the object that is looking for the resource), but +specific resource types may define other parameters that describe an instance of +that resource (such as file names, URLs, etc). + +An object looking for a resource invokes a resource resolver with an instance of +``Resource`` describing the resource it is after. The resolver goes through the +getters registered for that resource type in priority order attempting to obtain +the resource; once the resource is obtained, it is returned to the calling +object. If none of the registered getters could find the resource, +``NotFoundError`` is raised (or ``None`` is returned instead, if invoked with +``strict=False``). + +The most common kind of object looking for resources is a ``Workload``, and the +``Workload`` class defines +:py:meth:`wa.framework.workload.Workload.init_resources` method, which may be +overridden by subclasses to perform resource resolution. For example, a workload +looking for an executable file would do so like this:: + + from wa import Workload + from wa.import Executable + + class MyBenchmark(Workload): + + # ... + + def init_resources(self, resolver): + resource = Executable(self, self.target.abi, 'my_benchmark') + host_exe = resolver.get(resource) + + # ... + + +Currently available resource types are defined in :py:mod:`wa.framework.resources`. + +.. _deploying-executables: + +Deploying executables to a target +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Some targets may have certain restrictions on where executable binaries may be +placed and how they should be invoked. To ensure your plugin works with as +wide a range of targets as possible, you should use WA APIs for deploying and +invoking executables on a target, as outlined below. + +As with other resources, host-side paths to the executable binary to be deployed +should be obtained via the :ref:`resource resolver `. A +special resource type, ``Executable`` is used to identify a binary to be +deployed. This is similar to the regular ``File`` resource, however it takes an +additional parameter that specifies the ABI for which the executable was +compiled for. + +In order for the binary to be obtained in this way, it must be stored in one of +the locations scanned by the resource resolver in a directory structure +``/bin//`` (where ``root`` is the base resource location to +be searched, e.g. ``~/.workload_automation/dependencies/``, and +```` is the ABI for which the executable has been compiled, as returned by +``self.target.abi``). + +Once the path to the host-side binary has been obtained, it may be deployed +using one of two methods from a +`Target `_ instance -- +``install`` or ``install_if_needed``. The latter will check a version of that +binary has been previously deployed by WA and will not try to re-install. + +.. code:: python + + from wa import Executable + + host_binary = context.resolver.get(Executable(self, self.target.abi, 'some_binary')) + target_binary = self.target.install_if_needed(host_binary) + + +.. note:: Please also note that the check is done based solely on the binary name. + For more information please see the devlib + `documentation `_. + +Both of the above methods will return the path to the installed binary on the +target. The executable should be invoked *only* via that path; do **not** assume +that it will be in ``PATH`` on the target (or that the executable with the same +name in ``PATH`` is the version deployed by WA. + +For more information on how to implement this, please see the +:ref:`how to guide `. + + +Deploying assets +----------------- +WA provides a generic mechanism for deploying assets during workload initialization. +WA will automatically try to retrieve and deploy each asset to the target's working directory +that is contained in a workloads ``deployable_assets`` attribute stored as a list. + +If the parameter ``cleanup_assets`` is set then any asset deployed will be removed +again and the end of the run. + +If the workload requires a custom deployment mechanism the ``deploy_assets`` +method can be overridden for that particular workload, in which case, either +additional assets should have their on target paths added to the workload's +``deployed_assests`` attribute or the corresponding ``remove_assets`` method +should also be implemented. + +.. _plugin-parmeters: + +Parameters +---------- + +All plugins can be parametrized. Parameters are specified using +``parameters`` class attribute. This should be a list of +:class:`wa.framework.plugin.Parameter` instances. The following attributes can be +specified on parameter creation: + +:name: + This is the only mandatory argument. The name will be used to create a + corresponding attribute in the plugin instance, so it must be a valid + Python identifier. + +:kind: + This is the type of the value of the parameter. This must be an + callable. Normally this should be a standard Python type, e.g. ``int`` + or ``float``, or one the types defined in :mod:`wa.utils.types`. + If not explicitly specified, this will default to ``str``. + + .. note:: Irrespective of the ``kind`` specified, ``None`` is always a + valid value for a parameter. If you don't want to allow + ``None``, then set ``mandatory`` (see below) to ``True``. + +:allowed_values: + A list of the only allowed values for this parameter. + + .. note:: For composite types, such as ``list_of_strings`` or + ``list_of_ints`` in :mod:`wa.utils.types`, each element of + the value will be checked against ``allowed_values`` rather + than the composite value itself. + +:default: + The default value to be used for this parameter if one has not been + specified by the user. Defaults to ``None``. + +:mandatory: + A ``bool`` indicating whether this parameter is mandatory. Setting this + to ``True`` will make ``None`` an illegal value for the parameter. + Defaults to ``False``. + + .. note:: Specifying a ``default`` will mean that this parameter will, + effectively, be ignored (unless the user sets the param to ``None``). + + .. note:: Mandatory parameters are *bad*. If at all possible, you should + strive to provide a sensible ``default`` or to make do without + the parameter. Only when the param is absolutely necessary, + and there really is no sensible default that could be given + (e.g. something like login credentials), should you consider + making it mandatory. + +:constraint: + This is an additional constraint to be enforced on the parameter beyond + its type or fixed allowed values set. This should be a predicate (a function + that takes a single argument -- the user-supplied value -- and returns + a ``bool`` indicating whether the constraint has been satisfied). + +:override: + A parameter name must be unique not only within an plugin but also + with that plugin's class hierarchy. If you try to declare a parameter + with the same name as already exists, you will get an error. If you do + want to override a parameter from further up in the inheritance + hierarchy, you can indicate that by setting ``override`` attribute to + ``True``. + + When overriding, you do not need to specify every other attribute of the + parameter, just the ones you what to override. Values for the rest will + be taken from the parameter in the base class. + + +Validation and cross-parameter constraints +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A plugin will get validated at some point after construction. When exactly +this occurs depends on the plugin type, but it *will* be validated before it +is used. + +You can implement ``validate`` method in your plugin (that takes no arguments +beyond the ``self``) to perform any additional *internal* validation in your +plugin. By "internal", I mean that you cannot make assumptions about the +surrounding environment (e.g. that the device has been initialized). + +The contract for ``validate`` method is that it should raise an exception +(either ``wa.framework.exception.ConfigError`` or plugin-specific exception type -- see +further on this page) if some validation condition has not, and cannot, been met. +If the method returns without raising an exception, then the plugin is in a +valid internal state. + +Note that ``validate`` can be used not only to verify, but also to impose a +valid internal state. In particular, this where cross-parameter constraints can +be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on +another parameter, there is no way to express that declaratively when specifying +the parameters. In that case the dependent attribute should be left unspecified +on creation and should instead be set inside ``validate``. + +Logging +------- + +Every plugin class has it's own logger that you can access through +``self.logger`` inside the plugin's methods. Generally, a :class:`Target` will +log everything it is doing, so you shouldn't need to add much additional logging +for device actions. However you might what to log additional information, e.g. +what settings your plugin is using, what it is doing on the host, etc. +(Operations on the host will not normally be logged, so your plugin should +definitely log what it is doing on the host). One situation in particular where +you should add logging is before doing something that might take a significant +amount of time, such as downloading a file. + + +Documenting +----------- + +All plugins and their parameter should be documented. For plugins +themselves, this is done through ``description`` class attribute. The convention +for an plugin description is that the first paragraph should be a short +summary description of what the plugin does and why one would want to use it +(among other things, this will get extracted and used by ``wa list`` command). +Subsequent paragraphs (separated by blank lines) can then provide a more +detailed description, including any limitations and setup instructions. + +For parameters, the description is passed as an argument on creation. Please +note that if ``default``, ``allowed_values``, or ``constraint``, are set in the +parameter, they do not need to be explicitly mentioned in the description (wa +documentation utilities will automatically pull those). If the ``default`` is set +in ``validate`` or additional cross-parameter constraints exist, this *should* +be documented in the parameter description. + +Both plugins and their parameters should be documented using reStructureText +markup (standard markup for Python documentation). See: + +http://docutils.sourceforge.net/rst.html + +Aside from that, it is up to you how you document your plugin. You should try +to provide enough information so that someone unfamiliar with your plugin is +able to use it, e.g. you should document all settings and parameters your +plugin expects (including what the valid values are). + + +Error Notification +------------------ + +When you detect an error condition, you should raise an appropriate exception to +notify the user. The exception would typically be :class:`ConfigError` or +(depending the type of the plugin) +:class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`. +All these errors are defined in :mod:`wa.framework.exception` module. + +A :class:`ConfigError` should be raised where there is a problem in configuration +specified by the user (either through the agenda or config files). These errors +are meant to be resolvable by simple adjustments to the configuration (and the +error message should suggest what adjustments need to be made. For all other +errors, such as missing dependencies, mis-configured environment, problems +performing operations, etc., the plugin type-specific exceptions should be +used. + +If the plugin itself is capable of recovering from the error and carrying +on, it may make more sense to log an ERROR or WARNING level message using the +plugin's logger and to continue operation. + +.. _instrument-reference: + +Adding an Instrument +--------------------- +Instruments can be used to collect additional measurements during workload +execution (e.g. collect power readings). An instrument can hook into almost any +stage of workload execution. Any new instrument should be a subclass of +Instrument and it must have a name. When a new instrument is added to Workload +Automation, the methods of the new instrument will be found automatically and +hooked up to the supported signals. Once a signal is broadcasted, the +corresponding registered method is invoked. + +Each method in ``Instrument`` must take two arguments, which are ``self`` and +``context``. Supported methods and their corresponding signals can be found in +the :ref:`Signals Documentation `. To make +implementations easier and common, the basic steps to add new instrument is +similar to the steps to add new workload and an example can be found in the +:ref:`How To ` section. + +.. _instrument-api: + +To implement your own instrument the relevant methods of the interface shown +below should be implemented: + + :name: + + The name of the instrument, this must be unique to WA. + + :description: + + A description of what the instrument can be used for. + + :parameters: + + A list of additional :class:`Parameters` the instrument can take. + + :initialize(context): + + This method will only be called once during the workload run + therefore operations that only need to be performed initially should + be performed here for example pushing the files to the target device, + installing them. + + :setup(context): + + This method is invoked after the workload is setup. All the + necessary setup should go inside this method. Setup, includes + operations like clearing logs, additional configuration etc. + + :start(context): + + It is invoked just before the workload start execution. Here is + where instrument measurement start being registered/taken. + + :stop(context): + + It is invoked just after the workload execution stops and where + the measurements should stop being taken/registered. + + :update_output(context): + + It is invoked after the workload updated its result. + update_result is where the taken measures are added to the result so it + can be processed by Workload Automation. + + :teardown(context): + + It is invoked after the workload is torn down. It is a good place + to clean any logs generated by the instrument. + + :finalize(context): + + This method is the complement to the initialize method and will also + only be called once so should be used to deleting/uninstalling files + pushed to the device. + + +This is similar to a ``Workload``, except all methods are optional. In addition to +the workload-like methods, instruments can define a number of other methods that +will get invoked at various points during run execution. The most useful of +which is perhaps ``initialize`` that gets invoked after the device has been +initialised for the first time, and can be used to perform one-time setup (e.g. +copying files to the device -- there is no point in doing that for each +iteration). The full list of available methods can be found in +:ref:`Signals Documentation `. + +.. _prioritization: + +Prioritization +^^^^^^^^^^^^^^ + +Callbacks (e.g. ``setup()`` methods) for all instruments get executed at the +same point during workload execution, one after another. The order in which the +callbacks get invoked should be considered arbitrary and should not be relied +on (e.g. you cannot expect that just because instrument A is listed before +instrument B in the config, instrument A's callbacks will run first). + +In some cases (e.g. in ``start()`` and ``stop()`` methods), it is important to +ensure that a particular instrument's callbacks run a closely as possible to the +workload's invocations in order to maintain accuracy of readings; or, +conversely, that a callback is executed after the others, because it takes a +long time and may throw off the accuracy of other instruments. You can do +this by using decorators on the appropriate methods. The available decorators are: +``very_slow``, ``slow``, ``normal``, ``fast``, ``very_fast``, with ``very_fast`` +running closest to the workload invocation and ``very_slow`` running furtherest +away. For example:: + + from wa import very_fast + # .. + + class PreciseInstrument(Instrument) + + # ... + @very_fast + def start(self, context): + pass + + @very_fast + def stop(self, context): + pass + + # ... + +``PreciseInstrument`` will be started after all other instruments (i.e. +*just* before the workload runs), and it will stopped before all other +instruments (i.e. *just* after the workload runs). + +If more than one active instrument has specified fast (or slow) callbacks, then +their execution order with respect to each other is not guaranteed. In general, +having a lot of instruments enabled is going to negatively affect the +readings. The best way to ensure accuracy of measurements is to minimize the +number of active instruments (perhaps doing several identical runs with +different instruments enabled). + +Example +^^^^^^^ + +Below is a simple instrument that measures the execution time of a workload:: + + class ExecutionTimeInstrument(Instrument): + """ + Measure how long it took to execute the run() methods of a Workload. + + """ + + name = 'execution_time' + + def initialize(self, context): + self.start_time = None + self.end_time = None + + @very_fast + def start(self, context): + self.start_time = time.time() + + @very_fast + def stop(self, context): + self.end_time = time.time() + + def update_output(self, context): + execution_time = self.end_time - self.start_time + context.add_metric('execution_time', execution_time, 'seconds') + + +.. include:: developer_information/developer_reference/instrument_method_map.rst + +.. _adding-an-output-processor: + +Adding an Output processor +---------------------------- + +A output processor is responsible for processing the results. This may +involve formatting and writing them to a file, uploading them to a database, +generating plots, etc. WA comes with a few output processors that output +results in a few common formats (such as csv or JSON). + +You can add your own output processors by creating a Python file in +``~/.workload_automation/plugins`` with a class that derives from +:class:`wa.OutputProcessor `, and should +implement the relevant methods from the following interface: + + :name: + + The name of the output processor, this must be unique to WA. + + :description: + + A description of what the output processor can be used for. + + :parameters: + + A list of additional :class:`Parameters` the output processor can take. + + :initialize(): + + This method will only be called once during the workload run + therefore operations that only need to be performed initially should + be performed here. + + :process_job_output(output, target_info, run_ouput): + + This method should be used to perform the processing of the + output from an individual job output. This is where any + additional artifacts should be generated if applicable. + + :export_job_output(output, target_info, run_ouput): + + This method should be used to perform the exportation of the + existing data collected/generated for an individual job. E.g. + uploading them to a database etc. + + :process_run_output(output, target_info): + + This method should be used to perform the processing of the + output from the run as a whole. This is where any + additional artifacts should be generated if applicable. + + :export_run_output(output, target_info): + + This method should be used to perform the exportation of the + existing data collected/generated for the run as a whole. E.g. + uploading them to a database etc. + + :finalize(): + + This method is the complement to the initialize method and will also + only be called once. + + +The method names should be fairly self-explanatory. The difference between +"process" and "export" methods is that export methods will be invoked after +process methods for all output processors have been generated. Process methods +may generate additional artifacts (metrics, files, etc.), while export methods +should not -- they should only handle existing results (upload them to a +database, archive on a filer, etc). + +The output object passed to job methods is an instance of +:class:`wa.framework.output.JobOutput`, the output object passed to run methods +is an instance of :class:`wa.RunOutput `. + + +Adding a Resource Getter +------------------------ + +A resource getter is a plugin that is designed to retrieve a resource +(binaries, APK files or additional workload assets). Resource getters are invoked in +priority order until one returns the desired resource. + +If you want WA to look for resources somewhere it doesn't by default (e.g. you +have a repository of APK files), you can implement a getter for the resource and +register it with a higher priority than the standard WA getters, so that it gets +invoked first. + +Instances of a resource getter should implement the following interface:: + + class ResourceGetter(Plugin): + + name = None + + def register(self, resolver): + raise NotImplementedError() + +The getter should define a name for itself (as with all plugins), in addition it +should implement the ``register`` method. This involves registering a method +with the resolver that should used to be called when trying to retrieve a resource +(typically ``get``) along with it's priority (see `Getter Prioritization`_ +below. That method should return an instance of the resource that +has been discovered (what "instance" means depends on the resource, e.g. it +could be a file path), or ``None`` if this getter was unable to discover +that resource. + +Getter Prioritization +^^^^^^^^^^^^^^^^^^^^^ + +A priority is an integer with higher numeric values indicating a higher +priority. The following standard priority aliases are defined for getters: + + + :preferred: Take this resource in favour of the environment resource. + :local: Found somewhere under ~/.workload_automation/ or equivalent, or + from environment variables, external configuration files, etc. + These will override resource supplied with the package. + :lan: Resource will be retrieved from a locally mounted remote location + (such as samba share) + :remote: Resource will be downloaded from a remote location (such as an HTTP + server) + :package: Resource provided with the package. + +These priorities are defined as class members of +:class:`wa.framework.resource.SourcePriority`, e.g. ``SourcePriority.preferred``. + +Most getters in WA will be registered with either ``local`` or +``package`` priorities. So if you want your getter to override the default, it +should typically be registered as ``preferred``. + +You don't have to stick to standard priority levels (though you should, unless +there is a good reason). Any integer is a valid priority. The standard priorities +range from 0 to 40 in increments of 10. + +Example +^^^^^^^ + +The following is an implementation of a getter that searches for files in the +users dependencies directory, typically +``~/.workload_automation/dependencies/`` It uses the +``get_from_location`` method to filter the available files in the provided +directory appropriately:: + + import sys + + from wa import settings, + from wa.framework.resource import ResourceGetter, SourcePriority + from wa.framework.getters import get_from_location + + class UserDirectory(ResourceGetter): + + name = 'user' + + def register(self, resolver): + resolver.register(self.get, SourcePriority.local) + + def get(self, resource): + basepath = settings.dependencies_directory + directory = _d(os.path.join(basepath, resource.owner.name)) + return get_from_location(directory, resource) + +.. _adding_a_target: + +Adding a Target +--------------- + +In WA3, a 'target' consists of a platform and a devlib target. The +implementations of the targets are located in ``devlib``. WA3 will instantiate a +devlib target passing relevant parameters parsed from the configuration. For +more information about devlib targets please see `the documentation +`_. + +The currently available platforms are: + :generic: The 'standard' platform implementation of the target, this should + work for the majority of use cases. + :juno: A platform implementation specifically for the juno. + :tc2: A platform implementation specifically for the tc2. + :gem5: A platform implementation to interact with a gem5 simulation. + +The currently available targets from devlib are: + :linux: A device running a Linux based OS. + :android: A device running Android OS. + :local: Used to run locally on a linux based host. + :chromeos: A device running ChromeOS, supporting an android container if available. + +For an example of adding you own customized version of an existing devlib target, +please see the how to section :ref:`Adding a Custom Target `. + + +Other Plugin Types +--------------------- + +In addition to plugin types covered above, there are few other, more +specialized ones. They will not be covered in as much detail. Most of them +expose relatively simple interfaces with only a couple of methods and it is +expected that if the need arises to extend them, the API-level documentation +that accompanies them, in addition to what has been outlined here, should +provide enough guidance. + +:commands: This allows extending WA with additional sub-commands (to supplement + exiting ones outlined in the :ref:`invocation` section). +:modules: Modules are "plugins for plugins". They can be loaded by other + plugins to expand their functionality (for example, a flashing + module maybe loaded by a device in order to support flashing). + + +Packaging Your Plugins +---------------------- + +If your have written a bunch of plugins, and you want to make it easy to +deploy them to new systems and/or to update them on existing systems, you can +wrap them in a Python package. You can use ``wa create package`` command to +generate appropriate boiler plate. This will create a ``setup.py`` and a +directory for your package that you can place your plugins into. + +For example, if you have a workload inside ``my_workload.py`` and a result +processor in ``my_result_processor.py``, and you want to package them as +``my_wa_exts`` package, first run the create command :: + + wa create package my_wa_exts + +This will create a ``my_wa_exts`` directory which contains a +``my_wa_exts/setup.py`` and a subdirectory ``my_wa_exts/my_wa_exts`` which is +the package directory for your plugins (you can rename the top-level +``my_wa_exts`` directory to anything you like -- it's just a "container" for the +setup.py and the package directory). Once you have that, you can then copy your +plugins into the package directory, creating +``my_wa_exts/my_wa_exts/my_workload.py`` and +``my_wa_exts/my_wa_exts/my_result_processor.py``. If you have a lot of +plugins, you might want to organize them into subpackages, but only the +top-level package directory is created by default, and it is OK to have +everything in there. + +.. note:: When discovering plugins through this mechanism, WA traverses the + Python module/submodule tree, not the directory structure, therefore, + if you are going to create subdirectories under the top level directory + created for you, it is important that your make sure they are valid + Python packages; i.e. each subdirectory must contain a __init__.py + (even if blank) in order for the code in that directory and its + subdirectories to be discoverable. + +At this stage, you may want to edit ``params`` structure near the bottom of +the ``setup.py`` to add correct author, license and contact information (see +"Writing the Setup Script" section in standard Python documentation for +details). You may also want to add a README and/or a COPYING file at the same +level as the setup.py. Once you have the contents of your package sorted, +you can generate the package by running :: + + cd my_wa_exts + python setup.py sdist + +This will generate ``my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz`` package which +can then be deployed on the target system with standard Python package +management tools, e.g. :: + + sudo pip install my_wa_exts-0.0.1.tar.gz + +As part of the installation process, the setup.py in the package, will write the +package's name into ``~/.workoad_automation/packages``. This will tell WA that +the package contains plugin and it will load them next time it runs. + +.. note:: There are no uninstall hooks in ``setuputils``, so if you ever + uninstall your WA plugins package, you will have to manually remove + it from ``~/.workload_automation/packages`` otherwise WA will complain + about a missing package next time you try to run it. diff --git a/doc/source/developer_information/developer_reference.rst b/doc/source/developer_information/developer_reference.rst index 47f674af..58683ae4 100644 --- a/doc/source/developer_information/developer_reference.rst +++ b/doc/source/developer_information/developer_reference.rst @@ -13,16 +13,17 @@ Developer Reference ----------------- -.. include:: developer_information/developer_reference/writing_plugins.rst +.. include:: developer_information/developer_reference/plugins.rst ----------------- -.. include:: developer_information/developer_reference/contributing.rst +.. include:: developer_information/developer_reference/revent.rst ----------------- -.. include:: developer_information/developer_reference/revent.rst +.. include:: developer_information/developer_reference/serialization.rst ----------------- -.. include:: developer_information/developer_reference/serialization.rst +.. include:: developer_information/developer_reference/contributing.rst + diff --git a/doc/source/developer_information/developer_reference/plugins.rst b/doc/source/developer_information/developer_reference/plugins.rst new file mode 100644 index 00000000..638ff9fa --- /dev/null +++ b/doc/source/developer_information/developer_reference/plugins.rst @@ -0,0 +1,376 @@ +.. plugins: + + +Plugins +======= + +Workload Automation offers several plugin points (or plugin types). The most +interesting of these are + +:workloads: These are the tasks that get executed and measured on the device. These + can be benchmarks, high-level use cases, or pretty much anything else. +:targets: These are interfaces to the physical devices (development boards or end-user + devices, such as smartphones) that use cases run on. Typically each model of a + physical device would require its own interface class (though some functionality + may be reused by subclassing from an existing base). +:instruments: Instruments allow collecting additional data from workload execution (e.g. + system traces). Instruments are not specific to a particular workload. Instruments + can hook into any stage of workload execution. +:output processors: These are used to format the results of workload execution once they have been + collected. Depending on the callback used, these will run either after each + iteration and/or at the end of the run, after all of the results have been + collected. + +You can create a plugin by subclassing the appropriate base class, defining +appropriate methods and attributes, and putting the .py file containing the +class into the "plugins" subdirectory under ``~/.workload_automation`` (or +equivalent) where it will be automatically picked up by WA. + + +Plugin Basics +-------------- + +This section contains reference information common to plugins of all types. + +.. _metrics: + +Metrics +^^^^^^^ +This is what WA uses to store a single metric collected from executing a workload. + + :name: the name of the metric. Uniquely identifies the metric + within the results. + :value: The numerical value of the metric for this execution of a + workload. This can be either an int or a float. + :units: Units for the collected value. Can be None if the value + has no units (e.g. it's a count or a standardised score). + :lower_is_better: Boolean flag indicating where lower values are + better than higher ones. Defaults to False. + :classifiers: A set of key-value pairs to further classify this + metric beyond current iteration (e.g. this can be used + to identify sub-tests). + +Metrics can be added to WA output via the context: + + +.. code-block:: python + + context.add_metric("score", 9001) + context.add_metric("time", 2.35, "seconds", lower_is_better=True) + +You only need to specify the name and the value for the metric. Units and +classifiers are optional, and, if not specified otherwise, it will be assumed +that higher values are better (lower_is_better=False). + +The metric will be added to the result for the current job, if there is one; +otherwise, it will be added to the overall run result. + +.. _artifact: + +Artifacts +^^^^^^^^^ +This is an artifact generated during execution/post-processing of a workload. +Unlike :ref:`metrics `, this represents an actual artifact, such as a +file, generated. This may be "output", such as trace, or it could be "meta +data" such as logs. These are distinguished using the ``kind`` attribute, which +also helps WA decide how it should be handled. Currently supported kinds are: + + :log: A log file. Not part of the "output" as such but contains + information about the run/workload execution that be useful for + diagnostics/meta analysis. + :meta: A file containing metadata. This is not part of the "output", but + contains information that may be necessary to reproduce the + results (contrast with ``log`` artifacts which are *not* + necessary). + :data: This file contains new data, not available otherwise and should + be considered part of the "output" generated by WA. Most traces + would fall into this category. + :export: Exported version of results or some other artifact. This + signifies that this artifact does not contain any new data + that is not available elsewhere and that it may be safely + discarded without losing information. + :raw: Signifies that this is a raw dump/log that is normally processed + to extract useful information and is then discarded. In a sense, + it is the opposite of ``export``, but in general may also be + discarded. + + .. note:: whether a file is marked as ``log``/``data`` or ``raw`` + depends on how important it is to preserve this file, + e.g. when archiving, vs how much space it takes up. + Unlike ``export`` artifacts which are (almost) always + ignored by other exporters as that would never result + in data loss, ``raw`` files *may* be processed by + exporters if they decided that the risk of losing + potentially (though unlikely) useful data is greater + than the time/space cost of handling the artifact (e.g. + a database uploader may choose to ignore ``raw`` + artifacts, whereas a network filer archiver may choose + to archive them). + + .. note: The kind parameter is intended to represent the logical + function of a particular artifact, not it's intended means of + processing -- this is left entirely up to the output + processors. + +As with :ref:`metrics`, artifacts are added via the context: + +.. code-block:: python + + context.add_artifact("benchmark-output", "bech-out.txt", kind="raw", + description="stdout from running the benchmark") + +.. note:: The file *must* exist on the host by the point at which the artifact + is added, otherwise an error will be raised. + +The artifact will be added to the result of the current job, if there is one; +otherwise, it will be added to the overall run result. In some situations, you +may wish to add an artifact to the overall run while being inside a job context, +this can be done with ``add_run_artifact``: + +.. code-block:: python + + context.add_run_artifact("score-summary", "scores.txt", kind="export", + description=""" + Summary of the scores so far. Updated after + every job. + """) + +In this case, you also need to make sure that the file represented by the +artifact is written to the output directory for the run and not the current job. + +.. _metadata: + +Metadata +^^^^^^^^ + +There may be additional data collected by your plugin that you want to record as +part of the result, but that does not fall under the definition of a "metric". +For example, you may want to record the version of the binary you're executing. +You can do this by adding a metadata entry: + +.. code-block:: python + + context.add_metadata("exe-version", 1.3) + + +Metadata will be added either to the current job result, or to the run result, +depending on the current context. Metadata values can be scalars or nested +structures of dicts/sequences; the only constraint is that all constituent +objects of the value must be POD (Plain Old Data) types -- see :ref:`WA POD +types `. + +There is special support for handling metadata entries that are dicts of values. +The following call adds a metadata entry ``"versions"`` who's value is +``{"my_exe": 1.3}``: + +.. code-block:: python + + context.add_metadata("versions", "my_exe", 1.3) + +If you attempt to add a metadata entry that already exists, an error will be +raised, unless ``force=True`` is specified, in which case, it will be +overwritten. + +Updating an existing entry whose value is a collection can be done with +``update_metadata``: + +.. code-block:: python + + context.update_metadata("ran_apps", "my_exe") + context.update_metadata("versions", "my_other_exe", "2.3.0") + +The first call appends ``"my_exe"`` to the list at metadata entry +``"ran_apps"``. The second call updates the ``"versions"`` dict in the metadata +with an entry for ``"my_other_exe"``. + +If an entry does not exit, ``update_metadata`` will create it, so it's +recommended to always use that for non-scalar entries, unless the intention is +specifically to ensure that the entry does not exist at the time of the call. + +.. _classifiers: + +Classifiers +^^^^^^^^^^^ + +Classifiers are key-value pairs of tags that can be attached to metrics, +artifacts, jobs, or the entire run. Run and job classifiers get propagated to +metrics and artifacts. Classifier keys should be strings, and their values +should be simple scalars (i.e. strings, numbers, or bools). + +Classifiers can be thought of as "tags" that are used to annotate metrics and +artifacts, in order to make it easier to sort through them later. WA itself does +not do anything with them, however output processors will augment the output +they generate with them (for example, ``csv`` processor can add additional +columns for classifier keys). + +Classifiers are typically added by the user to attach some domain-specific +information (e.g. experiment configuration identifier) to the results, see +:ref:`using classifiers `. However, plugins can also attach +additional classifiers, by specifying them in ``add_metric()`` and +``add_artifacts()`` calls. + + +Metadata vs Classifiers +^^^^^^^^^^^^^^^^^^^^^^^ + +Both metadata and classifiers are sets of essentially opaque key-value pairs +that get included in WA output. While they may seem somewhat similar and +interchangeable, they serve different purposes and are handled differently by +the framework. + +Classifiers are used to annotate generated metrics and artifacts in order to +assist post-processing tools in sorting through them. Metadata is used to record +additional information that is not necessary for processing the results, but +that may be needed in order to reproduce them or to make sense of them in a +grander context. + +These are specific differences in how they are handled: + +- Classifiers are often provided by the user via the agenda (though can also be + added by plugins). Metadata in only created by the framework and plugins. +- Classifier values must be simple scalars; metadata values can be nested + collections, such as lists or dicts. +- Classifiers are used by output processors to augment the output the latter + generated; metadata typically isn't. +- Classifiers are essentially associated with the individual metrics and + artifacts (though in the agenda they're specified at workload, section, or + global run levels); metadata is associated with a particular job or run, and + not with metrics or artifacts. + + +.. _execution-decorators: + +Execution Decorators +--------------------- + +The following decorators are available for use in order to control how often a +method should be able to be executed. + +For example, if we want to ensure that no matter how many iterations of a +particular workload are ran, we only execute the initialize method for that instance +once, we would use the decorator as follows: + +.. code-block:: python + + from wa.utils.exec_control import once + + @once + def initialize(self, context): + # Perform one time initialization e.g. installing a binary to target + # .. + +@once_per_instance +^^^^^^^^^^^^^^^^^^ +The specified method will be invoked only once for every bound instance within +the environment. + +@once_per_class +^^^^^^^^^^^^^^^ +The specified method will be invoked only once for all instances of a class +within the environment. + +@once +^^^^^ +The specified method will be invoked only once within the environment. + +.. warning:: If a method containing a super call is decorated, this will also cause + stop propagation up the hierarchy, unless this is the desired + effect, additional functionality should be implemented in a + separate decorated method which can then be called allowing for + normal propagation to be retained. + + + + +Utils +^^^^^ + +Workload Automation defines a number of utilities collected under +:mod:`wa.utils` subpackage. These utilities were created to help with the +implementation of the framework itself, but may be also be useful when +implementing plugins. + +-------------------- + +Workloads +--------- + +All of the type inherit from the same base :class:`Workload` and its API can be +seen in the :ref:`API ` section. + +Workload methods (except for ``validate``) take a single argument that is a +:class:`wa.framework.execution.ExecutionContext` instance. This object keeps +track of the current execution state (such as the current workload, iteration +number, etc), and contains, among other things, a +:class:`wa.framework.output.JobOutput` instance that should be populated from +the ``update_output`` method with the results of the execution. For more +information please see `the context`_ documentation. :: + + # ... + + def update_output(self, context): + # ... + context.add_metric('energy', 23.6, 'Joules', lower_is_better=True) + + # ... + +.. _workload-types: + +Workload Types +^^^^^^^^^^^^^^^^ + +There are multiple workload types that you can inherit from depending on the +purpose of your workload, the different types along with an output of their +intended use cases are outlined below. + +.. _basic-workload: + +Basic (:class:`wa.Workload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This type of the workload is the simplest type of workload and is left the to +developer to implement its full functionality. + + +.. _apk-workload: + +Apk (:class:`wa.ApkWorkload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This workload will simply deploy and launch an android app in its basic form +with no UI interaction. + +.. _uiautomator-workload: + + +UiAuto (:class:`wa.UiautoWorkload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This workload is for android targets which will use UiAutomator to interact with +UI elements without a specific android app, for example performing manipulation +of android itself. This is the preferred type of automation as the results are +more portable and reproducible due to being able to wait for UI elements to +appear rather than having to rely on human recordings. + +.. _apkuiautomator-workload: + +ApkUiAuto (:class:`wa.ApkUiautoWorkload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The is the same as the UiAuto workload however it is also associated with an +android app e.g. AdobeReader and will automatically deploy and launch the +android app before running the automation. + +.. _revent-workload: + +Revent (:class:`wa.ReventWorkload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Revent workloads are designed primarily for games as these are unable to be +automated with UiAutomator due to the fact that they are rendered within a +single UI element. They require a recording to be performed manually and +currently will need re-recording for each different device. For more +information on revent workloads been please see :ref:`revent_files_creation` + +.. _apkrevent-workload: + +APKRevent (:class:`wa.ApkReventWorkload `) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The is the same as the Revent workload however it is also associated with an +android app e.g. AngryBirds and will automatically deploy and launch the android +app before running the automation. diff --git a/doc/source/developer_information/developer_reference/writing_plugins.rst b/doc/source/developer_information/developer_reference/writing_plugins.rst deleted file mode 100644 index de66486d..00000000 --- a/doc/source/developer_information/developer_reference/writing_plugins.rst +++ /dev/null @@ -1,1315 +0,0 @@ -.. _writing-plugins: - - -Writing Plugins -================ - -Workload Automation offers several plugin points (or plugin types). The most -interesting of these are - -:workloads: These are the tasks that get executed and measured on the device. These - can be benchmarks, high-level use cases, or pretty much anything else. -:targets: These are interfaces to the physical devices (development boards or end-user - devices, such as smartphones) that use cases run on. Typically each model of a - physical device would require its own interface class (though some functionality - may be reused by subclassing from an existing base). -:instruments: Instruments allow collecting additional data from workload execution (e.g. - system traces). Instruments are not specific to a particular workload. Instruments - can hook into any stage of workload execution. -:output processors: These are used to format the results of workload execution once they have been - collected. Depending on the callback used, these will run either after each - iteration and/or at the end of the run, after all of the results have been - collected. - -You can create a plugin by subclassing the appropriate base class, defining -appropriate methods and attributes, and putting the .py file containing the -class into the "plugins" subdirectory under ``~/.workload_automation`` (or -equivalent) where it will be automatically picked up by WA. - - -Plugin Basics --------------- - -This sub-section covers things common to implementing plugins of all types. It -is recommended you familiarize yourself with the information here before -proceeding onto guidance for specific plugin types. - -.. _context: - -The Context -^^^^^^^^^^^ - -The majority of methods in plugins accept a context argument. This is an -instance of :class:`wa.framework.execution.ExecutionContext`. It contains -information about the current state of execution of WA and keeps track of things -like which workload is currently running. - -Notable methods of the context are: - -context.add_artifact(name, host_file_path, kind, description=None, classifier=None) - Plugins can add :ref:`artifacts ` of various kinds to the run - output directory for WA and associate them with a description and/or - :ref:`classifier `. - -context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None) - This method should be used to add :ref:`metrics ` that have been - generated from a workload, this will allow WA to process the results - accordingly depending on which output processors are enabled. - -Notable attributes of the context are: - -context.workload - :class:`wa.framework.workload` object that is currently being executed. - -context.tm - This is the target manager that can be used to access various information - about the target including initialization parameters. - -context.current_job - This is an instance of :class:`wa.framework.job.Job` and contains all - the information relevant to the workload job currently being executed. - -context.current_job.spec - The current workload specification being executed. This is an - instance of :class:`wa.framework.configuration.core.JobSpec` - and defines the workload and the parameters under which it is - being executed. - -context.current_job.current_iteration - The current iteration of the spec that is being executed. Note that this - is the iteration for that spec, i.e. the number of times that spec has - been run, *not* the total number of all iterations have been executed so - far. - -context.current_job_output - This is the result object for the current iteration. This is an instance - of :class:`wa.framework.output.JobOutput`. It contains the status - of the iteration as well as the metrics and artifacts generated by the - workload. - - -In addition to these, context also defines a few useful paths (see below). - - -Paths -^^^^^ - -You should avoid using hard-coded absolute paths in your plugins whenever -possible, as they make your code too dependent on a particular environment and -may mean having to make adjustments when moving to new (host and/or device) -platforms. To help avoid hard-coded absolute paths, WA defines a number of -standard locations. You should strive to define your paths relative -to one of these. - -On the host -~~~~~~~~~~~ - -Host paths are available through the context object, which is passed to most -plugin methods. - -context.run_output_directory - This is the top-level output directory for all WA results (by default, - this will be "wa_output" in the directory in which WA was invoked. - -context.output_directory - This is the output directory for the current iteration. This will an - iteration-specific subdirectory under the main results location. If - there is no current iteration (e.g. when processing overall run results) - this will point to the same location as ``root_output_directory``. - - -Additionally, the global ``wa.settings`` object exposes on other location: - -settings.dependency_directory - this is the root directory for all plugin dependencies (e.g. media - files, assets etc) that are not included within the plugin itself. - -As per Python best practice, it is recommended that methods and values in -``os.path`` standard library module are used for host path manipulation. - -On the target -~~~~~~~~~~~~~ - -Workloads and instruments have a ``target`` attribute, which is an interface to -the target used by WA. It defines the following location: - -target.working_directory - This is the directory for all WA-related files on the target. All files - deployed to the target should be pushed to somewhere under this location - (the only exception being executables installed with ``target.install`` - method). - -Since there could be a mismatch between path notation used by the host and the -target, the ``os.path`` modules should *not* be used for on-target path -manipulation. Instead target has an equipment module exposed through -``target.path`` attribute. This has all the same attributes and behaves the -same way as ``os.path``, but is guaranteed to produce valid paths for the target, -irrespective of the host's path notation. For example: - -.. code:: python - - result_file = self.target.path.join(self.target.working_directory, "result.txt") - self.command = "{} -a -b -c {}".format(target_binary, result_file) - -.. note:: Output processors, unlike workloads and instruments, do not have their - own target attribute as they are designed to be able to be ran offline. - -.. _metrics: - -Metrics -^^^^^^^ -This is what WA uses to store a single metric collected from executing a workload. - - :name: the name of the metric. Uniquely identifies the metric - within the results. - :value: The numerical value of the metric for this execution of a - workload. This can be either an int or a float. - :units: Units for the collected value. Can be None if the value - has no units (e.g. it's a count or a standardised score). - :lower_is_better: Boolean flag indicating where lower values are - better than higher ones. Defaults to False. - :classifiers: A set of key-value pairs to further classify this - metric beyond current iteration (e.g. this can be used - to identify sub-tests). - -Metrics can be added to WA output via the context: - - -.. code-block:: python - - context.add_metric("score", 9001) - context.add_metric("time", 2.35, "seconds", lower_is_better=True) - -You only need to specify the name and the value for the metric. Units and -classifiers are optional, and, if not specified otherwise, it will be assumed -that higher values are better (lower_is_better=False). - -The metric will be added to the result for the current job, if there is one; -otherwise, it will be added to the overall run result. - -.. _artifact: - -Artifacts -^^^^^^^^^ -This is an artifact generated during execution/post-processing of a workload. -Unlike :ref:`metrics `, this represents an actual artifact, such as a -file, generated. This may be "output", such as trace, or it could be "meta -data" such as logs. These are distinguished using the ``kind`` attribute, which -also helps WA decide how it should be handled. Currently supported kinds are: - - :log: A log file. Not part of the "output" as such but contains - information about the run/workload execution that be useful for - diagnostics/meta analysis. - :meta: A file containing metadata. This is not part of the "output", but - contains information that may be necessary to reproduce the - results (contrast with ``log`` artifacts which are *not* - necessary). - :data: This file contains new data, not available otherwise and should - be considered part of the "output" generated by WA. Most traces - would fall into this category. - :export: Exported version of results or some other artifact. This - signifies that this artifact does not contain any new data - that is not available elsewhere and that it may be safely - discarded without losing information. - :raw: Signifies that this is a raw dump/log that is normally processed - to extract useful information and is then discarded. In a sense, - it is the opposite of ``export``, but in general may also be - discarded. - - .. note:: whether a file is marked as ``log``/``data`` or ``raw`` - depends on how important it is to preserve this file, - e.g. when archiving, vs how much space it takes up. - Unlike ``export`` artifacts which are (almost) always - ignored by other exporters as that would never result - in data loss, ``raw`` files *may* be processed by - exporters if they decided that the risk of losing - potentially (though unlikely) useful data is greater - than the time/space cost of handling the artifact (e.g. - a database uploader may choose to ignore ``raw`` - artifacts, whereas a network filer archiver may choose - to archive them). - - .. note: The kind parameter is intended to represent the logical - function of a particular artifact, not it's intended means of - processing -- this is left entirely up to the output - processors. - -As with :ref:`metrics`, artifacts are added via the context: - -.. code-block:: python - - context.add_artifact("benchmark-output", "bech-out.txt", kind="raw", - description="stdout from running the benchmark") - -.. note:: The file *must* exist on the host by the point at which the artifact - is added, otherwise an error will be raised. - -The artifact will be added to the result of the current job, if there is one; -otherwise, it will be added to the overall run result. In some situations, you -may wish to add an artifact to the overall run while being inside a job context, -this can be done with ``add_run_artifact``: - -.. code-block:: python - - context.add_run_artifact("score-summary", "scores.txt", kind="export", - description=""" - Summary of the scores so far. Updated after - every job. - """) - -In this case, you also need to make sure that the file represented by the -artifact is written to the output directory for the run and not the current job. - -.. _metadata: - -Metadata -^^^^^^^^ - -There may be additional data collected by your plugin that you want to record as -part of the result, but that does not fall under the definition of a "metric". -For example, you may want to record the version of the binary you're executing. -You can do this by adding a metadata entry: - -.. code-block:: python - - context.add_metadata("exe-version", 1.3) - - -Metadata will be added either to the current job result, or to the run result, -depending on the current context. Metadata values can be scalars or nested -structures of dicts/sequences; the only constraint is that all constituent -objects of the value must be POD (Plain Old Data) types -- see :ref:`WA POD -types `. - -There is special support for handling metadata entries that are dicts of values. -The following call adds a metadata entry ``"versions"`` who's value is -``{"my_exe": 1.3}``: - -.. code-block:: python - - context.add_metadata("versions", "my_exe", 1.3) - -If you attempt to add a metadata entry that already exists, an error will be -raised, unless ``force=True`` is specified, in which case, it will be -overwritten. - -Updating an existing entry whose value is a collection can be done with -``update_metadata``: - -.. code-block:: python - - context.update_metadata("ran_apps", "my_exe") - context.update_metadata("versions", "my_other_exe", "2.3.0") - -The first call appends ``"my_exe"`` to the list at metadata entry -``"ran_apps"``. The second call updates the ``"versions"`` dict in the metadata -with an entry for ``"my_other_exe"``. - -If an entry does not exit, ``update_metadata`` will create it, so it's -recommended to always use that for non-scalar entries, unless the intention is -specifically to ensure that the entry does not exist at the time of the call. - -Classifiers -^^^^^^^^^^^ - -Classifiers are key-value pairs of tags that can be attached to metrics, -artifacts, jobs, or the entire run. Run and job classifiers get propagated to -metrics and artifacts. Classifier keys should be strings, and their values -should be simple scalars (i.e. strings, numbers, or bools). - -Classifiers can be thought of as "tags" that are used to annotate metrics and -artifacts, in order to make it easier to sort through them later. WA itself does -not do anything with them, however output processors will augment the output -they generate with them (for example, ``csv`` processor can add additional -columns for classifier keys). - -Classifiers are typically added by the user to attach some domain-specific -information (e.g. experiment configuration identifier) to the results, see -:ref:`classifiers`. However, plugins can also attach additional classifiers, by -specifying them in ``add_metric()`` and ``add_artifacts()`` calls. - - -Metadata vs Classifiers -^^^^^^^^^^^^^^^^^^^^^^^ - -Both metadata and classifiers are sets of essentially opaque key-value pairs -that get included in WA output. While they may seem somewhat similar and -interchangeable, they serve different purposes and are handled differently by -the framework. - -Classifiers are used to annotate generated metrics and artifacts in order to -assist post-processing tools in sorting through them. Metadata is used to record -additional information that is not necessary for processing the results, but -that may be needed in order to reproduce them or to make sense of them in a -grander context. - -These are specific differences in how they are handled: - -- Classifiers are often provided by the user via the agenda (though can also be - added by plugins). Metadata in only created by the framework and plugins. -- Classifier values must be simple scalars; metadata values can be nested - collections, such as lists or dicts. -- Classifiers are used by output processors to augment the output the latter - generated; metadata typically isn't. -- Classifiers are essentially associated with the individual metrics and - artifacts (though in the agenda they're specified at workload, section, or - global run levels); metadata is associated with a particular job or run, and - not with metrics or artifacts. - - -.. _resource-resolution: - -Dynamic Resource Resolution -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The idea is to decouple resource identification from resource discovery. -Workloads/instruments/devices/etc state *what* resources they need, and not -*where* to look for them -- this instead is left to the resource resolver that -is part of the execution context. The actual discovery of resources is -performed by resource getters that are registered with the resolver. - -A resource type is defined by a subclass of -:class:`wa.framework.resource.Resource`. An instance of this class describes a -resource that is to be obtained. At minimum, a ``Resource`` instance has an -owner (which is typically the object that is looking for the resource), but -specific resource types may define other parameters that describe an instance of -that resource (such as file names, URLs, etc). - -An object looking for a resource invokes a resource resolver with an instance of -``Resource`` describing the resource it is after. The resolver goes through the -getters registered for that resource type in priority order attempting to obtain -the resource; once the resource is obtained, it is returned to the calling -object. If none of the registered getters could find the resource, -``NotFoundError`` is raised (or ``None`` is returned instead, if invoked with -``strict=False``). - -The most common kind of object looking for resources is a ``Workload``, and the -``Workload`` class defines -:py:meth:`wa.framework.workload.Workload.init_resources` method, which may be -overridden by subclasses to perform resource resolution. For example, a workload -looking for an executable file would do so like this:: - - from wa import Workload - from wa.import Executable - - class MyBenchmark(Workload): - - # ... - - def init_resources(self, resolver): - resource = Executable(self, self.target.abi, 'my_benchmark') - host_exe = resolver.get(resource) - - # ... - - -Currently available resource types are defined in :py:mod:`wa.framework.resources`. - -.. _deploying-executables: - -Deploying executables to a target -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some targets may have certain restrictions on where executable binaries may be -placed and how they should be invoked. To ensure your plugin works with as -wide a range of targets as possible, you should use WA APIs for deploying and -invoking executables on a target, as outlined below. - -As with other resources, host-side paths to the executable binary to be deployed -should be obtained via the :ref:`resource resolver `. A -special resource type, ``Executable`` is used to identify a binary to be -deployed. This is similar to the regular ``File`` resource, however it takes an -additional parameter that specifies the ABI for which the executable was -compiled for. - -In order for the binary to be obtained in this way, it must be stored in one of -the locations scanned by the resource resolver in a directory structure -``/bin//`` (where ``root`` is the base resource location to -be searched, e.g. ``~/.workload_automation/dependencies/``, and -```` is the ABI for which the executable has been compiled, as returned by -``self.target.abi``). - -Once the path to the host-side binary has been obtained, it may be deployed -using one of two methods from a -`Target `_ instance -- -``install`` or ``install_if_needed``. The latter will check a version of that -binary has been previously deployed by WA and will not try to re-install. - -.. code:: python - - from wa import Executable - - host_binary = context.resolver.get(Executable(self, self.target.abi, 'some_binary')) - target_binary = self.target.install_if_needed(host_binary) - - -.. note:: Please also note that the check is done based solely on the binary name. - For more information please see the devlib - `documentation `_. - -Both of the above methods will return the path to the installed binary on the -target. The executable should be invoked *only* via that path; do **not** assume -that it will be in ``PATH`` on the target (or that the executable with the same -name in ``PATH`` is the version deployed by WA. - -For more information on how to implement this, please see the -:ref:`how to guide `. - - -Deploying assets ------------------ -WA provides a generic mechanism for deploying assets during workload initialization. -WA will automatically try to retrieve and deploy each asset to the target's working directory -that is contained in a workloads ``deployable_assets`` attribute stored as a list. - -If the parameter ``cleanup_assets`` is set then any asset deployed will be removed -again and the end of the run. - -If the workload requires a custom deployment mechanism the ``deploy_assets`` -method can be overridden for that particular workload, in which case, either -additional assets should have their on target paths added to the workload's -``deployed_assests`` attribute or the corresponding ``remove_assets`` method -should also be implemented. - -Parameters -^^^^^^^^^^ - -All plugins can be parametrized. Parameters are specified using -``parameters`` class attribute. This should be a list of -:class:`wa.framework.plugin.Parameter` instances. The following attributes can be -specified on parameter creation: - -:name: - This is the only mandatory argument. The name will be used to create a - corresponding attribute in the plugin instance, so it must be a valid - Python identifier. - -:kind: - This is the type of the value of the parameter. This must be an - callable. Normally this should be a standard Python type, e.g. ``int`` - or ``float``, or one the types defined in :mod:`wa.utils.types`. - If not explicitly specified, this will default to ``str``. - - .. note:: Irrespective of the ``kind`` specified, ``None`` is always a - valid value for a parameter. If you don't want to allow - ``None``, then set ``mandatory`` (see below) to ``True``. - -:allowed_values: - A list of the only allowed values for this parameter. - - .. note:: For composite types, such as ``list_of_strings`` or - ``list_of_ints`` in :mod:`wa.utils.types`, each element of - the value will be checked against ``allowed_values`` rather - than the composite value itself. - -:default: - The default value to be used for this parameter if one has not been - specified by the user. Defaults to ``None``. - -:mandatory: - A ``bool`` indicating whether this parameter is mandatory. Setting this - to ``True`` will make ``None`` an illegal value for the parameter. - Defaults to ``False``. - - .. note:: Specifying a ``default`` will mean that this parameter will, - effectively, be ignored (unless the user sets the param to ``None``). - - .. note:: Mandatory parameters are *bad*. If at all possible, you should - strive to provide a sensible ``default`` or to make do without - the parameter. Only when the param is absolutely necessary, - and there really is no sensible default that could be given - (e.g. something like login credentials), should you consider - making it mandatory. - -:constraint: - This is an additional constraint to be enforced on the parameter beyond - its type or fixed allowed values set. This should be a predicate (a function - that takes a single argument -- the user-supplied value -- and returns - a ``bool`` indicating whether the constraint has been satisfied). - -:override: - A parameter name must be unique not only within an plugin but also - with that plugin's class hierarchy. If you try to declare a parameter - with the same name as already exists, you will get an error. If you do - want to override a parameter from further up in the inheritance - hierarchy, you can indicate that by setting ``override`` attribute to - ``True``. - - When overriding, you do not need to specify every other attribute of the - parameter, just the ones you what to override. Values for the rest will - be taken from the parameter in the base class. - - -Validation and cross-parameter constraints -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A plugin will get validated at some point after construction. When exactly -this occurs depends on the plugin type, but it *will* be validated before it -is used. - -You can implement ``validate`` method in your plugin (that takes no arguments -beyond the ``self``) to perform any additional *internal* validation in your -plugin. By "internal", I mean that you cannot make assumptions about the -surrounding environment (e.g. that the device has been initialized). - -The contract for ``validate`` method is that it should raise an exception -(either ``wa.framework.exception.ConfigError`` or plugin-specific exception type -- see -further on this page) if some validation condition has not, and cannot, been met. -If the method returns without raising an exception, then the plugin is in a -valid internal state. - -Note that ``validate`` can be used not only to verify, but also to impose a -valid internal state. In particular, this where cross-parameter constraints can -be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on -another parameter, there is no way to express that declaratively when specifying -the parameters. In that case the dependent attribute should be left unspecified -on creation and should instead be set inside ``validate``. - -Logging -^^^^^^^ - -Every plugin class has it's own logger that you can access through -``self.logger`` inside the plugin's methods. Generally, a :class:`Target` will -log everything it is doing, so you shouldn't need to add much additional logging -for device actions. However you might what to log additional information, e.g. -what settings your plugin is using, what it is doing on the host, etc. -(Operations on the host will not normally be logged, so your plugin should -definitely log what it is doing on the host). One situation in particular where -you should add logging is before doing something that might take a significant -amount of time, such as downloading a file. - - -Documenting -^^^^^^^^^^^ - -All plugins and their parameter should be documented. For plugins -themselves, this is done through ``description`` class attribute. The convention -for an plugin description is that the first paragraph should be a short -summary description of what the plugin does and why one would want to use it -(among other things, this will get extracted and used by ``wa list`` command). -Subsequent paragraphs (separated by blank lines) can then provide a more -detailed description, including any limitations and setup instructions. - -For parameters, the description is passed as an argument on creation. Please -note that if ``default``, ``allowed_values``, or ``constraint``, are set in the -parameter, they do not need to be explicitly mentioned in the description (wa -documentation utilities will automatically pull those). If the ``default`` is set -in ``validate`` or additional cross-parameter constraints exist, this *should* -be documented in the parameter description. - -Both plugins and their parameters should be documented using reStructureText -markup (standard markup for Python documentation). See: - -http://docutils.sourceforge.net/rst.html - -Aside from that, it is up to you how you document your plugin. You should try -to provide enough information so that someone unfamiliar with your plugin is -able to use it, e.g. you should document all settings and parameters your -plugin expects (including what the valid values are). - - -Error Notification -^^^^^^^^^^^^^^^^^^ - -When you detect an error condition, you should raise an appropriate exception to -notify the user. The exception would typically be :class:`ConfigError` or -(depending the type of the plugin) -:class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`. -All these errors are defined in :mod:`wa.framework.exception` module. - -A :class:`ConfigError` should be raised where there is a problem in configuration -specified by the user (either through the agenda or config files). These errors -are meant to be resolvable by simple adjustments to the configuration (and the -error message should suggest what adjustments need to be made. For all other -errors, such as missing dependencies, mis-configured environment, problems -performing operations, etc., the plugin type-specific exceptions should be -used. - -If the plugin itself is capable of recovering from the error and carrying -on, it may make more sense to log an ERROR or WARNING level message using the -plugin's logger and to continue operation. - -.. _decorators: - -Execution Decorators ---------------------- -The following decorators are available for use in order to control how often a -method should be able to be executed. - -For example, if we want to ensure that no matter how many iterations of a -particular workload are ran, we only execute the initialize method for that instance -once, we would use the decorator as follows: - -.. code-block:: python - - from wa.utils.exec_control import once - - @once - def initialize(self, context): - # Perform one time initialization e.g. installing a binary to target - # .. - -@once_per_instance -^^^^^^^^^^^^^^^^^^ -The specified method will be invoked only once for every bound instance within -the environment. - -@once_per_class -^^^^^^^^^^^^^^^ -The specified method will be invoked only once for all instances of a class -within the environment. - -@once -^^^^^ -The specified method will be invoked only once within the environment. - -.. warning:: If a method containing a super call is decorated, this will also cause - stop propagation up the hierarchy, unless this is the desired - effect, additional functionality should be implemented in a - separate decorated method which can then be called allowing for - normal propagation to be retained. - - - - -Utils -^^^^^ - -Workload Automation defines a number of utilities collected under -:mod:`wa.utils` subpackage. These utilities were created to help with the -implementation of the framework itself, but may be also be useful when -implementing plugins. - -Workloads ---------- - -.. _workload-types: - -Workload Types -^^^^^^^^^^^^^^^^ - -.. _basic-workload: - -Basic (:class:`wa.Workload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This type of the workload is the simplest type of workload and is left the to -developer to implement its full functionality. - - -.. _apk-workload: - -Apk (:class:`wa.ApkWorkload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This workload will simply deploy and launch an android app in its basic form -with no UI interaction. - -.. _uiautomator-workload: - - -UiAuto (:class:`wa.UiautoWorkload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This workload is for android targets which will use UiAutomator to interact with -UI elements without a specific android app, for example performing manipulation -of android itself. This is the preferred type of automation as the results are -more portable and reproducible due to being able to wait for UI elements to -appear rather than having to rely on human recordings. - -.. _apkuiautomator-workload: - -ApkUiAuto (:class:`wa.ApkUiautoWorkload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The is the same as the UiAuto workload however it is also associated with an -android app e.g. AdobeReader and will automatically deploy and launch the -android app before running the automation. - -.. _revent-workload: - -Revent (:class:`wa.ReventWorkload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Revent workloads are designed primarily for games as these are unable to be -automated with UiAutomator due to the fact that they are rendered within a -single UI element. They require a recording to be performed manually and -currently will need re-recording for each different device. For more -information on revent workloads been please see :ref:`revent_files_creation` - -.. _apkrevent-workload: - -APKRevent (:class:`wa.ApkReventWorkload `) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The is the same as the Revent workload however it is also associated with an -android app e.g. AngryBirds and will automatically deploy and launch the android -app before running the automation. - - -.. _workload-interface: - -Workload Interface -^^^^^^^^^^^^^^^^^^^ -The workload interface should be implemented as follows: - -.. class:: (TargetedPlugin) - - .. attribute:: name - - This identifies the workload (e.g. it is used to specify the - workload in the :ref:`agenda `). - - .. method:: init_resources(context) - - This method may be optionally overridden to implement dynamic - resource discovery for the workload. This method executes - early on, before the device has been initialized, so it - should only be used to initialize resources that do not - depend on the device to resolve. This method is executed - once per run for each workload instance. - - .. method:: validate( - This method can be used to validate any assumptions your workload - makes about the environment (e.g. that required files are - present, environment variables are set, etc) and should raise a - :class:`wa.WorkloadError ` - if that is not the case. The base class implementation only makes - sure sure that the name attribute has been set. - - .. method:: initialize(context) - - This method is decorated with the ``@once_per_instance`` decorator, - (for more information please see `Execution Decorators`_) - therefore it will be executed exactly once per run (no matter - how many instances of the workload there are). It will run - after the device has been initialized, so it may be used to - perform device-dependent initialization that does not need to - be repeated on each iteration (e.g. as installing executables - required by the workload on the device). - - .. method:: setup(context) - - Everything that needs to be in place for workload execution should - be done in this method. This includes copying files to the device, - starting up an application, configuring communications channels, - etc. - - .. method:: setup_rerun(context) - - Everything that needs to be in place for workload execution should - be done in this method. This includes copying files to the device, - starting up an application, configuring communications channels, - etc. - - .. method:: run(context) - - This method should perform the actual task that is being measured. - When this method exits, the task is assumed to be complete. - - .. note:: Instruments are kicked off just before calling this - method and disabled right after, so everything in this - method is being measured. Therefore this method should - contain the least code possible to perform the operations - you are interested in measuring. Specifically, things like - installing or starting applications, processing results, or - copying files to/from the device should be done elsewhere if - possible. - - .. method:: extract_results(context) - - This method gets invoked after the task execution has finished and - should be used to extract metrics from the target. - - .. method:: update_output(context) - - This method should be used to update the output within the specified - execution context with the metrics and artifacts from this - workload iteration. - - .. method:: teardown(context) - - This could be used to perform any cleanup you may wish to do, e.g. - Uninstalling applications, deleting file on the device, etc. - - .. method:: finalize(context) - - This is the complement to ``initialize``. This will be executed - exactly once at the end of the run. This should be used to - perform any final clean up (e.g. uninstalling binaries installed - in the ``initialize``). - -Workload methods (except for ``validate``) take a single argument that is a -:class:`wa.framework.execution.ExecutionContext` instance. This object keeps -track of the current execution state (such as the current workload, iteration -number, etc), and contains, among other things, a -:class:`wa.framework.output.JobOutput` instance that should be populated from -the ``update_output`` method with the results of the execution. For more -information please see `the context`_ documentation. :: - - # ... - - def update_output(self, context): - # ... - context.add_metric('energy', 23.6, 'Joules', lower_is_better=True) - - # ... - -.. _ReventWorkload: - -Adding Revent Workload ------------------------ - -There are two base classes that can be subclassed to create Revent based workloads -depending on whether the workload is associated with an android Apk or not -:class:`wa.ApkReventWorkload ` and -:class:`wa.ReventWorkload ` respectively. -They both implement all the methods needed to push the files to the device and run -them. - -The revent workload classes define the following interfaces:: - - class ReventWorkload(Workload): - - name = None - - class ApkReventWorkload(Workload): - - name = None - package_names = [] - -The interface should be implemented as follows - - :name: This identifies the workload (e.g. it used to specify it in the - :ref:`agenda `. - :package_names: This is a list of the android application apk packages names that - are required to run the workload. - - -.. _instrument-reference: - -Adding an Instrument ---------------------- -Instruments can be used to collect additional measurements during workload -execution (e.g. collect power readings). An instrument can hook into almost any -stage of workload execution. Any new instrument should be a subclass of -Instrument and it must have a name. When a new instrument is added to Workload -Automation, the methods of the new instrument will be found automatically and -hooked up to the supported signals. Once a signal is broadcasted, the -corresponding registered method is invoked. - -Each method in ``Instrument`` must take two arguments, which are ``self`` and -``context``. Supported methods and their corresponding signals can be found in -the :ref:`Signals Documentation `. To make -implementations easier and common, the basic steps to add new instrument is -similar to the steps to add new workload and an example can be found in the -:ref:`How To ` section. - -.. _instrument-api: - -To implement your own the relevant methods of the interface shown below should be implemented: - -.. class:: Instrument(TargetedInstrument) - - .. attribute:: name - - The name of the instrument, this must be unique to WA. - - .. attribute:: description - - A description of what the instrument can be used for. - - .. attribute:: parameters - - A list of additional :class:`Parameters` the instrument can take. - - .. method:: initialize(context): - - This method will only be called once during the workload run - therefore operations that only need to be performed initially should - be performed here for example pushing the files to the target device, - installing them. - - .. method:: setup(context): - - This method is invoked after the workload is setup. All the - necessary setup should go inside this method. Setup, includes - operations like clearing logs, additional configuration etc. - - .. method:: start(context): - - It is invoked just before the workload start execution. Here is - where instrument measures start being registered/taken. - - .. method:: stop(context): - - It is invoked just after the workload execution stops. The measures - should stop being taken/registered. - - .. method:: update_output(context): - - It is invoked after the workload updated its result. - update_result is where the taken measures are added to the result so it - can be processed by Workload Automation. - - .. method:: teardown(context): - - It is invoked after the workload is torn down. It is a good place - to clean any logs generated by the instrument. - - .. method:: finalize(context): - - This method is the complement to the initialize method and will also - only be called once so should be used to deleting/uninstalling files - pushed to the device. - - -This is similar to a ``Workload``, except all methods are optional. In addition to -the workload-like methods, instruments can define a number of other methods that -will get invoked at various points during run execution. The most useful of -which is perhaps ``initialize`` that gets invoked after the device has been -initialised for the first time, and can be used to perform one-time setup (e.g. -copying files to the device -- there is no point in doing that for each -iteration). The full list of available methods can be found in -:ref:`Signals Documentation `. - -.. _prioritization: - -Prioritization -^^^^^^^^^^^^^^ - -Callbacks (e.g. ``setup()`` methods) for all instruments get executed at the -same point during workload execution, one after another. The order in which the -callbacks get invoked should be considered arbitrary and should not be relied -on (e.g. you cannot expect that just because instrument A is listed before -instrument B in the config, instrument A's callbacks will run first). - -In some cases (e.g. in ``start()`` and ``stop()`` methods), it is important to -ensure that a particular instrument's callbacks run a closely as possible to the -workload's invocations in order to maintain accuracy of readings; or, -conversely, that a callback is executed after the others, because it takes a -long time and may throw off the accuracy of other instruments. You can do -this by using decorators on the appropriate methods. The available decorators are: -``very_slow``, ``slow``, ``normal``, ``fast``, ``very_fast``, with ``very_fast`` -running closest to the workload invocation and ``very_slow`` running furtherest -away. For example:: - - from wa import very_fast - # .. - - class PreciseInstrument(Instrument) - - # ... - @very_fast - def start(self, context): - pass - - @very_fast - def stop(self, context): - pass - - # ... - -``PreciseInstrument`` will be started after all other instruments (i.e. -*just* before the workload runs), and it will stopped before all other -instruments (i.e. *just* after the workload runs). - -If more than one active instrument has specified fast (or slow) callbacks, then -their execution order with respect to each other is not guaranteed. In general, -having a lot of instruments enabled is going to negatively affect the -readings. The best way to ensure accuracy of measurements is to minimize the -number of active instruments (perhaps doing several identical runs with -different instruments enabled). - -Example -^^^^^^^ - -Below is a simple instrument that measures the execution time of a workload:: - - class ExecutionTimeInstrument(Instrument): - """ - Measure how long it took to execute the run() methods of a Workload. - - """ - - name = 'execution_time' - - def initialize(self, context): - self.start_time = None - self.end_time = None - - @very_fast - def start(self, context): - self.start_time = time.time() - - @very_fast - def stop(self, context): - self.end_time = time.time() - - def update_output(self, context): - execution_time = self.end_time - self.start_time - context.add_metric('execution_time', execution_time, 'seconds') - - -.. include:: developer_information/developer_reference/instrument_method_map.rst - -.. _adding-an-output-processor: - -Adding an Output processor ----------------------------- - -A output processor is responsible for processing the results. This may -involve formatting and writing them to a file, uploading them to a database, -generating plots, etc. WA comes with a few output processors that output -results in a few common formats (such as csv or JSON). - -You can add your own output processors by creating a Python file in -``~/.workload_automation/plugins`` with a class that derives from -:class:`wa.OutputProcessor `, and should - implement the relevant methods from the following interface: - -.. class:: OutputProcessor(Plugin): - - .. attribute:: name - - The name of the output processor, this must be unique to WA. - - .. attribute:: description - - A description of what the output processor can be used for. - - .. attribute:: parameters - - A list of additional :class:`Parameters` the output processor can take. - - .. method:: initialize(): - - This method will only be called once during the workload run - therefore operations that only need to be performed initially should - be performed here. - - .. method:: process_job_output(output, target_info, run_ouput): - - This method should be used to perform the processing of the - output from an individual job output. This is where any - additional artifacts should be generated if applicable. - - .. method:: export_job_output(output, target_info, run_ouput): - - This method should be used to perform the exportation of the - existing data collected/generated for an individual job. E.g. - uploading them to a database etc. - - .. method:: process_run_output(output, target_info): - - This method should be used to perform the processing of the - output from the run as a whole. This is where any - additional artifacts should be generated if applicable. - - .. method:: export_run_output(output, target_info): - - This method should be used to perform the exportation of the - existing data collected/generated for the run as a whole. E.g. - uploading them to a database etc. - - .. method:: finalize(): - - This method is the complement to the initialize method and will also - only be called once. - - -The method names should be fairly self-explanatory. The difference between -"process" and "export" methods is that export methods will be invoked after -process methods for all output processors have been generated. Process methods -may generate additional artifacts (metrics, files, etc.), while export methods -should not -- they should only handle existing results (upload them to a -database, archive on a filer, etc). - -The output object passed to job methods is an instance of -:class:`wa.framework.output.JobOutput`, the output object passed to run methods -is an instance of :class:`wa.RunOutput `. - - -Adding a Resource Getter ------------------------- - -A resource getter is a plugin that is designed to retrieve a resource -(binaries, APK files or additional workload assets). Resource getters are invoked in -priority order until one returns the desired resource. - -If you want WA to look for resources somewhere it doesn't by default (e.g. you -have a repository of APK files), you can implement a getter for the resource and -register it with a higher priority than the standard WA getters, so that it gets -invoked first. - -Instances of a resource getter should implement the following interface:: - - class ResourceGetter(Plugin): - - name = None - - def register(self, resolver): - raise NotImplementedError() - -The getter should define a name for itself (as with all plugins), in addition it -should implement the ``register`` method. This involves registering a method -with the resolver that should used to be called when trying to retrieve a resource -(typically ``get``) along with it's priority (see `Getter Prioritization`_ -below. That method should return an instance of the resource that -has been discovered (what "instance" means depends on the resource, e.g. it -could be a file path), or ``None`` if this getter was unable to discover -that resource. - -Getter Prioritization -^^^^^^^^^^^^^^^^^^^^^ - -A priority is an integer with higher numeric values indicating a higher -priority. The following standard priority aliases are defined for getters: - - - :preferred: Take this resource in favour of the environment resource. - :local: Found somewhere under ~/.workload_automation/ or equivalent, or - from environment variables, external configuration files, etc. - These will override resource supplied with the package. - :lan: Resource will be retrieved from a locally mounted remote location - (such as samba share) - :remote: Resource will be downloaded from a remote location (such as an HTTP - server) - :package: Resource provided with the package. - -These priorities are defined as class members of -:class:`wa.framework.resource.SourcePriority`, e.g. ``SourcePriority.preferred``. - -Most getters in WA will be registered with either ``local`` or -``package`` priorities. So if you want your getter to override the default, it -should typically be registered as ``preferred``. - -You don't have to stick to standard priority levels (though you should, unless -there is a good reason). Any integer is a valid priority. The standard priorities -range from 0 to 40 in increments of 10. - -Example -^^^^^^^ - -The following is an implementation of a getter that searches for files in the -users dependencies directory, typically -``~/.workload_automation/dependencies/`` It uses the -``get_from_location`` method to filter the available files in the provided -directory appropriately:: - - import sys - - from wa import settings, - from wa.framework.resource import ResourceGetter, SourcePriority - from wa.framework.getters import get_from_location - - class UserDirectory(ResourceGetter): - - name = 'user' - - def register(self, resolver): - resolver.register(self.get, SourcePriority.local) - - def get(self, resource): - basepath = settings.dependencies_directory - directory = _d(os.path.join(basepath, resource.owner.name)) - return get_from_location(directory, resource) - -.. _adding_a_target: - -Adding a Target ---------------- - -In WA3, a 'target' consists of a platform and a devlib target. The -implementations of the targets are located in ``devlib``. WA3 will instantiate a -devlib target passing relevant parameters parsed from the configuration. For -more information about devlib targets please see `the documentation -`_. - -The currently available platforms are: - :generic: The 'standard' platform implementation of the target, this should - work for the majority of use cases. - :juno: A platform implementation specifically for the juno. - :tc2: A platform implementation specifically for the tc2. - :gem5: A platform implementation to interact with a gem5 simulation. - -The currently available targets from devlib are: - :linux: A device running a Linux based OS. - :android: A device running Android OS. - :local: Used to run locally on a linux based host. - :chromeos: A device running ChromeOS, supporting an android container if available. - -For an example of adding you own customized version of an existing devlib target, -please see the how to section :ref:`Adding a Custom Target `. - - -Other Plugin Types ---------------------- - -In addition to plugin types covered above, there are few other, more -specialized ones. They will not be covered in as much detail. Most of them -expose relatively simple interfaces with only a couple of methods and it is -expected that if the need arises to extend them, the API-level documentation -that accompanies them, in addition to what has been outlined here, should -provide enough guidance. - -:commands: This allows extending WA with additional sub-commands (to supplement - exiting ones outlined in the :ref:`invocation` section). -:modules: Modules are "plugins for plugins". They can be loaded by other - plugins to expand their functionality (for example, a flashing - module maybe loaded by a device in order to support flashing). - - -Packaging Your Plugins ----------------------- - -If your have written a bunch of plugins, and you want to make it easy to -deploy them to new systems and/or to update them on existing systems, you can -wrap them in a Python package. You can use ``wa create package`` command to -generate appropriate boiler plate. This will create a ``setup.py`` and a -directory for your package that you can place your plugins into. - -For example, if you have a workload inside ``my_workload.py`` and a result -processor in ``my_result_processor.py``, and you want to package them as -``my_wa_exts`` package, first run the create command :: - - wa create package my_wa_exts - -This will create a ``my_wa_exts`` directory which contains a -``my_wa_exts/setup.py`` and a subdirectory ``my_wa_exts/my_wa_exts`` which is -the package directory for your plugins (you can rename the top-level -``my_wa_exts`` directory to anything you like -- it's just a "container" for the -setup.py and the package directory). Once you have that, you can then copy your -plugins into the package directory, creating -``my_wa_exts/my_wa_exts/my_workload.py`` and -``my_wa_exts/my_wa_exts/my_result_processor.py``. If you have a lot of -plugins, you might want to organize them into subpackages, but only the -top-level package directory is created by default, and it is OK to have -everything in there. - -.. note:: When discovering plugins through this mechanism, WA traverses the - Python module/submodule tree, not the directory structure, therefore, - if you are going to create subdirectories under the top level directory - created for you, it is important that your make sure they are valid - Python packages; i.e. each subdirectory must contain a __init__.py - (even if blank) in order for the code in that directory and its - subdirectories to be discoverable. - -At this stage, you may want to edit ``params`` structure near the bottom of -the ``setup.py`` to add correct author, license and contact information (see -"Writing the Setup Script" section in standard Python documentation for -details). You may also want to add a README and/or a COPYING file at the same -level as the setup.py. Once you have the contents of your package sorted, -you can generate the package by running :: - - cd my_wa_exts - python setup.py sdist - -This will generate ``my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz`` package which -can then be deployed on the target system with standard Python package -management tools, e.g. :: - - sudo pip install my_wa_exts-0.0.1.tar.gz - -As part of the installation process, the setup.py in the package, will write the -package's name into ``~/.workoad_automation/packages``. This will tell WA that -the package contains plugin and it will load them next time it runs. - -.. note:: There are no uninstall hooks in ``setuputils``, so if you ever - uninstall your WA plugins package, you will have to manually remove - it from ``~/.workload_automation/packages`` otherwise WA will complain - about a missing package next time you try to run it. diff --git a/doc/source/developer_information/how_tos/adding_plugins.rst b/doc/source/developer_information/how_tos/adding_plugins.rst index 6a9f6d05..8d1209f7 100644 --- a/doc/source/developer_information/how_tos/adding_plugins.rst +++ b/doc/source/developer_information/how_tos/adding_plugins.rst @@ -7,7 +7,7 @@ Installing binaries for a particular plugin should generally only be performed once during a run. This should typically be done in the ``initialize`` method, if the only functionality performed in the method is to install the required binaries then the ``initialize`` method should be decorated with the ``@once`` -:ref:`decorator ` otherwise this should be placed into a dedicated +:ref:`decorator ` otherwise this should be placed into a dedicated method which is decorated instead. Please note if doing this then any installed paths should be added as class attributes rather than instance variables. As a general rule if binaries are installed as part of ``initialize`` then they @@ -75,7 +75,7 @@ to compress a file of a particular size on the device. .. note:: This is intended as an example of how to implement the Workload - :ref:`interface `. The methodology used to + :ref:`interface `. The methodology used to perform the actual measurement is not necessarily sound, and this Workload should not be used to collect real measurements. diff --git a/doc/source/developer_reference.rst b/doc/source/developer_reference.rst deleted file mode 100644 index fdf74173..00000000 --- a/doc/source/developer_reference.rst +++ /dev/null @@ -1,29 +0,0 @@ -.. _developer_reference: - -==================== -Developer Reference -==================== - -.. contents:: - :depth: 3 - :local: - --------------------------------------------------------------------------------- - -.. include:: developer_reference/execution_model.rst - ------------------ - -.. include:: developer_reference/writing_plugins.rst - ------------------ - -.. include:: developer_reference/contributing.rst - ------------------ - -.. include:: developer_reference/revent.rst - ------------------ - -.. include:: developer_reference/serialization.rst diff --git a/doc/source/migration_guide.rst b/doc/source/migration_guide.rst index ca0de1cb..0d51220f 100644 --- a/doc/source/migration_guide.rst +++ b/doc/source/migration_guide.rst @@ -167,9 +167,10 @@ Python Workload Structure any results from the target back to the host system and to update the output with any metrics or artefacts for the specific workload iteration respectively. -- WA now features :ref:`decorators ` which can be used to allow for more efficient - binary deployment and that they are only installed to the device once per run. For - more information of implementing this please see +- WA now features :ref:`execution decorators ` which can + be used to allow for more efficient binary deployment and that they are only + installed to the device once per run. For more information of implementing + this please see :ref:`deploying executables to a target `. diff --git a/doc/source/user_information/how_tos/agenda.rst b/doc/source/user_information/how_tos/agenda.rst index da05fcbe..5f9dfec8 100644 --- a/doc/source/user_information/how_tos/agenda.rst +++ b/doc/source/user_information/how_tos/agenda.rst @@ -332,7 +332,7 @@ parameters used. name: cyclictest iterations: 10 -.. _classifiers: +.. _using-classifiers: Classifiers ------------ -- cgit v1.2.3