SearchBestie can be executed by executing cz.vutbr.fit.sbestie.Manager class.

 
Usage: Manager [-options]
  supported options:
   -config <filename> - XML config file to be used instead of default 'sbestie.xml'.
   -import <filename> - XML file containing exported results which should be imported before execution.
                        If  also -onlyImport is specified results are imported and exploration is not performed.

• If -config is not defined, Manager tries to find sbestie.xml file in the CLASSPATH and use it.
• If -import is defined, Manager creates state space according to configuration file and then fill it with content of specified file. Then, normal exploration is executed if not -onlyImport is used.

Configuration file is an XML file that defines behavior of SearchBestie infrastructure, state space, and exploration engines. Configuration file must follow format defined in sbestie.dtd file (located in the lib subdirectory by default). Examples of such XML files can be found also in the lib subdirectory.

The configuration file has the following structure:

 
<searchbestie>
  <searchengine name="some name" class="class to be used"> 
    <parameters>
      <parameter key="name of parameter" value="value of parameter">
    </parameters>
  </searchengine>

  <executor> 
    <iterationchecker> ... </iterationchecker>
    <timeout time="number of seconds">
    <pretask> ... </pretask>
    <posttask> ... </posttask>
  </executor>

  <statespace addpolicy="chosen policy"> 
    <dimension name="some identifier" ordinal="true or false">
      ...
    </dimension>
    <dimension name="some identifier" ordinal="true or false">
      ...
    </dimension>
    ...
    <testdimension name="some identifier" ordinal="true or false">
      <test name="some identifier">
      </test>
      ...
    </testdimension>
    
    <result>
      <resultitem name="some identifier" class="some Java class">
	 <listen>
	  <event name="event identifier" />
          ...
	</listen>
      </resultitem>
      <resultitem name="some identifier" class="some Java class">
	 <listen>
	  <event name="event identifier" />
          ...
	</listen>
      </resultitem>
      ...
    </result>

    <fitness name="some identifier" exp="math expression over resultitem names" />
  </statespace>
</searchbestie>

The configuration file has the following three parts:

Controls which state of the state space is explored in the following step. One has to provide class which contains search algorithm and optionally can specify parameters of this file. Currently, only a few search engines are implemented:

• RandomSearch - Randomly choose the next state to be explored. One can specify a seed that is used to initialize pseudorandom generator (if you use the same seed, you get the same sequence of pseudorandom numbers). You can also specify whether the random generator should choose next state from all possible states or only from yet unexplored.
• SequentialSearch - Does what you expect. Starts at zero position in the state space and sequentially goes through the state space.

Controls exploration process (main loop). executor part is optional and can contain parameters of executor. Currently, four parameters are available:

• iterationchecker controls number of times the searchengine is asked for a state to explore. There are multiple iteration checkers available - look bellow.
• timeout controls maximal time for which the SearchBestie exploration loop is allowed to run - then interruption mechanism is used to stop exploration.
• pretask and posttask are executed only once during the exploration process. pretask is executed before exploration loop is entered and can be used for instance to prepare testing, posttask is executed after exploration.

Defines state space including testdimension that contains tests to be used for exploration of a given state. State space contains n dimensions of various type and one testdimension.

Statespace has one parameter that determines how state space behaves when a new result is being added to a state that has been already explored. Several methods are available:

• REPLACE - the old value is replaced by the new one.
• CUMULANDREPLACE - the old value is replaced by cumulated value (from old and new value).
• ADDTOSET - the new value is added into a list of results for that state (order of adding preserves).
• CUMULANDADDTOSET - a cumulated value (from old and new value) is added into a list of results for that state.

Each dimension can be set to be ordinal or not. Dimensions can be of the following types:

• constant - such domain has one specific value of specific type (string, boolean, integer).
• boolean - such domain has two possible values (true, false).
• integer - such domain has n possible values delimited by min and max values (including these two values).
• enum - such domain has n possible values of possibly different types (string, boolean, integer).
• testdimension is specific because contains configuration of actions that are performed when a particular state in the state space is explored. This dimension must contain at least one test.

There are several building blocks available for easier development of tests (or experimenting with them). You can combine these building blocks in order to get complex tests:

• javatest - Java test explores the given state using a Java class specified as parameter. Test class perform exploration and generates result (see below). Optionally, you could specify a list of parameters for this class.
• processtest - executes a new process (command) and generates IntResult out of process return value. You could also detect liveness of such process or filter its output. For further information see API documentation.
• javaprocesstest - executes a new Java process (class) and generates IntResult out of process return value. You could also detect liveness of such process or filter its output. For further inforomation see API documentation.
• repeatedtest - This kind of exploration can be used if you want to repeat exploration until some condition (checked by iterationchecker) is reached. repeatedtest contains two parts:
  • iterationchecker - which is used to control termination of iterative exploration of the specified state, and
  • test - (could be again one of items listed here) which is used for exploration.
• composedtest - Contains three parts. Pretask and posttask are optional and can perform actions that must be executed before (after) test. One possible scenario is to perform exploration within pretask and use test for processing output and generating result.
  • pretask
  • test (could be again one of items listed here)
  • posttask
• timedtest - Encapsulates another test (could be again one of items listed here) and takes as a parameter timeout (a number specifying number of seconds given to the test to run). If timeout is reached and inner test does not finnish its execution, interruption is generated and exploration is finished.

Defines infrastructure responsible for collecting of results. Result is in general a vector of resultitem elements (integer, long, coverage information, …). Each task or test used in the test dimension usually can produce one or more types of events (messages describing something that happen before, during, or after the test). Each resultitem can be bound to a specific event type by its name. The name contains two parts delimited by ':' character. First part contains identification of the test or task, second identify event type produced by that test or task. For instance, if there is a test with the name parameter set to 'myTest' and the class is able to produce an event of type 'result', the identifier of the event is 'myTest:result'. Of course, multiple resultitem elements can listen to the same event or multiple tests can send events to one resultitem element. There are also some specific events related to test begin and end that could be used withing resultitem producers. Some resultitems already available are:

• DurationResultItem - simply counts time difference of begin testing and end testing events.
• IntResultItem - counts number of events (e.g., each representing an error in the test).
• CoverageResultItem - is able to handle coverage information collected during execution of the test.

Fitness is represented by a math expression constructed over resultitems. The expression can contain floating point constants, brackets, unary operator -, binary operators +,-,*,/ and identifiers of resultitems.

Gives decision whether to continue in exploration. It can be used either in executor (to control number of execution of the exploration loop) or in repeated test (to control number of times the state should be explored). Only a few are available right now:

• FixedIterChecker - this iteration checker allows to specify certain number of repetition.
• SimpleDeltaIterChecker - this iteration checker compare integer values of last two consecutive results and stop iteration if their difference is bellow predefined number.

Task is similarly to test a unit of computation used during exploration process. The difference between test and task is, that test generates result and task does not. Therefore, tasks can be used to prepare testing environment or to clean after test has been done. Similarly to javatest, there are a few general implementations of task available:

• processtask - executes a new process (command). You could also detect liveness of such process or filter its output. For further information see API documentation.
• javaprocesstest - executes a new Java process (class). You could also detect liveness of such process or filter its output. For further inforomation see API documentation.

When executed, SearchBestie informs you about initialized state space and its setting. Then, the exploration loop is executed end standard output could be used by exploration algorithms. Finally, SearchBestie prints out statistics concerning explored state space and export the state space into sbestie_output output directory. A simple example of output follows:

SBestie INFO:  Search-Based Testing Environment - SearchBestie 
SBestie INFO:   version: 0.4 Beta, any sugestions and comments: iletko@fit.vutbr.cz 
SBestie INFO:  Reading input file with parameters...
SBestie INFO:   Found 1 search engine.
SBestie INFO:   Found 4 dimensions.
SBestie INFO:   Found 1 test dimension.
SBestie INFO:  Initalizing state space storage module...
SBestie INFO:   Created integer  dimension 'dimension1' (1,100) ordinal: true
SBestie INFO:   Created integer  dimension 'dimension2' (1,100) ordinal: true
SBestie INFO:   Created integer  dimension 'dimension3' (1,100) ordinal: true
SBestie INFO:   Created integer  dimension 'dimension4' (1,100) ordinal: true
SBestie INFO:   Cteated test     dimension 'testdimension' (1 tests) ordinal: false
SBestie INFO:   Results AddPolicy is set to ADDTOSET.
SBestie INFO:   ResultItemCreator 'resItem' created.
SBestie INFO:   Fitness function 'MyFitness' created.
SBestie INFO:  Initializing search module...
SBestie INFO:   HillClimbingSearch: Using seed 123456789.
SBestie INFO:   HillClimbingSearch: First-choice hill climbing: false.
SBestie INFO:   HillClimbingSearch: Neighborhood size: 1.
SBestie INFO:   HillClimbingSearch: Complex neighborhood: false.
SBestie INFO:   HillClimbingSearch: Performing hill climbing dimension by dimension: false.
SBestie INFO:   Created search engine 'hill climbing'.
SBestie INFO:  Initializing exploration module...
SBestie INFO:   Exploration timeout set to 60 seconds.
SBestie INFO:  Entering state space exploration loop...

...output produced during exploration...

SBestie INFO:  State space exploration loop exited...
SBestie INFO:  State space statistics:
SBestie INFO:    States(Total/Explored): 100000000/1076
SBestie INFO:    Results(Total/Pass/Interrupted/Fail): 1076/1076/0/0
SBestie INFO:    Best (State: Value): [99, 99, 99, 99, 0]: {401.0;PASS;L358;I401}: 1
SBestie INFO:    Worst (State: Value): [65, 0, 83, 43, 0]: {196.0;PASS;L365;I196}: 1
SBestie INFO:    Cumulated value: {310178.0;PASS;L387391;I310178}
SBestie INFO:  Exporting results...
SBestie INFO:   Creating file: '/somewhere/sbestie_output/sbestie_1292598162.xml'
SBestie INFO:  Finalizing...
SBestie INFO:  Exiting...

Currently, SearchBestie does not provide logging into file or any debug level options.

Exported XML file contains definition of state space and collected results. The state space additionally contains information what kind of results were used. Currently, exported state space does not contain exact information concerning test dimension (each test is replaced by an Empty test with the same identifier as original test). The exported results can be then imported by -import command line parameter.