Tracking Evaluator

Fig. 1: Tracking Evaluator application view.

Department of Computer Graphics and Multimedia (DCGM)

Víťa Beran

Program description

According to AMI published paper proposing the method for multi-object tracking evaluation, we have implemented the application Evaluator and tested on AMI data.

The Tracking Evaluator is a Windows platform application for evaluation of tracked video data. It is based on the evaluation core, composed by structures and functions, provide reading of data source accepting several data formats, converting data sets into other formats and evaluating the results according to defined methodology. The core is implemented using C language, portable into Linux without further changes.

Two data sets are necessary at least. One, the annotated data referred to as the Ground Truth (GT) objects, and second, the output of an image-based tracking system referred to as the Estimations (E). An output of the evaluation is error values (defined and described in previous document part). The configuration errors can be evaluated for particular frame and the identification errors for entire sequence. Error values are displayed on interactive graph, which allows finding trouble frames effectively. The most important and controlling data set is the Ground Truth one. According to this set, the numbers and amount of frames in the sequence are initialized and also consequent browsing is allowed only through frames occurring in GT set.

Fig 1 shows the application GUI. Data in GT or E sets are managed by load, reset or save functions provided by menu or buttons.

When loading events from the source file, it is possible to re-index event's IDs and also process the interpolation of event object parameters. For particular sequence, such functionality provides possibility to compose several annotations of different people which are annotated in different frames.

Data formats

Event sources

The Evaluator is able to import/export events from text and XML file.
The XML files must follow format:

<!ELEMENT AVEvents (EventGroups?, EventTypes?, File?, Editor?)>

<!ELEMENT EventGroups (Group+)>
<!ELEMENT EventTypes (Type+)>

<!ELEMENT Group (ID, Name, Meaning?, Enabled?)>
<!ELEMENT ID (#PCDATA)>
<!ELEMENT Name (#PCDATA)>
<!ELEMENT Meaning (#PCDATA)>
<!ELEMENT Enabled (#PCDATA)>

<!ELEMENT Type (ID, Name, Key?, Group?, GroupIndex?, Offset?, 
                    Parameters?, Secondary*)>
<!ELEMENT Key (#PCDATA)>
<!ELEMENT Group (#PCDATA)>
<!ELEMENT GroupIndex (#PCDATA)>
<!ELEMENT Offset (#PCDATA)>
<!ELEMENT Parameters EMPTY>

<!ELEMENT Secondary (Key, Offset?, Parameters?)>

<!ELEMENT File (Source*, TimeFormat?, Event*, Title*)>
<!ELEMENT Source (#PCDATA)>
<!ELEMENT TimeFormat (#PCDATA)>

<!ELEMENT Event (ID, Time, Text?, Parameters?)>
<!ELEMENT Time (#PCDATA)>
<!ELEMENT Text (#PCDATA)>

<!ELEMENT Title (Time, Text)>

<!ELEMENT Editor ANY>
                

The TXT files can be one of followed formats: format 1

frame    frameID
object   objectID   BoxCenterX   BoxCenterY   Width/2   Height/2                
                

where all spaces are tabs, or format 2

image[frameID].* objectID minXPos minYPos maxXPos maxYPos
                

or format 3

frameID objectID visibility minXPos minYPos maxXPos maxYPos
                

Evaluation results

The results of evaluation might be reported in two ways - full and brief. The full version reports all types of errors, tables with F-measure values for each E and GT event combination, errors for each frame, etc. The brief report print out only main errors in one line using CSV format:

Sequence; F-Measure; FN; FP; MT; MO; CD; FNbar; FPbar; MTbar; MObar; CDbar; 
                     FIT; FIO; FITbar; FIObar; TPbar; OPbar
                

Documentation

Method's brief overview is covered by first part of this document. Document also contains the definitions of configuration and identification errors with examples.

Second part of the document describes the application, used data formats and graphical user interface.

Evaluator core manual

The Evalulator core structures and functions are described here.

Download (version 1.4)

• AMI Tracking Evaluator Full (1.42MB) - all files
• AMI Tracking Evaluator Console (230kB) - command line version
• AMI Tracking Evaluator Light (630kB) - only application
• AMI Tracking Evaluator DLLs (200kB) - necessary libraries
• AMI Tracking Evaluator Help (370kB) - help files
• Presentation (620kB)
• Documentation (250kB)

History

version 1.4   (22.02.2006)
--------------------------------------

- Evaluation Core changed
    - FIT/FIO errors evaluated different way
    - TP/OP errors normalized only over frames where GT occured

- Evaluation results
    - GUI changes (Configuration and Indentification Errors)
    - Brief output format is now same as IDIAP tool


version 1.3   (16.02.2006)
--------------------------------------

- Evaluation results
    - save results in brief format at the end of selected file


version 1.2   (01.02.2006)
--------------------------------------

- Evaluation Core repaired 
  - TP/TO and FIT/FIO remedied
  - all GT frames loaded, not only non-empty frames
  
- Graphs 
   - FIT and FIO graphs added
   
- Evaluation results
    - F-measures and Object and Tracker Purity repaired


version 1.1   (20.01.2006)
--------------------------------------

- Evaluation results
   - save results into text file
   - errors sums, not only normalized values

- Graphs 
   - graphs area also sensitive to mouse-moving with a 
     left or right button pressed

- Help 
   - contains history and new about versions


version 1.0
--------------------------------------
...