Hacking

I’m not a fan of PHP. The language has many shortcomings that make it difficult to write nicely encapsulated code. I didn’t even try but started the project with one big index.php file that included everything, and then refractored some code later on into separate .php files to prevent the file from bloating above 1000 lines of code.

Overview

The file index.php contains all the logic and structured into the following sections (VIM can expand/collapse the corresponding code folds).

  1. Pre-initialization, includes : Things common to all instances.
  2. Set up instance : Find out form cookies or parameters which instance should be used and set up paths accordingly.
  3. Load configuration from config.xls into global $user. See file config.php.
  4. Login: Show the login box if the user is not currently logged in and compare username and password to configuration options during login action. Then load user information (access rights etc) into global $user.
  5. Connect to database, load forms. The excel forms from the directory instances/XXX/forms are loaded and compared with the local ODK database. See file``odk_form.php``.
  6. Actions : Do stuff, such as file upload or download. Note that so far no HTML output has been generated (if the user is logged in).
  7. Start HTML output
  8. Show the menu, depending on user’s access rights.
  9. Display : Generate the main output
    1. show=overview : Show the tabular overview over all data contained in the database. See overview.php.
    2. show=form : Show the content of a form
    3. show=forms : List the contents of the instances/XXX/forms directory.
    4. show=admin : Administrator view with upload/download of config.xls
  10. Footer : Show some more information if &test is specified as a URL parameter and the current user has test rights.

The file cron.php finally contains code that can be run without any user interaction and is normally executed from the operating system’s job scheduler.

Globals and Utilities

The following globals are noteworthy

  • $config : An ExcelConfig object – see file config.php. This object is generated from the file instances/XXX/config/config.xls
  • $user : An array that is set to one of the values in $config->users after successful login. With keys rights and access that describe this users access permissions.
  • $forms : An OdkDirectory object – see file odk_form.php. This object is generated from the files instances/XXX/forms/*.xls. Every object OdkForm in the dictionary $forms->forms contains information about the form as described in its .xls file and can be used to read the data from the database. See Accessing the Database.
  • $show : The name of the current view (see above in Overview).
  • $hooks : Plugins use this object in order to install hooks.

And some utility functions

  • log_add($name, $message) : Adds a message to the specified log file.
  • alert($html, $class) : Displays the html snippet inside an alert box with the given class (success, info, danger, error).
  • profile_start($name) and profile_stop($name) : Measure time spent for $name (can be called multiple times, e.g. inside a recursive function). The footer displays the total time spent in every $name (if the &test URL parameter is specified).

Accessing the Database

The file odk_form.php provides methods to access data in the ODK database using identifiers from .xls files that were used as input to XLSForm. Please use the file’s API doc for reference.

Developing a Plugin

odk_planner comes with a simple plugin called doughnut that adds a page with overview plots of data in the database. This section will walk you step by step through the creation of the doughnut plugin that is the file plugins/doughnut.php. The four steps outlined below build successively on top of each other and introduce every time some new API to give a quick an dirty introduction to the internals described above. Note that for every step the whole source code of the plugin is included. Overwrite the file plugins/doughnut.php with the source listing from the different steps to see what changes and play around with the code.

The idea of this plugin is pretty simple : Specify fields and values in the configuration (sheet doughnut) and let the plugin generate doughnut plots that show the distribution of the field of interest over the whole study population.

Step 1 : Hooking

The plugin needs to hook into the normal program flow at multiple points. First it needs to include the javascript plotting library chart.js into the header of the html page. Then it needs to change the menu (“augment the views”) with a new menu entry doughnut. And finally it needs to render the plots when the view doughnut is active.

The file plugins.php lists some 10 different hooks together with an explanation when they are called and the arguments. In the code below we use the three hooks dump_headers (to add chart.js), augment_views (to add the new menu point), and display (to render a static doughnut plot).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<?php

function doughnut_dump_headers($args) {
    echo '<script src="plugins/doughnut_chart.js"></script>';
}

function doughnut_display($args) {
    global $show;
    if ($show !== 'doughnut') return;
?>
    <div class="row">
        <br><br>
        <div class="span12">
            <center>
                <canvas id="doughnut" style="width:100%"></canvas>
            </center>
            <h4 style="text-align:center">doughnut</h4>
            <script type="text/javascript">
                // Data copied from www.chartjs.org/docs/#doughnut-pie-chart
                new Chart(document.getElementById('doughnut').getContext('2d'))
                    .Doughnut([{ value: 300, color:"#F7464A", highlight: "#FF5A5E", label: "Red" }, { value: 50, color: "#46BFBD", highlight: "#5AD3D1", label: "Green" }, { value: 100, color: "#FDB45C", highlight: "#FFC870", label: "Yellow" }]);
            </script>
        </div>
    </div>
<?php
}

function doughnut_augment_views($args) {
    $views =& $args['views'];
    array_push($views, 'doughnut');
}

$hooks->register('dump_headers', 'doughnut_dump_headers');
$hooks->register('augment_views', 'doughnut_augment_views');
$hooks->register('display', 'doughnut_display');

Step 2 : Access

We don’t want just any user to see our plugin, but only users with the access right (as described in user configuration). This is easily implemented by checking for the right in the $user global that contains the configuration for the currently logged in user. We add the check before rendering the plot but also when the menu is constructed so users without the needed access right will neither see the plugin in the menu nor be able to access the plot by manually tweaking the URL.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<?php

function doughnut_dump_headers($args) {
    echo '<script src="plugins/doughnut_chart.js"></script>';
}

function doughnut_display($args) {
    global $show, $user;
    if (!in_array('data', $user['rights'])) return;
    if ($show !== 'doughnut') return;
?>
    <div class="row">
        <br><br>
        <div class="span12">
            <center>
                <canvas id="doughnut" style="width:100%"></canvas>
            </center>
            <h4 style="text-align:center">doughnut</h4>
            <script type="text/javascript">
                // Data copied from www.chartjs.org/docs/#doughnut-pie-chart
                new Chart(document.getElementById('doughnut').getContext('2d'))
                    .Doughnut([{ value: 300, color:"#F7464A", highlight: "#FF5A5E", label: "Red" }, { value: 50, color: "#46BFBD", highlight: "#5AD3D1", label: "Green" }, { value: 100, color: "#FDB45C", highlight: "#FFC870", label: "Yellow" }]);
            </script>
        </div>
    </div>
<?php
}

function doughnut_augment_views($args) {
    global $user;
    if (!in_array('data', $user['rights'])) return;
    $views =& $args['views'];
    array_push($views, 'doughnut');
}

$hooks->register('dump_headers', 'doughnut_dump_headers');
$hooks->register('augment_views', 'doughnut_augment_views');
$hooks->register('display', 'doughnut_display');

Step 3 : Configuration

This snippet iterates through the configuration key/value pairs in the lines 54-63. Every value is parsed with the new function doughnut_config (by the way : it’s good practise to prepend all functions in the plugin with the plugins name to avoid name space collisions). If an error occurs during the parsing (quite possible since Excel lets you save just about any invalid setting imaginable) the doughnut is not rendered for that row and an error is displayed to the user.

The function doughnut_config expects value that specifies the field of interest and the form this field can be found in (FORM\FIELD). The form is then looked up in the global associative array $forms->forms and the specified field is verified to exist in the $form->mapping (read more above). The name of the MySQL table and column that represent the field are then returned and used in the next step to extract the data from the MySQL database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
<?php

function doughnut_dump_headers($args) {
    echo '<script src="plugins/doughnut_chart.js"></script>';
}

function doughnut_render($name, $config) {

    $id = 'doughnut_' . $name;
?>
    <div class="row">
        <br><br>
        <div class="span12">
            <center>
                <canvas id="<?php echo $id; ?>" style="width:100%"></canvas>
            </center>
            <h4 style="text-align:center"><?php echo $name; ?></h4>
            <script type="text/javascript">
                // Data copied from www.chartjs.org/docs/#doughnut-pie-chart
                new Chart(document.getElementById('<?php echo $id; ?>').getContext('2d'))
                    .Doughnut([{ value: 300, color:"#F7464A", highlight: "#FF5A5E", label: "Red" }, { value: 50, color: "#46BFBD", highlight: "#5AD3D1", label: "Green" }, { value: 100, color: "#FDB45C", highlight: "#FFC870", label: "Yellow" }]);
            </script>
        </div>
    </div>
<?php
}

function doughnut_config($config) {
    global $forms;
    // We only expect one part but allow more values for later versions.
    $parts = explode(' ', $config);
    if (count($parts) < 1) return 'expected "FORM\\FIELD ..."';
    $form_field = explode('\\', $parts[0]);
    if (count($form_field) !== 2) return 'expected "FORM\\FIELD ..."';

    // $forms->forms is array that maps formid to OdkForm (see odk_form.php).
    $form = @$forms->forms[$form_field[0]];
    if (!$form) return 'unknown form "' . $form_field[0] . '"';
    // OdkForm::mapping maps form field to table/column (see odk_form.php).
    $table_column = @$form->mapping[$form_field[1]];
    if (!$table_column) return 'unknown field "' . implode('\\', $form_field) . '"';

    return array(
        'table' => $table_column[0],
        'column' => $table_column[1],
    );
}

function doughnut_display($args) {
    global $show, $user, $config;
    if (!in_array('data', $user['rights'])) return;
    if ($show !== 'doughnut') return;
    // Parse doughnut config lines.
    foreach($config->plugins['doughnut'] as $name=>$config_string) {
        $config_or_error = doughnut_config($config_string);
        if (gettype($config_or_error) === 'string') {
            // Function returns string in case of parse error.
            alert('Error config doughnut ' . $name . ' : ' .
                  $config_or_error, 'error');
        } else {
            doughnut_render($name, $config_or_error);
        }
    }
}

function doughnut_augment_views($args) {
    global $user;
    if (!in_array('data', $user['rights'])) return;
    $views =& $args['views'];
    array_push($views, 'doughnut');
}

$hooks->register('dump_headers', 'doughnut_dump_headers');
$hooks->register('augment_views', 'doughnut_augment_views');
$hooks->register('display', 'doughnut_display');

Step 4 : Data

Having all the necessary hooks, the doughnut plot, some access control and the configuration parsing in place, the only thing that need to be done is to connect the plot to the actual data from the database.

The function doughnut_query constructs a MySQL query using the MySQL table and column name. The query has the following form:

SELECT column AS value, COUNT(column) AS count
FROM table GROUP BY column ORDER BY column

where column and table will be replace with the actual values. If the query were generated for the field LRF1\COLONY_COUNT from the example dataset, the following table would be the reply from the query:

value count
negative 3
1+ 2
2+ 3
3+ 2

The function doughnot_json merely translates the MySQL result to a JSON in the format expected by chart.js. Note that we call mysql_query_ so the query gets logged (calling the standard mysql_query would also work fine). If MySQL returns an error no plot is displayed but the error is shown to the user by calling alert (from util.php).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
<?php

function doughnut_dump_headers($args) {
    echo '<script src="plugins/doughnut_chart.js"></script>';
}

function doughnut_query($config) {
    $query = 'SELECT $column AS value, COUNT($column) AS count ' .
             'FROM $table GROUP BY $column ORDER BY $column';
    $query = str_replace('$table', $config['table'], $query);
    $query = str_replace('$column', $config['column'], $query);
    return $query;
}

function doughnut_json($curs) {
    $data = array();
    $colors = ['red', 'green', 'blue', 'yellow', 'magenta', 'purple', 'orange'];
    $i = 0;
    while($row = mysql_fetch_assoc($curs)) {
        array_push($data, array(
            'value' => $row['count'],
            'label' => trim($row['value']),
            'color' => $colors[$i++ % count($colors)]
        ));
    }
    return json_encode($data);
}

function doughnut_render($name, $config) {
    global $conn;

    $id = 'doughnut_' . $name;

    $query = doughnut_query($config);
    $curs = mysql_query_($query, $conn);
    if ($curs === FALSE) {
        alert('MySQL error : ' . mysql_error(), 'error');
        return;
    }
    $json = doughnut_json($curs);
?>
    <div class="row">
        <br><br>
        <div class="span12">
            <center>
                <canvas id="<?php echo $id; ?>" style="width:100%"></canvas>
            </center>
            <h4 style="text-align:center"><?php echo $name; ?></h4>
            <script type="text/javascript">
                // See www.chartjs.org/docs/#doughnut-pie-chart
                new Chart(document.getElementById('<?php echo $id; ?>').getContext('2d'))
                    .Doughnut(<?php echo $json; ?>);
            </script>
        </div>
    </div>
<?php
}

function doughnut_config($config) {
    global $forms;
    // We only expect one part but allow more values for later versions.
    $parts = explode(' ', $config);
    if (count($parts) < 1) return 'expected "FORM\\FIELD ..."';
    $form_field = explode('\\', $parts[0]);
    if (count($form_field) !== 2) return 'expected "FORM\\FIELD ..."';

    // $forms->forms is array that maps formid to OdkForm (see odk_form.php).
    $form = @$forms->forms[$form_field[0]];
    if (!$form) return 'unknown form "' . $form_field[0] . '"';
    // OdkForm::mapping maps form field to table/column (see odk_form.php).
    $table_column = @$form->mapping[$form_field[1]];
    if (!$table_column) return 'unknown field "' . implode('\\', $form_field) . '"';

    return array(
        'table' => $table_column[0],
        'column' => $table_column[1],
    );
}

function doughnut_display($args) {
    global $show, $user, $config;
    if (!in_array('data', $user['rights'])) return;
    if ($show !== 'doughnut') return;
    // Parse doughnut config lines.
    foreach($config->plugins['doughnut'] as $name=>$config_string) {
        $config_or_error = doughnut_config($config_string);
        if (gettype($config_or_error) === 'string') {
            // Function returns string in case of parse error.
            alert('Error config doughnut ' . $name . ' : ' .
                  $config_or_error, 'error');
        } else {
            doughnut_render($name, $config_or_error);
        }
    }
}

function doughnut_augment_views($args) {
    global $user;
    if (!in_array('data', $user['rights'])) return;
    $views =& $args['views'];
    array_push($views, 'doughnut');
}

$hooks->register('dump_headers', 'doughnut_dump_headers');
$hooks->register('augment_views', 'doughnut_augment_views');
$hooks->register('display', 'doughnut_display');

Step 5 : Bucketing

Now compare the source from the last step with the plugin as included in the plugins/doughnut.php (if you overwrite it get it back from github: doughnut.php). The configuration was extended to list possible values after FORM\FIELD. If these values have the form of a range of numbers (e.g. 10-20) then the MySQL query will summarize the values of some ordinal datapoint into the buckets thus specified.

By the way : there is also a simple test suite for this plugin. The file test/test_doughnut.py checks that the access restrictions work as expected, that the configuration parsing alerts user if invalid values are specified, and that a new plot can be added modifying the configuration.