From field data to business decisions:
what XLSForm taught me about data systems

Written by Denzy Samba, Product Manager at Sustaain

I discovered XLSForm about five years ago, while working for an agroforestry organization. I was looking for a solution that could do two things at once: standardize and scale the way we integrated data, including field data from our agroforestry projects, and be easy to build with. Easy enough that if I had teams at origin, they could be trained on it quickly.

XLSForm is an open standard that turns a simple Excel file into a mobile field questionnaire. Using it taught me three things that now shape how we build at Sustaain: design the data model before the questionnaire, govern data end to end from field capture to key performance indicators (KPIs), and rely on open standards so that traders, NGOs and field organizations can share the same data, whatever their technical level.

An open-source community that answers

When I was benchmarking tools, I was in touch with the French-speaking ODK community. Later, I spoke with Yaw Anokwa, the project’s co-founder. What struck me first was that he took the time to talk with me. He was transparent about the roadmap, and the team's answers were technically strong. On the KoboToolbox community forum, I asked a lot of questions. I always got answers, and solutions.

From theory of change to a field data model

The real challenge is linking what we call the theory of change with questionnaire design, questionnaire creation, how the questionnaire is used, and the governance of data integration and management behind it. That takes a real methodology: how you design the questionnaire, how you build it, how you name the fields.

I color-code my Excel files so that I keep a technical data model in mind while I'm designing a monitoring and evaluation (M&E) framework. The colors show the scale of the data: data at the level of the region, of the farmer, of the plot. These levels are the core objects of the data model. A well-structured spreadsheet is what keeps the M&E world and the data world compatible.

GPS accuracy starts with questionnaire design

Geospatial data raised its own challenge: how to calibrate and properly format the questions around geospatial data capture.

The hardest part is GPS accuracy, and it depends on the context: the project area, and how digitally comfortable the people using the tool are. XLSForm lets you capture either points or polygons, either by hand or as a GPS trace: you walk with the phone in your pocket and it records the track. Part of the accuracy depends on the phone, but a lot depends on how you design the questionnaire.

One trick I used: let the GPS calibrate at the start of the questionnaire, and don't take the track right away. Then there is the reliability of the information, the checks you can do afterwards, and compatibility with QGIS and other open-source tools. Those checks matter: at Sustaain, our geospatial data quality framework found quality issues in 35.6% of 330,000 plot geometries.

End-to-end data governance, from questionnaire to KPI

Data governance covers questionnaire design, security and sharing. It's really about managing data collection end to end: who designs the questionnaire, how it is designed, how it is deployed in the field, who checks the data, and who says whether a piece of data is good or not in terms of quality. That is what gives you reliable data from end to end.

And then there is the mapping of each individual field to KPIs, which may or may not be standard ones. In other words, the link between the raw data and the KPI that helps people make decisions.

Reusing M&E field data for EUDR compliance

With the EU Deforestation Regulation (EUDR), the focus is now on origin data, especially data about farmers, which relates to several different parts of the EUDR. There is the plot itself, for deforestation. Then there are risk mitigation measures and social data.

The work is to think in blocks. These blocks come from data collection that originally has nothing to do with the EUDR: it serves operations and M&E. The question is how to make these two functions compatible: the same data, used differently. M&E means a huge number of questions and a lot of data collected at one point in time. And it turns out that some of this data, grouped together, will be used at different stages of EUDR compliance.

One open standard for traders, NGOs and field teams

Interoperability is very important to me. The same XLSForm can exist at a trader, at an agroforestry organization, or at an NGO, precisely because it is open source and an open standard.

That is the power of open source and community: tools used at the same time by the private sector, the public sector and completely different stakeholders, with very different levels of skills and in many different languages. We should encourage this kind of initiative, with the funding it needs, so that this interoperability can continue.

What field data taught me about building Sustaain