Redesigning geospatial data management

Background

Koordinates’ core product for business and enterprise customers is a branded Koordinates site. Several well-known New Zealand open data providers publish their data using a Koordinates site, including Land Information New Zealand and Stats NZ.

The product is also used by private businesses, who can choose to open their site and data to the public or use it completely privately, depending on their needs. Some engineering firms use the product to manage and distribute geospatial data internally, for instance.

In 2017, we designed and built the first iteration of data management tools for Koordinates customers.

These data management tools helped our customers to:

Manage data sources: Upload files, or connect to remote data sources like AWS, ArcGIS REST APIs, or CIFS servers, and scan them for geospatial datasets to import.

Import datasets: Importing from a data source to Koordinates translates the data into a proprietary storage format, enabling end users of the data to visualise it on a map and export it in any format or projection.

Publish datasets: Publish geospatial datasets as exportable map layers in a Koordinates site instance, and make them available to the public, or privately within an organisation.

Maintain datasets: Update data, metadata, and permissions on each geospatial dataset.

What we learned

After several years in the hands of existing and prospective customers, feedback and support tickets highlighted some customer pain points. By fixing these issues we could reduce our support workload and help the sales team to close new business:

Unclear workflow: The product lacked a clear thread from uploading files or connecting a remote data source, to importing the data, to publishing a completed geospatial dataset. We were burning sales and support time to train new customers.

Bulk updates: In 2018, only mature customers needed to make updates to multiple datasets, and they were able to script the updates using the Koordinates API. Over time more customers wanted to make batch updates, and they often lacked the resource to do it programatically.

Data quality: Very few customers are in possession of data that strictly meets every standard that the Koordinates data importer is built around. Customer data imports often failed, and each failure required engineering and support time to resolve.

Human error: Customers sometimes made geospatial datasets public, or chose an incorrect license, by mistake. It was hard to isolate which datasets were affected. The product did not do much to prevent either of these mistakes from occurring, or help people to rectify them when they did.

Storage costs: Storing a copy of your organisation’s geospatial data inside Koordinates didn’t quite work for some prospective customers. Either they had to migrate entirely away from their existing data storage solutions, or maintain two copies of the data and pay for the storage of both.

Opportunities

Two developments since 2018 would change our approach to the design of Koordinates’ geospatial data management tools.

Kart: Around 2020, Koordinates released Kart, an open-source geospatial data version control tool built on top of git. Kart is not picky about data quality, since there’s no import process — all it does is track changes to the data. And with Kart, customers can use a Koordinates site to catalogue and distribute their geospatial data, while keeping their existing data storage solution in place.

Larger customers: Our sales team learned that larger potential customers might have a need to run multiple Koordinates sites, and share data between them. They might have a central location for shared datasets, and run sites for teams, projects or regional offices. Koordinates could cater to those larger potential customers by creating an organisation concept and inter-site publishing tools and permissions.

Design objectives

With our understanding of shortcomings in the existing solution, the introduction of Kart, and the need to flesh out the product for larger customers, we established some product design objectives.

Improve workflow efficiency: Streamline data upload, import, and publishing tasks into one efficient workflow, to reduce customer onboarding and support workload.

Drive adoption of Kart: Promote Kart as the default storage format to minimise import errors and supercharge data management for customers, allowing them to keep their data off-site without migrating or duplicating it to Koordinates.

Implement multi-site support: Develop permissions and publishing tools to enable one organisation to manage multiple Koordinates sites for teams, projects, or branches, and share data between them.

Solutions

Edit everything in one place: Present all datasets the user can manage in a spreadsheet-like list view. Customisable columns allow customers to configure the view to suit different tasks. We also highlight ‘upload’ as a key first action for new customers, simplifying onboarding and reducing support workload.

Clearer workflows: Streamline data management by re-imagining uploading, importing, and publishing as a series of steps in a single workflow, rather than separate product areas, helping new customers find their way through this process on their own.

Kart by default: In addition to unlocking version control for data, using Kart for storage minimises import errors and reduces running costs by allowing customers to keep their data off-site without migrating or duplicating it to Koordinates. The proposed design encourages the use of Kart repositories as a default choice.

Easy data source management: Seamlessly add, configure and browse external data sources “on the fly” as part of the data management workflow.

Quickly locate and correct issues: For example, filter down to dataset permissions; quickly identify the most used datasets; or immediately know when scripted updates have failed.

Easy derivations: Enable the creation of new geospatial datasets by joining or merging data from external sources, simplifying the publishing process and reducing storage costs for customers.

Results

At the time of writing, design work is still underway and we’re just beginning to put some of the pieces in place to build the improved data and source management tools. After it is validated, built, and available to customers, we’d hope to see:

Reduced support workload: Less time spent training new customers to use the system, thanks to improved usability; fewer data import errors requiring engineering and support time to resolve, thanks to increased Kart adoption; and reduced user error in setting permissions and selecting licenses.

More data in the platform: More public and private data in the platform, updated more frequently, thanks to reduced friction in data management tools and bulk import and edit functionality.

Increased adoption of Kart: Increased uptake by paying customers would lead to more organic growth of Kart as an open source tool, perhaps creating a positive feedback loop in which more companies working with geospatial data are aware of Koordinates.

More sales closed: Larger numbers of PoCs and sales leads converting to business or enterprise plans, because of support for multi-site organisations and remote data support in Kart.

Process

At Koordinates, we worked fully remotely and found FigJam to be an invaluable collaboration tool. Often I quickly sketched out concepts and shared them with product and design colleagues for review, gathered feedback and quickly iterated an idea. Here are some examples.