Changelog, Monday 16 September 2024

Data Pipelines, Code Sharing, and Appraise

Hi there, Elliot here this week to share with you some of the progress we've made on building out our products, data processing, machine learning capabilities, and internal tooling. It's been a productive week completing major milestones supporting the upcoming launch of the latest version of Appraise.

Data Pipelines

Tract processes vast amounts of data in the background, ranging from planning applications to local authorities' plan documents and every road and junction in the UK. Many of these datasets require multiple jobs executed in a specific order for full processing.

For example:

1. Find all planning applications in Barrow-in-Furness
2. Download PDF documents for each planning application
3. Use machine learning models to convert PDF documents into markdown
4. Use machine learning models to extract application summaries from markdown
5. Populate the database with results

Managing asynchronous jobs within a pipeline in a specific order can be complex. We've been using River with custom logic to determine job order based on the success or failure of previous jobs.

This week, we upgraded our background processing workflows to River Pro, which provides a native method for chaining job dependencies. Henry has been implementing River Pro and enhancing our machine learning processing capabilities.

The purpose of this is to provide more data for our Appraise product, and give landowners more information about their land when their appraisal is generated.

Code Sharing

Our API and background processing codebases are both written in Go. These services share similar low-level operations, such as logging, database connections, and queries. This week, Jamie has focused on eliminating code duplication and creating a shared codebase for these common tasks between our API and background processing.

This kind of work can often be a thankless task, but it is extremely important as our codebase grows, and allows us to more easily build complex and nuanced processes. (Jamie was thanked for this).

Appraise

This week, I've been developing a new version of Appraise, our tool for generating land appraisals for landowners. The redesigned version aligns with our new website's style and offers more readable appraisals. It will include additional land data such as heritage sites, proximity to power infrastructure, and the relevant local planning authority for the land (all of which is provided by our Data Pipelines mentioned earlier).

The new version of Appraise will also feature an improved mapping UI, and a method to refresh previously generated appraisals with new data as we fold it into our pipelines.

We're planning to launch the new version of Appraise very soon, and it will remain free to use!