Validating geospatial data at scale: a human-in-the-loop ML approach
Every data migration has to bring legacy data forward. For anyone who manages buildings, that legacy data includes thousands of floor plans, which are not always intended to be displayed on a map. To integrate them with larger systems and allow geospatial analysis, each of these files first has to be geolocalized, and large vendor tools often lack optimization or validation features that can scale to a whole estate.
What we typically want is a method that scales and is explicit about its failure modes. We don’t expect full automation, but if we can use machine learning to cut the cost by 80% and clearly flag problematic cases for human validation, we can keep the whole pipeline in-house instead of off-shored, which has two main benefits: sensitive data (such as school plans or military bases) never leaves the organisation, and checking it ourselves improves our understanding of our own data.
Cal Poly, San Luis Obispo
To showcase a possible approach to this class of problems, we turn to the university campus of Cal Poly in San Luis Obispo, California. This campus publishes 74 of its building plans (159 individual footprints), all drawn in different Coordinate Reference Systems (CRS), which means different units, origins and orientations. These plans cannot readily be laid out together on a single map- they need to be brought to a consistent CRS first.
Using geometry and a couple optimisation tricks, we can actually automate the process of geolocating each plan. See the result of our algorithm in Map 1: press Aligned and watch each plan find its geolocalized building on OpenStreetMap. The algorithm figures out where a plan goes by looking only at its shape: the outlines of the footprints in the plan and of the real buildings on OpenStreetMap (OSM). From that alone, 64 of the 74 buildings end up in the right place.
Geolocating a plan
Those plans are drawn in computer-aided design (CAD) software, and a CAD plan is made to build or refurbish a building, not to sit on a map: the architect draws in whatever units suit them, from whatever corner is convenient, in an orientation that fits the building shape. To use a plan in a geographic information system (GIS), someone has to geolocate it: find the building on a real map, scale the plan footprints, turn them, nudge them into place. You can get a sense of how much time that takes by hand for one plan (8 footprints) in Fig 1. Multiply by thousands of plans, and remember that some buildings have since been extended, split or demolished.
Geolocating a footprint means finding a projection (a mathematical operation that scales, rotates and shifts a building until its coordinates are expressed in a certain CRS) from the plan’s CRS to a target CRS (for San Luis obispo, we’d probably use EPSG 10925). Try it yourself in Fig 1, on building 172: a cluster of seven residence halls + 1 parking lot. The score at the top is how well your aligned plan and the OSM map agree: the area where they overlap, divided by the area they cover together. 1 is a perfect fit. How high a score can you get?
Fiddly, isn’t it? At scale it’s an even more challenging task: every footprint needs its own transformation. Automated tools can find a transformation from an official CRS to another, so we can cross our fingers and hope that all these plans are expressed in an official CRS somehow, but even then it gets costly fast: there are over 6000 CRS projections worldwide, and 130+ for California alone. And what do we do if a plan’s coordinate system is not an official CRS? Existing solutions rely on a GIS expert manually finding that transformation (like you just did) using a tool like FME, which is highly unscalable.
Now press Search. Our algorithm starts from an educated guess, made from the plans and the buildings taken as wholes: scale the plan juntil it covers the same area as its OSM counterpart, line up their long axes, put their centres on top of each other. That gets a score of 0.30. It then tries 1,000 random variations around that guess, which only gets to 0.46. The real work is the polishing: the best few guesses are refined with a classic optimiser (Nelder-Mead in this case), and that takes it up to 0.74.
The scale it settles on is 0.0834 map feet per plan unit: exactly one twelfth. The plans are drawn in inches! Across the campus, the median scale it finds is within 1.4% of 1/12.
When the site doesn’t move as one
Plans can feature one or more building footprints, sometimes adjacent, sometimes relatively far apart. Should we move them all as one or one by one? Moving all footprints as one block (call it whole site) assumes the relative positions of buildings in a single plan are correct, but sometimes they aren’t. In a few Cal Poly buildings (9, 27 and 28 in Fig 2), some plan files are already on real-world coordinates and others aren’t, so no single move fits them all. For those, there is a fallback, footprint by footprint: geolocate each footprint on its own. Scale it to the area of a real building, line up their long axes, and pair each footprint with the OSM building it fits best.
The obvious recipe is to run both and keep whichever scores higher (call it best score), but Map 2 shows why it’s a trap. On building 172, placing each footprint on its own scores 0.91, higher than the whole site’s 0.74, but because the halls all look alike, it mismatches six of them out of seven!
If the whole site approach gets a decent score, we should always prefer it: relative footprint positions bring more information and prevent mismatches or misorientations (if a building is a rectangle, there are 2 possible transformations. If it’s a square, there are 4. But if we move these 2 footprints as one, then there is only one). Back on the whole campus, most buildings now do travel as one block of footprints. Map 3 replays Map 1, this time with footprints groups travelling together: 67 of the 74 move as a single piece, and only seven fall back to footprint by footprint.
But how do we know where the plans truly belong? In real life, we’ll almost always have metadata to help us narrow the search for a match in the first place (e.g in OSM, many of the Cal Poly buildings were tagged with their real-life building number). We are lucky enough that we can build an automated testing set: 75 of the 159 footprints turn out to be in real-world coordinates already, just multiplied by 12 (inches again). It gives us 32 buildings for which we know exactly where every plan should go: Fig 2.
We can now test different approaches: Best score gets 25 of the 32 buildings right, fewer than whole site alone (27). The rule that works, whole site first, only falls back to footprint by footprint when whole site clearly fails (a score below 0.4). That gets 30 of 32 correct matches, and it’s the approach we’d always recommend with this method.
The lesson here is a central idea in machine learning: choosing a metric for our algorithm amounts to picking the problem we want it to solve, so we have to be mindful about that. Whole site measures the quality of a site-wide fit, whereas footprint by footprint measures the average quality of individual fits.
Put a human in the loop
No method gets every building right, so we want to use the score as a proxy for model confidence and a threshold for human validation. The dataset here is small enough that we can assess each placement by eye. Fig 3 shows us the score distribution along with the match correctness.
With a confidence threshold at 0.7, 84% of the matches are readily accepted, and 61 of those 62 (98%) are correct. The one that slips through, building 52, scores 0.90: two plans for one building, placed close to their true position but not close enough to count.
That threshold also sends some correct placements to review, which is due to differences in data providers: building 172 scores 0.74 while every footprint is in the right place, because OSM draws the halls more coarsely than the plan does and they disagree on the shape of the parking block. Moving that threshold trades analyst time against a higher risk of mismatch: it is a business choice for our data owner. Depending on the scale of our estate and the sensitivity of our data, it might make sense to invest a little time developping a nice UI for our analysts to save a few precious seconds per match validation.
The bottom line: performance, validation and deployment
On Cal Poly, our algorithm processes 74 buildings in 13 minutes on a laptop with an out-of-the-box matching accuracy of 86%, and using a reasonable confidence threshold, we send a dozen matches out for human validation. If the validation UI contains something like the alignment tool above and is hooked to a staging environment, we could get to 98% accuracy in a very short amount of time.
What this algorithm cannot solve but will surface are the proverbial data issues: a demolished wing, an extension, or a plan that was never right in the first place will all be flagged for human validation, to the extent that they damage the data enough the match is assigned a poor score. In a sense, an algorithm like this one makes the most sense when it’s shipped as part of a more complete validation platform, although that platform can remain extremely light. Tools like Docker, FastAPI and Streamlit allow building a CRUD app in a matter of minutes, and if it never leaves your environment, deployment constraints can be relatively minimal.
Building plans from Cal Poly Administration & Finance, Facilities. Building outlines © OpenStreetMap contributors, available under the Open Database License.