Democratizing Digital Maps

Veit Schiele

• Founder and managing director of cusy GmbH, Berlin
• Author of the Python for Data Science Tutorials
• Author of the Python-Basics-Tutorial
• Contact: cusy.io/veit

The market for digital map services has so far been dominated by proprietary providers such as Mapbox – with high licence fees, limited control over the data and a strong vendor lock-in.

In this talk, Protomaps is presented as an open source alternative and its potential for various use cases is analysed.

cusy supports research groups with a research platform for developing their software. In this context, we have been using Protomaps for some time now to visualise geodata.

Contents

1. What is Protomaps? History, architecture, data basis and processing, checking the data

Firstly, there is a brief introduction to vector tiles, the basic idea behind Protomaps and the most important differences to conventional raster or proprietary tile services. The complete Protomaps tool stack – from the open source datasets from OpenStreetMap, to efficient tile generation, to delivery via HTTP/2 and the lean PMTiles format - is then described in more detail.

2. Deployment with S3, Caddy or nginx

A special focus is on deployment.

3. Integration with common front-end libraries such as MapLibre³, Leaflet⁴ and OpenLayers³⁶

Practical examples show how to generate your own tiles, how to host Protomaps yourself and how to fulfil data protection requirements.

4. What can you use protomaps for … and what not?

Finally, the most important advantages over Mapbox are discussed – free of charge, data control and offline capability – and a brief outlook on future developments in the open source mapping ecosystem is given.

5. Protomaps sprint

We are planning a sprint in the spring of next year. The date and location are yet to be announced.

6. Read more

The focus of Protomaps is exclusively on tile-based cartography and interactive visualisation.

However, there are also extensions for geocoding and routing.

1. What is Protomaps?

Protomaps¹ is an open source ecosystem for

While conventional map services such as Mapbox⁵ market their data as a proprietary service, Protomaps provides a complete open source solution:

1. Creation

Developers can import geodata for example from OpenStreetMap², Natural Earth or GeoJSON or similar

2. Storage

Storage takes place in the efficient PMTiles format

3. Delivery of vector tiles

The resulting tiles can be delivered via a lightweight HTTP/2 server.

1. What is Protomaps?

1.A How it all began

The Protomaps project was started by one person, Brandon Liu (@bdon⁷), who’s still the main person keeping it going.

The project was funded by the NGI Zero Core Fund⁸.

… and what happened next

Now, GitHub Sponsors covers the costs for stuff like running the API and cloud storage.

1. What is Protomaps?

1.B Architecture

A Protomaps service consists of three main components:

Component	Purpose
Create PMTiles	Einlesen von Geo‑Daten und Generieren der Tiles, z.B. in einer CI/CD‑Pipeline
Store PMTiles	Persistent storage of *.pmtiles files, for example, in an S3-compatible storage platform that supports HTTP range requests and Cross-Origin Resource Sharing (CORS)
PMTiles Server	HTTP/2-based web server that supports caching and range requests, for example, Caddy or nginx

This is a great example of how Protomaps implements the Unix philosophy

“Do one thing and do it well.”

The PMTiles format is central, while other well-known technologies are used for storage and web servers.

1. What is Protomaps?

1.C Data basis and processing

• PMTiles is a single-file archive format for pyramid-tiled data.

• The tiled data can be addressed via Z/X/Y coordinates in HTTP range requests.

• The tiles and directories are arranged in such a way that the effort required for moving and zooming is minimised.

  • In our example, zooming in loads eight additional resources with a size of 760 KB in 620 ms.

• What you see here is the cartographic base maps. However, you can also integrate additional resources into a Protomaps file, for example:

  • remote sensing data
  • JPEG images
  • and so on

• However, PMTiles is a read-only format, which means that it is not possible to update an archive without rewriting the entire file.

1. What is Protomaps?

1.D Data sources

• Base maps from OpenStreetMap⁶ and Natural Earth¹³ data

The base maps provided use OpenStreetMap and Natural Earth data to generate the *.pmtiles files.

However, it does not contain all data and tags from OSM, but attempts to strike a balance between tile size and completeness so that it can be used as a general map.

• Create your own maps with github.com/protomaps/basemaps

The organisation of the layers and tags is based on the open source project Tilezen and is documented in Basemap Layers.

The repository contains a Java Planetiler profile for creating PMTiles. For example, you can also include house numbers with addr_housenumber for kind=address.

• Excerpts from these maps with pmtiles extract¹⁸

• pmtiles convert¹⁹ converts MBTiles archives to PMTiles

• tippecanoe²⁰ can also convert GeoJSON, FlatGeobuf and other file formats

• rio-pmtiles²¹ is a plugin for the Rasterio²² Python library that can be used to convert GeoTIFFs

• GDAL ≥ 3.8.0 offers native support for PMTiles²³

1. What is Protomaps?

1.E Checking the data

Check the created Protomaps files with pmtiles verify {NAME}.pmtiles

Display header data with

$ pmtiles show NAME.pmtiles  --header-json

{
    "tile_compression": "gzip",
    "tile_type": "mvt",
    "minzoom": 0,
    "maxzoom": 7,
    "bounds": [
        -176.684714,
        -14.37374,
        145.830418,
        71.341223
    ],
    "center": [
        -82.96875,
        37.71024,
        7
    ]
}

1. What is Protomaps?

1.E Checking the data

Display meta data with

$ pmtiles show NAME.pmtiles --metadata

{"description":"NAME.mbtiles","format":"pbf","generator":"tippecanoe v2.8.1","generator_options":"/Users/veit/workspace/protomaps/tippecanoe/tippecanoe -zg '--projection=EPSG:4326' -o NAME.mbtiles -l zcta NAME.json","maxzoom":"7","minzoom":"0","name":"NAME.mbtiles",… "vector_layers":[{"description":"","fields":{"AFFGEOID10":"String","ALAND10":"Number","AWATER10":"Number","GEOID10":"String","NAME10":"String"},"id":"name","maxzoom":7,"minzoom":0}],"version":"2"}

1.E Checking the data

If you want to view your PMTiles data, you can use the PMTiles Viewer²⁵:

2. Deployment

S3

• PMTiles works on any S3-compatible cloud storage that supports HTTP range requests.
• In addition, Cross-Origin Resource Sharing (CORS) is required if your map frontend is hosted on a different domain than your storage.
• With pmtiles upload, you can move PMTiles to S3-compatible cloud storage:

$ pmtiles upload INPUT.pmtiles REMOTE.pmtiles --bucket=s3://BUCKET_NAME

2. Deployment

HTTP2 server

Caddy

map.cusy.io {
	### CORS Config ###
    header @map_origins Access-Control-Allow-Origin "https://map.cusy.io"
	header @map_origins Access-Control-Allow-Methods "GET, HEAD"
	header @map_origins Access-Control-Allow-Headers "Range, If-Match"
	header @map_origins Access-Control-Expose-Headers "ETag"

	### Tiles ###
	handle_path /tiles/* {
		root * {{ tiles_dir }}
		file_server
	}
}

However, you can also set up your own HTTP server, for example with Caddy²⁷. In addition to the file_server configuration for your *.pmtiles from a static directory /tiles/, you then only need to set the CORS headers.

2. Deployment

HTTP2 server

nginx

if ($request_method = "OPTIONS") {
  add_header "Access-Control-Max-Age" 3600;
  add_header "Content-Type" "text/plain charset=UTF-8";
  add_header "Access-Control-Allow-Origin" "*" always;
  add_header "Access-Control-Allow-Headers" "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range" always;
  add_header "Content-Length" 0;
  return 204;
}

Integration

MapLibre GL JS

PMTiles can also be rendered directly in the browser by the MapLibre GL JS²⁸ renderer, either for thematic overlay tilesets or base map tilesets.

Integration

MapLibre GL JS

$ npm install pmtiles

You can also render a map with your own style using the NPM package @protomaps/basemaps³⁵.

This gives you comprehensive control with minimal code.

import { Protocol } from "pmtiles";

let protocol = new Protocol();
maplibregl.addProtocol("pmtiles",protocol.tile);

To read PMTiles directly, you need the JavaScript library PMTiles²⁹.

It contains a plugin for MapLibre GL that uses its addProtocol function³⁰.

Integration

MapLibre GL JS

  getStyle(): StyleSpecification {
    const colorScheme = this.getActiveColorScheme();
    const flavors = this.normalizeFlavors(this.options.flavors);
    const sprites =
      colorScheme == "dark" && flavors.invertDark ? "light" : colorScheme;
    return {
      version: 8,
      glyphs: "https://map.cusy.io/fonts/{fontstack}/{range}.pbf",
      sprite: "https://map.cusy.io/sprites/v4/${sprites}",
      sources: {
        [PMTILES_SOURCE]: {
          type: "vector",
          url: "pmtiles://${this.tilesURL}",
          attribution:
            "Map data: <a href="https://www.openstreetmap.org/copyright" target="_blank">OpenStreetMap</a>",
        },
      },
      layers: layers(PMTILES_SOURCE, flavors[colorScheme], DEFAULT_LAYERS_OPTS),
    };
  }

layers from @protomaps/basemaps is a function with the following three arguments:

• The source name of the base map
• A flavour object; the default settings can be retrieved under namedFlavor with light, dark, white, black and grayscale.
• An options object for labels, language keys and labels.

Integration

Leaflet⁴

As an alternative to MapLibre, PMTiles can also be rendered by Leaflet⁴ in the browser.

$ npm install protomaps-leaflet

<script src="protomaps-leaflet.js"></script>
<script>
    const map = L.map("map")
    var layer = protomapsL.leafletLayer({url:"FILE.pmtiles", flavor: "light", lang: "en"})
    layer.addTo(map)
</script>

Integration

OpenLayers³⁶

$ npm install ol-pmtiles

import "./style.css";
import { Map, View } from "ol";
import VectorTile from "ol/layer/VectorTile";
import { PMTilesVectorSource } from "ol-pmtiles";
import { Style, Stroke, Fill } from "ol/style";
import { useGeographic } from "ol/proj";

Integration

OpenLayers

const vectorLayer = new VectorTile({
  declutter: true,
  source: new PMTilesVectorSource({
    url: "https://r2-public.protomaps.com/protomaps-sample-datasets/nz-buildings-v3.pmtiles",
    attributions: ["© Land Information New Zealand"],
  }),
  style: new Style({
    stroke: new Stroke({
      color: "gray",
      width: 1,
    }),
    fill: new Fill({
      color: "rgba(20,20,20,0.9)",
    }),
  }),
});

Integration

OpenLayers

useGeographic();

const map = new Map({
  target: "map",
  layers: [vectorLayer],
  view: new View({
    center: [172.606201,-43.556510],
    zoom: 7,
  }),
});

4. What can you use Protomaps for?

Let’s take a look at various applications where Protomaps can prove transformative.

• Cities and municipalities

Cities and municipalities can replace their commercial mapping system with a Protomaps-based solution to display

• area information
• infrastructure projects and
• municipal resources.

The self-hosted implementation

• eliminates recurring licence costs
• and adds the ability to add local landmarks and municipal boundaries to the maps, which were previously difficult or impossible to mark with commercial services.

• Offline maps

Protomaps enables the creation of offline-capable map tools in areas with sporadic or no internet connection.

By distributing PMTiles files containing detailed local and regional maps, interactive map tools can be accessed without the need for constant internet access.

This is ideal for crisis areas where the internet is down.

• Privacy-critical applications

For example, healthcare providers can use Protomaps to create tools for locating facilities.

The location data can remain on the end user’s device.

• Specialised tools

PMTiles enables companies to create highly specialised maps with industry-specific symbols and data visualisations that cannot be offered by commercial map providers.

At the same time, it ensures that the maps are accessible.

4. What can you use Protomaps for… and what can’t you use them for?

PMTiles is intended for the web-based display of large, static data sets, which means

• they are based on a web platform and not on a local desktop application.

• the information totals more than a few megabytes, which is more than can be sensibly loaded at one time.

• the data set changes at most daily or never.

4. What can you use Protomaps for… and what can’t you use them for?

If your application does not have these three features, there are simpler alternatives to PMTiles:

• GeoJSON

If

• you are creating a web-based map with static information,
• but your data is small, you should provide it as a single GeoJSON file.

This saves you the hassle of converting your data into tiles.

You can use the same map design and interaction techniques as with tiled data.

• PostGIS

If you are creating a web-based map for a large dataset that is dynamic and frequently updated, you should store your features in a transactional database.

While it is possible to update a PMTiles file regularly, this requires reloading the file each time it is saved. This may be fine for daily updates, but any higher frequency requires inefficient data transfer.

• GeoParquet

If you are exploring large, static datasets but do not need to publish them on the internet, you can avoid tiling and visualise the files directly with desktop software.

Tiling with tools such as tippecanoe requires the prior calculation of general overview tiles and is designed for retrieving small, optimised pieces of data over the internet. If the network is not the bottleneck and you have your dataset locally, QGIS is an excellent open-source solution for visualisation and map creation.

GeoParquet is a relatively new format that can efficiently store large data sets and is interoperable with open-source data tools.

• Lonboard

• Tools such as Lonboard enable the visualisation of GeoParquet in Jupyter notebooks.
• However, for local data, GeoParquet and Lonboard are a great solution for exploratory data analysis.
• They save you the trouble of converting to a network-optimised tile format.

5. Protomaps sprint

Next spring.

Stay tuned and check our events page from time to time:

cusy.io/en/about/events/

6. Read more

¹ Protomaps: https://protomaps.com

² OpenStreetMap: https://www.openstreetmap.org

³ MapLibre: https://maplibre.org

⁴ Leaflet: https://leafletjs.com

⁵ Mapbox: https://www.mapbox.com

⁶ OpenStreetMap (OSM): https://www.openstreetmap.org

⁷ NGI0 Core Fund: https://nlnet.nl/core

⁸ NLnet Foundation: https://nlnet.nl/project/Protomaps/

⁹ HTTP Range Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

¹⁰ Cross-Origin Resource Sharing (CORS): https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

¹¹ Base maps: https://maps.protomaps.com/builds/

¹² OpenStreetMap: https://openstreetmap.org/

6. Read more

¹³ Natural Earth: https://naturalearthdata.com/

¹⁴ basemaps repository: https://github.com/protomaps/basemaps

¹⁵ Planetiler: https://github.com/onthegomap/planetiler

¹⁶ Basemap Layers: https://docs.protomaps.com/basemaps/layers

¹⁷ Tilezen: https://tilezen.readthedocs.io/en/latest/layers/

¹⁸ pmtiles extract: https://docs.protomaps.com/pmtiles/cli#extract

¹⁹ pmtiles convert: https://docs.protomaps.com/pmtiles/cli#convert

²⁰ tippecanoe: https://github.com/felt/tippecanoe

²¹ rio-pmtiles: https://pypi.org/project/rio-pmtiles/

²² Rasterio: https://rasterio.readthedocs.io/en/stable/

²³ GDAL docs: https://gdal.org/en/stable/drivers/vector/pmtiles.html

²⁴ tilemaker: https://github.com/systemed/tilemaker

6. Read more

²⁵ PMTiles Viewer: https://pmtiles.io/

²⁶ RClone: https://rclone.org/

²⁷ Caddy: https://caddyserver.com/

²⁸ MapLibre GL JS: https://maplibre.org/maplibre-gl-js/docs/

²⁹ PMTiles: https://www.npmjs.com/package/pmtiles

³⁰ MapLibre GL addProtocol-Funktion: https://maplibre.org/maplibre-gl-js/docs/API/functions/addProtocol/

³¹ MapLibre fontstack: https://maplibre.org/maplibre-style-spec/glyphs/

³² MapLibre spritesheet assets: https://maplibre.org/maplibre-style-spec/sprite/

³³ Basemaps assets repository: http://github.com/protomaps/basemaps-assets

³⁴ MapLibre style: https://maplibre.org/maplibre-style-spec/

³⁵ @protomaps/basemaps: https://www.npmjs.com/package/@protomaps/basemaps

³⁶ OpenLayers: https://openlayers.org

6. Read more

³⁷ Nominatim: https://nominatim.org/

³⁸ Photon: https://photon.komoot.io/

³⁹ Pelias: https://pelias.io/

⁴⁰ Valhalla: https://valhalla.github.io/valhalla/

⁴¹ OpenTripPlanner: https://www.opentripplanner.org/