In this post, I will describe how to create a Protomaps PMTiles file by starting off with an .osm.pbf extract from OpenStreetMap. The heavy lifting is done by Tilemaker which, with some scripting, enables you to turn .osm.pbf to an .mbtiles file, which in turn can be converted to a .pmtiles file.

Motivation

OpenTrailMap is a recent project to depict foot and cycle related paths. The map is bare-bones by design, ensuring that only those paths stand out in their contrasting (to the background) colours. They, however, do not seem to include sidewalks. Knowledge of sidewalks is important to me. Being a pedestrian (or in my case, a runner) in suburbia USA is unpleasant at best. Hostile is a better term. Knowing there is a sidewalk means knowing you can definitely stay safe on that stretch of road.

Eager to have a similar map as OpenTrailMap, but with sidewalks included, I figured it should be easy with the Protomaps setup I detailed before. Indeed, I can just follow my post about styling Protomaps and end up with something functional.

I create the following rule and push it into the end of the my_paint_rules array from the styling post. All the footways and cycleways that I am interested in, seem to be in the roads layer in a standard Protomaps file. They also have the property pmap:kind=path as a key-value pair.

let footway_rule = {
  dataLayer: "roads",
  symbolizer: new protomapsL.LineSymbolizer({
    color: "green",
    width: 2,
  }),
  filter: (zoom, feature) => {
    return feature.props["pmap:kind"] === "path";
  },
};

This works, but one thing is off. Zoom out even a little and most the paths start disappearing. This despite our rule not filtering on zoom at all. The issue is with the source .pmtiles file. The data in the PMTiles file is stored by zoom level. Most people are not interested in seeing sidewalks and footpaths from a very zoomed out vantage point. Thus to save space, most of those features are simply not included at those zoom levels. You cannot render data that is not there!

To get around this, creating my own PMTiles file seems like the way to go. Now, Protomaps does publish their Planetiler code. However, by the looks of it, that involves managing a Java setup. If I can avoid that, I happily will. So I look for another approach.

Approach

I will create a .pmtiles file consisting only of those footways, cycleways, and similar ways. For the rest of the map rendering, I will rely on the existing .pmtiles file from the intro post.

My approach consists of the following steps.

  1. Get an .osm.pbf file for the region of your choice from Geofabrik.
  2. Optional. If that region is still too large, you can carve out a smaller region from it. I detail how I do this at the end of a much larger post about a faulty .osm.pbf extract, specifically the “Extracting Pennsylvania Ourselves” section.
  3. Create a Tilemaker JSON configuration file.
  4. Create a Tilemaker Lua processing script.
  5. Have Tilemaker convert .osm.pbf to .mbtiles. At the time of writing, there is no built-in Protomaps support. Follow issue #445 for any updates on that front.
  6. Use pmtiles convert to turn .mbtiles to .pmtiles.

I use the following Makefile. It assumes your output file will have the same name as your input file (not counting extensions). So in my case I have a PA.osm.pbf file and want to obtain a PA.pmtiles file. PA stands for Pennsylvania, USA.

pmfile = PA.pmtiles
tilemaker-config = ./tilemaker/config.json
tilemaker-process = ./tilemaker/process.lua
tilemaker-exec = ./tilemaker/tilemaker

all: $(pmfile)

%.pmtiles: %.mbtiles
	pmtiles convert $< $@

%.mbtiles: %.osm.pbf $(tilemaker-config) $(tilemaker-process)
	$(tilemaker-exec) --input $< --output $@ --config $(tilemaker-config) --process $(tilemaker-process)

clean:
	rm $(pmfile)

Tilemaker

Tilemaker distributes as a single binary. To use it, you provide it three files: the .osm.pbf file, a JSON configuration file, and a Lua processing file.

Layer Setup

I believe I managed to trim this config.json file down to its essentials.

{
  "layers": {
    "roads": {
      "minzoom": 4,
      "maxzoom": 18,
      "simplify_below": 11,
      "simplify_level": 0.0003
    }
  },
  "settings": {
    "maxzoom": 14,
    "basezoom": 14,
    "include_ids": false,
    "combine_below": 14,
    "name": "Footways and friends in PA, USA",
    "version": "3.0",
    "description": "Only footways, cycleways, and similar enough stuff that is interesting to a runner are included. Based on data in a PA osm.pbf extract from Geofabrik.",
    "compress": "gzip"
  }
}

In it, you define what layers you want to exist in the final file. Compare with the dataLayer property in the paint rule above. In this case, I will stick to the Protomaps convention of a roads layer. This is the only layer I will have Tilemaker create. However, since the only style we will be applying is our own style, you have absolute freedom in the name you choose. If you think you might want to use the Protomaps default style at any point though, then you will want to stick to their scheme.

For a layer, you define the minimum and maximum zoom levels. This is important to ensure the data exists at the zoom levels you are interested in. Here I notice a weird discrepancy when rendering in Leaflet later on. While the data seems to be stored for the right zoom levels, rendering has an off-by-one error. A min zoom of 10 here, was a min zoom of 11 in that render. I am yet to figure out whether I am doing something wrong or whether there is a bug in the Protomaps Leaflet library.

I will also quote the Tilemaker documentation for this note about zoom levels. From it I conclude min zoom is important to ensure data starts being available, but max zoom does not matter as much.

Because vector tiles are so efficiently encoded, you generally don’t need to create tiles above (say) zoom level 14. Instead, your renderer will use the data in the z14 tiles to generate z15, z16 etc. (This is called ‘overzooming’.) So when you set a maximum zoom level of 14 in tilemaker, this doesn’t mean you’re restricted to displaying maps at z14. It just means that tilemaker will create z14 tiles, and it’s your renderer’s job to use these tiles to draw the most detailed maps.

You may also specify a simplify_below property. This will simplify your geometries once zoomed out past that level. This can help save some space in your .pmtiles file. It will essentially cut out some points from your paths that it thinks do not matter at that zoom, making them less smooth. You will still see the feature, it may just look a bit choppier.

The settings property is general metadata for your resulting file. I have not quite yet figured out all the meanings and which matter. My final result seems to work sufficiently for me to not care about it.

Processing Script

The real work happens in the Lua processing file which I, inspired creative soul that I am, name process.lua. I piece this file together while heavily relying on the example processing files that come with Tilemaker as well as their configuration documentation file. Note that all their documentation is in the docs/ folder in their repository.

Hooks

The Lua script is expected to contain some certain variables and functions. These are the hooks that Tilemaker needs to do any processing. You can, evidently, define as much extra stuff as you need to create a coherent program. The following three are required to be present in the file.

  • node_keys: a list of OSM tags you are interested in, used to filter nodes before passing them on to node_function.
  • node_function(node): function that gets called for every node that is not filtered out.
  • way_function(way): function that gets called for every way

There are other optional hooks. At the time of writing I spot init_function, exit_function, relation_scan_function, and relation_function. If they pique your interest, peruse the Tilemaker docs. They are not required for this post.

Nodes

Two of the required hooks are not needed. I will only be showing ways, so nodes are unnecessary. Thus, the following minimal implementation suffices for them.

-- Required by tilemaker, list of OSM keys indicating a node should be processed
node_keys = {}

-- Required by tilemaker, called for every node that is kept by node_keys
function node_function(node) end

Ways

The important thing to realise here is that interaction with both the input and output is via the way object that way_function receives as an argument. With a method like Find, you can read in information from the OSM data. For example, way:Find("highway") will return the value of the highway tag for that way. With the Layer method, you write out the way to a certain data layer. That data layer has to be a layer you defined in the JSON configuration file. In our case, we will be writing out to the roads layer. With the Attribute method, you write out a key-value tag for that feature. This way, you can pass on an OSM tag or add your own extra information.

With that in mind, let’s get started.

First off, I follow the example processing file that comes with Tilemaker and define a Set ADT. A set here is a table where every key you give it is set to true. I then create a set of values that match values in the OSM highway key. Specifically, I care about footway, cycleway, sidewalk, and the like. The set ADT feels a bit redundant here, since I only create one such set. If you, however, end up expanding this processing script, I imagine it will come in handy.

-- Implement Sets in tables
function Set(list)
	local set = {}
	for _, l in ipairs(list) do
		set[l] = true
	end
	return set
end

-- Relevant values for OSM `highway=*`
pedestrian_road_values = Set({
	"footway",
	"cycleway",
	"path",
	"steps",
	"pedestrian",
	"sidewalk",
	-- Do the following really fit here?
	"bridleway",
	"track",
})

Next, I have two helper functions. Again, these feel redundant since I use each of them once in this particular script, but for a bigger script, you will find yourself happily using them.

The first writes out the minimum zoom at which the given object should be included in the resulting file. There is a discrepancy between the zoom I have the script write out and the zoom at which it becomes available when rendering it in Leaflet. If I write out minimum zoom 8 here, then it does not show up till zoom 10 in Leaflet. Thus this function simply writes out a minimum zoom that is 2 lower than the minimum zoom you provide it. That way you do not always have to think about this difference. As with the off-by-one in the JSON configuration file, I am not aware whether I am messing up or whether there is some bug in the Protomaps Leaflet library.

function write_minzoom(obj, minzoom)
	-- For some reason writing a value n results in it not being available
	-- in pmtiles till zoom n+2. So here we make it fit.
	-- Note: This mismatch only seems to happen with leaflet. In the
	-- PMTiles Viewer the number we end up writing here _is_ respected.
	obj:MinZoom(minzoom - 2)
end

The second helper function reads the name from an OSM object and, if there is a name, writes it out to the resulting object.

function write_name(obj)
	local name = obj:Find("name")
	if name then
		obj:Attribute("name", name)
	end
end

Putting those things together at last gives us a way_function. I have the way_function just call a process_highways function, again with the goal of possible future expansion: I could now, for example, add a process_rivers and call that too in way_function.

process_highways finds the value for the OSM highway tag. It then ensures the value matches one of those we defined earlier. If it matches, then the way is written to the roads layer of the resulting file from the given minimum zoom onwards. Note that the minimum zoom mentioned in the JSON configuration also applies! I also have the script write out a pmap:kind tag on the way with value path. I do this to remain consistent with the official PMTiles build file and, consequently, the default styling file. Just in case I want to have all these paths styled the default way.

function way_function(way)
	process_highways(way)
end

function process_highways(way)
	local highway = way:Find("highway")

	if highway ~= "" then
		-- Filter for the highway=* values we listed
		if not pedestrian_road_values[highway] then
			return
		end

		-- Write out to roads layer
		way:Layer("roads", false)
		-- Ensure the way is available early enough. Note that config
		-- minzoom also applies.
		write_minzoom(way, 8)
		-- Set metadata
		write_name(way)
		way:Attribute("highway", highway)
		-- This pmap:kind=path is what Protomaps uses for these kinds
		-- of ways. I do too for consistency and to use default styling
		-- if so desired.
		way:Attribute("pmap:kind", "path")
	end
end

Now I can call tilemaker as shown in the Makefile earlier. If that is a bit hard for you to read, the call looks something like this.

tilemaker --input PA.osm.pbf --output PA.mbtiles --config config.json --process process.lua

This gives us a .mbtiles file. Almost there!

Convert to PMTiles

Converting the .mbtiles file is another long process. Just kidding it is one simple call of the pmtiles binary (remember, the one I used to extract a section from the official build in my intro post).

pmtiles convert PA.mbtiles PA.pmtiles

And there you go, one artisanal homemade .pmtiles file.

Putting It On The Map

Now just add that .pmtiles file as another Leaflet layer (again, see the intro post). For paint_rules, use an array with as only element the footway_rule I defined way back when at the beginning of this post (remember those times?).

let paths_paint_rules = [footway_rule];
let paths_layer = protomapsL.leafletLayer({
  url: "PA.pmtiles",
  paint_rules: paths_paint_rules,
  label_rules: [],
});
paths_layer.addTo(map);

If you added a Control to the map, as I did in the GoldenCheetah to PMTiles post, you can now expand it like this.

// Easy layer switching
L.control
  .layers(
    // base layers: only one activated (radiobox)
    { pmtiles: layer, osm: osm },
    // supplemental layers: checkboxes
    { activities: gc_layer, paths: paths_layer },
    // options
    { collapsed: false },
  )
  .addTo(map);

And here is what the final result looks like.

pathsimg

Future Considerations

  • OpenTrailMap also shows trail heads and hiking routes. These could be interesting to add (and would tackle parsing of nodes and relations and thus be informative subjects).
  • Also interesting might be things like (free) parking spaces, toilets, water fountains.
  • Including some data like water or tree cover from natural earth and friends requires a different layering approach. That could probably be a post on itself. If you need this now, try to figure it out with their example files.
  • You might not want to have a separate .pmtiles file at all. Redoing everything from Protomaps’ Planetiler setup might be a bit of a big task though. (It is, I half started it once, it is quickly not fun any more). Good luck to you if you strike out on that endeavour. Be sure to share it with the world.