Making an App – Part 1

I've been playing Zelda: Breath of the Wild with my daughter recently. Since she's kind of a "completionist" and wants to complete all the hundreds of shrines, quests, and extras in the game, I went looking for a decent map with locations called out.

We found Zelda Maps which is decent and even installs as an app on an iPad. Unfortunately, my daughter's iPad is one of the original iPad mini models, which is basically an iPad 2 in a smaller case, and the site really just doesn't run well enough to be usable. Even on my iPad Air it runs OK, but still not great. The UI is geared towards a mouse and keyboard with very small controls which is also problematic on a small iPad and the site has ads and ad trackers running.

After poking around the Javascript and using Safari's web inspector, I could see how the app worked so I thought I'd use the site's data to make a native iPad app for her that would run well and maybe have some extra features to boot.

Making an App … In Public

I thought it might be interesting to post about the process of making the app, especially since it's not really destined for the AppStore, and not as a neatly packaged, finished product, but the way most apps get made – a little bit at a time and sometimes not so pretty.

Checking Out the Data

Looking inside the web site, it's built using the Javascript map library Leaflet. The map imagery is tiled with 256x256 image tiles at nine zoom levels, from level 0 which is a single, 1x1 tile set, up through level 8, which is a massive 188x160 tile set of 48k x 40k pixel image.

The markers are pretty easy, too, since they in basically two JSON feeds, one for all of the individual markers and another for the categories of markers.

That's nearly 1gb of map data and over 3,000 map markers … no wonder the web app doesn't run so well on an iPad mini.

Showing a large, zoomable and scrollable image with overlays is well within the kind of thing iOS and UIKit can do well, so this shouldn't be all that hard.

Getting the Data

Considering we have nearly 50,000 files to download and some images and JSON data manipulate, we've got some coding to do. You might be tempted to write some Python, Ruby, Perl, or even Swift to do it, but I always try to stick with shell scripts for this kind of thing where I can – Bash is very capable and there some excellent command line tools for image and JSON manipulation.

I recommend curl, ImageMagick, and jq. If you don't have them, install Homebrew and then install them.

When I need to run some commands several times, I like to write a script. First, let's setup a project directory:

$ mkdir botw
$ cd botw
$ mkdir tiles
$ mkdir scripts
$ mate scripts/download

I tend to use Textmate for editing scripts and other non-Xcode file, but vi works fine, too (or any other text editor you prefer). Start with a shebang and assume we're getting a zoom level, a min & max column, and a min & max row. Experience and Cocoa have taught me that calling things what they are with minimal abbreviation is always worth the effort.

#!/bin/bash

scale=$1
col_min=$2
col_max=$3
row_min=$4
row_max=$5

width=$(( col_max-col_min+1 ))
height=$(( row_max-row_min+1 ))

We'll need to know the width and height later, so a little arithmetic here. If this were destructive script, I'd put some validation and argument parsing in, but we're going to inch our way to a solution so not really important now.

I usually will "dry run" my scripts by just echoing the scripts first. We need to download a bunch of PNG files in this form:

(scale)_(column)_(row).png

To keep things easy, we'll name them the same. So a loop in a loop should do the trick:

for col in `seq ${col_min} ${col_max}`
do
  for row in `seq ${row_min} ${row_max}`
  do
    pattern=${scale}_${col}_${row}.png
    echo curl https://zeldamaps.com/tiles/botw/hyrule/${pattern} -o ${pattern} -s
  done
done

Save it and back in the terminal make the command executable and try it out. I happen to know that scale 1 has tiles from 0 to 1 in rows and columns, so let's try it:

$ chmod +x scripts/download
$ ./scripts/download 1 0 1 0 1

That produces this:

curl https://zeldamaps.com/tiles/botw/hyrule/1_0_0.png -o 1_0_0.png -s
curl https://zeldamaps.com/tiles/botw/hyrule/1_0_1.png -o 1_0_1.png -s
curl https://zeldamaps.com/tiles/botw/hyrule/1_1_0.png -o 1_1_0.png -s
curl https://zeldamaps.com/tiles/botw/hyrule/1_1_1.png -o 1_1_1.png -s

Let's try one of those commands out:

$ curl https://zeldamaps.com/tiles/botw/hyrule/1_0_0.png -o 1_0_0.png -s
$ identify 1_0_0.png
1_0_0.png PNG 256x256 256x256+0+0 8-bit sRGB 45494B 0.010u 0:00.009

Looks like we got a good download, so let's delete test file, remove the echo from the curl command, add some extra status messaging, and add an extra parameter for a prefix so we can direct where the files download to.

#!/bin/bash

scale=$1
col_min=$2
col_max=$3
row_min=$4
row_max=$5
prefix=$6

width=$(( col_max-col_min+1 ))
height=$(( row_max-row_min+1 ))

for col in `seq ${col_min} ${col_max}`
do
  for row in `seq ${row_min} ${row_max}`
  do
    pattern=${scale}_${col}_${row}.png
    echo -n "Downloading ${pattern} ... "
    curl https://zeldamaps.com/tiles/botw/hyrule/${pattern} -o ${prefix}${pattern} -s
    echo "OK"
  done
done

Now let's run it ...

$ ./scripts/download 1 0 1 0 1 tiles/
Downloading 1_0_0.png ... OK
Downloading 1_0_1.png ... OK
Downloading 1_1_0.png ... OK
Downloading 1_1_1.png ... OK

Of course, some of these downloads are going to be pulling down hundreds or thousands of files … would be nice if it could be started and stopped without starting all over. Let's put a check in for already downloaded files and test it out

#!/bin/bash

scale=$1
col_min=$2
col_max=$3
row_min=$4
row_max=$5
prefix=$6

width=$(( col_max-col_min+1 ))
height=$(( row_max-row_min+1 ))

for col in `seq ${col_min} ${col_max}`
do
  for row in `seq ${row_min} ${row_max}`
  do
    pattern=${scale}_${col}_${row}.png
    if [ -f ${prefix}${pattern} ]
    then
      echo "Skipping ${pattern}"
    else
      echo -n "Downloading ${pattern} ... "
      curl https://zeldamaps.com/tiles/botw/hyrule/${pattern} -o ${prefix}${pattern} -s
      echo "OK"
    fi
  done
done

Now testing ...

$ rm tiles/1_0_0.png 
$ ./scripts/download 1 0 1 0 1 tiles/
Downloading 1_0_0.png ... OK
Skipping 1_0_1.png
Skipping 1_1_0.png
Skipping 1_1_1.png

Works nicely, time to download 10,097 files (I decided to skip level 8 since it had no actual additional detail, just level 7 doubled).

Assembling the Tiles

While it's possible to display large, tiled views with multiple zoom levels using CATiledLayer, I do want to checkout that the tiles I've pulled down look good together and it might help to use a single, assembled map as a starting point so we don't have to implement tiling right away.

ImageMagick has a tool that does this easily, montage. It takes many options and can produce many kinds of output, but we just want to assemble the tiles in order to produce the original, untiled image. To do this, we simply need to pass the list of images, how much padding to use, and the output file name.

Like so, using zoom level 2 which is a 4x4 grid.

$ montage tiles/2_*.png -geometry +0+0 2.png

montage can usually figure out layout, height and width, but looking at the output it's not working right; the tiles that should be going down the left are going across the top. It's mixing the rows and columns. Turns out that montage is wanting the tiles to be sorted into rows, but our tiles are sorting by column since the column is first in the name.

Fortunately, sorting in the shell is pretty easy; we want to sort the file name by row which is the third number in the name and then by the row. A quick look at the manual page for sort shows we need to pass a separator and keys. This seems to give the correct result (the n in -k3n -k2n is important since it indicates the values are numbers so that we don't get "1, 11, 12, … 2, 21, … 3, …", but "1, 2, 3, … 11, 12, …, 21, …"):

$ ls tiles/2_* | sort -t'_' -k3n -k2n

Now we give this to montage and we get the output we want:

$ montage `ls tiles/2_* | sort -t'_' -k3n -k2n` -geometry +0+0 2.png

Unfortunately, starting at level 5 the grid isn't square so montage guesses wrong on the layout. We need to help it out. It looks like the map is 24 tiles wide at level 5 so we can hint it (and double that at 6 and 7, so 48x and 96x):

$ montage `ls tiles/5_* | sort -t'_' -k3n -k2n` -geometry +0+0 -tile 24x 5.png

These larger maps can really chew up some CPU and run for a bit so you might want to use both nice and caffeinate to lower the priority of the job and keep the power saver from kicking in while it's running.

Once that's done, that's a pretty decent stopping point. Next post will pull down and clean up the JSON data.

Home