The following is a guest post by Jacob Hands, Creator of FactorioMaps.com. He is building a community site for the game Factorio centered around sharing user creations.
Factorio is a game about building and maintaining factories. Players mine resources, research new technology and automate production. Resources move along the production line through multiple means of transportation such as belts and trains. Once production starts getting up to speed, alien bugs start to attack the factory requiring strong defenses.
A Factorio factory producing many different items.
A Factorio military outpost fighting the alien bugs.
A Factorio map view of a small factory, that’s still too big to easily share fully with screenshots.
At FactorioMaps.com, I am building a place for the community of Factorio players to share their factories as interactive Leaflet maps. Due to the size and detail of the game, it can be difficult to share an entire factory through a few screenshots. A Leaflet map provides a Google Maps-like experience allowing viewers to pan and zoom throughout the map almost as if they are playing the game.
Leaflet maps contain thousands of small images for X/Y/Z coordinates. Amazon S3 and Google Cloud Storage are the obvious choices for low-latency object storage. However, after 3.5 months in operation, FactorioMaps.com contains 17 million map images (>1TB). For this use-case, $0.05 per 10,000 upload API calls and $0.08 to 0.12/GB for egress would add up quickly. Backblaze B2 is a better fit because upload API calls are free, egress bandwidth is $0.00/GB to Cloudflare, and storage is 1/4th the price of the competition.
Backblaze B2 requires a prefix of /file/bucketName on all public files, which I don’t want. To remove it, I added a VPS proxy to rewrite paths and add a few 301 redirects. Unfortunately, the latency from the user -> VPS -> B2 was sub-par averaging 800-1200ms in the US.
A Closer Look At Leaflet
Leaflet maps work by loading images at the user’s X/Y/Z coordinates to render the current view. As a map is zoomed in, it requires 4x as many images to show the same area. That means 75% of a map’s images are in the max rendered zoom level.
A diagram of how each zoom level is 4x larger than the previous
With hosting working, it’s time to start making the site faster. The majority of image requests come from the first few zoom levels, representing less than 25% of a given map’s images. Adding a local SSD cache on the VPS containing all except the last 1-3 zoom levels for each map reduces latency for 66% of requests. The problem with SSD storage is it’s difficult to scale with ever-increasing data and is still limited to the network and CPU performance of the server it occupies.
Going Serverless with Cloudflare Workers
While Google Cloud Storage is more expensive than B2, it has much lower latency to the US and worldwide destinations because of their network and multi-regional object storage. However, it’s not time to move the whole site over to GCS just yet; the upload API calls alone would cost $85 for 17 million files.
Multi-Tier Object Storage
The first few zoom levels are stored in GCS, while the rest are in B2. Cloudflare Workers figure out where files are located by checking both sources simultaneously. By doing this, 66% of requested files come from GCS with a mean latency of