Building a tile server on Ubuntu- 13.04

Manually building a tile server (13.04)

This page describes how to install, setup and configure all the necessary software to operate your own tile server. The step-by-step instructions are written for Ubuntu Linux 13.04 Software installation

The OSM tile server stack is a collection of programs and libraries that work together to create a tile server. As so often with OpenStreetMap, there are many ways to achieve this goal and nearly all of the components have alternatives that have various specific advantages and disadvantages. This tutorial describes the most standard version that is also used on the main OpenStreetMap.org tile server.

It consists of 5 main components: Mod_tile, renderd, mapnik, osm2pgsql and a postgresql/postgis database. Mod_tile is an apache module, that serves cached tiles and decides which tiles need re-rendering – either because they are not yet cached or because they are outdated. Renderd provides a priority queueing system for rendering requests to manage and smooth out the load from rendering requests. Mapnik is the software library that does the actual rendering and is used by renderd.

In order to build these components, a variety of dependencies need to be installed first.

Mapnik requires a recent version of Boost, which can be installed on Ubuntu like this:

sudo add-apt-repository ppa:mapnik/boost
sudo apt-get update
sudo apt-get install libboost-dev libboost-filesystem-dev libboost-program-options-dev libboost-python-dev libboost-regex-dev libboost-system-dev libboost-thread-dev

The remaining dependencies can be installed with the following instructions

sudo apt-get install subversion git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libpq-dev libbz2-dev proj munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libpng12-dev libtiff4-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont    (IN red color not installed for 11.10) (Files in Blue are not installed during installation for 13.04 rest of files installed)

**** For Proj -> sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable . AND apache2-dev is just a virtual package and provided by apache2-threaded-dev,

Installing postgresql / postgis

PostgreSQL is a powerful and reliable object-relational database system. It’s a great alternative for MySQL. It is as easy to set up, performs better and offers far more features.

1. Make sure you already have install python-software-properties

$sudo apt-get install python-software-properties

2.Add PPA repository to my Ubuntu.

$sudo add-apt-repository ppa:pitti/postgresql

3. After adding PPA, update your system apt:

$sudo apt-get update

4.Finally install postgresql-9.1:

$sudo apt-get install postgresql or sudo apt-get install postgresql-9.1-postgis postgresql-contrib-9.1 postgresql-server-dev-9.1

if you having any error, make sure you already install libpq-dev.The libpq-dev package is for compiling wrappers/clients against libpq.

5.Now check it out installation is successful or…..

$ locate postgresql

6. If done!!!, Cheers …………..                      Check the install version.

$psql -V

7.Now let’s take look to postgres console

$su postgres

7(a) Setup Root User ‘postres’

$sudo passwd postgres
give the postgres user a (unix) password,Now we can switch to the user postgres using command
$ su postgres

please check if

On Ubuntu there are pre-packaged versions of both postgis and postgresql, so these can simply be installed via the ubuntu package manager.

sudo apt-get install postgresql-9.1-postgis postgresql-contrib-9.1 postgresql-server-dev-9.1

Now you need to create a postgis database. The defaults of various programs assume the database is called gis and we will use the same convention in this tutorial, although this is not necessary. Substitute your username for username in the two places below. This should be the username that will render maps with Mapnik.

sudo -u postgres -i
createuser postgres # answer yes for superuser (although this isn’t strictly necessary)
createdb -E UTF8 -O postgres  gis
exit

Set up PostGIS on the postgresql database: (using postgres user)

psql -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -d gis

This should respond with many lines ending with:


CREATE FUNCTION
COMMIT

DROP FUNCTION

Give your newly-created user permission to access some of the PostGIS extensions’ data. Make sure you replace username with your user’s name:

psql -d gis -c “ALTER TABLE geometry_columns OWNER TO postgres; ALTER TABLE spatial_ref_sys OWNER TO postgres;”

Installing osm2pgsql

Although there might be a osm2pgsql package in the repository, it is likely rather old and so we need to compile a newer one from source

 ( Using  unprivileged user i.e. not by postgres but by using abc$)

add repo

$sudo add-apt-repository ppa:ubuntugis/ppa    {Part of ubuntugis-stable problem occurred  added the required files manually. It May not be required if the system is updated during installation of ubuntu}

mkdir ~/src
cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
./autogen.sh
./configure   {Problem was as the required packages were not found so ./configure was not executed It Worked after installing required packages}

make
sudo make install
psql -f /usr/local/share/osm2pgsql/900913.sql -d gis    ( Using  postgres user i.e. privileged user)

Install Mapnik library

Next, we need to install the Mapnik library. Mapnik is used to render the OpenStreetMap data into the tiles used for an OpenLayers web map.

Build the Mapnik library from source:

cd ~/src
git clone git://github.com/mapnik/mapnik
cd mapnik
git branch 2.0 origin/2.0.x
git checkout 2.0
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
python scons/scons.py     { Took to much time for installation  It worked but with few errors }
sudo python scons/scons.py install
sudo ldconfig

Verify that Mapnik has been installed correctly:

python
>>> import mapnik
>>>

If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations!   { Shown >>> during installation

Install mod_tile and renderd

Compile the mod_tile source code: {unpriviliged user}

cd ~/src
git clone git://github.com/openstreetmap/mod_tile.git
cd mod_tile
./autogen.sh
./configure    {Again giving error for ubuntu 13.04 but resolved after installing sudo apt-get install apache2-prefork-dev}

make

sudo make install     {Again giving error for ubuntu 13.04}

sudo make install-mod_tile
sudo ldconfig

Install Mapnik stylesheet

Next, we need to install the OpenStreetMap Mapnik tools, which include the default style file and tools to help Mapnik render OpenStreetMap data:

cd ~/src
svn co http://svn.openstreetmap.org/applications/rendering/mapnik mapnik-style

Mapnik uses prepared files to generate coastlines and ocean areas for small scale maps since it is faster than reading the entire database for this information. Downloading the coastline data requires about 400 Mb of download.

cd ~/src/mapnik-style
sudo ./get-coastlines.sh /usr/local/share

Software configuration

Now that all of the necessary software is installed, you will need to configure some of the software to work correctly.

Configure mapnik style-sheet

In order for mapnik to find the correct postgis database and the coast line data, you will need to configure the mapnik style-sheet to your local settings. There are a number of different style-sheets publically available and you can of course create your own ones. Each style-sheet might be slightly different in how to configure it and so this page describes how to configure the “default” OpenStreetMap style-sheet that was installed with the mapnik-tools.

In your style-sheet directory (e.g. ~/src/mapnik-style) there should be a directory inc. There are a number of files you need to adapt in this directory:

cd inc
cp fontset-settings.xml.inc.template fontset-settings.xml.inc
cp datasource-settings.xml.inc.template datasource-settings.xml.inc
cp settings.xml.inc.template settings.xml.inc

Now you need to modify each of these files:

settings.xml.inc { which is in the same foolder}

replace

<!ENTITY symbols “%(symbols)s”>

with

<!ENTITY symbols “symbols”>

replace

<!ENTITY osm2pgsql_projection “&srs%(epsg)s;”>

with (assuming you will be using the defaul projection 900913)

<!ENTITY osm2pgsql_projection “&srs900913;”>

replace

<!ENTITY dwithin_node_way “&dwithin_%(epsg)s;”>

with (assuming you will be using the default projection 900913)

<!ENTITY dwithin_node_way “&dwithin_900913;”>

replace

<!ENTITY world_boundaries “%(world_boundaries)s”>

with (assuming you installed the coast line data in /usr/local/share/world_boundaries)

<!ENTITY world_boundaries “/usr/local/share/world_boundaries”>

replace

<!ENTITY prefix “%(prefix)s”>

with (assuming you are using the default database table prefix)

<!ENTITY prefix “planet_osm”>

datasource-settings.xml.inc

In this file you will need to enter your database settings. You are running postgresql on the same machine as the rendering stack, so you can comment out the parameters “password”, “host” and “port” with an HTML-style comment. This will enable mapnik to use the “unix local user” as an authentication method.

Change the “dbname” from “%(dbname)s” to “gis”, “estimate_extent” to “false”, and “extent” to “-20037508,-19929239,20037508,19929239″ so that the file now reads:

<!–
Settings for your postgres setup.

Note: feel free to leave password, host, port, or use blank
–>

<Parameter name=”type”>postgis</Parameter>
<!– <Parameter name=”password”>%(password)s</Parameter> –>
<!– <Parameter name=”host”>%(host)s</Parameter> –>
<!– <Parameter name=”port”>%(port)s</Parameter> –>
<!– <Parameter name=”user”>%(user)s</Parameter> –>
<Parameter name=”dbname”>gis</Parameter>
<!– this should be ‘false’ if you are manually providing the ‘extent’ –>
<Parameter name=”estimate_extent”>false</Parameter>
<!– manually provided extent in epsg 900913 for whole globe –>
<!– providing this speeds up Mapnik database queries –>
<Parameter name=”extent”>-20037508,-19929239,20037508,19929239</Parameter>

fontset-settings.xml.inc

This file contains font definitions, and information about how to change the default font.  It is recommended that you don’t edit this right now.

Configure rendered

Change the the renderd settings by editing the  /usr/local/etc/renderd.conf {open this file using gedit} and change the following lines like so (remember to change username to your user’s name):

socketname=/var/run/renderd/renderd.sock
plugins_dir=/usr/local/lib/mapnik/input
font_dir=/usr/share/fonts/truetype/ttf-dejavu
XML=/home/abc/src/mapnik-style/osm.xml     {is username at college desktop}
HOST=localhost

Create the files required for the mod_tile system to run (remember to change username to your user’s name):

sudo mkdir /var/run/renderd
sudo chown postgres /var/run/renderd
sudo mkdir /var/lib/mod_tile
sudo chown postgres /var/lib/mod_tile

Configure mod_tile

Next, we need to tell the Apache web server about our new mod_tile installation by creating the file /etc/apache2/conf.d/mod_tile (Created file using gedit) and adding one line:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

Also, Apache’s default website configuration file needs to be modified to include mod_tile settings. Modify the file /etc/apache2/sites-available/default to include the following lines immediately after the admin e-mail address line:

LoadTileConfigFile /usr/local/etc/renderd.conf
ModTileRenderdSocketName /var/run/renderd/renderd.sock
# Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
# Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30

Tuning your system

A tile server can put a lot of load on hard- and software. The default settings may therefore not be appropriate and a significant improvement can potentially be achieved through tuning various parameters.

Tuning postgresql

The default configuration for PostgreSQL 9.1 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/9.1/main/postgresql.conf and make the following changes:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

These changes require a kernel configuration change, which needs to be applied every time that the computer is rebooted. As root, edit /etc/sysctl.conf and add these lines near the top after the other “kernel” definitions:

# Increase kernel shared memory segments – needed for large databases
kernel.shmmax=268435456

Reboot your computer. Run this:

sudo sysctl kernel.shmmax

and verify that it displays as 268435456.

Loading data into your server

Get the latest OpenStreetMap data

Retrieve a piece of OpenStreetMap data from http://planet.openstreetmap.org/. Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. Since it is smaller and faster to interact with, the PBF file format is preferable when available. In this case we’ll download the entire planet file by issuing the following command:

cd planet
wget http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

OR country extracts can be found e.g. on download.geofabrik.de). e.g:

wget http://download.geofabrik.de/openstreetmap/europe/ireland-and-northern-ireland.osm.pbf

i downloaded Indian data from the website

wget http://download.geofabrik.de/asia/india-latest.osm.pbf    ( worked)

Importing data into the database

With the conversion tool compiled and the database prepared, the following command will insert the OpenStreetMap data you downloaded earlier into the database. This step is very disk I/O intensive and will take anywhere from 10 hours on a fast server with SSDs to several days for the full planet depending on the speed of the computer performing the import. For smaller extracts the import time is much faster accordingly, and you may need to experiment with different -C values to fit within your machine’s available memory.

Change directory to where you unzipped osm2pgsql:

cd PATH_TO_OSM2PGSQL  {usinh postgres user}

Import OSM data:

osm2pgsql –slim -d gis -C 1000 –number-processes 3 ~/india-map/india-latest.osm.pbf  {work but may try 1000 is size of RAM . for laptop i used RAM 500}

osm2pgsql –slim -d gis -C 500 –number-processes 3  /home/a/src/indiamap/punjab.osm

osm2pgsql -U postgres -s -S ./default.style  /home/abc/india-map/pbadm.osm  {worked for me as i downloaded punjab data from cloudmade site}

osm2pgsql -U postgres -s -S ./default.style  /home/abc/india-map/india.osm  {worked for me as i downloaded data from cloudmade site}

{Using postgres user to open this user first enter root user then use su postgre becoz it will not demand password then}

Worked for .osm format as .osm.pbf was not supported format. I got

Starting your tileserver

Now that everything is installed, set-up and loaded, you can start up your tile server and hopefully everything is working.  We’ll run it interactively first, just to make sure that everything’s working properly:   {try to run this using root}

To Start Renderd

$$sudo service renderd start

and to stop if use

$$sudo service renderd stop

renderd -f -c /usr/local/etc/renderd.conf

On my laptop i was able to see the http://localhost/osm_tiles/0/0/0.png but unable to see http://localhost/osm/slippymap.html as then i copied slippymap.html file from mod_tile folder to /var/www and edited the local tile server line to http://localhost/osm_tiles/S{z}/${y}/${x} and it worked.

and on a different session:

sudo /etc/init.d/apache2 restart

If any FATAL errors occur you’ll need to double-check any edits that you made earlier.

If not, try and browse to http://localhost/ to see if a small picture of the world appears.

If it does, you can stop the interactive renderd process and configure it to run automatically at machine startup as a daemon.

sudo cp  ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd

Edit the /etc/init.d/renderd file as root – you’ll need to make a couple of changes to the DAEMON and DAEMON_ARGS lines so that they read:

DAEMON=/usr/local/bin/$NAME
DAEMON_ARGS=”-c /usr/local/etc/renderd.conf”

Also, you’ll need to change references to www-data so that they match your username – change “www-data” to what you changed “username” to in other files.

You should now be able to start mapnik by doing the following:

sudo /etc/init.d/renderd start

and stop it:

sudo /etc/init.d/renderd stop

Logging information is now written to /var/log/daemon.log instead of to the terminal.

Next, add a link to the interactive startup directory so that it starts automatically:

sudo ln -s /etc/init.d/renderd /etc/rc2.d/S20renderd

and then restart your server, browse to http://yourserveraddress/osm_tiles/0/0/0.png and everything should be working! You can also go to the page http://localhost/mod_tile which should give you some stats about your tileserver.

Next, you might want to have a look at something in the using tiles section to create a map that uses your new tile server.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s