Storage

Note

The cache is the physical location of the layers tiles, whether it is on the local file system, an S3 bucket, a pure in-memory cache, or anything else. A “blobstore” is a software component that provides the operations to store and retrieve tiles to and from a given storage mechanism.

Cache

Starting with version 1.8.0, there are two types of persistent storage mechanisms for tiles:

  • File blob store: stores tiles in a directory structure consisting of various image files organized by layer and zoom level.

  • S3 blob store: stores tiles in an Amazon Simple Storage Service bucket, as individual “objects” following a TMS-like key structure.

Zero or more blobstores can be configured in the configuration file to store tiles at different locations and on different storage back-ends. One of the configured blobstores will be the default one. Meaning that it will be used to store the tiles of every layer whose configuration does not explicitly indicate which blobstore shall be used.

Note

there will always be a “default” blobstore. If a blobstore to be used by default is not explicitly configured, one will be created automatically following the legacy cache location lookup mechanism used in versions prior to 1.8.0.

Configuration File

The location of the configuration file, geowebcache.xml, will be defined by the GEOWEBCACHE_CACHE_DIR application argument.

There are a few ways to define this argument:

  • JVM system environment variable

  • Servlet context parameteter

  • Operating system environment variable

The variable in all cases is defined as GEOWEBCACHE_CACHE_DIR.

To set as a JVM system environment variable, add the parameter -DGEOWEBCACHE_CACHE_DIR=<path> to your servlet startup script. In Tomcat, this can be added to the Java Options (JAVA_OPTS) variable in the startup script.

To set as a servlet context parameter, edit the GeoWebCache web.xml file and add the following code:

<context-param>
  <param-name>GEOWEBCACHE_CACHE_DIR</param-name>
  <param-value>PATH</param-value>
</context-param>

where PATH is the location of the cache directory.

To set as an operating system environment variable, run one of the the following commands:

Windows:

> set GEOWEBCACHE_CACHE_DIR=<path>

Linux/OS X:

$ export GEOWEBCACHE_CACHE_DIR=<path>

Finally, although not recommended, it is possible to set this location directly in the geowebcache-core-context.xml file. Uncomment this code:

<!-- bean id="gwcBlobStore" class="org.geowebcache.storage.blobstore.file.FileBlobStore" destroy-method="destroy">
  <constructor-arg value="/tmp/gwc_blobstore" />
</bean -->

making sure to edit the path. As usual, any changes to the servlet configuration files will require Reloading Configuration.

Note

if GEOWEBCACHE_CACHE_DIR is not provided by any of the above mentioned methods, the directory will default to the temporary storage folder specified by the web application container. (For Tomcat, this is the temp directory inside the root.) The directory created will be called geowebcache. If this directory is not available, GeoWebCache will attempt to create a new geowebcache directory in the location specified by the TEMP system environment variable. It is highly recommended to explicitly define the location of the configuration/cache directory.

BlobStore configuration

A basic installation does not require to configure a blobstore. One will be created automatically following the same cache location lookup mechanism as for versions prior to 1.8.0, meaning that a file blobstore will be used at the directory defined by the GEOWEBCACHE_CACHE_DIR application argument.

Starting with 1.8.0, it is possible to configure multiple blobstores, which provides several advantages:

  • Allow to decouple the location of the configuration and the storage;

  • Allow for multiple cache base directories;

  • Allow for alternate storage mechanisms than the current FileBlobStore;

  • Allow for different storage mechanisms to coexist;

  • Allow to chose which “blob store” to save tiles to on a per “tile layer” basis;

  • Allow serving pre-seeded caches directly from S3.

The geowebcache.xml file must be edited to configure blob stores.

The following is the excerpt of the schema definition that allows to configure blob stores: BlobStores XML schema

Between the formatModifiers and gridSets elements of the root gwcConfiguration element, a list of blob stores can be configured as children of the blobStores element. For example:

<gwcConfiguration>
  ...
  <formatModifiers>...</formatModifiers>

  <blobStores>
   <FileBlobStore default="true"><id>default_cache</id><enabled>true</enabled>...</FileBlobStore>
   <S3BlobStore><id>default_cache</id><enabled>true</enabled>...</S3BlobStore>
   <FileBlobStore><id>default_cache</id><enabled>false</enabled>...</FileBlobStore>
  </blobStores>

  <gridSets>...</gridSets>
  ...
</gwcConfiguration>

Common properties

All blob stores have the default, id, and enabled properties.

  • default is an optional attribute which defines if the blob store is the default one. Only one blob store can have this attribute set to true. Having more than one blob store with default="true" will raise an exception at startup time. Yet, if no blob store has default="true", a default FileBlobStore will be automatically created at the directory specified by the GEOWEBCACHE_CACHE_DIR application argument for backwards compatibility.

  • id is a mandatory string property defining a unique identifier for the blobstore for the geowebcache instance. Not defining a unique id for a blobstore, or configuring more than one with the same id, will raise an exception at application startup time. This identifier can then be referred to by the blobStoreId element of a wmsLayer in the same configuration file, in order to explicitly state which blob store to use for a given layer.

  • enabled is an optional attribute that defaults to true. If a blobstore is not enabled (i.e. <enabled>false</enabled>), then it cannot be used and any attempt to store or retrieve a tile from it will result in a runtime exception making the operation fail. Note that it is invalid to have the default="true" and <enabled>false</enabled> properties at the same time, resulting in a startup failure.

Besides these common properties, each kind of blob store defines its own, as follows:

File Blob Store

The file blob store saves tiles on disk following the traditional geowebcache cache directory layout.

Example:

<FileBlobStore default="false">
  <id>defaultCache</id>
  <enabled>false</enabled>
  <baseDirectory>/opt/defaultCache</baseDirectory>
  <fileSystemBlockSize>4096</fileSystemBlockSize>
</FileBlobStore>

Properties:

  • baseDirectory: Mandatory. The absolute path for the cache’s root directory.

  • fileSystemBlockSize: Optional, defaults to 4096. A positive integer representing the file system block size (usually 4096, 8292, or 16384, depending on the file system where the base directory resides. This value is used to pad the size of tile files to the actual size of the file on disk before notifying the internal blob store listeners when tiles are stored, deleted, or updated. This is useful, for example, for the “disk-quota” subsystem to correctly compute the cache’s disk usage.

Amazon Simple Storage Service (S3) Blob Store

The following documentation assumes you’re familiar with the Amazon Simple Storage Service.

This blob store allows to configure a cache for layers on an S3 bucket with the following TMS-like key structure:

[prefix]/<layer id>/<gridset id>/<format id>/<parameters hash | “default”>/<z>/<x>/<y>.<extension>

  • prefix: if provided in the configuration, it will be used as the “root path” for tile keys. Otherwise the keys will be built starting at the bucket’s root.

  • layer id: the unique identifier for the layer. Note it equals to the layer name for standalone configured layers, but to the geoserver catalog’s object id for GeoServer tile layers.

  • gridset id: the name of the gridset of the tile

  • format id: the gwc internal name for the tile format. E.g.: png, png8, jpeg, etc.

  • parameters hash: if the request that originated that tiles included parameter filters, a unique hash code of the set of parameter filters, otherwise the constant default.

  • z: the z ordinate of the tile in the gridset space.

  • x: the x ordinate of the tile in the gridset space.

  • y: the y ordinate of the tile in the gridset space.

  • extension: the file extension associated to the tile format. E.g. png, jpeg, etc. (Note the extension is the same for the png and png8 formats, for example).

Support for S3-compatible servers other than Amazon is also present.

Configuration example:

<S3BlobStore default="false">
  <id>myS3Cache</id>
  <enabled>false</enabled>
  <bucket>put-your-actual-bucket-name-here</bucket>
  <prefix>test-cache</prefix>
  <awsAccessKey>putYourActualAccessKeyHere</awsAccessKey>
  <awsSecretKey>putYourActualSecretKeyHere</awsSecretKey>
  <access>private</access>
  <maxConnections>50</maxConnections>
  <useHTTPS>true</useHTTPS>
  <endpoint>http://putYourServerEndpointHereOrLeaveOutIfUsingAmazon:9000</endpoint>
  <proxyDomain></proxyDomain>
  <proxyWorkstation></proxyWorkstation>
  <proxyHost></proxyHost>
  <proxyPort></proxyPort>
  <proxyUsername></proxyUsername>
  <proxyPassword></proxyPassword>
  <useGzip>true</useGzip>
</S3BlobStore>

Properties:

  • bucket: Mandatory. The name of the AWS S3 bucket where to store tiles.

  • prefix: Optional. A prefix path to use as the “root folder” to store tiles at. For example, if the bucket is bucket.gwc.example and prefix is “mycache”, all tiles will be stored under bucket.gwc.example/mycache/{layer name} instead of bucket.gwc.example/{layer name}.

  • awsAccessKey: Mandatory. The public access key the client uses to connect to S3.

  • awsSecretKey: Mandatory. The secret key the client uses to connect to S3.

  • access: Optional. Whether direct access in S3 will be readable by the public or only to the owner of the bucket. Defaults to public, set to private to disable public access.

  • maxConnections: Optional, default: 50. Maximum number of concurrent HTTP connections the S3 client may use.

  • useHTTPS: Optional, default: true. Whether to use HTTPS when connecting to S3 or not.

  • endpoint: Optional. Endpoint of the server, if using an alternative S3-compatible server instead of Amazon.

  • proxyDomain: Optional. The Windows domain name for configuring an NTLM proxy. If you are not using a Windows NTLM proxy, you don’t need to set this property.

  • proxyWorkstation: Optional. The Windows domain name for configuring an NTLM proxy. If you are not using a Windows NTLM proxy, you don’t need to set this property.

  • proxyHost: Optional. The proxy host the client will connect through.

  • proxyPort: Optional. The proxy port the client will connect through.

  • proxyUsername: Optional. The proxy user name to use if connecting through a proxy.

  • proxyPassword: Optional. The proxy password to use when connecting through a proxy.

  • useGzip: Optional, default: true. Whether gzip compression should be used when transferring tiles to/from S3.

Note: It is possible to set above properties from environment variable as long as they are of string type. In the example below, The awsAccessKey is set from environment variable named AWS_ACCESS_KEY

<awsAccessKey>${AWS_ACCESS_KEY}</awsAccessKey>

Additional Information:

The S3 objects for tiles are created with public visibility to allow for “standalone” pre-seeded caches to be used directly from S3 without geowebcache as middleware. In the future this behavior could be disabled through a configuration option.

Beware of amazon services costs. Especially in terms of bandwidth usage when serving tiles out of the Amazon cloud, and S3 storage prices. We haven’t conducted a thorough assessment of costs associated to seeding and serving caches. Yet we can provide some general purpose advise:

  • Do not seed at high zoom levels (except if you know what you’re doing). The number of tiles grow exponentially as the zoom level increases.

  • Use the tile format that produces the smalles possible tiles. For instance, png8 is a great compromise for quality/size. Keep in mind that the smaller the tiles the bigger the size difference between two identical caches on S3 vs a regular file system. The S3 cache takes less space because the actual space used for each tile is not padded to a file system block size. For example, the topp:states layer seeded up to zoom level 10 for EPSG:4326 with png8 format takes roughly 240MB on an Ext4 file system, and about 21MB on S3.

  • Use in-memory caching. When serving S3 tiles from GeoWebcache, you can greately reduce the number of GET requests to S3 by configuring an in-memory cache as described in the “In-Memory caching” section bellow. This will allow for frequently requested tiles to be kept in memory instead of retrieved from S3 on each call.

The following is an example OpenLayers 3 HTML/JavaScript to set up a map that fetches tiles from a pre-seeded geowebcache layer directly from S3. We’re using the typical GeoServer topp:states sample layer on a fictitious my-geowebcache-bucket bucket, using test-cache as the cache prefix, png8 tile format, and EPSG:4326 CRS.

<div class="row-fluid">
  <div class="span12">
    <div id="map" class="map"></div>
  </div>
</div>
var map = new ol.Map({
  target: 'map',
  controls: ol.control.defaults(),
  layers: [
    new ol.layer.Tile({
      source: new ol.source.XYZ({
        projection: "EPSG:4326",
        url: 'http://my-geowebcache-bucket.s3.amazonaws.com/test-cache/topp%3Astates/EPSG%3A4326/png8/default/{z}/{x}/{-y}.png'
      })
    })
  ],
  view: new ol.View({
    projection: "EPSG:4326",
    center: [-104, 39],
    zoom: 2
  })
});

Microsoft Azure Blob Store

The following documentation assumes you’re familiar with the Azure BLOB storage.

This blob store allows to configure a cache for layers on an Azure container with the following TMS-like key structure:

[prefix]/<layer id>/<gridset id>/<format id>/<parameters hash | “default”>/<z>/<x>/<y>.<extension>

  • prefix: if provided in the configuration, it will be used as the “root path” for tile keys. Otherwise the keys will be built starting at the bucket’s root.

  • layer id: the unique identifier for the layer. Note it equals to the layer name for standalone configured layers, but to the geoserver catalog’s object id for GeoServer tile layers.

  • gridset id: the name of the gridset of the tile

  • format id: the gwc internal name for the tile format. E.g.: png, png8, jpeg, etc.

  • parameters hash: if the request that originated that tiles included parameter filters, a unique hash code of the set of parameter filters, otherwise the constant default.

  • z: the z ordinate of the tile in the gridset space.

  • x: the x ordinate of the tile in the gridset space.

  • y: the y ordinate of the tile in the gridset space.

  • extension: the file extension associated to the tile format. E.g. png, jpeg, etc. (Note the extension is the same for the png and png8 formats, for example).

Configuration example:

<AzureBlobStore default="false">
  <id>myAzureCache</id>
  <enabled>false</enabled>
  <container>put-your-actual-container-name-here</container>
  <prefix>test-cache</prefix>
  <accountName>putYourActualAccountNameHere</accountName>
  <accountKey>putYourActualAccountKeyHere</accountKey>
  <maxConnections>100</maxConnections>
  <useHTTPS>true</useHTTPS>
  <serviceURL>http://putYourServerEndpointHereOrLeaveOutIfUsing.blob.core.windows.net</serviceURL>
  <proxyHost></proxyHost>
  <proxyPort></proxyPort>
  <proxyUsername></proxyUsername>
  <proxyPassword></proxyPassword>
</AzureBlobStore>

Properties:

  • container: Mandatory. The name of the Azure container where to store tiles. The code will try to create it if missing.

  • prefix: Optional. A prefix path to use as the “root folder” to store tiles at. For example, if the container is gwc.example and prefix is “mycache”, all tiles will be stored under gwc.example/mycache/{layer name} instead of gwc.example/{layer name}.

  • accountName: Mandatory. The account name used to connect to Azure storage (found in the archiving account, choose the account, and then access keys).

  • accountKey: Mandatory. The secret key the client uses to connect to S3.

  • maxConnections: Optional, default: 100. Maximum number of concurrent HTTP connections the client may use. The more the merrier, as Azure REST API does not have support for bulk deletes, so each tile needs to be deleted in a separate request on cleanup.

  • useHTTPS: Optional, default: true. Whether to use HTTPS when connecting to Azure or not.

  • serviceURL: Optional. The full service URL, in case the default one is not suitable. The default is build using the account name, e.g. https://accountName.blob.core.windows.net

  • proxyHost: Optional. The proxy host the client will connect through.

  • proxyPort: Optional. The proxy port the client will connect through.

  • proxyUsername: Optional. The proxy user name to use if connecting through a proxy.

  • proxyPassword: Optional. The proxy password to use when connecting through a proxy.

Unlike S3, access level in Azure can be set at the container level only, so if you desired to pre-seed a publicly available cache, please create a container that has “public” or “BLOB” access level. The access level can be modified also after the container creation.

Additional Information:

The Azure objects for tiles are created with public visibility to allow for “standalone” pre-seeded caches to be used directly from Azure without GeoWebCache as middleware. If

Beware of Azure services costs. Especially in terms of bandwidth usage when serving tiles out of the Azure cloud, and Azure storage prices. We haven’t conducted a thorough assessment of costs associated to seeding and serving caches. Yet we can provide some general purpose advise:

  • Do not seed at high zoom levels (except if you know what you’re doing). The number of tiles grow exponentially as the zoom level increases.

  • Use the tile format that produces the smalles possible tiles. For instance, png8 is a great compromise for quality/size. Keep in mind that the smaller the tiles the bigger the size difference between two identical caches on Azure vs a regular file system. The Azure cache takes less space because the actual space used for each tile is not padded to a file system block size.

  • Use in-memory caching. When serving Azure Blob tiles from GeoWebcache, you can greatly reduce the number of GET requests to Azure by configuring an in-memory cache as described in the “In-Memory caching” section below. This will allow for frequently requested tiles to be kept in memory instead of retrieved from Azure on each call.

The following is an example OpenLayers 3 HTML/JavaScript to set up a map that fetches tiles from a pre-seeded geowebcache layer directly from Azure, assuming that the container access level has been set to “public” or “blob”, so that direct access to the blobs is possible. We’re using the typical GeoServer topp:states sample layer on a fictitious my-geowebcache-container bucket, using test-cache as the cache prefix, png8 tile format, and EPSG:4326 CRS.

<div class="row-fluid">
  <div class="span12">
    <div id="map" class="map"></div>
  </div>
</div>
var map = new ol.Map({
  target: 'map',
  controls: ol.control.defaults(),
  layers: [
    new ol.layer.Tile({
      source: new ol.source.XYZ({
        projection: "EPSG:4326",
        url: 'https://<accountNameHere>.blob.core.windows.net/<containerNameHere>/<prefixIfAny>/<layerId>/EPSG:4326/png8/default/{z}/{x}/{-y}.png'
      })
    })
  ],
  view: new ol.View({
    projection: "EPSG:4326",
    center: [-104, 39],
    zoom: 2
  })
});

The prefix needs to be filled only if used otherwise that part of the path should be empty. The layerId is the layer identifier. In GWC it has been hand-assigned during configuration, if using GWC inside GeoServer it will be the internal layer identifier, e.g., something like LayerInfoImpl--5f036b28:16bbda57c0e:-7ffc which can be retrieved by checking the GeoServer configuration files for the layer in question.

MBTiles Blob Store

This blob store allow us to store tiles using the MBTiles specification (version 1.1) which defines a schema for storing tiles in an SQLite database with some restrictions regarding tiles formats and projections.

MBTiles specification only supports JPEG and PNG formats and projection EPSG:3857 is assumed. The implemented blob store will read and write MBTiles files compliant with the specification but will also be able to write and read MBTiles files that use others formats and projections.

Using the MBTiles blob store will bring several benefits at the cost of some performance loss. The MBTiles storage uses a significantly smaller number of files, which results in easier data handling (e.g., backups, moving tiles between environments). In some cases the stored data will be more compact reducing the size of the data on disk.

When compared to the file blob store this store has two limitations:

  • This store does not integrate with disk quota, this is a consequence of using database files.

  • This store cannot be shared among several GeoWebCache instances.

Note

If disk quota is activated the stored stats will not make much sense and will not reflect the actual disk usage, the size of the database files cannot be really controlled.

Database files cannot be managed as simple files. When connections to a database are open the associated file should not be deleted, moved or switched or the database file may become corrupted. Databases files can also become fragmented after deleting an huge amount of data or after frequent inserts, updates or delete operations.

File Path Templates

An MBTiles file will correspond to an SQLite database file. In order to limit the amount of contention on each single database file users will be allowed to decide the granularity of the databases files. When GeoWebCache needs to map a tile to a database file it will only look at the databases files paths, it will not take in account the MBTiles metadata (this is why this store is able to handle others formats and projections).

To configure the databases files granularity the user needs to provide a file path template. The default file path template for the MBTiles blob store is this one:

{layer}/{grid}{format}{params}/{z}-{x}-{y}.sqlite

This file template will stores all the tiles belonging to a certain layer in a single folder that will contain sub folders for each given format, projection and set of parameters and will group tiles with the same zoom level, column range and row range in a SQLite file. The column and row range values are passed by configuration, by default those values are equal to 250. The provided files paths templates will always be considered relative to the root directory provided as a configuration option.

Follows an example of what the blob store root directory structure may look like when using the default path template:

.
|-- nurc_Pk50095
|   `-- EPSG_4326image_pngnull
|       |-- 11_2000_1500.sqlite
|       `-- 12_4250_3000.sqlite
`-- topp_states
    |-- EPSG_900913image_jpeg7510004a12f49fdd49a2ba366e9c4594be7e4358
    |   |-- 6_250_500.sqlite
    |   `-- 7_0_0.sqlite
    `-- EPSG_900913image_jpegnull
        |-- 3_500_0.sqlite
        |-- 4_0_250.sqlite
        `-- 8_750_500.sqlite

If no parameters were provided null string will be used. Is the responsibility of the user to define a file path template that will avoid collisions.

The terms that can be used in the file path template are:

  • grid: the grid set id

  • layer: the name of the layer

  • format: the image format of the tiles

  • params: parameters unique hash

  • x: column range, computed based on the column range count configuration property

  • y: row range, computed based on the row range count configuration property

  • z: the zoom level

It is also possible to use parameters values, like style for example. If the parameter is not present null will be used.

Note

Characters \ and / can be used as path separator, they will be translated to the operating system specific one (\ for Linux and / for Windows). Any special char like \, /, : or empty space used in a term value will be substituted with an underscore.

MBTiles Metadata

A valid MBTiles file will need some metadata, the image format and layer name will be automatically added when an MBTiles file is created. The user can provide the remaining metadata using a properties file whose name must follow this pattern:

<layerName>.metadata

As an example, to add metadata description and attribution entries to layer tiger_roads a file named tiger_roads.properties with the following content should be present in the metadata directory:

description=ny_roads
attribution=geoserver

The directory that contains this metadata files is defined by a configuration property.

Vector Tile Compression

Some non-standard MBTiles files contain vector tiles, and these are sometimes compressed using gzip. A gzipVector entry to the the store configuration with a value of true will enable this behaviour. Raster tiles will not be affected.

Expiration Rules

The MBTiles specification don’t give information about when a tile was created. To allow expire rules, an auxiliary table is used to store tile creation time. In the presence of an MBTiles file generated by a third party tool it is assumed that the creation time of a tile was the first time it was accessed. This feature can be activated or deactivated by configuration. Note that this will not break the MBTiles specification compliance.

Eager Truncate

When performing a truncate of the cache the store will try to remove the whole database file avoiding to create fragmented space. This is not suitable for all the situations and is highly dependent on the database files granularity. The configuration property eagerDelete allows the user to disable or deactivate this feature which is disabled by default.

When a truncate request by tile range is received all the the databases files that contains tiles that belong to the tile range are identified. If eager delete is set to true those databases files are deleted otherwise a single delete query for each file is performed.

Configuration Example

Follows as an example the default configuration of the MBTiles store:

<MbtilesBlobStore default="true">
  <id>mbtiles-store</id>
  <enabled>true</enabled>
  <rootDirectory>/tmp/gwc-mbtiles</rootDirectory>
  <templatePath>{grid}/{layer}/{format}/{params}/{z}/tiles_{x}_{y}.sqlite</templatePath>
  <rowRangeCount>250</rowRangeCount>
  <columnRangeCount>250</columnRangeCount>
  <poolSize>1000</poolSize>
  <poolReaperIntervalMs>500</poolReaperIntervalMs>
  <eagerDelete>false</eagerDelete>
  <useCreateTime>true</useCreateTime>
  <executorConcurrency>5</executorConcurrency>
  <mbtilesMetadataDirectory>/tmp/gwc-mbtiles/layersMetadata</mbtilesMetadataDirectory>
</MbtilesBlobStore>

The rootDirectory property defines the location where all the files produced by this store will be created. The templatePath property is used to control the granularity of the database files (see section above). Properties rowRangeCount and columnRangeCount will be used by the path template to compute tile ranges.

The poolSize property allows to control the max number of open database files, when defining this property the user should take in account the number open files allowed by the operating system. The poolReaperIntervalMs property controls how often the pool size will be checked to see if some database files connections need to be closed.

Property eagerDelete controls how the truncate operation is performed (see section above). The property useCreateTime can be used to activate or deactivate the insertion of the tile creation time (see section above). Property executorConcurrency controls the parallelism used to perform certain operations, like the truncate operation for example. Property mbtilesMetadataDirectory defines the directory where the store will look for user provided MBTiles metadata.

Note

Since the connection pool eviction happens at a certain interval, it means that the number of files open concurrently can go above the threshold limit for a certain amount of time.

Replace Operation

As said before, if the cache is running SQLite files cannot be simply switched, first all connections need to be closed. The replace operation was created for this propose. The replace operation will first copy the new file side by side the old one, then block the requests to the old file, close the connections tot he old file, delete the old one, rename the new file to current one, reopen the new db file and start serving requests again. Should be almost instant.

A REST entry point for this operation is available, it will be possible to submit a ZIP file or a single file along with the request. The replace operation can also use an already present file or directory. When using a directory the directory structure will be used to find the destination of each file, all the paths will be assumed to be relative to the store root directory. This means that is possible to replace a store content with another store content (a seeded one for example) by zipping the second store root directory and send it as a replacement.

Note

When using a local directory or submitting a zip file all the file present in the directory will be considered.

There is four ways to invoke this operation. Follows an example of all those variants invocations using CURL.

Replace a single file uploading the replacement file:

curl -u geowebcache:secured -H 'Content-Type: multipart/form-data'
  -F "file=@tiles_0_0.sqlite"
  -F "destination=EPSG_4326/sf_restricted/image_png/null/10/tiles_0_0.sqlite"
  -F "layer=sf:restricted"
  -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace'

Replace a single file using a file already present on the system:

curl -u geowebcache:secured -H 'Content-Type: multipart/form-data'
  -F "source=/tmp/tiles_0_0.sqlite"
  -F "destination=EPSG_4326/sf_restricted/image_png/null/10/tiles_0_0.sqlite"
  -F "layer=sf:restricted"
  -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace'

Replace multiple files uploading a ZIP file:

curl -u geowebcache:secured -H 'Content-Type: multipart/form-data'
  -F "file=@tiles.zip"
  -F "layer=sf:restricted"
  -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace'

Replace multiple files using a directory already present on the system:

curl -u geowebcache:secured -H 'Content-Type: multipart/form-data'
  -F "source=/tmp/tiles"
  -F "layer=sf:restricted"
  -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace'

The layer parameter identifies the layer whose associated blob store content should be replaced. The file parameter is used to upload a single file or a ZIP file. The source parameter is used to reference an already present file or directory. The destination parameter is used to define the file that should be replaced with the provided file.

This are the only valid combinations of this parameters other combinations will ignore some of the provided parameters or will throw an exception.

In-Memory caching

Default blobstore can be changed with a new one called MemoryBlobStore, which allows in memory tile caching. The MemoryBlobStore is a wrapper of a blobstore implementation, which can be the default one(FileBlobStore) or another one. For using the new blobstore implementation, the user have to modify the blobstore bean associated to the gwcStorageBroker bean (inside the Application Context file geowebcache-core-context.xml) by setting gwcMemoryBlobStore instead of gwcBlobStore.

The configuration of a MemoryBlobStore requires a blobstore to wrap and a CacheProvider object. This one provides the caching mechanism for saving input data in memory. User can define different caching objects but can only inject one of them inside the MemoryBlobStore. More information about the CacheProvider can be found in the next section.

An example of MemoryBlobStore configuration can be found beow:

<bean id="gwcMemoryBlobStore" class="org.geowebcache.storage.blobstore.memory.MemoryBlobStore" destroy-method="destroy">
  <property name="store" ref="gwcBlobStore" />
      <!-- "cacheProviderName" is optional. It is the name of the bean associated to the cacheProvider object used by this MemoryBlobStore-->
  <property name="cacheBeanName" value="cacheProviderName" />
      <!-- "cacheProvider" is optional. It is the Reference to a CacheProvider bean in the application context. -->
      <property name="cacheProvider" ref="ExampleCacheProvider" />
</bean>

Note

Note that cacheProviderName/cacheProvider cannote be used together, if a cacheProvider is defined, the cacheProviderName is not considered. If cacheProviderName/cacheProvider are not defined, the MemoryBlobStore will internally search for a suitable CacheProvider.

CacheProvider configuration

A CacheProvider object should be configured with an input object called CacheConfiguration. CacheConfiguration parameters are:

  • hardMemoryLimit : which is the cache size in Mb

  • policy : which can be LRU, LFU, EXPIRE_AFTER_WRITE, EXPIRE_AFTER_ACCESS, NULL

  • evitionTime : which is the cache eviction time in seconds

  • concurrencyLevel : which is the cache concurrency level

These parameters must be defined as properties in the cacheConfiguration bean in the Spring Application Context (like geowebcache-core-context.xml).

At the time of writing there are two implementations of the CacheProvider interface:

  • GuavaCacheProvider

  • HazelcastCacheProvider

GuavaCacheProvider

GuavaCacheProvider provides local in-memory caching by using a Guava Cache for storing the various GWC Tiles locally on the machine. For configuring a GuavaCacheProvider the user must create a new bean in the Application Context file (like geowebcache-core-context.xml) and then add a reference to a CacheConfiguration instance.

Here is an example of configuration:

<bean id="cacheConfiguration" class="org.geowebcache.storage.blobstore.memory.CacheConfiguration">
  <property name="hardMemoryLimit" value="64"/> <!-- 64 Mb -->
      <property name="policy" value="EXPIRE_AFTER_ACCESS"/> <!-- Cache Eviction Policy is EXPIRE_AFTER_ACCESS. Other values are EXPIRE_AFTER_WRITE, NULL(LRU eviction based on cache size) -->
      <property name="evitionTime" value="240"/> <!-- Eviction time is 240 seconds -->
      <property name="concurrencyLevel" value="4"/> <!-- Concurrency Level of the cache is 4 -->
</bean>

<bean id="guavaCacheProvider" class="org.geowebcache.storage.blobstore.memory.guava.GuavaCacheProvider">
  <property name="configuration" ref="cacheConfiguration"/> <!-- Setting of the configuration -->
</bean>

HazelcastCacheProvider

HazelcastCacheProvider is useful for implementing distributed in memory caching for clustering. It internally uses Hazelcast for handling distributed caching. The HazelcastCacheProvider configuration requires another object called HazelcastLoader. This object accepts an Hazelcast instance or loads a file called hazelcast.xml from a proper directory defined by the property “hazelcast.config.dir”. If none of them is present, the CacheProvider object cannot be used.

The user must follow these rules for configuring the Hazelcast instance:

  1. The Hazelcast configuration requires a Map object with name CacheProviderMap

  2. Map eviction policy must be LRU or LFU

  3. Map configuration must have a fixed size defined in Mb

  4. Map configuration must have USED_HEAP_SIZE as MaxSizePolicy

Here the user can find both examples:

  • From hazelcast.xml:

    <hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-2.3.xsd"
                       xmlns="http://www.hazelcast.com/schema/config"
                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <group>
            <name>cacheCluster</name>
            <password>geoserverCache</password>
      </group>
    
      <network>
            <!--
                    Typical usage: multicast enabled with port auto-increment enabled
                    or tcp-ip enabled with port auto-increment disabled. Note that you
                    must choose between multicast and tcp-ip. Another option could be
                    aws, but will not be described here.
    
            -->
            <port auto-increment="false">5701</port>
                    <join>
                             <multicast enabled="false">
                                    <multicast-group>224.2.2.3</multicast-group>
                                    <multicast-port>54327</multicast-port>
                            </multicast>
                            <tcp-ip enabled="true">
                                    <interface>192.168.1.32</interface>
                                    <interface>192.168.1.110</interface>
                            </tcp-ip>
                    </join>
      </network>
      <map name="CacheProviderMap">
                    <eviction-policy>LRU</eviction-policy>
                    <max-size policy="USED_HEAP_SIZE">16</max-size>
      </map>
    
    </hazelcast>
    

    And the related application context will be:

    <bean id="HazelCastLoader1"
            class="org.geowebcache.storage.blobstore.memory.distributed.HazelcastLoader">
    </bean>
    
    <bean id="HazelCastCacheProvider1"
            class="org.geowebcache.storage.blobstore.memory.distributed.HazelcastCacheProvider">
            <constructor-arg ref="HazelCastLoader1" />
    </bean>
    

    Note

    Remember that in this case the user must define the hazelcast.config.dir property when starting the application.

  • From application context (See Hazelcast documentation for more info):

    <hz:hazelcast id="instance1">
            <hz:config>
                    <hz:group name="dev" password="password" />
                    <hz:network port="5701" port-auto-increment="true">
                            <hz:join>
                                    <hz:multicast enabled="true" multicast-group="224.2.2.3"
                                            multicast-port="54327" />
                            <hz:tcp-ip enabled="false">
                              <hz:members>10.10.1.2, 10.10.1.3</hz:members>
                            </hz:tcp-ip>
                            </hz:join>
                    </hz:network>
                    <hz:map name="CacheProviderMap" max-size="16" eviction-policy="LRU"
                            max-size-policy="USED_HEAP_SIZE" />
            </hz:config>
    </hz:hazelcast>
    
    <bean id="HazelCastLoader1"
            class="org.geowebcache.storage.blobstore.memory.distributed.HazelcastLoader">
            <property name="instance" ref="instance1" />
    </bean>
    
    <bean id="HazelCastCacheProvider1"
            class="org.geowebcache.storage.blobstore.memory.distributed.HazelcastCacheProvider">
            <constructor-arg ref="HazelCastLoader1" />
    </bean>
    

Optional configuration parameters

In this section are described other available configuration parameters to configure:

  • Cache expiration time:

    <map name="CacheProviderMap">
    ...
    
        <time-to-live-seconds>0</time-to-live-seconds>
        <max-idle-seconds>0</max-idle-seconds>
    
    </map>
    

    Where time-to-live-seconds indicates how many seconds an entry can stay in cache and max-idle-seconds indicates how many seconds an entry may be not accessed before being evicted.

  • Near Cache.

    <map name="CacheProviderMap">
    ...
    <near-cache>
      <!--
        Same configuration parameters of the Hazelcast Map. Note that size indicates the maximum number of
        entries in the near cache. A value of Integer.MAX_VALUE indicates no limit on the maximum
        size.
      -->
      <max-size>5000</max-size>
      <time-to-live-seconds>0</time-to-live-seconds>
      <max-idle-seconds>60</max-idle-seconds>
      <eviction-policy>LRU</eviction-policy>
    
      <!--
        Indicates if a cached entry can be evicted if the same value is modified in the Hazelcast Map. Default is true.
      -->
      <invalidate-on-change>true</invalidate-on-change>
    
      <!--
        Indicates if local entries must be cached. Default is false.
      -->
      <cache-local-entries>false</cache-local-entries>
    </near-cache>
    
    </map>
    

    Near Cache is a local cache for each cluster instance which is used for caching entries in the other cluster instances. This behaviour avoids to request those entries each time by executing a remote call. This feature could be helpful in order to improve Hazelcast Cache performances.

    Note

    A value of max-size bigger or equal to Integer.MAX_VALUE cannot be used in order to avoid an uncontrollable growth of the cache size.

OpenStack Swift (Swift) Blob Store

The following documentation assumes you’re familiar with the Openstack Swift Documentation.

This blob store allows to configure a cache for layers using a Swift container with the following TMS-like key structure:

[prefix]/<layer id>/<gridset id>/<format id>/<parameters hash | “default”>/<z>/<x>/<y>.<extension>

  • prefix: if provided in the configuration, it will be used as the “root path” for tile keys. Otherwise the keys will be built starting at the bucket’s root.

  • layer id: the unique identifier for the layer. Note it equals to the layer name for standalone configured layers, but to the geoserver catalog’s object id for GeoServer tile layers.

  • gridset id: the name of the gridset of the tile

  • format id: the gwc internal name for the tile format. E.g.: png, png8, jpeg, etc.

  • parameters hash: if the request that originated that tiles included parameter filters, a unique hash code of the set of parameter filters, otherwise the constant default.

  • z: the z ordinate of the tile in the gridset space.

  • x: the x ordinate of the tile in the gridset space.

  • y: the y ordinate of the tile in the gridset space.

  • extension: the file extension associated to the tile format. E.g. png, jpeg, etc. (Note the extension is the same for the png and png8 formats, for example).

Configuration example:

<SwiftBlobStore default="true">
    <id>ObjectStorageCache</id>
    <enabled>true</enabled>
    <container>put-your-actual-container-name-here</container>
    <prefix>test-cache</prefix>
    <endpoint>endpoint</endpoint>
    <provider>openstack-swift</provider>
    <region>put-region-here</region>
    <keystoneVersion>3</keystoneVersion>
    <keystoneScope>project</keystoneScope>
    <keystoneDomainName>Default</keystoneDomainName>
    <identity>put-tenant-name-here:put-username-here</identity>
    <password>put-password-here</password>
</SwiftBlobStore>

Properties:

  • container: Mandatory. The name of the Swift container where to store tiles.

  • prefix: Optional. A prefix path to use as the “root folder” to store tiles at. For example, if the bucket is bucket.gwc.example and prefix is “mycache”, all tiles will be stored under bucket.gwc.example/mycache/{layer name} instead of bucket.gwc.example/{layer name}.

  • endpoint: Manditory. Endpoint of the server

  • provider: Mandatory. Jclouds provider (shouldn’t need modifying)

  • region: Mandatory. Swift region for container.

  • keystoneVersion: Mandatory. Keystone version

  • keystoneScope: Optional. For scoped keystone authorization (project or domain scoped)

  • keystoneDomainName: Optional. Keystone domain name (if different than the user domain)

  • identity: Mandatory. Identity used to authenticate with the swift API (format - tenantName:username)

  • password: Mandatory. Password used to authenticate with the swift API.

Additional Information:

Some links that might be useful: