org.openstreetmap.atlas.generator.tools.caching

Class HadoopAtlasFileCache

java.lang.Object

org.openstreetmap.atlas.utilities.caching.ConcurrentResourceCache

org.openstreetmap.atlas.generator.tools.caching.HadoopAtlasFileCache

All Implemented Interfaces:

org.openstreetmap.atlas.utilities.caching.ResourceCache

public class HadoopAtlasFileCache
extends org.openstreetmap.atlas.utilities.caching.ConcurrentResourceCache

Cache an atlas file stored in the standard way (parentpath/COUNTRY/COUNTRY_z-x-y.atlas) to a system temporary location. This cache is designed to be used with atlases that can be fetched from a hadoop file system. This cache is threadsafe. You can optionally associate a namespace with a HadoopAtlasFileCache at creation time. This will create a unique container per-namespace for cached content.

For example, consider the following code:

ResourceCache cache1 = new HadoopAtlasFileCache(parentPath, "namespace1", config);
ResourceCache cache2 = new HadoopAtlasFileCache(parentPath, "namespace2", config);

// We will be fetching resource behind URI "parentPath/AAA/AAA_1-1-1.atlas"
Resource r1 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();
// Assume some event changes the contents behind the URI between get() calls
Resource r2 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();

// This fails since the caches have different namespaces and the resource contents changed
Assert.assertEquals(r1.all(), r2.all());

// Now we invalidate cache1's copy of the resource
cache1.invalidate(getURIForResource(r1));
// This call to cache1.get() will re-fetch since we invalidated
r1 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();

// Now this passes since cache1 was refreshed
Assert.assertEquals(r1.all(), r2.all());


The key takeaway here is that r1 and r2 are not guaranteed to have the same contents, even though their URIs are the same. Since the two cache instances have different namespaces, their underlying stores do not intersect. It is also worth noting that the HadoopAtlasFileCache cannot detect if the resource behind a URI has changed. You must call ConcurrentResourceCache.invalidate() or ConcurrentResourceCache.invalidate(URI) to force an update.

Constructor Summary

Constructors

Constructor and Description
HadoopAtlasFileCache(java.lang.String parentAtlasPath, java.util.Map<java.lang.String,java.lang.String> configuration) Create a new cache.
HadoopAtlasFileCache(java.lang.String parentAtlasPath, SlippyTilePersistenceScheme atlasScheme, java.util.Map<java.lang.String,java.lang.String> configuration) Create a new cache.
HadoopAtlasFileCache(java.lang.String parentAtlasPath, java.lang.String namespace, java.util.Map<java.lang.String,java.lang.String> configuration) Create a new cache.
HadoopAtlasFileCache(java.lang.String parentAtlasPath, java.lang.String namespace, SlippyTilePersistenceScheme atlasScheme, java.util.Map<java.lang.String,java.lang.String> configuration) Create a new cache.

Method Summary

Modifier and Type	Method and Description
java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>	get(java.lang.String country, org.openstreetmap.atlas.geography.sharding.Shard shard) Get an Optional of an atlas resource specified by the given parameters.
void	invalidate(java.lang.String country, org.openstreetmap.atlas.geography.sharding.Shard shard) Invalidate a given shard for a given country.

Methods inherited from class org.openstreetmap.atlas.utilities.caching.ConcurrentResourceCache

get, getCacheID, getStrategyName, invalidate, invalidate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.openstreetmap.atlas.utilities.caching.ResourceCache

get

Constructor Detail

HadoopAtlasFileCache

public HadoopAtlasFileCache(java.lang.String parentAtlasPath,
                            java.util.Map<java.lang.String,java.lang.String> configuration)

Create a new cache.

Parameters:

parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files

configuration - The configuration map

HadoopAtlasFileCache

public HadoopAtlasFileCache(java.lang.String parentAtlasPath,
                            SlippyTilePersistenceScheme atlasScheme,
                            java.util.Map<java.lang.String,java.lang.String> configuration)

Create a new cache.

Parameters:

parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files

atlasScheme - The scheme used to locate atlas files based on slippy tiles

configuration - The configuration map

HadoopAtlasFileCache

public HadoopAtlasFileCache(java.lang.String parentAtlasPath,
                            java.lang.String namespace,
                            java.util.Map<java.lang.String,java.lang.String> configuration)

Create a new cache.

Parameters:

parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files

namespace - The namespace for this cache's resoures

configuration - The configuration map

HadoopAtlasFileCache

public HadoopAtlasFileCache(java.lang.String parentAtlasPath,
                            java.lang.String namespace,
                            SlippyTilePersistenceScheme atlasScheme,
                            java.util.Map<java.lang.String,java.lang.String> configuration)

Create a new cache.

Parameters:

parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files

namespace - The namespace for this cache's resoures

atlasScheme - The scheme used to locate atlas files based on slippy tiles

configuration - The configuration map

Method Detail

get

public java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> get(java.lang.String country,
                                                                                   org.openstreetmap.atlas.geography.sharding.Shard shard)

Get an Optional of an atlas resource specified by the given parameters.

Parameters:

country - The ISO country code of the desired shard

shard - The Shard object representing the shard

Returns:

an Optional wrapping the shard

invalidate

public void invalidate(java.lang.String country,
                       org.openstreetmap.atlas.geography.sharding.Shard shard)

Invalidate a given shard for a given country. It is highly recommended to use this implementation of invalidate(String, Shard) over ConcurrentResourceCache.invalidate(URI).

Parameters:

country - the country

shard - the shard to invalidate