Package org.openstreetmap.atlas.generator.tools.caching

Class HadoopAtlasFileCache

  • All Implemented Interfaces:
    org.openstreetmap.atlas.utilities.caching.ResourceCache
    public class HadoopAtlasFileCache
extends org.openstreetmap.atlas.utilities.caching.ConcurrentResourceCache
    Cache an atlas file stored in the standard way (parentpath/COUNTRY/COUNTRY_z-x-y.atlas) to a system temporary location. This cache is designed to be used with atlases that can be fetched from a hadoop file system. This cache is threadsafe. You can optionally associate a namespace with a HadoopAtlasFileCache at creation time. This will create a unique container per-namespace for cached content.

    For example, consider the following code:

    ResourceCache cache1 = new HadoopAtlasFileCache(parentPath, "namespace1", config);
    ResourceCache cache2 = new HadoopAtlasFileCache(parentPath, "namespace2", config);

    // We will be fetching resource behind URI "parentPath/AAA/AAA_1-1-1.atlas"
    Resource r1 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();
    // Assume some event changes the contents behind the URI between get() calls
    Resource r2 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();

    // This fails since the caches have different namespaces and the resource contents changed
    Assert.assertEquals(r1.all(), r2.all());

    // Now we invalidate cache1's copy of the resource
    cache1.invalidate(getURIForResource(r1));
    // This call to cache1.get() will re-fetch since we invalidated
    r1 = cache1.get("AAA", new SlippyTile(1, 1, 1)).get();

    // Now this passes since cache1 was refreshed
    Assert.assertEquals(r1.all(), r2.all());

    The key takeaway here is that r1 and r2 are not guaranteed to have the same contents, even though their URIs are the same. Since the two cache instances have different namespaces, their underlying stores do not intersect. It is also worth noting that the HadoopAtlasFileCache cannot detect if the resource behind a URI has changed. You must call ConcurrentResourceCache.invalidate() or ConcurrentResourceCache.invalidate(URI) to force an update.

    • Constructor Summary

      Constructors 
      Constructor Description
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, java.lang.String namespace, java.util.Map<java.lang.String,​java.lang.String> configuration)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, java.lang.String namespace, SlippyTilePersistenceScheme atlasScheme, java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, java.lang.String namespace, SlippyTilePersistenceScheme atlasScheme, java.util.Map<java.lang.String,​java.lang.String> configuration)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, java.util.Map<java.lang.String,​java.lang.String> configuration)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, SlippyTilePersistenceScheme atlasScheme, java.util.Map<java.lang.String,​java.lang.String> configuration)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, SlippyTilePersistenceScheme atlasScheme, org.openstreetmap.atlas.utilities.caching.strategies.CachingStrategy cachingStrategy, java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
      Create a new cache.
      HadoopAtlasFileCache​(java.lang.String parentAtlasPath, org.openstreetmap.atlas.utilities.caching.strategies.CachingStrategy cachingStrategy, java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
      Create a new cache.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> get​(java.lang.String country, org.openstreetmap.atlas.geography.sharding.Shard shard)
      Get an Optional of an atlas resource specified by the given parameters.
      void invalidate​(java.lang.String country, org.openstreetmap.atlas.geography.sharding.Shard shard)
      Invalidate a given shard for a given country.

      • Methods inherited from class org.openstreetmap.atlas.utilities.caching.ConcurrentResourceCache

        get, getCacheID, getStrategyName, invalidate, invalidate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.openstreetmap.atlas.utilities.caching.ResourceCache

        get

    • Constructor Detail

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            java.util.Map<java.lang.String,​java.lang.String> configuration)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        configuration - The configuration map

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        fetcher - Function to fetch an atlas resource in case of cache miss

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            org.openstreetmap.atlas.utilities.caching.strategies.CachingStrategy cachingStrategy,
                            java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        cachingStrategy - Strategy to cache an atlas resource
        fetcher - Function to fetch an atlas resource in case of cache miss

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            SlippyTilePersistenceScheme atlasScheme,
                            java.util.Map<java.lang.String,​java.lang.String> configuration)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        atlasScheme - The scheme used to locate atlas files based on slippy tiles
        configuration - The configuration map

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            java.lang.String namespace,
                            java.util.Map<java.lang.String,​java.lang.String> configuration)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        namespace - The namespace for this cache's resources
        configuration - The configuration map

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            java.lang.String namespace,
                            SlippyTilePersistenceScheme atlasScheme,
                            java.util.Map<java.lang.String,​java.lang.String> configuration)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        namespace - The namespace for this cache's resoures
        atlasScheme - The scheme used to locate atlas files based on slippy tiles
        configuration - The configuration map

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            java.lang.String namespace,
                            SlippyTilePersistenceScheme atlasScheme,
                            java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        namespace - The namespace for this cache's resoures
        atlasScheme - The scheme used to locate atlas files based on slippy tiles
        fetcher - Function to fetch an atlas resource in case of cache miss

      • HadoopAtlasFileCache

        public HadoopAtlasFileCache​(java.lang.String parentAtlasPath,
                            SlippyTilePersistenceScheme atlasScheme,
                            org.openstreetmap.atlas.utilities.caching.strategies.CachingStrategy cachingStrategy,
                            java.util.function.Function<java.net.URI,​java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>> fetcher)
        Create a new cache.
        Parameters:
        parentAtlasPath - The parent path to the atlas files. This might look like hdfs://some/path/to/files
        atlasScheme - The scheme used to locate atlas files based on slippy tiles
        cachingStrategy - Strategy to cache an atlas resource
        fetcher - Function to fetch an atlas resource in case of cache miss

    • Method Detail

      • get

        public java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> get​(java.lang.String country,
                                                                                   org.openstreetmap.atlas.geography.sharding.Shard shard)
        Get an Optional of an atlas resource specified by the given parameters.
        Parameters:
        country - The ISO country code of the desired shard
        shard - The Shard object representing the shard
        Returns:
        an Optional wrapping the shard

      • invalidate

        public void invalidate​(java.lang.String country,
                       org.openstreetmap.atlas.geography.sharding.Shard shard)
        Invalidate a given shard for a given country. It is highly recommended to use this implementation over ConcurrentResourceCache.invalidate(URI).
        Parameters:
        country - the country
        shard - the shard to invalidate