Class S100CatalogueFeedIndexer

Namespace

EncDotNet.S100.Collections.Indexing

Assembly

EncDotNet.S100.Collections.dll

Indexes an S100CatalogueFeedSource (issue #685): fetches (or revalidates) a remote S-100 exchange catalogue — such as NOAA's S-102 bathymetry on AWS Open Data — and lists the datasets in the selected folders as online items, each downloading as its own file.

public sealed class S100CatalogueFeedIndexer : ICollectionSourceIndexer

Inheritance

object

S100CatalogueFeedIndexer

Implements

ICollectionSourceIndexer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Caching follows NoaaEncFeedIndexer: one raw copy per URL, revalidated with conditional requests, served from cache when offline. Gzip-encoded catalogues are kept compressed.

S-100 catalogues carry no file sizes. When the catalogue is in an Amazon S3 bucket, the selected folders are listed (one ListObjectsV2 request per thousand files) for sizes and dates; elsewhere sizes stay unknown. Folders become the index's Groups.

Constructors

S100CatalogueFeedIndexer(HttpClient, string, FeedCacheOptions?, TimeProvider?)

Creates an indexer.

public S100CatalogueFeedIndexer(HttpClient httpClient, string cacheDirectory, FeedCacheOptions? options = null, TimeProvider? timeProvider = null)

Parameters

httpClient HttpClient

The client used to fetch catalogues and list buckets.

cacheDirectory string

Where raw catalogues are cached.

options FeedCacheOptions

Cache options; defaults to Default.

timeProvider TimeProvider

The clock; defaults to System.

Methods

CanIndex(CollectionSource)

Returns true when this indexer handles source.

public bool CanIndex(CollectionSource source)

Parameters

source CollectionSource

Returns

bool

DownloadFolderFor(Uri)

The managed folder, relative to the downloads root, that datasets of the catalogue at catalogUri download into: one per catalogue, e.g. catalogues/noaa-s102-pds.s3.amazonaws.com-3f2a9c1b.

public static string DownloadFolderFor(Uri catalogUri)

Parameters

catalogUri Uri

Returns

string

GetCatalogueAsync(Uri, bool, CancellationToken)

Returns the (cached or freshly fetched) catalogue at catalogUri, unfiltered and without sizes, for example to choose folders.

public Task<RemoteS100Catalogue> GetCatalogueAsync(Uri catalogUri, bool forceRevalidate = false, CancellationToken cancellationToken = default)

Parameters

catalogUri Uri

forceRevalidate bool

cancellationToken CancellationToken

Returns

Task<RemoteS100Catalogue>

Exceptions

HttpRequestException

The server could not be reached and nothing is cached.

XmlException

The document is not an S-100 exchange catalogue.

GetFingerprintAsync(CollectionSource, CancellationToken)

Cheaply computes a fingerprint of the source's current state. When it equals a previous index's Fingerprint, that index is still valid. Returns null when the state cannot be determined (for example the source is unavailable).

public ValueTask<string?> GetFingerprintAsync(CollectionSource source, CancellationToken cancellationToken)

Parameters

source CollectionSource

cancellationToken CancellationToken

Returns

ValueTask<string>

HealthOf(Uri)

How the last attempt to fetch the catalogue at catalogUri went; null before the first attempt.

public FeedHealth? HealthOf(Uri catalogUri)

Parameters

catalogUri Uri

Returns

FeedHealth

IndexAsync(CollectionSource, IProgress<IndexProgress>?, CancellationToken)

Indexes source.

public ValueTask<SourceIndex> IndexAsync(CollectionSource source, IProgress<IndexProgress>? progress, CancellationToken cancellationToken)

Parameters

source CollectionSource

progress IProgress<IndexProgress>

cancellationToken CancellationToken

Returns

ValueTask<SourceIndex>

ListAsync(RemoteS100Catalogue, IEnumerable<string>, CancellationToken)

Lists the files under folders of catalogue (folders relative to its root; empty for the whole set), keyed by URL — or returns null when the catalogue is not in an S3 bucket, so sizes cannot be listed. Listings are reused for a few minutes.

public Task<IReadOnlyDictionary<Uri, S3Object>?> ListAsync(RemoteS100Catalogue catalogue, IEnumerable<string> folders, CancellationToken cancellationToken = default)

Parameters

catalogue RemoteS100Catalogue

folders IEnumerable<string>

cancellationToken CancellationToken

Returns

Task<IReadOnlyDictionary<Uri, S3Object>>

Exceptions

HttpRequestException

The bucket could not be listed.