Class S100ForecastFeedIndexer

Namespace

EncDotNet.S100.Collections.Indexing

Assembly

EncDotNet.S100.Collections.dll

Indexes an S100ForecastFeedSource (#685) — for example NOAA's S-111 surface currents on AWS — by reading each chosen model's catalogue, which lists only its latest run.

public sealed class S100ForecastFeedIndexer : ICollectionSourceIndexer

Inheritance

object

S100ForecastFeedIndexer

Implements

ICollectionSourceIndexer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Each run is stamped on its items: RunProperty, ValidToProperty (run time plus the model's horizon) and, as LastModified, the run time — runs have no editions, so a downloaded run is outdated once the catalogue lists a later one (IsOlderThan(CollectionItem)). Items keep their identity across runs (the run time is dropped from their names), so a new run's download replaces the previous one.

Catalogues are cached like S100CatalogueFeedIndexer's; a model whose catalogue cannot be read is reported and skipped. When the feed is in an S3 bucket, each run's folder is listed for sizes.

Constructors

S100ForecastFeedIndexer(HttpClient, string, FeedCacheOptions?, TimeProvider?)

Creates an indexer.

public S100ForecastFeedIndexer(HttpClient httpClient, string cacheDirectory, FeedCacheOptions? options = null, TimeProvider? timeProvider = null)

Parameters

httpClient HttpClient

The client used to fetch catalogues and list buckets.

cacheDirectory string

Where raw catalogues are cached.

options FeedCacheOptions

Cache options; defaults to DefaultCacheOptions.

timeProvider TimeProvider

The clock; defaults to System.

Fields

ModelProperty

The item property naming its model (e.g. cbofs).

public const string ModelProperty = "model"

Field Value

string

RunProperty

The item property holding its run time (ISO 8601, UTC).

public const string RunProperty = "run"

Field Value

string

ValidToProperty

The item property holding the end of its run's valid window (ISO 8601, UTC).

public const string ValidToProperty = "validTo"

Field Value

string

Properties

DefaultCacheOptions

The default revalidation interval: one minute. A model's catalogue is overwritten with every run, so a Refresh ("Check for new runs") must actually ask; an unchanged catalogue answers with a cheap 304. Hosts refresh only on start and on request, never in the background.

public static FeedCacheOptions DefaultCacheOptions { get; }

Property Value

FeedCacheOptions

Methods

CanIndex(CollectionSource)

Returns true when this indexer handles source.

public bool CanIndex(CollectionSource source)

Parameters

source CollectionSource

Returns

bool

CatalogUri(Uri, ForecastModel)

The catalogue of model under modelsUri.

public static Uri CatalogUri(Uri modelsUri, ForecastModel model)

Parameters

modelsUri Uri

model ForecastModel

Returns

Uri

DownloadFolderFor(Uri)

The managed folder, relative to the downloads root, that a feed's runs download into: one per feed, e.g. forecasts/noaa-s111-pds.s3.amazonaws.com-3f2a9c1b.

public static string DownloadFolderFor(Uri modelsUri)

Parameters

modelsUri Uri

Returns

string

GetFingerprintAsync(CollectionSource, CancellationToken)

Cheaply computes a fingerprint of the source's current state. When it equals a previous index's Fingerprint, that index is still valid. Returns null when the state cannot be determined (for example the source is unavailable).

public ValueTask<string?> GetFingerprintAsync(CollectionSource source, CancellationToken cancellationToken)

Parameters

source CollectionSource

cancellationToken CancellationToken

Returns

ValueTask<string>

GetModelsAsync(Uri, IReadOnlyList<ForecastModel>, CancellationToken)

Summarises the latest run of each of models under modelsUri (run time, tiles, sizes and domain), for choosing models; a model that cannot be read carries its error.

public Task<IReadOnlyList<ForecastModelSummary>> GetModelsAsync(Uri modelsUri, IReadOnlyList<ForecastModel> models, CancellationToken cancellationToken = default)

Parameters

modelsUri Uri

models IReadOnlyList<ForecastModel>

cancellationToken CancellationToken

Returns

Task<IReadOnlyList<ForecastModelSummary>>

HealthOf(S100ForecastFeedSource)

How the last attempt to reach feed's catalogues went: the first failing model's, else the most recent check; null before the first attempt.

public FeedHealth? HealthOf(S100ForecastFeedSource feed)

Parameters

feed S100ForecastFeedSource

Returns

FeedHealth

IndexAsync(CollectionSource, IProgress<IndexProgress>?, CancellationToken)

Indexes source.

public ValueTask<SourceIndex> IndexAsync(CollectionSource source, IProgress<IndexProgress>? progress, CancellationToken cancellationToken)

Parameters

source CollectionSource

progress IProgress<IndexProgress>

cancellationToken CancellationToken

Returns

ValueTask<SourceIndex>

RunOf(CollectionItem)

The run time of a forecast item, from RunProperty.

public static DateTimeOffset? RunOf(CollectionItem item)

Parameters

item CollectionItem

Returns

DateTimeOffset?

ValidToOf(CollectionItem)

The end of a forecast item's valid window, from ValidToProperty.

public static DateTimeOffset? ValidToOf(CollectionItem item)

Parameters

item CollectionItem

Returns

DateTimeOffset?