ConsistencyResampling

Consistency resampling of calibrated predictions

Overview

This Julia package contains an implementation of consistency resampling.[BS07]

Consistency resampling is a resampling technique that generates a calibrated set of predictions and corresponding targets from a set of predictions. The procedure consists of two steps:

  • predictions are resampled with replacement
  • synthetic targets are sampled according to the resampled predictions.

This resampling procedure ensures that the predictions are calibrated for the artificial targets.

Usage

The API consists only of Consistent.

ConsistencyResampling.Consistent — Type
Consistent(predictions::AbstractVector)

Create an object that can be used for resampling predictions and corresponding targets.

The set of predictions has to be provided as a vector consisting of

  • probabilities,
  • vectors of probabilities summing up to 1,
  • or distributions that can be sampled from.

Consistency resampling can be performed with rand and rand!, as described in the documentation of Random.

Examples

julia> predictions = rand(100);

julia> consistent = Consistent(predictions);

julia> rand(consistent) isa Tuple{Float64,Bool}
true

julia> first(rand(consistent)) in predictions
true
Note

One can use Random.Sampler to create a sampler that is optimized for the generation of multiple samples, e.g., by building an alias table:

julia> using Random: Sampler, GLOBAL_RNG

julia> sampler = Sampler(GLOBAL_RNG, Consistent(rand(100)));

julia> rand(sampler) isa Tuple{Float64,Bool}
true

However, for small number of samples the setup cost can outweigh the improved sampling efficiency.

Note

Multiple samples are returned as an array of tuples of predictions and corresponding targets. If you prefer a tuple of arrays of predictions and targets instead, you can use a package such as StructArrays and perform in-place sampling:

julia> using StructArrays, Random

julia> predictions = Vector{Float64}(undef, 100);

julia> targets = Vector{Bool}(undef, 100);

julia> rand!(StructVector((predictions, targets)), Consistent(rand(100)));

References

Bröcker, J. and Smith, L.A., 2007. Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), pp. 651-661.

source