API

reliability(probabilities, frequencies; deviation=true, kwargs...)
reliability(probabilities, frequencies, low_high; deviation=true, kwargs...)
reliability(probabilities, frequencies, low, high; deviation=true, kwargs...)

Plot a reliability diagram of the observed frequencies versus the predicted probabilities, optionally with consistency bars ranging from low to high for each probability.

If deviation is true (default), the difference frequencies - probabilities is plotted versus probabilities. This can be helpful for inspecting how closely the observed frequencies match the predicted probabilities.

Attributes

Available attributes and their defaults for Combined{ReliabilityDiagrams.reliability} are:

  barscolor         :black
  barscolormap      :viridis
  barscolorrange    MakieCore.Automatic()
  barslinewidth     1.5
  barsvisible       true
  barswhiskerwidth  0
  color             :black
  colormap          :viridis
  colorrange        MakieCore.Automatic()
  cycle             [:color]
  deviation         true
  inspectable       true
  linestyle         "nothing"
  linewidth         1.5
  marker            :circle
  markercolor       :black
  markercolormap    :viridis
  markercolorrange  MakieCore.Automatic()
  markersize        12
  markervisible     true
  strokecolor       :black
  strokewidth       0
  visible           true

General Axis Keywords

• xlabel: Label of x axis (default: confidence)
• ylabel: Label of y axis (default: if deviation == true, then empirical deviation, otherwise empirical frequency)

Examples

julia> probabilities = sort(rand(10));

julia> frequencies = rand(10);

julia> reliability(probabilities, frequencies);

julia> # plot frequencies instead of deviations
       reliability(probabilities, frequencies; deviation=false);

julia> # use different colors for markers
       reliability(probabilities, frequencies; markercolor=:red);

julia> # random consistency bars
       low = max.(0, probabilities .- rand.());

julia> high = min.(1, probabilities .+ rand.());

julia> # plot pre-computed consistency bars
       reliability(probabilities, frequencies, low, high);

julia> # alternatively consistency bars can be specified as vector of tuples
       reliability(probabilities, frequencies, map(tuple, low, high));

References

Bröcker, J. and Smith, L.A. (2007). Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), pp. 651-661.

reliability(
    probabilities::AbstractVector{<:Real},
    outcomes::AbstractVector{Bool};
    binning=EqualMass(),
    consistencybars=ConsistencyBars(),
    kwargs...,
)

Plot a reliability diagram of the observed frequencies versus the mean probabilities in different clusters together with a set of consistency bars.

Optionally, the binning algorithm that is used to form the clusters and the consistency bars can be configured with the binning and consistencybars keyword arguments. If consistencybars === nothing, then no consistency bars are computed.

Examples

julia> probabilities = rand(10);

julia> outcomes = rand(Bool, 10);

julia> # default binning and consistency bars
       reliability(probabilities, outcomes);

julia> # custom options: without consistency bars
       reliability(probabilities, outcomes; consistencybars=nothing);

See also: EqualMass, EqualSize, ConsistencyBars

reliabilityplot(probabilities, frequencies; deviation=true, kwargs...)
reliabilityplot(probabilities, frequencies, low_high; deviation=true, kwargs...)
reliabilityplot(probabilities, frequencies, low, high; deviation=true, kwargs...)

Plot a reliability diagram of the observed frequencies versus the predicted probabilities, optionally with consistency bars ranging from low to high for each probability.

If deviation is true (default), the difference frequencies - probabilities is plotted versus probabilities. This can be helpful for inspecting how closely the observed frequencies match the predicted probabilities.

Examples

julia> probabilities = sort(rand(10));

julia> frequencies = rand(10);

julia> reliabilityplot(probabilities, frequencies);

julia> # plot frequencies instead of deviations
       reliabilityplot(probabilities, frequencies; deviation=false);

julia> # use different colors for markers
       reliabilityplot(probabilities, frequencies; markercolor=:red);

julia> # random consistency bars
       low = max.(0, probabilities .- rand.());

julia> high = min.(1, probabilities .+ rand.());

julia> # plot pre-computed consistency bars
       reliabilityplot(probabilities, frequencies, low, high);

julia> # alternatively consistency bars can be specified as vector of tuples
       reliabilityplot(probabilities, frequencies, map(tuple, low, high));

References

Bröcker, J. and Smith, L.A. (2007). Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), pp. 651-661.

reliabilityplot(
    probabilities::AbstractVector{<:Real},
    outcomes::AbstractVector{Bool};
    binning=EqualMass(),
    consistencybars=ConsistencyBars(),
    kwargs...,
)

Plot a reliability diagram of the observed frequencies versus the mean probabilities in different clusters together with a set of consistency bars.

Optionally, the binning algorithm that is used to form the clusters and the consistency bars can be configured with the binning and consistencybars keyword arguments. If consistencybars === nothing, then no consistency bars are computed.

Examples

julia> probabilities = rand(10);

julia> outcomes = rand(Bool, 10);

julia> # default binning and consistency bars
       reliabilityplot(probabilities, outcomes);

julia> # custom options: without consistency bars
       reliabilityplot(probabilities, outcomes; consistencybars=nothing);

See also: EqualMass, EqualSize, ConsistencyBars

EqualMass(; n::Int=10)

Create binning algorithm with n bins of (approximately) equal mass.

EqualSize(; n::Int=10)

Create binning algorithm with n bins of the same size.

ConsistencyBars(; samples::Int=1_000, coverage::Real=0.95)

Create consistency bars that cover a proportion of coverage samples obtained by consistency resampling with samples resampling steps.

References

Bröcker, J. and Smith, L.A. (2007). Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), pp. 651-661.