RegressionTests

differentiate(f, g)

Determine if f() and g() have different distributions.

Returns false if f and g have the same distribution and usually returns true if they have different distributions.

Properties

• If f and g have the same distribution, the probability of a false positive is 1e-10.
• If f and g have distributions that differ* by at least .05, then the probability of a false negative is .05.
• If f and g have the same distribution, f and g will each be called 53 times, on average
• f and g will each be called at most 300 times.

The difference between distributions is quantified as the integral from 0 to 1 of (cdf(g)(invcdf(f)(x)) - x)^2 where cdf and invcdf are higher order functions that compute the cumulative distribution function and inverse cumulative distribution function, respectively.

More generally, for any k > .025, recall loss is according to empirical estimation, at most max(1e-4, 20^(1-k/.025)). So, for example, a regression with k = .1, will be escape detection at most 1 out of 8000 times.

runbenchmarks()

When called in test/runtests.jl while pwd() points to test, this function runs the benchmarks in bench/runbenchmarks.jl.

There are some keyword arguments, but they are not public.

test(skip_unsupported_platforms=false)

When called in testing, runs regression tests, reports all changes, and throws if there are regressions.

Set skip_unsupported_platforms to true to skip the test (quietly pass) on platforms that are not supported.

trackable(x) -> Union{Float64, NamedTuple{<:Any, NTuple{<:Any, Float64}}}

Convert an object into a Float64 or NamedTuple of Float64s for tracking. Called automatically by @track expr on the result of expr.

Define new methods for this function to track non-Real types.

@group expr

Group multiple tracked values together with setup code so that they may all be omitted if the first several trials do not indicate a plausible change in any of the grouped tracked values.

Example

@group begin
    x = rand(100)
    sm = sum(x)
    @track abs(sm - foldl(+, x))
    @track sm / mean(x)
end

@track expr

Track the return value of expr for regressions. expr must evaluate to a number that can be converted to a Float64.

If the first several trials do not indicate a plausible change in the tracked value then subsequent trials may skip evaluating expr. Do not put code in an @track expression that has side effects needed later on.

Should be used in or included by a runbenchmarks.jl file.

Examples

@track begin
    x = rand(100)
    abs(sum(x) - foldl(+, x))
end

y = rand(100)
@track abs(sum(y) - foldl(+, y))
@track sum(y) / mean(y)