Ttdfjt

Assuming that we have N data points $x_1, ldots, x_N$ and a known distribution $F_0(x)$ (not a Normal Distribution), i.e. we want to check if the system under study behaves 'typically'.

I would like to quantify the probability that those $N$ datapoints originate from this distribution. For large sample sizes, there are various statistical hypothesis tests that could be applied (e.g. chi-squared). However, if the sample size is small, the chi-square test will fail.

Intuitively, it should be possible to detect if those data points are clearly deviating, even with a small sample size. However, is there a mathematical test designed for these conditions?

Maybe Kolmogorov-Smirnov test with correction for small samples provided by Jan Vrbik in: Vrbik, Jan (2018). "Small-Sample Corrections to Kolmogorov–Smirnov Test Statistic". Pioneer Journal of Theoretical and Applied Statistics. 15 (1–2): 15–23.

Correction itself is also described on Wikipedia site for Kolmogorov-Smirnov test: replace $D_N$ with

$$ D_N+frac 16sqrt N+frac D_N-14N$$

where $D_N$ is standard Kolmogorov-Smirnov statistic.

Use R to generate 10 observations from a standard uniform
distribution:

set seed(722) # for reproducibility
x = runif(10)
summary(x); sd(x)
 Min. 1st Qu. Median Mean 3rd Qu. Max. 
 0.1270 0.4940 0.7454 0.6627 0.9070 0.9477 
[1] 0.293335 # SD

Use the Kolmogorov-Smirnov test to see if the sample is consistent with standard uniform. Appropriately, the answer is Yes because data were sampled from a standard uniform distribution: Large P-value, no rejection.

ks.test(x, punif)

 One-sample Kolmogorov-Smirnov test

data: x
D = 0.31507, p-value = 0.2217
alternative hypothesis: two-sided

Is the sample also consistent with $mathsfNorm(.5, sqrt1/12)?$ The mean and variance match, but
shapes differ. Notice that the parameters mean and standard deviation are specified. Again consistent, but we know the normal distribution is not correct.

 ks.test(x, pnorm, .5, sqrt(1/12))

 One-sample Kolmogorov-Smirnov test

data: x
D = 0.36246, p-value = 0.1104
alternative hypothesis: two-sided

However, the K-S test easily rejects that this sample is from $mathsfExp(rate=2),$ which has mean $1/2 = 0.5,$ but the wrong SD. This exponential distribution has almost 14% of its probability above $1,$ but our sample has no observation above 0.948.

ks.test(x, dexp, 2)

 One-sample Kolmogorov-Smirnov test

data: x
D = 1.5513, p-value < 2.2e-16
alternative hypothesis: two-sided

Notes: (1) See other pages on this site and the Internet, including
the relevant Wikipedia page, which has a brief explanation of the test and some remarks about cases in which parameters must be estimated from data.

(2) Several well-known statistical software programs have
procedures that check a sample against a list of often used
distributions to estimate parameters and see if any distribution is a fit. Often these are called 'distribution ID' procedures
and sometimes they are restricted to non-negative data.

For example, when the distribution ID procedure in Minitab
is asked to compare the small sample above to normal, lognormal, Weibull, and gamma families, here are the parameter estimates:

ML Estimates of Distribution Parameters

Distribution Location Shape Scale 
Normal* 0.66265 0.29334
Lognormal* -0.55937 0.66158
Weibull 2.62094 0.74268
Gamma 3.53947 0.18722

* Scale: Adjusted ML estimate

And here are appropriate probability plots with P-values of Anderson-Darling goodness-of-fit tests in
legends. The data are clearly inconsistent with distributions in the lognormal family.

(2) For very large sample sizes, Kolmogorov-Smirnov, Anderson-Darling and other goodness-of-fit tests can reject some
distributions as not fitting---even when the fit might be good
enough for some practical applications.