Evaluation of the proficiency testing

Preliminary remarks regarding the analysis software

The IFA conducts the whole statistical analysis. The data received is immediately anonymised at the IFA. Since 2011, QuoData's (QuoData GmbH Quality & Statistics, 01309 Dresden, https://quodata.de/de) Prolab Plus software has been used for interpretation of the results of the PT schemes . The interpretation meth¬ods employed in the software fully satisfy the requirements of DIN ISO 13528, DIN ISO 5725-2, -3, -5; and DIN 38402 A 45 [1-5].

General statistical values

The PT schemes are evaluated in accordance with DIN ISO 5725-2. The basis for scoring of the participants is the evaluation by the "z-score" [2].

Previous to the calculation of the statistical values the outliers are determined. For determi-nation of the outliers the Grubbs test is used.

For the PT scheme VOC the participants are supposed to indicate more than one individual value.

In this case, before calculating the individual mean value, the Cochran-test is used. I.e . if the difference between the results of a laboratory is too large, it is possible to disregard these results for further evaluation. The remaining individual results are averaged for each hazardous substance and weighed statistically, if applicable (see DIN ISO 5725-2 section 7.5.6 [4]).

Individual mean value

Individual mean values are only calculated for proficiency testing schemes with more than one individual value (PT scheme VOC).

Cochran outlier test

The dispersion of the individual values of laboratories are analysed with the outlier test according to Cochran:


Total mean value

The total mean value Ck is then calculated, which generally serves as the reference value during the evaluation. At PT schemes with low number of participants the mean concen¬tration found while analysing quality control samples can be defined as target concentration. This value is used for further statistical analyses such as the total standard deviation Sk, Grubbs test and z-score. Where certified materials are employed, the concentration stated on the certificate is defined as the reference value. The total mean value and total standard deviation are calculated from the data.


Grubbs test for outliers

It is generally assumed that the body of data is subject to normal distribution. The outlier test is then performed at the 95% level (both sides α = 2.5%). The elimination of outliers ulti¬mately produces a mean value which closely approximates to the "true value" of the sample.

In the Grubbs outlier test, the procedure is for all laboratory mean values to be calculated in the first instance and the total mean value and total standard deviation then to be determined from them. The difference is then formed between the individual mean values and the refer-ence value/total mean value, the individual mean value with the greatest difference to the reference value/total mean value being substituted marked * in the formula of the Grubbs outlier test. The test value t is compared to the tabular value for a significance level of α = 2.5% when considered from both sides (see above).

If it is confirmed as an outlier, it is removed from the body of data, and the total mean value and total standard deviation recalculated. The difference between the new total mean value/ reference value and the individual mean values is calculated and the greatest difference substituted in the Grubbs formula. If this value is also confirmed as an outlier, it is likewise removed. The procedure is repeated until no more outliers can be eliminated.


Should the data exhibit such strong variation that no outliers can be identified by means of the Grubbs test, they are defined by the deviation from the reference value. An outlier is recognized as such in this case when the deviation > ± 36%. Where the data material is heterogeneous, the limit can be increased to 50%, or identification of the outliers not be performed.

z-Score analysis

All individual mean values are considered in the z-score analysis, including the values identified as outliers in the Grubbs test.

The z-score of all participating laboratories, including those eliminated as outliers, is calculated with the aid of the outlier-free mean value and the outlier-free standard deviation.

The z-score can be regarded as the quality characteristic of the mean value of the individual laboratories (see below).

The z-score analysis is based upon the following formula:


A permissible deviation of 10% is assumed for s during analysis of the PT scheme. Where data material exhibits strong scatter the permissible deviation may be increased to up to 20%. The individual results are then evaluated as shown below:

  • |z| ≤ 1 Good result
  • 1 ≤ |z| ≤ 2 Satisfactory result
  • 2 ≤ |z| ≤ 3 Questionable result
  • 3 ≤ |z| Extremely questionable result

A result for which |z| ≤ 2 is deemed satisfactory, i.e. the PT is deemed passed. Should the |z| score exceed 2, review of the analysis method employed is advisable.

The z-score can be used to determine further statistically important parameters, which shall not however be considered in any greater detail at this point.

Evaluation Report, Confirmation of Participation and Certificate

After the evaluation has been completed, each participating measuring station receives a notification of the results. In the online portal, the participants are provided with an evaluation report with the confirmation of participation and an individual certificate for download.

The evaluation report contains the following information:

  • the confirmation of participation
  • tabular and graphical presentation of the entire analysis results and the outliers
  • tabular and graphical presentation of the z-score analysis
  • list of the analysis methods employed by the participants
  • alphabetical list of all participants

The integrated confirmation of its participation contains the following information:

  • laboratory number
  • own laboratory result and z-score
  • mean value of the samples following elimination of outliers (Grubbs test)
  • reproducibility standard deviation
  • reference value (mean value of the control samples for quality control)

The certificate (PDF, 656 kB, non-accessible) is produced individually from each participant's results. The certificate is produced in accordance with the following rules:

  • All substances are stated on the certificate for which, averaged over all samples, a z-score ≤ 2 was attained.
    Calculation for the individual substance:
    Of three samples, at least two must yield individual z-scores < 2; a z-score distribution such as "2.9, 2.8, 0", which purely mathematically would yield a mean value of 1.9, results in participation for the substance being deemed unsuccessful.
    No single sample may yield a z-score > 3.
  • Participation in the PT is deemed successful when a z-score ≤ 2 (see above) is obtained for over 50% of the analysed substances.
  • Should z-scores ≤ 2 be obtained for fewer than 50% of the analysed substances, the certifi¬cate merely confirms participation in the PT.

[1] DIN ISO 13528: Statistical methods for use in proficiency testing by interlaboratory comparisons (09.20). Beuth, Berlin 2020

[2] DIN ISO 5725-2: Accuracy (trueness and precision) of measurement methods and results - Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method (05.22). Beuth, Berlin 2022

[3] DIN ISO 5725-3: Accuracy (trueness and precision) of measurement methods and results - Part 3: Intermediate measures of the precision of a standard measurement method (02.03). Beuth, Berlin 2003

[4] DIN ISO 5725-5: Accuracy (trueness and precision) of measurement methods and results - Part 5: Alternative methods for the determination of the precision of a standard measurement method (11.02). Beuth, Berlin 2002

[5] DIN 38402-45: German standard methods for the examination of water, waste water and sludge - General information (group A) - Part 45: Interlaboratory comparisons for proficiency testing of laboratories (A 45); Berlin, 2014

Contact

Department Chemical and Biological Hazards

Krista Gusbeth
Brigitte Maybaum
Franziska Nürnberger
Ronja Schneck