"If data repeatability is the cornerstone of your lab then data reproducibility is the main pillar of our science"
Reproducibility has a global appeal and it carries the power of universal recognition and adoption of key findings in your lab. Unfortunately, data reproducibility is not a natural process due to “best practices” that varies between the labs. However, the scientific community has come together to limit these variations between the labs and this is expressed in “reproducibility error” of a test method/ test instrument. In order to improve quality of science, our labs should be regularly tested for its "reproducibility error". Recently, we have tested our lab performance and this is how we did it.
American Standards for Test Methods (ASTM)
American Standards for Test Methods (ASTM) is one of the several organizations that has taken the responsibility to define “reproducibility" through “global comparison of friction and wear tests from tribology labs”. ASTM Proficiency Testing Program (PTP) conducts this study regularly that compares lab results to obtain statistical quality assurance for a lab in terms of data reproducibility. As an example, we have elucidated an ASTM PTP study that described the "reproducibility error" for our lab using the Ducom Four Ball Tester (FBT-3) for friction and wear tests of lubricants.
Proficiency Testing of Four Ball Test Methods
Four ball test (FBT-3) is a common method to evaluate lubricant performance metrics such as extreme pressure load, non-seizure load, load wear index (ASTM D2783) and wear preventive property expressed in wear scar diameter/MWSD (ASTM D4172). ASTM PTP’s for four ball test methods are typically conducted twice a year using a reference oil from a known batch that is supplied to the participating test labs (or a group). A statistical analysis on reproducibility for each individual lab is reported. In the following sections, ASTM PTP report for years 2017 and 2019 is summarized, the group score and individual score are described. Figure 1 shows 15-20 participating labs per test cycle in the years 2017 and 2019.
Figure 1. Number of participants in each test cycle
PTP Assessment Metrics
Following the guidelines of ASTM D7372, PTP results are analyzed and interpreted by assessing the collective lab results for a given test method, as represented by TPI and each lab result compared to the collective PTP results, as represented by Z-score.
Test Performance Index (TPI): TPI analyses the reproducibility of the whole group data. ASTM D7372 categorizes TPI into three different ranges as summarized in this Table.
"A TPI > 1.2 and Z < ±2 implies that a laboratory is performing effectively and the results from the test method is well benchmarked relative to the global results."
Z-score: Z score describes the data reproducibility for an individual lab when compared to the entire group. Ducom’s Z-score value over different test cycles is presented in Figure 3. The results show that Ducom FBT-3 performance for different test methods was within the control limits for the PTP. Furthermore, the Z-score over a rolling period has shown continuous improvements across all test methods to achieve a value of ± 1 for the latest test cycle.
Figure 3. Z score for our lab using Ducom FBT-3.
Conclusions: ASTM PTP can determine lab performance by reporting the "reproducibility" from your lab. Because reproducibility is the pillar of science, we strongly recommend tribology labs to sign up for a PTP study. If you need any assistance in registration then please reach out to Ducom.
ASTM PTP describes reproducibility in terms of TPI and Z-score, that measures the lab performance in relation to measuring instruments and its use in wear tests. From our participation in ASTM PTP, the data from Ducom Four Ball Tester (FBT-3) was "reproducible" within the ASTM precision limits. Note that data from Ducom Instruments is always globally benchmarked with similar instruments in multiple labs.
"ASTM PTP study can resolve the data reproducibility crisis in science."
As a standard practice in purchasing of tribometers, we must make it a habit to request the ASTM PTP report from the manufacturer.
Future: A minor drawback in this PTP study is that it assumes zero human error in data reporting. However, human error is unavoidable due to high volume of data. Such unintentional human errors can be resolved by automating the data flow, as demonstrated by MOOHA, that is powered by IoT and A.I. We look forward to implementing MOOHA into the ASTM PTP study.