Guidelines for the Evaluation of Direct Reading Monitors

Transcription

1 Guidelines for the Evaluation of Direct Reading Monitors ER Kennedy, ML Woebkenberg, PC Schlecht, DL Bartley, S Shulman, HA Feng, R Song NIOSH C Cowherd, Jr., MA Grelinger, and KM Bauer Midwest Research Institute 1

2 Direct Reading Monitor (DRM) Guidelines DRM Guidelines were developed to parallel active sampling & analytical method development guidelines Provide manufacturers with scheme to test monitors Provide end users with information on the performance testing of gas & vapor monitors Procedure to evaluate bias, precision and accuracy of monitors Incorporate consensus standards 2

3 Components of the DRM Guidelines Background information and definitions Overview of monitor types and operation principles Suggested components in monitor testing Physical characteristics size, shape, weight, documentation and manuals Operational characteristics ease of use, maintenance, calibration, ruggedness, alarms, readout, data output Performance characteristics sensitivity, specificity, response, recovery, interferences, uncertainty of measurement Reliability intra- and inter-day, inter-monitor Evaluation documentation 3

4 Components of the DRM Guidelines Addendum (continued) Ensure First Responders have portable monitors suitable for operation in harsh environments. Compatibility with protective equipment Requisite data quality needs Applicability testing of DRMs in first responder environments Ease-of-use testing Performance tests based on DRM Guidelines 4

5 Experimental Summary Response time Calibration Linearity Drift Range Environmental effects Extreme temp and RH for Addendum testing Limit of measurement Interferences Chemical Smoke, fog and dust for Addendum testing Reliability RFI Drop test Sensor life Step change Supply voltage Long term stability Remote sampling Precision Bias Accuracy Monitor uncertainty Quality system requirements 5

6 Carbon Monoxide Monitors Tested Used by first responders CO monitors commercially available Two different types of sensors Electrochemical Direct readout (2) Alarm only Colorimetric 6

7 Monitor Evaluation under the Guidelines and Addendum Performance testing Test the Guidelines and not the monitors Testing not performed at extreme concentration to limit damage to monitors Based on selected tests from Guidelines CO concentrations generated from: Gas cylinders/traceable reference methods for dilution Gas cylinders/dilution system & continuous emission monitor 7

8 Monitor Response Times Monitors exposed to challenge concentration. Response time calculated based on time to rise from 0% to 90% of the challenge concentration. Monitors removed from challenge concentration. Response time calculated based on time to decrease from 100% to 10% of challenge concentration. Only data from monitors with digital readout are reported. Alarm only data was too erratic. 8

9 Monitor Response Times 0-90% Full Scale % Full Scale sec 90 M1 sec 90 Monitor=M-1 M CO concentration (ppm) CO concentration (ppm) sec 90 sec M CO concentration (ppm) 30 0 M2 M CO concentration (ppm) 9

10 Range Test Data Monitors calibrated using 35 ppm CO Monitors exposed to 4, 18, 35, 70 and 350 ppm CO Graph depicts two brands with 4 replicates While intramonitor variability is small, intermonitor variability can be large Positive bias noted at high concentration At lower concentrations (4, 18 and 35 ppm), negative bias > 10% noted ppm ppm Range Test Data 350 ppm M1 M2 Range Test Data at 70 ppm M1 M2 A B 60 10

11 Drift Test Digital monitors exposed to 0 and 90% of upper range for each of five days. Each monitor was consistent Variability was high between monitors Some monitors indicated a full scale response at 90% of full scale concentration Drift at 0% was less than 10 ppm CO Alarm only monitors exposed to 90 and 110% of alarm set point over 4 days. All monitors did not respond at 35 ppm set point on all days (2 false 32 ppm; 23 false 39 ppm) Monitors were inconsistent at 100 ppm set point on all days (4 false 90 ppm; 7 false ppm) 11

12 Interferent testing Test conditions 110 ppm SO 2 only 110 ppm CO ppm SO 2 for digital & alarm only monitors Digital readout monitor No false positives at 110 ppm SO 2 Results tended to read high (ca. 120 ppm) in presence of interferent Alarm only monitor False positives at 0 ppm CO and 110 ppm SO 2 (35 ppm set point) Only low level alarm was triggered (35 ppm set point) High level alarm (100 ppm set point) did not sound at 0 ppm CO No effect of interferent on 110 ppm CO exposures 12

13 Linearity Test M1 monitors exposed to 0, 300, 600, 900 and 1200 ppm CO. M2 monitors calibrated and exposed to 0, 150, 350, 550 and 700 ppm CO. Monitor results plotted versus actual concentration (XY plot with linear regression) Results showed linear response All intercepts not different from 0 All but one slope is different from 1 M1 35% of results fell within ± 10% of calibration line M2 80% of results fell within ± 10% of calibration line 13

14 Monitor Accuracy Estimate Monitor accuracy improves at intermediate concentrations Optimal accuracy observed around PEL/REL range (35 to 50 ppm Low bias observed at low concentrations High bias observed at high concentrations % NIOSH Accuracy M2 M ppm CO 14

15 Ease of Use Test Results Based on input from panel (4 members) Panel members evaluated monitors in semi-random fashion Monitors ranked on scale of 1-5 (1=best) Use conditions Noise Limited visibility Limited tactile sensation Assessment of DRM documentation Calibration/ operation Sum Weighted Score Final Weighted Scoring M-1 M-2 M-3 M-4 Monitor ID 15

16 Summary All tests were not applicable to all monitors Alarm only monitors Pumped tubes Field calibrated vs factory calibrated Found that some tests could be combined for efficiency Response time and linearity tests Guidelines found inter-monitor variability Individual monitor performance tended to be precise Inter-monitor variability found between and within manufacturers Monitors tend to show high bias at higher concentrations Monitor performance degraded under extreme environmental conditions Ease of use test demonstrated ability to rate monitors Guidelines document and addendum are currently in NIOSH review 16

17 Disclaimer Mention of company names or products does not constitute endorsement by the Centers for Disease Control and Prevention. The findings and conclusions in this presentation have not been formally disseminated by the National Institute for Occupational Safety and Health and should not be construed to represent any agency determination or policy. 17

18 The Tricorder! The ultimate direct reading monitor Questions? 18