Project

General

Profile

Actions

Bug #441

open

F3200 filter

Added by Anonymous about 13 years ago. Updated about 13 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
QC_data_checker
Start date:
09/14/2011
Due date:
% Done:

0%

Estimated time:

Description

This is a feature request, not a bug.

I have seen a large number of F3200 "suspecting replicated data" warnings, most of them spurious. HDH eliminated about 90% of the unnecessary F3200 warnings by providing an option to issue no F3200's for 0-D variables. Even so, in my experience the majority of F3200's have been spurious.

The F3200 exception is based on a comparing few indicators such as mean and maximum. For my own use, I wrote (in Python) a filter to read the suspect data a second time and compare the entire data arrays, item by item. For the comparison I used the method favored by numerical analysts and implemented by the function numpy.ma.allclose(): scalars a and b are close if |a-b|<atol+rtol*max(a,b). My default tolerances are atol=1.e-9 and rtol=1.e-6, but any small numbers would work. So far I have sent hundreds of data sets through this filter, and now I have zero spurious warnings.

I think that a better place for this second-pass check would be the QC tool.

Actions

Also available in: Atom PDF