Automatically cleaning up erroneous logged data

Hi,

Don’t know if I’m alone getting strange values logged sometimes.

Here one of my temperature sensors logged a very low value (below absolute zero!) :stuck_out_tongue:
This messes up the graphs pretty much… I would really like to either remove them manually in some relatively simple way or having dataMine skip values that are statistically "outliers"or whatever you call them.

root@MiOS:/dataMine# more 61\ PoolGarage\ -\ CurrentTemperature\ \[R2261\].txt
1367454314,11
1367457344,11
1367476620,-21474826
1367476620,11
1367501537,12
1367522159,12

No indeed. I’ve had several outrageously large power meter readings - maybe some sort of Z-Wave transmission error - I don’t know.

Certainly does muck up the plot scales. Would be nice to clean this up.

I’ll take a look at this. I have some ideas on setting a min/max filter value - intially I was going to do this in the GUI, but the problem there is that when I add more stats and history, it becomes impossible to filter it out, so it needs to be done before history records are collated. I’ve always had a philosophy of recording the raw data without modification, and then changing it later - if you change, or delete, the raw data, it becomes impossible to get back to it if you change your mind. There are times when filtering is obvious (ie when the temperature is 2 million below 0!), but other times when it’s not so obvious and you may want to revery…

Anyway, I’ll take a look at this in a (near) future version.

Cheers
Chris

Chris

I've always had a philosophy of recording the raw data without modification, and then changing it later

100% with you on that. Perhaps just a flag somewhere to say that this particular sample is to be ignored. From where I’m looking, this is not something that happens often and I don’t mind a fully manual method.

Still, thanks a million for your continuing work on all dataMine features - sterling work.
(obviously not to be confused with “thanks a million in stirling” - couldn’t afford it - but thanks anyway!)

Sounds great Chris.

[quote=“akbooer, post:4, topic:175242”]Still, thanks a million for your continuing work on all dataMine features - sterling work.
(obviously not to be confused with “thanks a million in stirling” - couldn’t afford it - but thanks anyway!)[/quote]
Damn - I thought it was Christmas already :slight_smile:

Cheers
Chris