Vera bug with handling large numbers?

Plugins & Plugin Development General Plugin Discussion
1

I’m trying to do some math on large numbers - specifically unix timestamps, and it seems they are not being handled correctly.

Firstly, it seems that some math isn’t being handled correctly, and the second problem is that string.format seems to be doing something strange!

I had a test that read -:
if A < (B-C) then …
A and B were large numbers (i.e. times) and C is small. This would not work correctly. If I change this to -:
if (B-A) > C
then it works. In the second case, the test is only against a number like 50, rather than doing the comparison of two large numbers (e.g. 1315000000 and 1315000010, which can produce a difference of 26 rather than 10!)

A bit of an example of the issue is shown below. This really illistrate the second problem where string.format is printing rubish (seamingly rounded numbers), but I suspect that it’s related.

Start = 1315000000 luup.log( "1: " .. Start .. " " .. string.format("%0.f", Start)) Start = Start-1 luup.log("2: " .. Start .. " " .. string.format("%0.f", Start)) Start = Start+0.9 luup.log("3: " .. Start .. " " .. string.format("%0.f", Start)) Start = Start+0.1 luup.log("4: " .. Start .. " " .. string.format("%0.f", Start))

Produces the output -:

1: 1315000000 1315000064 2: 1314999999 1314999936 3: 1.315e+09 1314999936 4: 1.315e+09 1314999936

I haven’t tried to work out the relationship. I understand that Lua should be using floating point doubles for all numbers, so this really shouldn’t happen. I don’t know if Vera is using short floats to represent numbers, or if I’m simply using “format.string” incorrectly (although the conditional tests are also showing similar output). I need to either use format.string or something similar in order to get the resolution since once you’ve performed a mathmatical function on a number, it’s printed out in engineering format by default.

Does anyone have any experience of this? Any comments or pointers?

Cheers
Chris

2

See bug #1451 and the references to the forum there.

3

Thanks. I was hoping for a different response, but unfortunately that’s pretty much what I expected. Unfortunately it does make time comparisson pretty damn difficult :frowning:

Cheers
Chris

4

I had this as well: http://forum.micasaverde.com/index.php?topic=6957.0

Everyone seems to run into this and I think it is an important issue (As result, a system for timing stuff cant keep proper time). However, if I interpret the comments at bug 1451 it seems that it is not classified as important and may not be solved for some time.

5

It’s mostly possible to work around the bug with additional effort in your Lua code.

What you have to do is write code that doesn’t invoke Lua’s normal “promotion” of integers to floats. Most of the math.* functions are off-limits, even math.floor(). Arithmetic such as + - / * % are fine, as long as your number fits in a 32-bit signed int, and it never got secretly promoted to float beforehand.

In your example, I bet that C somehow got converted to a float, and it polluted the timestamps in A and B, promoting them to floats. Your workaround of diffing the timestamps first to produce a small offset is the right one. The computer scientist in me wants to tell you that it’s what you should always be doing when you are working with numbers. (And the software engineer in me wants to tell you you should be using os.difftime() and not subtraction.)

I just did some experimenting, and provided that you keep your timestamps in integers or even strings, functions like os.difftime() will do the right thing. os.time(), likewise, will build a correct timestamp from year-month-date-hour-minute-second.

It seems that the only function that won’t play ball is the two-argument os.date(), where you give it a deliberate timestamp to separate into year-month-day-hour-minute-second. No matter what I feed in, the number gets converted to a float, and it loses precision. That seems to correspond with the Lua source: it converts the input to a LUA_NUMBER (float), then to a time_t, losing precision. Maybe in my copious free time I will code up a pure-integer-Lua replacement for os.date(“*t”).

6

@futzle,

you are right in that it is possible to work around the problem (this is what I do). I can as well write my own libraries which do not have the problem (or use the ones you will create). However, if I have to write everything from scratch I do not have to buy a vera. Such stuff should work.

7

Thanks for the comments. I wasn’t aware of os.difftime() - I’m rather new to Vera and Lua although have been programming for many years in C/C++. I’ll have another go at working around the problem by keeping things as longs as you suggest, but I’m not sure I will manage this as I do a / which ultimately ends up as a float.
Is there any way to cast numbers back to ints (I assume not)?

Regarding the priority on getting it fixed - I would have thought it should be high as it’s quite a fundimental issue. Yes, it might be possible to work around in many/most cases, but I spent a lot of time trying to work out why my code was doing daft things for what seemed like simple math (being new to Lua I thought it was me being stupid!). Vera should be able to handle times reliably - it’s a fundimental part of the system.

Cheers
Chris

8

[quote=“Chris, post:7, topic:169072”]I’m not sure I will manage this as I do a / which ultimately ends up as a float.
Is there any way to cast numbers back to ints (I assume not)?[/quote]

Division should be OK. It’s like in C: if both arguments are integers then the result is an integer.

I tested tonumber() on a string “1234567890” and got back an unmunged result, so it would seem that converting a number into a string and back again will intify a number. This is backed up by a browse through the source code.

Since posting my earlier reply in this thread, I searched some more. Lua natively doesn’t have any integer support at all. That the Lua interpreter in Vera has any knowledge of integers at all is because of an extensive patch to Lua 5.1 called LNUM. Vera’s Lua comes straight from OpenWrt, which applies LNUM to its Lua as a 100-kB patch. It’s the version of OpenWrt that Vera uses which made Lua’s Number type a C float, so this is a bug that Vera inherited. The bug wasn’t discovered and fixed in OpenWrt for a few months, and that’s the window during which MCV took the OpenWrt trunk.

The fact that LNUM doesn’t patch os.date() is arguably a bug. I don’t think it’s been reported to the LNUM project. That project seems to be inactive, FWIW. In any case, using a more recent OpenWrt would be just as good.

Regarding the priority on getting it fixed - I would have thought it should be high as it's quite a fundimental issue.

MCV’s handling of bugs smacks of triage, which I suppose is true of any software company. We developers can fend for ourselves, and there are (let’s be honest) far more serious bugs that demand MCV staff time.

Let’s say MCV dropped everything and fixed this bug tomorrow. MCV would have to get users to upgrade their Veras’ firmware to a later version of OpenWrt. Say 1% of users somehow mess up the upgrade. And say 10% of users don’t dare upgrade, because they’re happy with their current setup or got scared by the vocal 1% complaining on the forum. Now there are two problems instead of one: MCV has a support nightmare, and we programmers still have to work around the bug because not every user has the double-precision Lua that we want them to have.

For such minimal returns, I’d prefer that MCV spent their efforts somewhere else.

Edit: spelling.

9

Thanks for the pointers. From my previous reading, I understood that while LUA didn’t have ints, it should use doubles, which would have been fine as we’d have better than 32 bits precision. I’m trying to avoid any unnecessary data manipulation as it’s time consuming (I’m crunching quite a lot of data).

I’m sure you’re right about priority, although given that time is quite important in a control system, I’m surprised this hasn’t come up more often.

Cheers
Chris

10

I agree that MCV has to set priorities (And you cannot cater for all user wishes). However, as I understand they have, for example, people currently working on a new UI. I personally would fix basic issues first such as this number bug before creating a new UI (But then, I am not a marketing person and from that perspective such priority list may make sense).

11

The truth is that many users complained that the interface (UI4) was too complicated and that the information was not available in only one place. So UI5’s main purpose was to unify all that information, so the user won’t have to go to cp.mios.com to see notifications, energy usage, etc.

OTOH less than 10 users complained about the Lua precision bug, so you can see that the priorities have been set based on what the majority of users asked.

12

I would argue that many users interact with the UI and only a few interact with the programming interface. As consequence, more people will have some opinion on UI then on the programming side (In addition, the people that observe programming issues have generally the ability to handle them and find a workaround). I think that the number of requests does not necessarily relate to the importance of a problem. If the platform has less problems programmers have to find a workaround for then programmers can create new features quicker which would be a benefit for all the people using the system. So, fixing issues only a few complain about may in the long run be not necessarily a bad investment of time.

I understand mcvs point, I stop now, I will be careful adding numbers until this bug gets some attention :slight_smile: