USB Drive logging

kskenyon · May 3, 2011, 6:35pm

I was able to configure a reformatted USB drive for extended logging and storage using 1234 on a Vera 2. After upgrading to 1245 that no longer works and I cannot enable a new MSDOS/FAT USB drive either. I tried several and they all end as FAILED after Vera tries to format the drive.

Is USB logging working for anyone on 1245?

Kevin

guessed · May 3, 2011, 8:41pm

Yes. Note that USB Logging is fussy however, and tends not to like re-used drives.

I recently fried the 1st one that I had in Vera, and have moved onto the second… I suspect I burned out the original as I run permanently with verbose logging enabled (the “[tt]logread[/tt]” command was reporting SQUASHFS errors", and Vera was being more grumpy than usual)

Anyhow, both Production drives I’ve used were new, unformatted, direct-from-packaging drives when I put them into Vera, to avoid headaches. I keep a small collection of them just for this purpose.

During it’s early development, CJ had lots of problems getting my “reused” drives to behave correctly, so I recommend people avoid them.

sparkejj · May 17, 2011, 3:01am

@guessed,
I got USB logging working with a 128MB USB stick, after manually partitioning and formatting it. It seems the problem was with the partitioning. Don’t know why yet. Possibly because Vera attempts to create a 512MB partition.

Have you always used sticks >= 512MB?

guessed · May 17, 2011, 3:14am

[quote=“oTi@, post:3, topic:168250”]…
Have you always used sticks >= 512MB?[/quote]
Yup. All 512MB or greater. The “reused” ones were from trade-shows, the new ones were a Maxell 1GB, the current one is a 512MB Kingston DataTraveler.

… and yes, pre-existing partitions were most of the problem, since the tools present on Vera weren’t particularly flexible in handling this.

sparkejj · May 17, 2011, 9:59am

Exactly; that’s my observation as well. I’ll see if there’s some manual steps that can be posted here if the automatic script fails.

Currently logging to a 128MB PNY Attache.

lourens.henk · May 17, 2011, 3:54pm

Awaiting your “howto” @oTi@ as im trying to get my fat partitioned 1Gb stick to be accepted by vera.
Ill dig through the archives to see what i can find to contribute.

—edit—

Got mine to work! <df -h> returns the following;

/dev/scsi/host0/bus0/target0/lun0/part1 503.9M 20.2M 458.2M 4% /tmp/log/cmh

Since the stick is a 1Gb one, it shows only that 512Mb partition. Seems to be fixed that way. Maybe we get a more flexible solution in the near future. But for now… it wil do.

sparkejj · May 17, 2011, 4:40pm

[quote=“Henk, post:6, topic:168250”][…]it shows only that 512Mb partition. Seems to be fixed that way. Maybe we get a more flexible solution in the near future. But for now… it wil do.
[…]working on the “USB howto”, @oTi@, your steps to use a smaller device than 512Mb would be appreciated)[/quote]
Well, presumably it is fixed to 512MB because that is a, probably generous, worst-case scenario before logs are rotated.

lourens.henk · May 17, 2011, 4:59pm

HOWTO get your vera log an attached USB drive.

This wil work for:

VERA 2 only
firmware 1.1.1245 and above

First of all, the release notes for version 1.1.1245 state:

“2. There is an option with Vera2 (Currently not working with Vera1) on Advanced, Logs to enable logging to a USB stick. This frees up quite a bit of memory, so if you have a lot of devices or plugins and your Vera is running slow or crashing because it’s running out of memory, this is a solution. Note that after you check the box it can take up to 30 minutes to format the USB drive, and bootup will take a couple minutes longer with the USB drive installed. It will continue to boot up fine without the USB drive connected, just don’t remove it while Vera is running. IMPORTANT: Use a new USB drive that hasn’t already been formatted. In particular, don’t use a USB drive that is already formatted for Mac.”

This is true so if youre new to all of this and want to avoid headaches, follow the advice and get an NEW usb drive.
I recommend it being not smaller than 512Mb and not much bigger than 1Gb. Vera will create its own partition on it which will be 512Mb in size and use that to write logfiles to it.

For the less feintharted amongst us, if you DO want to use a drive that you have lying around, i suggest you clean it (backup and remove your files) and reformat it using a standard FAT partition before you go ahead.

Then

After you got your usb-drive connected to the USB port on the backside of your vera.
Please be patient, it might not work the first time, but it will work. And very important,
DO NOT REMOVE THE DRIVE WHILE VERA boots/reboots or powercycles.
Log into the UI
Select “advanced” on the sop right (next to “Mios developers” in the toolbox)
Select the "Logs"tab
Activate the “Store logs on USB device*” option and READ the text there:
[quote[
Please connect an USB drive first then check the checkbox.
WARNING: The usb drive will be erased. Wait 20 minutes after clicking OK.
Status
[/quote]
Click the “OK” button.

Wait until vera reports progress.

If vera reports “FAILURE” go through step 2 through 6 again.
If vera reports succes, you are done.

For those who would like to check:

SSH into your vera (using Putty or any other SSH client)
Login as root with the password thats on the back of your box.

At the prompt type:

<df -h>

If you find something like:

/dev/scsi/host0/bus0/target0/lun0/part1 503.9M 20.2M 458.2M 4% /tmp/log/cmh

Then your USB drive is used for storing logs.

For the enthousiasts, i stumbled on an older thread where @guessed and @Ap15e contributed here:
http://forum.micasaverde.com/index.php?topic=5405.0

Mixing the original code with some common sence gave me this:

-- Code by Ap15e: -- os.execute( 'uptime >> /tmp/log/cmh/LuaTerminated.log;'.. 'grep "LuaUPnP Terminated with" /var/log/cmh/LuaUPnP.log >> /tmp/log/cmh/LuaTerminated.log;'.. 'cat /tmp/log/cmh/LuaTerminated.log | sort | uniq >/tmp/log/cmh/LuaTerminated.log') -- -- logs uptime and Lua restarts to /tmp/log/cmh/LuaTerminated.log on the USB drive. -- Just paste the code into the 'Luup' tab of a scene and set an appropriate timer.

Any feedback is appreciated!
@oTi@, your steps to use a smaller device than 512Mb would be appreciated

sparkejj · May 17, 2011, 5:10pm

[quote=“Henk, post:8, topic:168250”]'cat /tmp/log/cmhLuaTerminat/ed.log | sort | uniq >/tmp/log/cmh/LuaTerminated.log')[/quote]
That fourth slash seems out of place?

lourens.henk · May 17, 2011, 5:13pm

[quote=“oTi@, post:9, topic:168250”][quote=“Henk, post:8, topic:168250”]'cat /tmp/log/cmhLuaTerminat/ed.log | sort | uniq >/tmp/log/cmh/LuaTerminated.log')[/quote]
That fourth slash seems out of place?[/quote]

Repaired!
Thanks, thats what you get from cutting and pasting, very observing, just the feedback i appreciate!
As for my question, how did you get your 128Mb stick to work?

Dav1d · May 17, 2011, 5:33pm

I see;

root@MiOS_1XXXX:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 6.3M 6.3M 0 100% /rom
tmpfs 14.9M 416.0K 14.5M 3% /tmp
/dev/mtdblock/4 8.9M 4.8M 4.1M 54% /overlay
mini_fo:/overlay 6.3M 6.3M 0 100% /
[font=Verdana]/dev/scsi/host0/bus0/target0/lun0/part1
495.8M 23.0M 447.3M 5% /tmp/log/cmh[/font]
/dev/loop/0 3.0M 3.0M 0 100% /mios
root@MiOS_1XXXX:~#
But if I use WinSCP I see Received error message from remote side: ‘scp: part1: not a regular file’

lourens.henk · May 17, 2011, 5:45pm

@Dav1d

As far as i can tell you dont have an USB drive attached, or at least, vera was unable to partition it.
As for WinSCP, its used to acces your vera for putting and getting files with a graphical UI.

For root commandline options simply use Putty

Dav1d · May 17, 2011, 5:56pm

[quote=“Henk, post:12, topic:168250”]@Dav1d

As far as i can tell you dont have an USB drive attached, or at least, vera was unable to partition it.
As for WinSCP, its used to acces your vera for putting and getting files with a graphical UI.

For root commandline options simply use Putty[/quote]

From I believe your post 7

If you find something like: Quote /dev/scsi/host0/bus0/target0/lun0/part1 503.9M 20.2M 458.2M 4% /tmp/log/cmh
Then your USB drive is used for storing logs.

I see where I posted above,

/dev/scsi/host0/bus0/target0/lun0/part1
495.8M 23.0M 447.3M 5% /tmp/log/cmh

Dav1d · May 17, 2011, 6:11pm

To actually get to the log files in a readable form one must navigate to /root/tmp/log/cmh

lourens.henk · May 17, 2011, 6:31pm

@Dav1d

my bad…i was multitasking again being on here, in my vera’s UI as well as editing on the wiki and replying to different posts.

I simply overlooked your bold tekst.

your path

/root/tmp/log/cmh

Is that in your vera or on the usbdrive? (i presume the usbdrive)

sparkejj · May 17, 2011, 6:43pm

Right. The log files are at /tmp/log/cmh on Vera. The stuff quoted below talks about disks/filesystems; it is not stuff you want to directly navigate to for file access.
(It says that /tmp/log/cmh is located at partition 1 on the usb stick, and shows the size and usage of the partition.)

/dev/scsi/host0/bus0/target0/lun0/part1 503.9M 20.2M 458.2M 4% /tmp/log/cmh

lourens.henk · May 22, 2011, 6:32pm

Now heres a next issue,

Ive been downloading some logs to my client and try to get smarter from them.
The fact is, i dont read unformatted data easily.

Is there a way to get the most used logs in an easier readable format?
Ive tried renaming the log to xml, html and csv (fiddling with several delimeters)
and nothing gave soothing results. Not notepad, not my browser, not excel.

So if anyone has an idea…
Let me know…

Hope i havent missed one…

guessed · May 22, 2011, 6:59pm

@Henk,
They expect you to read them in a Terminal program, like the SSH into Vera - at least for [tt]LuaUPnP.log[/tt]. There are various parts where they’re explicitly putting in terminal-escape sequences to make the output Red, Blue, etc.

The lines all “start” with a numeric, which are [loosely] described here:

http://wiki.micasaverde.com/index.php/Luup_Debugging

You can use that to “filter out” the different streams of logging going on (ZWave IO, Serial IO, Luup code execution, Scene execution, etc, etc)

lourens.henk · May 22, 2011, 7:04pm

[quote=“guessed, post:18, topic:168250”]@Henk,
They expect you to read them in a Terminal program, like the SSH into Vera - at least for [tt]LuaUPnP.log[/tt]. There are various parts where they’re explicitly putting in terminal-escape sequences to make the output Red, Blue, etc.

The lines all “start” with a numeric, which are [loosely] described here:

http://wiki.micasaverde.com/index.php/Luup_Debugging

You can use that to “filter out” the different streams of logging going on (ZWave IO, Serial IO, Luup code execution, Scene execution, etc, etc)[/quote]

Ugh… sounds timeconsuming… any other ways?

—edit—
How bout this?
http://wiki.micasaverde.com/index.php/View_Logfiles

guessed · May 22, 2011, 9:04pm

Yup, that’ll work also… for Windows folks 8)

It’s a little dated in it’s description since it refers to [tt]telnet[/tt] (etc), which used to be the “Default” way to get in. The SSH stuff you’ve documented is more common in the newer MiOS revs.

For Windows folks, they can also use winscp, that comes with the Putty toolset, to give them a semi-graphical way of pulling the files out of Vera to their Windows box (they can also use it to push stuff over)