Capturing Vera logs to my own FTP server instead of Vera

Community Playground General Discussions
1

I have been a little frustrated with the lack of flexibility in how Vera logging. However, for some reason I could never easily capture a log entry of a problem I am trying to troubleshoot even with USB logging since for some reason only 1 archive of the log is kept. ( If someone knows of a simple way to keep 10 archives of logs on USB, please share).

In searching forums, I could not find if others have shared this previously. Taking advantage of the complete lack of security with vera around this, I discovered an extremely simple way of capturing all of logs on my server vs sending to Vera. This basically tricks Vera in to thinking your FTP server is their own.

  1. Log in to your Vera and copy the FTP login credentials with the logs section of /etc/cmh/servers.conf

[code]root@MiOS_PRIVATE:/etc/cmh# more servers.conf | grep Log

Log

Server_Log=vera-us-oem-log11.mios.com
Server_Log_Alt=vera-us-oem-log12.mios.com
Use_Server_Log_Alt=0
Server_Log_User=logs
Server_Log_Pass=XXXX[/code]

Using whatever FTP server you want to use, create a login with user “logs” and password with whatever you found in the above file.
In the FTP root directory, create a subdirectory named the serial# of your Vera found in the Unit Settings section or from the home portal . The name should just be the serial#. Make sure this is writable by the “logs” user.

Back on the Vera, edit the /etc/hosts file and add entries for the log server names and use your server IP address as the destination. I also added the old names for good measure :

root@MiOS_PRIVATE:/etc# cat /etc/hosts 127.0.0.1 localhost 192.168.14.94 local.mios.com local.my.mios.com 192.168.1.5 logs1.mios.com 192.168.1.5 logs2.mios.com 192.168.1.5 vera-us-oem-log11.mios.com 192.168.1.5 vera-us-oem-log12.mios.com root@MiOS_PRIVATE:/etc#

Next, if not already selected within the logs section of the Vera GUI, select “Archive old logs on MiOS (recommended)”

To test, force a log Rotate which should push logs to your server:

root@MiOS_PRIVATE:~# /usr/bin/Rotate_Logs.sh --force === Start rotating logs PID=8911, force rotate=1, free space=127232 KB, min=5000 KB, cmh logs size=40 KB, max=19154 KB, now=1459704372, last rotation=1459704355, diff=0 min 17 sec, max=12 hours === === Successfully uploaded logs for serial=PRIVATE, version=1.7.1754, server=vera-us-oem-log11.mios.com, date=2016-04-03_10-26-12 ===

That is all. You will start seeing logs populate every few minutes within your FTP server. Since there is no true server authentication taking place, your Vera will happily send your logs to your own server.

Note: I doubt this will survive a firmware upgrade (I have not tested yet). After a firmware upgrade you may need to update your /etc/hosts file again to point to your own server. If you ever want to restore back and archive to Vera, just comment out the entries in the /etc/hosts file.

Hope this helps

2

I took a slightly different tack.
Modify /etc/cmh/servers.conf

Log

Server_Log=ftp.ipaddress.whatever
#Server_Log_Alt=oem-log4.mios.com
Use_Server_Log_Alt=0
Server_Log_User=xxxxxx
Server_Log_Pass=xxxxxx

BUT this file might be re-written at Lua-restart. Or at least partially as I’ve just gone to write this and teh ip address has been altered back to the FQDN of the vera server. User / Pass were unaltered.

Also I had to modify Rotate_Logs.sh to account for the login location of my ftp server:

log “Sending file with CMD: curl -s -S -f --connect-timeout 60 --max-time 300 -u "$ftpUser:$ftpPass" -T "${logFile}.gz" "ftp://$ftpServer/array1/Catman/VeraLogs/${pkap}/${pkap}_$(echo $logFile | awk -F ‘/’ ‘{print $NF}’).gz"”
sendOk=0
try=0
while [ $((++try)) -le 3 ]; do
log “Try $try/3”
curl -s -S -f --connect-timeout 60 --max-time 300 -u “$ftpUser:$ftpPass” -T “${logFile}.gz” “ftp://$ftpServer/array1/Catman/VeraLogs/${pkap}/${pkap}_$(echo $logFile | awk -F ‘/’ ‘{print $NF}’).gz” 1>>$log_file 2>&1

I’ll check after the next restart. The resolv.conf option may be better!

Cheers

C

3

Hi,

After disabling the option Archive old logs on MiOS and since then it keeps ten log files in gz format. Not just the LuaUPnP.log but a whole bunch.

So if you just need an occasional check you can think of toggeling that option.

Cheers Rene

4

[quote=“reneboer, post:3, topic:191848”]Hi,

After disabling the option Archive old logs on MiOS and since then it keeps ten log files in gz format. Not just the LuaUPnP.log but a whole bunch.

So if you just need an occasional check you can think of toggeling that option.

Cheers Rene[/quote]

Indeed. Less of a problem now I have SSD but not a massive fan of the rotation anyway. Working out which one you need to gunzip etc etc. 10 logs is not that far on the normal rotation schedule either.
Under FTP I’ve changed (I hope) to 24 hour rotation and I’m getting datestamped logs up on my SAN which also makes it quick and easier to do analysis as the versions of grep etc on my other hosts are rather better than on the Plus itself :wink:

C

5

Oh I might make some changes to the naming format as well, and possibly the upload directory. Having every log prefaced with the unit name is clearly a win for Mios support. Not so much for a single user.
Datestamp postfix less useful as well but…

C

6

Several good ways ways to solve the problem. For me, it was easier to just tinker with the hosts file and only manage that file when FW updates invariably change the contents. On my server, I just had the FTP file root for the user to be the directory I wanted the logs deposited.

Good call on the increasing the rotation length since you have an SSD already. Reduces the number of files filling up the FTP server. Also, I have had a few instances where a script lock file sticks, and rotations fails until I manually delete it.

7

I’ve used scp with keys. Created a script in /tmp/log/cmh that copies the files and then executes rotate_logs. Then changed the cron to execute my script instead of rotate_logs. I also changed the time from every minute to every 5 minutes and everything works perfect.

8

My version (Florin posted while I was writing–our approaches are similar):

Goal: copy log files during normal log rotation to local *nix server via SCP, to unprivileged (with login/shell-disabled) account on server.

Prerequisites: Linux server to receive files, account created with restrictions mentioned above, and you have your SSH server set up for public key authentication (many have this on by default).

  1. SSH into Vera and make a new key for this purpose:
    [tt]dropbearkey -t rsa -f /etc/dropbear/rigpapa_rsa_key[/tt]
    This will create a file in /etc/dropbear/ with the private key, and print the public key on screen.

  2. Copy-pasta the public key into [tt]~/.ssh/authorized_keys[/tt] on the Linux server, creating the subdirectory and file if necessary (~ refers to the home directory of the account created to receive the files).

  3. Test (from Vera): [tt]scp /var/log/cmh/LuaUPnP.log user@host:~/[/tt]
    This should work without the need to enter a password for host.

  4. Create the following script file in [tt]/etc/cmh-ludl/[/tt]. I call mine [tt]/etc/cmh-ludl/local-upload-logs.sh[/tt] (and the balance of these instructions assumes that name):

[code]#!/bin/sh

Change the rigpapa_rsa_key to whatever filename you used in step 1.

scp -i /etc/dropbear/rigpapa_rsa_key /var/log/cmh/LuaUPnP.log user@host:~/LuaUPnP-date +%F.%T.log[/code]
Note the use of back-ticks around the [tt]date[/tt] command–that’s not a single quote, it’s ASCII 96. I recommend copy-paste from here to get the right character.

  1. Remove the symbolic link for the system’s [tt]Rotate_Logs.sh[/tt] and replace it with the real file so we can edit it (by default, it is linked to a file on a read-only filesystem):
    [tt]cd /usr/bin[/tt]
    [tt]cp Rotate_Logs.sh Rotate_Logs.sh-orig[/tt]
    [tt]rm Rotate_Logs.sh[/tt]
    [tt]mv Rotate_Logs.sh-orig Rotate_Logs.sh[/tt]

  2. Now modify the [tt]Rotate_Logs.sh[/tt] file to run the script created in step 4. Open the file in your favorite available text editor (I personally have no trouble with [tt]vi[/tt] as I have decodes of muscle-memory connected to it):
    a) Find the line that reads: [tt]log “Start log rotation”[/tt]
    b) Add the following line immediately after:
    [tt][ -x “/etc/cmh-ludl/local-upload-log.sh” ] && /etc/cmh-ludl/local-upload-log.sh[/tt]

  3. Make the rotate script executable: [tt]chmod a+rx /etc/cmh-ludl/local-upload-log.sh[/tt]

8.) Test: [tt]Rotate_Logs.sh 1[/tt] or [tt]Rotate_Logs.sh --force[/tt]
You should find a fresh copy of the log file on your Linux server in the target account’s home directory.

I put the actual copy command in a separate script because I often like to run it manually just to push the current log up to the server without actually doing a full rotation (so I can just type [tt]/etc/cmh-ludl/local-upload-logs.sh[/tt] as a command and done).

9

I was thinking of adding an FTP server to openLuup and redirecting Vera’s output to there. Could then be browsed in the same way as openLuup’s own logs.

10

My 2 cents: is there any option to change the ftp url to something local? So I can alter my internal DNS server to my ftp server? And optionally to change the ftp server address to my internally in dns registered ftp server…?

11

Thanks for detailing the solution :smiley:
The difference in my version is that I don’t touch the rotate_logs.sh script. I just call it at the end of my script (local-upload-logs.sh in your version) instead of modifying it to include my code.

And the crontab line
#*/1 * * * * /usr/bin/Rotate_Logs.sh #Rotate_Logs
was replaced with
*/5 * * * * /tmp/log/cmh/copy-log.sh #copy&rotate logs

I saw some old threads with a similar subject. Some users complained that by changing the rotate logs script, the logs either stopped rotating or they were touched every minute but not emptied. I figured I can skip some problems by letting that script untouched.

Copying works like a charms and already I have a ton of logs to parse.

12

Do you mean the Vera log server URL? Then yes, it’s trivial

Edit /etc/hosts on your Vera

local.ip.address oem-log3.mios.com
local.ip.address oem-log5.mios.com

You’ll need to check in /etc/cmh/servers.conf (Log section) that that’s the FQDN of the servers you’re configured to use.

Then you either set the user / pass on your local ftp server to the same as that specified in the Log section of servers.conf, or edit servers.conf to meet your user / pass on the ftp server.

So sounds actually a bit more than trivial, but it is really :slight_smile:

Does that help?

C

13

Hi, I meant that if akboer would make some alterations on vera I would be able to change the IP or DNS names of the vera logservers more easy (via altui) instead of chaging config files in vera…

14

Ahh I see!

C

15

Nice job on the SCP option. If only Vera used SCP to upload logs like you have done with the trivial changes, instead of the unencrypted FTP they have (and still use) for years to their servers. I have posted in the past a detailed breakdown of the type of data that is sent in the logs, and it is quite sensitive information (including security system pin codes)

16

If you’re uploading locally, it’s not (IMHO) a huge issue. Unless your data is encrypted at rest it’s readable by anyone that’s breached your LAN. Clearly if you’re talking about SCP to Vera’s servers then yes, it would be a better idea than FTP. Really surprised that FTP is still used by anyone.

C

17

I was referring to to MCV’s servers.
I am also much less concerned for a local lan which is why I never considered putting in the effort to change the scripts. The host file changes was most easily repairable if something breaks when I originally did this years ago. I hope that Vera takes your (and others here) ideas and moves to SCP or similar soon for their log backup service.

Two years ago I had to ‘prove’ to them that not everything leaving vera is secure and encrypted as they claim. Supposedly, there was going to be some analysis and maybe fix, but as of today, nothing has changed. It is still unencrypted FTP. I highly doubt MCV has any ‘security team’ if they feel after all this time, its perfectly acceptable to use FTP for logs.