[SOLVED] Vera3 Upgrading failed... not enough space

Vera Controllers, Services & Devices Vera Controllers & Cameras
#1

Filesystem Size Used Available Use% Mounted on /dev/root 4.5M 4.5M 0 100% /rom tmpfs 62.2M 6.8M 55.5M 11% /tmp tmpfs 512.0K 0 512.0K 0% /dev /dev/mtdblock7 11.0M 6.6M 4.4M 60% /overlay overlayfs:/overlay 11.0M 6.6M 4.4M 60% / /dev/mtdblock8 5.5M 5.5M 0 100% /mios

2016-08-25_17:49:16 -[21365]- ==fu== Report start firmware upgrade ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== BEGIN MiOS UPGRADE ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== Check if /tmp/newmios.squashfs exists ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== found ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== Check /tmp/newmios.squashfs md5sum ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== md5sum ok ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== Renaming old files created by mios_mount script from: /etc/cmh-firmware/1.7.760/ ==fu== 2016-08-25_17:49:18 -[21365]- ==fu== Remove old squashfs: /etc/cmh-firmware/mios.squashfs ==fu== 2016-08-25_17:49:22 -[21365]- ==fu== Check free space for copy: /tmp/newmios.squashfs to: /etc/cmh-firmware/mios.squashfs ==fu== 2016-08-25_17:49:22 -[21365]- ^[[1;33mWARNING: Waiting 0/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m 2016-08-25_17:49:52 -[21365]- ^[[1;33mWARNING: Waiting 1/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m 2016-08-25_17:50:22 -[21365]- ^[[1;33mWARNING: Waiting 2/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m 2016-08-25_17:50:52 -[21365]- ^[[1;33mWARNING: Waiting 3/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m 2016-08-25_17:51:22 -[21365]- ^[[1;31mERROR: Not enough free space, free=4476 KB, required=6136 KB^[[1;00m 2016-08-25_17:51:22 -[21365]- ==fu== Finished MiOS firmware upgrade with failure=1 ==fu== 2016-08-25_17:51:22 -[21365]- ^[[1;33mWARNING: Skipped reporting end firmware upgrade because StartUpgID is NULL^[[1;00m 2016-08-25_17:51:22 -[21365]- ^[[1;31mERROR: Firmware upgrade failed^[[1;00m 2016-08-25_17:51:22 -[21365]- ==fu== Rotating logs... ==fu== ~

Was trying to upgrade to the latest firmware and got a message that the upgade failed.

I looked around the logs and found the above info.

Anyone have any ideas on how to deal with this?

#2

I have exactly the same situation, both /mios and /rom directories reported as being full. If anyone knows which files can safely be deleted, that would be great. So far on all the posts I can see its a case of leaving it to support, which is very unsatisfactory.

A simple listing of the files which can be removed would allow this to be resolved.

#3

OK - I managed to upgrade… here’s what I did… cleared the vera, (after taking a backup). Then I did the firmware upgrade, and then restored my backup.

So dumb, but at least I didn’t have to wait for literally eternity to get a response from MCV.

#4

Well, my Vera Edge didn’t survive the upgrade somehow. It is in a continuous boot-loop. I guess they made a mess of the last firmware???

Jaap

#5

I have had a similar problem, but not quite. I’m getting LUUP startup failure for exit code 16 after “upgrading” to 1.7.855 - will not be doing this again… I can SSH into the Vera, but the UI is on the infinite spinner on the local net and unreachable on the WAN port even though it has an IP and I can ping out to external IPs – so L2-3 is up and running. Anyone know which directories to clear to free up space? I might resort to reflashing the firmware using the Vera Firmware Flash tool if MCV doesn’t respond soon…

I’ve tried a factory reset and even using the command line to reset the Vera, but it always comes back up in this state.

Hints and suggestions welcomed!

#6

/etc/cmh-firmware holds the “huge” (by Vera standards) firmware download file. That file can safely be deleted (somehow the upgrade scripts sometimes fail to do this after an upgrade).

But I do not know if this space issue is the root cause of your problems

#7

@logread
did you delete the mios.squashfs from the location you mentioned on your device? Is it stil working properly?

I have not managed to do one “normal” firmware upgrade since I went to UI7 on my VeraLite.
This last firmware I uploaded to the Veralite via WinSCP and started the upgrade procedure using SSH (got the instructions from Vera support).

I will try deleting the file you mentioned next time there is a new firmware.

#8

Yes. But again not sure this is the root cause

#9

For the record (who knows if it will help anyone) I have an old Vera Lite on which I haven’t yet applied the upgrade and this also shows /mios and /rom as 100% full. It is on UI7, but only on 1.7.758.

The UI is still responding and offering to upgrade to the latest UI7. (Ha! I don’t think so.)
My Vera Edge is just reporting the Lua engine startup problem since the upgrade. Great update guys. I presume this is why I still haven’t heard back from support.

Anyhow, 100% full looks like situation normal on /mios and /rom.

Just my $0.02.

#10

I’m getting the error code 16 error also, with the additional info that it can’t find file libjpeg.so.61, not sure why that would no longer be present unless the firmware has removed it. Any thoughts?

#11

Vera Support (John M, many thanks) has been able to login to my Edge and restore to Factory blank. As part of that he has fixed the error and all is well in the world again.

I had noticed that in one of the log files (/tmp/log.Init-LuaUPnP) an error message was showing that a library file ?libjpeg.so.62? as reported an issue. which was stopping Lua from initiating. Space was also cleared up which seems to have been at the root of the problems.

#12

Re: update fails with a message along the lines of “Critical error. Update date failed. Call support”. These comments refer to a Vera 3 and firmware 1.7.902:

The firmware updater does an initial check for space and may indicate there is not enough space or Datamine needs to be backed up and removed. If it finds it has enough space it will allow the update to proceed. After the user starts the update, the updater makes a back up file that is stored in /etc/cmh-backup/backup.tgz for its own purposes. It’s deleted at some point after the flash occurs. Refer to the updater script /usr/bin/cmh-upgrade.sh where backup-store.sh is executed on line 263. If you regularly refresh the directory /etc/cmh-backup/backup.tgz you will see the backup pop up during a firmware upgrade.

The backup file created can take up a lot of space, especially if the user has large plugins installed such as AltUI or PLEG. So for example AltUI will take up a total of 1,728 KB (actual measured value) - that’s the original files plus the files in the tarred zipped backup. So deleting all the AltUI files in /etc/cmh-ludl/ makes “twice” the difference. Note that on rebooting, the firmware will download any missing plugin files that where originally downloaded from the store (ie not those manually installed - they need to be backed up by you). Do deletions at your own risk.

The “Critical error. Update date failed. Call support” occurs because the available space is rechecked again after the back up is created. Refer to the updater script /usr/bin/cmh-upgrade.sh line 512. Looks like MCV thinks it’s a garbage collection problem when it’s actually huge backups causing problems.

It would be useful to check the backup size before the user initiates the firmware update by creating it earlier.

Another problem occurs in that the upgrade log file is destroyed immediately after the upgrade fails - see /tmp/log.upgrade, because the logs are rotated. Refer to /usr/bin/cmh-upgrade.sh line 695. So trying to work out what went wrong is not easy.

Here is a list of all the files (and folders) found in my backup file:

[code]…\Backup contents\etc\cmh
…\Backup contents\etc\cmh-ludl
…\Backup contents\etc\cmh-ra
…\Backup contents\etc\config
…\Backup contents\etc\crontabs
…\Backup contents\etc\dropbear
…\Backup contents\etc\ethers
…\Backup contents\etc\filelist.txt
…\Backup contents\etc\lighttpd.users
…\Backup contents\etc\mios_backup.info
…\Backup contents\etc\openwrt_release
…\Backup contents\etc\passwd
…\Backup contents\etc\TZ
…\Backup contents\etc\TZ-full
…\Backup contents\etc\cmh\alerts.json
…\Backup contents\etc\cmh\cmh.conf
…\Backup contents\etc\cmh\datetime
…\Backup contents\etc\cmh\devices
…\Backup contents\etc\cmh\dongle.3.20.dump.0
…\Backup contents\etc\cmh\dongle.3.20.dump.1
…\Backup contents\etc\cmh\dongle.dump
…\Backup contents\etc\cmh\ergy_key
…\Backup contents\etc\cmh\first_boot
…\Backup contents\etc\cmh\HW_Key
…\Backup contents\etc\cmh\HW_Key2
…\Backup contents\etc\cmh\keys
…\Backup contents\etc\cmh\last_report
…\Backup contents\etc\cmh\lighttpd_failures
…\Backup contents\etc\cmh\lock_log_levels
…\Backup contents\etc\cmh\PK_AccessPoint
…\Backup contents\etc\cmh\servers.conf.default
…\Backup contents\etc\cmh\srv_timestamp
…\Backup contents\etc\cmh\sync_kit
…\Backup contents\etc\cmh\sync_rediscover
…\Backup contents\etc\cmh\user_data.json.luup.lzo
…\Backup contents\etc\cmh\user_data.json.lzo
…\Backup contents\etc\cmh\user_data.json.lzo.1
…\Backup contents\etc\cmh\user_data.json.lzo.2
…\Backup contents\etc\cmh\user_data.json.lzo.3
…\Backup contents\etc\cmh\user_data.json.lzo.4
…\Backup contents\etc\cmh\user_data.json.lzo.5
…\Backup contents\etc\cmh\vera_model
…\Backup contents\etc\cmh\wan_failover
…\Backup contents\etc\cmh\zwave_locale
…\Backup contents\etc\cmh\wan_failover\check_internet.hosts

– all the files in \etc\cmh-ludl, which are the plugin files listed in the UI under “App—>Develop apps–>Lupp files”,
– are include in the back up. AltUI plugin is very large and PLEG is pretty big as well. They may cause problems.

…\Backup contents\etc\cmh-ra\cmh-ra.conf
…\Backup contents\etc\cmh-ra\keys
…\Backup contents\etc\cmh-ra\keys\cmh-ra-key.priv
…\Backup contents\etc\cmh-ra\keys\cmh-ra-key.pub
…\Backup contents\etc\config\ddns
…\Backup contents\etc\config\dhcp
…\Backup contents\etc\config\dropbear
…\Backup contents\etc\config\firewall
…\Backup contents\etc\config\luci
…\Backup contents\etc\config\luci_devinfo
…\Backup contents\etc\config\multiwan
…\Backup contents\etc\config\network
…\Backup contents\etc\config\ntpclient
…\Backup contents\etc\config\system
…\Backup contents\etc\config\timeserver
…\Backup contents\etc\config\ucitrack
…\Backup contents\etc\config\wireless
…\Backup contents\etc\config\wol
…\Backup contents\etc\crontabs\root
…\Backup contents\etc\dropbear\dropbear_dss_host_key
…\Backup contents\etc\dropbear\dropbear_rsa_host_key
[/code]

Here’s the upgrade log that almost succeeded. Note that the log reckons there is 8 KB to play with. It wasn’t enough:

[code]…
initial log stuff
…

2017-03-08_12:38:23 -[4498]- ==fu== BEGIN MiOS UPGRADE ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== Check if /tmp/newmios.squashfs exists ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== found ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== Check /tmp/newmios.squashfs md5sum ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== md5sum ok ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== Renaming old files created by mios_mount script from: /etc/cmh-firmware/1.7.902/ ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== Remove old squashfs: /etc/cmh-firmware/mios.squashfs ==fu==
2017-03-08_12:38:28 -[4498]- ==fu== Check free space for copy: /tmp/newmios.squashfs to: /etc/cmh-firmware/mios.squashfs ==fu==

…Note: we have 8 KB free!!!

2017-03-08_12:38:28 -[4498]- ==fu== We have 6120 KB jffs free space and required is 6112 KB ==fu==

2017-03-08_12:38:28 -[4498]- ==fu== Copy MiOS Squashfs into jffs2 partition ==fu==
cp: write error: No space left on device
cp: can’t preserve times of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve ownership of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve permissions of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
2017-03-08_12:38:42 -[4498]- e[1;31mERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 0 / 3. Wait 30 sece[1;00m
cp: write error: No space left on device
cp: can’t preserve times of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve ownership of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve permissions of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
2017-03-08_12:39:28 -[4498]- e[1;31mERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 1 / 3. Wait 30 sece[1;00m
cp: write error: No space left on device
cp: can’t preserve times of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve ownership of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve permissions of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
2017-03-08_12:40:13 -[4498]- e[1;31mERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 2 / 3. Wait 30 sece[1;00m
cp: write error: No space left on device
cp: can’t preserve times of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve ownership of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
cp: can’t preserve permissions of ‘/etc/cmh-firmware/mios.squashfs’: No space left on device
2017-03-08_12:40:58 -[4498]- e[1;31mERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 3 / 3. Wait 30 sece[1;00m
[/code]

#13

Thank you, all! This thread helped me tremendously when the Vera instructions on how to upgrade failed. (The Update button isn’t available) and all I had to do was clean out some crap to make enough room for the regular / normal update to work properly. Why Vera can’t break their updates down into a Part A and Part B is beyond me. Compress the images… download images / logos from their website to temp space / ram… compress some unneeded files, provide a “clear logs” button…something. Anything would have helped me get the 50KB that I needed to make this upgrade work with a lot less hassle.

#14

Will be taking note of this too