Guys, I'm bringing down TerraStorage to attempt a HDD recovery.
Oolite-update service will be unavailable.
I'll post an update later tonight.
I have managed to take a backup of the VMs and the latest scripts. However, if I could get an image of the source disk, it would avoid me the trouble of reinstalling everything from scratch. If you are wondering about the previous backups I have taken, well, the first one is on a disk that died and the second one is currently being taken.
The rate of imaging is dramatically low; without errors but we are talking about less than 128KB/s.
Unfortunately all the disks I have are in the Methuselah age category.
I am trying to get my hands to another/younger target disk to check if the issue is there.
I hope it is not due to the source disk.
No available hdd found yet to pinpoint the issue.
Good news. Just got a spare hdd to proceed with the troubleshooting.
Due to R/L priorities, I will be taking over these tasks this Monday.
Better news! It seems that the problem was not with worn out hdd,
but with some faulty Molex to SATA power cable adapters.
Currently cloning the server's disk.
Not THAT much better news. I managed to make a clone of the system disk.
I left it this morning trying to take a second clone.
It seems that at some point it finds some difficulty reading the source.
This means that the main disk will be dead soon.
I will try the clones the next couple of days and then will most probably proceed to disk replacement.
EDIT 2017-10-05 - Part II
I estimate that it will take 12 more hours to complete the second cloning.
This time I will let it complete. In the meantime I am looking into getting a disk.
The worst case scenario (please don't!) will call for three disks; one live, the second for backup and
the third for disaster recovery, physically far away from the server.
Second clone finished faster. This gave me the opportunity to test it as a primary before last nightly execution.
Currently this clone is the live one and I'm validating it while performing some operational tasks.
OK. The two disk-clones seem to be healthy. Today the source disk has been restored as live and terrastorage will remain up till the new replacement disk comes in.