BackUp Solutions/Suggestions for 8TB of Data?

Durahl

Member
Member
Greetings!

Today while casually browsing the iNet my Music suddenly stopped playing and while at first I believed it to be an issue with Xbox Music I later discovered that the Culprit was actually one of my RAID0 HDD's which decided to go AWOL - The Intel RST Manager confirmed this by showing that the 4th Drive ( I'm running two separate RAID0 with two drives each ) was missing.

A restart didn't solve the issue at first but physically removing the affected RAID and installing it back again using the SATA connectors of the other RAID0 for testing did. Interestingly enough even now that I installed everything back like it was before it appears to be fine so I'm not quite sure where I stand right now but suddenly I'm in the Market for BackUp Solutions, again.

Again? Yea... Every once in a while I test BackUp solutions but I'm never really happy with the outcome as with ~7TB of Data using USB3.0 or Ethernet it usually takes like forever to do such a task and this is where I usually go: "**** this rubbish..." /fliptable
I just crunched some numbers and, unless I'm terribly wrong, with the amount of Data I have it would even take me ~6h of BackUp time on a 420MB/sec ThunderBold 2 BackUp solution which is not possible as I cannot install the additional required Hardware thus making me cringe at the thought of using either USB3.0 or Ethernet which is magnitudes slower -_-

Right now I'm considering in converting my second PC into a BackUp System itself by getting large enough HDD's, install them into the primary PC first in order to quickly copy all the files and then move them into the secondary PC.
From there one I plan to use BTSync which is technically not a BackUp software but pretty much behaves like one as it simply copies all files of a selected folder from the primary PC to the secondary PC which it cannot find there in an actual version.

And it's easy to use as well.

So what's your opinion on this plan?
Maybe other suggestions?

Thanks in Advance :)
 

My Computer

System One

  • OS
    Microsoft Windows 10 Professional
    Computer type
    PC/Desktop
    System Manufacturer/Model
    Homemade
    CPU
    Intel Core i7-5930K
    Motherboard
    ASUS RAMPAGE V Extreme
    Memory
    32GB Corsair Dominator Platinum
    Graphics Card(s)
    2x EVGA GTX 980 SC
    Sound Card
    Creative Sound Blaster ZxR
    Monitor(s) Displays
    3x Dell UltraSharp U3011 / 1x Wacom Cintiq 24HD Touch
    Screen Resolution
    3x 2560x1600 / 1x 1920x1200
    Hard Drives
    1x Kingston HyperX Predator 480GB M.2 SSD
    4x Western Digital Caviar Green 1TB
    PSU
    Corsair AX1200i Digital ATX Power Supply
    Case
    Corsair 900D
    Cooling
    EKWB / Aquacomputer Custom Water Cooling
    Keyboard
    2x Logitech G710+ Mechanical Gaming Keyboard / 1x Logitech G13 Advanced Gameboard
    Mouse
    Logitech G700 Wireless Gaming Mouse / 3DConnexion Space Pilot PRO
    Internet Speed
    250'000 / 15'000kbit
    Browser
    Microsoft Internet Explorer 11
    Antivirus
    YOLO
Having that much data is going to be very hard to backup. I'm guessing that you have a ton of video, which isn't going to be changing all that often. Unfortunately, if this is the case, video is already compressed so you won't gain any speed benefits from compression in backup software.

Even if you had a ThunderBolt2 option, you would have to have source drives that could provide reading your data that fast and backup drives that could write the data that fast. This is where your bottleneck is going to likely be if you plan to backup faster than 100MB/sec.

For my home backups, I use robocopy to simply make a second copy of my data onto a pair of external drives, which I then keep offsite. The first time the copy is run, it takes a long time as it copies everything. All subsequent runs are much faster because it only copies the differences in the files. So, my backup drive looks just like my source drive. I use the /MIR switch which mirrors the files, so if I delete a file from the source, it deletes from the destination.

So, if I were you, I would
*Buy a handful of 2TB of 4TB hard drives
*Buy an external USB 3.0 dock
*Plug in a drive, and run robocopy against a source partition (that will fit on the external hard drive).
*Suffer through a very long backup the first time with robocopy.
*Every month or so, plug the drives back in and rerun the same robocopy command and let it gather any changes (which probably won't be extensive and will finish in just a few minutes)
*try to keep your backup drives offsite if possible. Take them to work with you, store at a friends house, etc. (if your house burns down, a backup stored onsite might do no good).
 

My Computer

System One

  • OS
    Windows 7
    System Manufacturer/Model
    Self-Built in July 2009
    CPU
    Intel Q9550 2.83Ghz OC'd to 3.40Ghz
    Motherboard
    Gigabyte GA-EP45-UD3R rev. 1.1, F12 BIOS
    Memory
    8GB G.Skill PI DDR2-800, 4-4-4-12 timings
    Graphics Card(s)
    EVGA 1280MB Nvidia GeForce GTX570
    Sound Card
    Realtek ALC899A 8 channel onboard audio
    Monitor(s) Displays
    23" Acer x233H
    Screen Resolution
    1920x1080
    Hard Drives
    Intel X25-M 80GB Gen 2 SSD
    Western Digital 1TB Caviar Black, 32MB cache. WD1001FALS
    PSU
    Corsair 620HX modular
    Case
    Antec P182
    Cooling
    stock
    Keyboard
    ABS M1 Mechanical
    Mouse
    Logitech G9 Laser Mouse
    Internet Speed
    15/2 cable modem
    Other Info
    Windows and Linux enthusiast. Logitech G35 Headset.
As the last poster mentioned, that much data is going to take a long time to make a fresh backup. If your data must be online all the time then you might want to look into a RAID1 or RAID5 type setup as it gives you a bit more redundancy.

As for off site, your best bet with that much data is as he noted, a good option might be to have a nas at some other location that your can sync/back to. This way the changes are migrated over as they happen or during a nightly job.

I can not say I have ever used robocopy, but syncbackpro is my favourite, it works with a number of services and protocols. Additionally it can integrate with the windows task manager to make backups an automated event.
 

My Computer

System One

  • OS
    Windows 8.1
Thanks!

Thanks for the input!

Going offsite is not something I'm considering at this point as I don't really have that option.
I was actually more looking for something that would just simply work with my current setup.
Should have clarified this.

Last time I tried external HDD's or an actual NAS I could afford I became increasingly frustrated at the terrible transfer rates and/or the overly complicated user interface which is why I'm trying to avoid such systems as best as I can or else I'm expecting to get the same outcome as last time - Not backing up at all *duh*

For now I decided to buy two 4TB Drives and use my secondary PC as a BackUp solution.
Once everything is ready I'm going to test Robocopy and see how well this performs as compared to BTSync which I'm currently using to get my daily fix of Animation from my primary work/storage PC to my secondary multimedia PC which can only hold about a week of video files worth.

On a little side-note:
I was just rereading some information on External TB2 HDD's which kind of bothered me before I even opened this thread.
The HDD's I was looking into were listing Transfer rates at ~420MB/s which did not quite compute with the 2'560MB/s ( 20Gb/s ) TB2 is supposed to deliver - I just now realized that first one is bottleneck of the HDD's while the other is the pass-through for daisy-chaining.

Needs more coffee I guess ;)
 

My Computer

System One

  • OS
    Microsoft Windows 10 Professional
    Computer type
    PC/Desktop
    System Manufacturer/Model
    Homemade
    CPU
    Intel Core i7-5930K
    Motherboard
    ASUS RAMPAGE V Extreme
    Memory
    32GB Corsair Dominator Platinum
    Graphics Card(s)
    2x EVGA GTX 980 SC
    Sound Card
    Creative Sound Blaster ZxR
    Monitor(s) Displays
    3x Dell UltraSharp U3011 / 1x Wacom Cintiq 24HD Touch
    Screen Resolution
    3x 2560x1600 / 1x 1920x1200
    Hard Drives
    1x Kingston HyperX Predator 480GB M.2 SSD
    4x Western Digital Caviar Green 1TB
    PSU
    Corsair AX1200i Digital ATX Power Supply
    Case
    Corsair 900D
    Cooling
    EKWB / Aquacomputer Custom Water Cooling
    Keyboard
    2x Logitech G710+ Mechanical Gaming Keyboard / 1x Logitech G13 Advanced Gameboard
    Mouse
    Logitech G700 Wireless Gaming Mouse / 3DConnexion Space Pilot PRO
    Internet Speed
    250'000 / 15'000kbit
    Browser
    Microsoft Internet Explorer 11
    Antivirus
    YOLO
There is no external hard drive that's going to deliver 420MB/sec. Unless it's an SSD.
 

My Computer

System One

  • OS
    Windows 7
    System Manufacturer/Model
    Self-Built in July 2009
    CPU
    Intel Q9550 2.83Ghz OC'd to 3.40Ghz
    Motherboard
    Gigabyte GA-EP45-UD3R rev. 1.1, F12 BIOS
    Memory
    8GB G.Skill PI DDR2-800, 4-4-4-12 timings
    Graphics Card(s)
    EVGA 1280MB Nvidia GeForce GTX570
    Sound Card
    Realtek ALC899A 8 channel onboard audio
    Monitor(s) Displays
    23" Acer x233H
    Screen Resolution
    1920x1080
    Hard Drives
    Intel X25-M 80GB Gen 2 SSD
    Western Digital 1TB Caviar Black, 32MB cache. WD1001FALS
    PSU
    Corsair 620HX modular
    Case
    Antec P182
    Cooling
    stock
    Keyboard
    ABS M1 Mechanical
    Mouse
    Logitech G9 Laser Mouse
    Internet Speed
    15/2 cable modem
    Other Info
    Windows and Linux enthusiast. Logitech G35 Headset.
Hi there

You WILL need an initial full backup - you can't avoid that - and depending on the nature of the music / multi-media files then the amount of compression you'd get from a backup might be small to non. I have nearly all my music as FLAC so the backup compresses it to around 55-60% but for mp3 etc which is already compressed the gain is smaller.

Afterwards simply use what's known as "Incrementals" (or some people might call them "Differentials") - not sure what the correct term is but decent professional backup products will produce a backup simply of changed and new files since the last full backup. With things like music I doubt if you make a huge amount of changes.

The other idea is to split the library up into "manageable pieces" - say by category, alphabetical order, favourite composers - or whatever and back these up in turns.

Over the years I have got a load of laptop HDD's which were replaced with SSD's etc. These are slow etc but IDEAL for storing backups - I just attach them to the computer via a USB==>sata connection. These are 320GB each so I try and break my library down into sections that are about 200GB each. You can get the smaller passport type USB3 drives that are 2TB as well which make for good backup media.

For this type of Backup -- actually we should really call it an ARCHIVE - there is a difference !! the speed and performance of the HDD's doesn't matter - you just want to STORE your data reliably. Banks and other big businesses often have whole departments dealing with this topic so it's not an easy one - especially for individuals.

Split the task down -- WHAT do I need to backup, What will I use to back it up, how do I organise the data into manageable pieces and how to I organise the backup media once I've got the backup.

VERY IMPORTANT -- if you are backing up to loads of media (I've got 10 X these old laptop HDD's) then LABEL THEM CAREFULLY - both SOFTWARE WISE and PHYSICALLY - with a felt pen / sticky (removable) label etc - and ideally keep details in a spreadsheet as a catalog.

Archiving large (at least by our standards as an individual users) volumes of data is a topic rarely touched upon and there really isn't currently any decent "management system" around that can handle this process so currently we have to make do with "backup software" and manual handling - but it's the best we can do currently.

I'm surprised that nobody has come up with say an idea where I could stick say the 10 ex laptop (or any other set of disks) disks in a box rather like the old tape systems and provide software which would automatically manage the archive - but how about it - any young budding entrepreneur out there. !! There are commercial solutions but I'm speaking of cheap reliable systems that us as CONSUMERS could use.

I don't think though using the Cloud for several TB of data is either a practical or cost effective solution.

(I've found also a decent source of HDD's is to go to the tip and see if there are any discarded computers -- you could pick up a few 500 GB HDD's for ZERO !!!! - remember all you want are these for ARCHIVE - they don't have to be fast etc - and run the INITIAL archive overnight - you van pause it during the day if required - until you've got the initial set.

It's NOT a trivial topic BTW - hopefully people will add to this thread if they have better ways of archiving LARGE volumes of data !!!

Note - this data is ARCHIVE DATA - I'm not referring to the music etc you have ONLINE - that's a different topic as to whether you are using NAS / RAID or whatever. My post here deals only with COPYING what you already have so you can recover it if required.

Cheers
jimbo
 

My Computer

System One

  • OS
    Linux Centos 7, W8.1, W7, W2K3 Server W10
    Computer type
    PC/Desktop
    Monitor(s) Displays
    1 X LG 40 inch TV
    Hard Drives
    SSD's * 3 (Samsung 840 series) 250 GB
    2 X 3 TB sata
    5 X 1 TB sata
    Internet Speed
    0.12 GB/s (120Mb/s)
Afterwards simply use what's known as "Incrementals" (or some people might call them "Differentials") - not sure what the correct term is but decent professional backup products will produce a backup simply of changed and new files since the last full backup. With things like music I doubt if you make a huge amount of changes.

An incremental backup means the 1st backup is a full, the second backup is just stuff that changed since the first full backup, the third backup is what has changed since the second backup, the forth backup is what changed since the third backup, and so on.

A differential backup means the 1st backup is a full, the second backup is stuff that changed since the last full backup. So, thus far the same as an incremental. However, the third backup is again everything that changed since the first full backup (so it includes the stuff from the 2nd and the 3rd backup. The forth backup, would include the 2nd, 3rd and 4th.

Incrementals run the fastest, since it's just the changes since the last incremental, but restores are slower as you have to restore all of them.

Differentials take longer to backup and consume more space, but are faster to restore...since you restore only the full backup and the last differential.
 

My Computer

System One

  • OS
    Windows 7
    System Manufacturer/Model
    Self-Built in July 2009
    CPU
    Intel Q9550 2.83Ghz OC'd to 3.40Ghz
    Motherboard
    Gigabyte GA-EP45-UD3R rev. 1.1, F12 BIOS
    Memory
    8GB G.Skill PI DDR2-800, 4-4-4-12 timings
    Graphics Card(s)
    EVGA 1280MB Nvidia GeForce GTX570
    Sound Card
    Realtek ALC899A 8 channel onboard audio
    Monitor(s) Displays
    23" Acer x233H
    Screen Resolution
    1920x1080
    Hard Drives
    Intel X25-M 80GB Gen 2 SSD
    Western Digital 1TB Caviar Black, 32MB cache. WD1001FALS
    PSU
    Corsair 620HX modular
    Case
    Antec P182
    Cooling
    stock
    Keyboard
    ABS M1 Mechanical
    Mouse
    Logitech G9 Laser Mouse
    Internet Speed
    15/2 cable modem
    Other Info
    Windows and Linux enthusiast. Logitech G35 Headset.
The way I'm planning for this to be done is just copying the files 1:1 without any form of compression from one PC to another using a daily scheduled Robocopy /mir task with the added benefit of being able to use the files directly on the 2nd PC which should come in handy in it's own way. Loosing half my storage capacity for peace of mind and accessibility - Ah well...

I guess I'll be able to start tomorrow once the HDD's have arrived by Mail.
Got to figure out how to automate this on both PC's for both being awake and ready when it's time.

Still wondering what has happened anyway... Everything's fine now.
 

My Computer

System One

  • OS
    Microsoft Windows 10 Professional
    Computer type
    PC/Desktop
    System Manufacturer/Model
    Homemade
    CPU
    Intel Core i7-5930K
    Motherboard
    ASUS RAMPAGE V Extreme
    Memory
    32GB Corsair Dominator Platinum
    Graphics Card(s)
    2x EVGA GTX 980 SC
    Sound Card
    Creative Sound Blaster ZxR
    Monitor(s) Displays
    3x Dell UltraSharp U3011 / 1x Wacom Cintiq 24HD Touch
    Screen Resolution
    3x 2560x1600 / 1x 1920x1200
    Hard Drives
    1x Kingston HyperX Predator 480GB M.2 SSD
    4x Western Digital Caviar Green 1TB
    PSU
    Corsair AX1200i Digital ATX Power Supply
    Case
    Corsair 900D
    Cooling
    EKWB / Aquacomputer Custom Water Cooling
    Keyboard
    2x Logitech G710+ Mechanical Gaming Keyboard / 1x Logitech G13 Advanced Gameboard
    Mouse
    Logitech G700 Wireless Gaming Mouse / 3DConnexion Space Pilot PRO
    Internet Speed
    250'000 / 15'000kbit
    Browser
    Microsoft Internet Explorer 11
    Antivirus
    YOLO
I prefer just having the raw files in a second location as well. I think your robocopy task will take care of this problem perfectly for you.

Storage is cheap, losing your files is terrible. If it's worth having the data, it's worth having a backup. And if you have data that's not worth backing up, then don't. For example, if you lose some TV shows or movies...it's likely not the end of the worth. If you lose your entire digital picture library of your children, that is a different story altogether.
 

My Computer

System One

  • OS
    Windows 7
    System Manufacturer/Model
    Self-Built in July 2009
    CPU
    Intel Q9550 2.83Ghz OC'd to 3.40Ghz
    Motherboard
    Gigabyte GA-EP45-UD3R rev. 1.1, F12 BIOS
    Memory
    8GB G.Skill PI DDR2-800, 4-4-4-12 timings
    Graphics Card(s)
    EVGA 1280MB Nvidia GeForce GTX570
    Sound Card
    Realtek ALC899A 8 channel onboard audio
    Monitor(s) Displays
    23" Acer x233H
    Screen Resolution
    1920x1080
    Hard Drives
    Intel X25-M 80GB Gen 2 SSD
    Western Digital 1TB Caviar Black, 32MB cache. WD1001FALS
    PSU
    Corsair 620HX modular
    Case
    Antec P182
    Cooling
    stock
    Keyboard
    ABS M1 Mechanical
    Mouse
    Logitech G9 Laser Mouse
    Internet Speed
    15/2 cable modem
    Other Info
    Windows and Linux enthusiast. Logitech G35 Headset.
@Durahl,
Calculate your power-consumption carefully. Use the maximum for disk-spinup.
Have a look at the maimum throughput of your Fuses.
The intermittant problem you described sounds like you encountered such a problem.
Regards,Leon
 

My Computer

System One

  • OS
    Windows 8.1 (355 + Auto-Update)
    Computer type
    Laptop
    System Manufacturer/Model
    Toshiba U500 17E
    CPU
    Intel Core 2 Duo T6600 2.2 Ghz
    Motherboard
    ?
    Memory
    4 GB
    Graphics Card(s)
    NVIDIA GeForce G210M
    Internet Speed
    180Mb/s
    Browser
    Internet Explorer 11.0.7 (auto update)
    Antivirus
    Norton Internet Security
Got to figure out how to automate this on both PC's for both being awake and ready when it's time.
You could use task scheduler to run a powershell script every hour of whatever with

program %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
arguments -noprofile -executionpolicy bypass -File "C:\Users\adam\Powershell\ScriptName.ps1"

make the script send a wake on lan packet, wait for response to ping and then issue your robocopy. Something like this (I've not tested it but do similar things like this)
Code:
[COLOR=#008000]## Server details[/COLOR]
$IPAddress="192.168.1.82"
$mac="00:11:22:33:44:55"
$port=9

[COLOR=#008000]## Wake server if not already running[/COLOR]
$Ping=New-ObjectSystem.Net.NetworkInformation.Ping
$webClient=New-ObjectSystem.Net.WebClient
$Echo=$Ping.Send($IPAddress)
if ($Echo.Status.ToString() -ne"Success")
{
  $broadcast=[Net.IPAddress]::Parse($IPAddress) 
  $mac=(($mac.replace(":","")).replace("-","")).replace(".","") 
  $target=0,2,4,6,8,10|% {[convert]::ToByte($mac.substring($_,2),16)} 
  $packet= (,[byte]255*6) + ($target*16) 
  $UDPclient=new-ObjectSystem.Net.Sockets.UdpClient
  $UDPclient.Connect($broadcast,$port) 
  [void]$UDPclient.Send($packet,102) 
}

[COLOR=#008000]## Wait for echo[/COLOR]
$Activity="Waking server"
$Status="Waiting for server to respond to echo"
$j=1
$TimeOut=30
$TriesPerSecond=2
do
{    
  $Echo=$Ping.Send($IPAddress)
  Write-Progress-ID2-ParentID1-Activity$Activity-Status$Status-PercentComplete (($j/$TriesPerSecond*100)/$TimeOut)
  Sleep (1/$TriesPerSecond)
  if ($j-eq$TimeOut*$TriesPerSecond)
  {
    Write-Host"Time out expired, aborting."
    Write-Host"Press any key to continue ..."
    $x=$host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
    exit
  }
  $j++
} while ($Echo.Status.ToString() -ne"Success")

[COLOR=#008000]## Issue robocopy[/COLOR]
$SourceDir="C:\Users\adam\SkyDrive"
$DestinationDir="E:\SkyDrive"
robocopy $SourceDir $DestinationDir /b /e /r:0 /w:0 /copyall /mt /log:$DestinationDir\logname.log
 

My Computer

System One

  • OS
    Windows 10 Pro Prieview x64
    Computer type
    Laptop
    System Manufacturer/Model
    MacBook Pro Core2Duo
    CPU
    T7600
    Memory
    3
    Graphics Card(s)
    ATI Radeon X1600
    Monitor(s) Displays
    Internal
    Screen Resolution
    1440 x 800
    Hard Drives
    40GB
    Keyboard
    Apple
    Mouse
    Apple
    Internet Speed
    Varies
    Browser
    Various
    Antivirus
    Defender
Afterwards simply use what's known as "Incrementals" (or some people might call them "Differentials") - not sure what the correct term is but decent professional backup products will produce a backup simply of changed and new files since the last full backup. With things like music I doubt if you make a huge amount of changes.

An incremental backup means the 1st backup is a full, the second backup is just stuff that changed since the first full backup, the third backup is what has changed since the second backup, the forth backup is what changed since the third backup, and so on.

A differential backup means the 1st backup is a full, the second backup is stuff that changed since the last full backup. So, thus far the same as an incremental. However, the third backup is again everything that changed since the first full backup (so it includes the stuff from the 2nd and the 3rd backup. The forth backup, would include the 2nd, 3rd and 4th.

Incrementals run the fastest, since it's just the changes since the last incremental, but restores are slower as you have to restore all of them.

Differentials take longer to backup and consume more space, but are faster to restore...since you restore only the full backup and the last differential.


Hi there

Thanks for the explanation.

Both systems obviously have their merits and disadvantages -- I'm used to the Incremental method - but especially for things like music that rarely change the Differential scheme might be better -- although it could get confusing using two different methods. For the OS though I always Image it fully.

Cheers
jimbo
 

My Computer

System One

  • OS
    Linux Centos 7, W8.1, W7, W2K3 Server W10
    Computer type
    PC/Desktop
    Monitor(s) Displays
    1 X LG 40 inch TV
    Hard Drives
    SSD's * 3 (Samsung 840 series) 250 GB
    2 X 3 TB sata
    5 X 1 TB sata
    Internet Speed
    0.12 GB/s (120Mb/s)
My personal favorite free backup utility can be found here:

Personal-Backup

Personal-Backup_1.png

Personal-Backup_2.png

Personal-Backup_3.png
 

My Computer

System One

  • OS
    Win8 Pro
Back
Top