Windows 8 and 8.1 Forums

BackUp Solutions/Suggestions for 8TB of Data?

  1. #1

    Nearby... In an astronomical sense.
    Posts : 43
    Microsoft Windows 10 Professional

    BackUp Solutions/Suggestions for 8TB of Data?


    Today while casually browsing the iNet my Music suddenly stopped playing and while at first I believed it to be an issue with Xbox Music I later discovered that the Culprit was actually one of my RAID0 HDD's which decided to go AWOL - The Intel RST Manager confirmed this by showing that the 4th Drive ( I'm running two separate RAID0 with two drives each ) was missing.

    A restart didn't solve the issue at first but physically removing the affected RAID and installing it back again using the SATA connectors of the other RAID0 for testing did. Interestingly enough even now that I installed everything back like it was before it appears to be fine so I'm not quite sure where I stand right now but suddenly I'm in the Market for BackUp Solutions, again.

    Again? Yea... Every once in a while I test BackUp solutions but I'm never really happy with the outcome as with ~7TB of Data using USB3.0 or Ethernet it usually takes like forever to do such a task and this is where I usually go: "**** this shit..." /fliptable
    I just crunched some numbers and, unless I'm terribly wrong, with the amount of Data I have it would even take me ~6h of BackUp time on a 420MB/sec ThunderBold 2 BackUp solution which is not possible as I cannot install the additional required Hardware thus making me cringe at the thought of using either USB3.0 or Ethernet which is magnitudes slower -_-

    Right now I'm considering in converting my second PC into a BackUp System itself by getting large enough HDD's, install them into the primary PC first in order to quickly copy all the files and then move them into the secondary PC.
    From there one I plan to use BTSync which is technically not a BackUp software but pretty much behaves like one as it simply copies all files of a selected folder from the primary PC to the secondary PC which it cannot find there in an actual version.

    And it's easy to use as well.

    So what's your opinion on this plan?
    Maybe other suggestions?

    Thanks in Advance :)

      My System SpecsSystem Spec

  2. #2

    Having that much data is going to be very hard to backup. I'm guessing that you have a ton of video, which isn't going to be changing all that often. Unfortunately, if this is the case, video is already compressed so you won't gain any speed benefits from compression in backup software.

    Even if you had a ThunderBolt2 option, you would have to have source drives that could provide reading your data that fast and backup drives that could write the data that fast. This is where your bottleneck is going to likely be if you plan to backup faster than 100MB/sec.

    For my home backups, I use robocopy to simply make a second copy of my data onto a pair of external drives, which I then keep offsite. The first time the copy is run, it takes a long time as it copies everything. All subsequent runs are much faster because it only copies the differences in the files. So, my backup drive looks just like my source drive. I use the /MIR switch which mirrors the files, so if I delete a file from the source, it deletes from the destination.

    So, if I were you, I would
    *Buy a handful of 2TB of 4TB hard drives
    *Buy an external USB 3.0 dock
    *Plug in a drive, and run robocopy against a source partition (that will fit on the external hard drive).
    *Suffer through a very long backup the first time with robocopy.
    *Every month or so, plug the drives back in and rerun the same robocopy command and let it gather any changes (which probably won't be extensive and will finish in just a few minutes)
    *try to keep your backup drives offsite if possible. Take them to work with you, store at a friends house, etc. (if your house burns down, a backup stored onsite might do no good).
      My System SpecsSystem Spec

  3. #3

    Posts : 16
    Windows 8.1

    As the last poster mentioned, that much data is going to take a long time to make a fresh backup. If your data must be online all the time then you might want to look into a RAID1 or RAID5 type setup as it gives you a bit more redundancy.

    As for off site, your best bet with that much data is as he noted, a good option might be to have a nas at some other location that your can sync/back to. This way the changes are migrated over as they happen or during a nightly job.

    I can not say I have ever used robocopy, but syncbackpro is my favourite, it works with a number of services and protocols. Additionally it can integrate with the windows task manager to make backups an automated event.
      My System SpecsSystem Spec

  4. #4

    Nearby... In an astronomical sense.
    Posts : 43
    Microsoft Windows 10 Professional


    Thanks for the input!

    Going offsite is not something I'm considering at this point as I don't really have that option.
    I was actually more looking for something that would just simply work with my current setup.
    Should have clarified this.

    Last time I tried external HDD's or an actual NAS I could afford I became increasingly frustrated at the terrible transfer rates and/or the overly complicated user interface which is why I'm trying to avoid such systems as best as I can or else I'm expecting to get the same outcome as last time - Not backing up at all *duh*

    For now I decided to buy two 4TB Drives and use my secondary PC as a BackUp solution.
    Once everything is ready I'm going to test Robocopy and see how well this performs as compared to BTSync which I'm currently using to get my daily fix of Animation from my primary work/storage PC to my secondary multimedia PC which can only hold about a week of video files worth.

    On a little side-note:
    I was just rereading some information on External TB2 HDD's which kind of bothered me before I even opened this thread.
    The HDD's I was looking into were listing Transfer rates at ~420MB/s which did not quite compute with the 2'560MB/s ( 20Gb/s ) TB2 is supposed to deliver - I just now realized that first one is bottleneck of the HDD's while the other is the pass-through for daisy-chaining.

    Needs more coffee I guess ;)
      My System SpecsSystem Spec

  5. #5

    There is no external hard drive that's going to deliver 420MB/sec. Unless it's an SSD.
      My System SpecsSystem Spec

  6. #6

    Hafnarfjörður IS
    Posts : 4,376
    Linux Centos 7, W8.1, W7, W2K3 Server W10

    Hi there

    You WILL need an initial full backup - you can't avoid that - and depending on the nature of the music / multi-media files then the amount of compression you'd get from a backup might be small to non. I have nearly all my music as FLAC so the backup compresses it to around 55-60% but for mp3 etc which is already compressed the gain is smaller.

    Afterwards simply use what's known as "Incrementals" (or some people might call them "Differentials") - not sure what the correct term is but decent professional backup products will produce a backup simply of changed and new files since the last full backup. With things like music I doubt if you make a huge amount of changes.

    The other idea is to split the library up into "manageable pieces" - say by category, alphabetical order, favourite composers - or whatever and back these up in turns.

    Over the years I have got a load of laptop HDD's which were replaced with SSD's etc. These are slow etc but IDEAL for storing backups - I just attach them to the computer via a USB==>sata connection. These are 320GB each so I try and break my library down into sections that are about 200GB each. You can get the smaller passport type USB3 drives that are 2TB as well which make for good backup media.

    For this type of Backup -- actually we should really call it an ARCHIVE - there is a difference !! the speed and performance of the HDD's doesn't matter - you just want to STORE your data reliably. Banks and other big businesses often have whole departments dealing with this topic so it's not an easy one - especially for individuals.

    Split the task down -- WHAT do I need to backup, What will I use to back it up, how do I organise the data into manageable pieces and how to I organise the backup media once I've got the backup.

    VERY IMPORTANT -- if you are backing up to loads of media (I've got 10 X these old laptop HDD's) then LABEL THEM CAREFULLY - both SOFTWARE WISE and PHYSICALLY - with a felt pen / sticky (removable) label etc - and ideally keep details in a spreadsheet as a catalog.

    Archiving large (at least by our standards as an individual users) volumes of data is a topic rarely touched upon and there really isn't currently any decent "management system" around that can handle this process so currently we have to make do with "backup software" and manual handling - but it's the best we can do currently.

    I'm surprised that nobody has come up with say an idea where I could stick say the 10 ex laptop (or any other set of disks) disks in a box rather like the old tape systems and provide software which would automatically manage the archive - but how about it - any young budding entrepreneur out there. !! There are commercial solutions but I'm speaking of cheap reliable systems that us as CONSUMERS could use.

    I don't think though using the Cloud for several TB of data is either a practical or cost effective solution.

    (I've found also a decent source of HDD's is to go to the tip and see if there are any discarded computers -- you could pick up a few 500 GB HDD's for ZERO !!!! - remember all you want are these for ARCHIVE - they don't have to be fast etc - and run the INITIAL archive overnight - you van pause it during the day if required - until you've got the initial set.

    It's NOT a trivial topic BTW - hopefully people will add to this thread if they have better ways of archiving LARGE volumes of data !!!

    Note - this data is ARCHIVE DATA - I'm not referring to the music etc you have ONLINE - that's a different topic as to whether you are using NAS / RAID or whatever. My post here deals only with COPYING what you already have so you can recover it if required.

      My System SpecsSystem Spec

  7. #7

    Quote Originally Posted by jimbo45 View Post
    Afterwards simply use what's known as "Incrementals" (or some people might call them "Differentials") - not sure what the correct term is but decent professional backup products will produce a backup simply of changed and new files since the last full backup. With things like music I doubt if you make a huge amount of changes.
    An incremental backup means the 1st backup is a full, the second backup is just stuff that changed since the first full backup, the third backup is what has changed since the second backup, the forth backup is what changed since the third backup, and so on.

    A differential backup means the 1st backup is a full, the second backup is stuff that changed since the last full backup. So, thus far the same as an incremental. However, the third backup is again everything that changed since the first full backup (so it includes the stuff from the 2nd and the 3rd backup. The forth backup, would include the 2nd, 3rd and 4th.

    Incrementals run the fastest, since it's just the changes since the last incremental, but restores are slower as you have to restore all of them.

    Differentials take longer to backup and consume more space, but are faster to restore...since you restore only the full backup and the last differential.
      My System SpecsSystem Spec

  8. #8

    Nearby... In an astronomical sense.
    Posts : 43
    Microsoft Windows 10 Professional

    The way I'm planning for this to be done is just copying the files 1:1 without any form of compression from one PC to another using a daily scheduled Robocopy /mir task with the added benefit of being able to use the files directly on the 2nd PC which should come in handy in it's own way. Loosing half my storage capacity for peace of mind and accessibility - Ah well...

    I guess I'll be able to start tomorrow once the HDD's have arrived by Mail.
    Got to figure out how to automate this on both PC's for both being awake and ready when it's time.

    Still wondering what has happened anyway... Everything's fine now.
      My System SpecsSystem Spec

  9. #9

    I prefer just having the raw files in a second location as well. I think your robocopy task will take care of this problem perfectly for you.

    Storage is cheap, losing your files is terrible. If it's worth having the data, it's worth having a backup. And if you have data that's not worth backing up, then don't. For example, if you lose some TV shows or's likely not the end of the worth. If you lose your entire digital picture library of your children, that is a different story altogether.
      My System SpecsSystem Spec

  10. #10

    Kerkrade, Limburg, Nethrlands
    Posts : 24
    Windows 8.1 (355 + Auto-Update)

    Calculate your power-consumption carefully. Use the maximum for disk-spinup.
    Have a look at the maimum throughput of your Fuses.
    The intermittant problem you described sounds like you encountered such a problem.
      My System SpecsSystem Spec

Page 1 of 2 12 LastLast
BackUp Solutions/Suggestions for 8TB of Data?
Related Threads
CMS Backup Solutions in Software and Apps
Is there anyone using the CMS Backup Ultimate ver 14? I have been reading about it and would like to hear from anyone who has used it. Thanks!
I've done a clean install of Windows 8 and updated to 8.1 via the store update on a new SSD about 1 month ago. Everything was fine back then. Regarding hardware or settings I've changed nothing since then. Since about 1 week I've got some serious problems related to WMI: event id 28 At...
Great Apps :) Backup and Restore all windows applications. How to Backup App Data in Windows 8
I have been using Syncback SE Free for a long time now with Win7. It works well. I like the mirror function and the ability for syncback to detect small changes and allows you to view different files. When I use the mirror function in syncback, it scans the drive/folder and only copy/delete the...
Backup Strategies with Large volumes of data in Performance & Maintenance
Hi there Just wondering how people handle backing up large volumes of DATA and performing recoveries if required from this data. Now as cheap 3TB disks are readily available and by using Spanning or Storage pools you can get very large data spaces of say 12 TB for example. Now things like...
I just performed a painful full system recovery and I am setting up my PC the way I want it before creating a backup image. I already have all my preferred programs installed. Does anyone have any suggestions what tweak ups I should also do (for faster and better performance) before creating an...
I wanted to have File History backup data that is not contained in the "standard" libraries so I set up a library for each volume that I wanted backed up. I just included the root of the volume as the only folder in the library. But File History just seems to ignore those libraries. Am I...
Eight Forums Android App Eight Forums IOS App Follow us on Facebook