Jump to content


Photo
- - - - -

RAID suggestions


  • Please log in to reply
20 replies to this topic

#1 katy-kid

katy-kid

    Lionfish

  • Member
  • PipPip
  • 64 posts
  • Gender:Female
  • Location:SWFL
  • Interests:FLORIDA!

Posted 26 June 2017 - 12:48 PM

Hi there, I have a 2013 macbook pro, and have a seagate hard drive and am looking into a RAID system. From my understanding RAID 5 or 6 is the best choice for image security and speed; does anyone have any brand recommendations they could offer? I'm looking into spending in the $500 range, not sure if this is even viable or if RAID is all it's cracked up to be. Thank you guys for your help, it's always really appreciated.


Katy Danca Galli

Atmospheric UW Photography

http://katydancagalli.com


#2 TimG

TimG

    Sperm Whale

  • Moderator
  • 1844 posts
  • Gender:Male
  • Location:Amsterdam
  • Interests:Sunlight reefs, warm seas, fine wine, beautiful women. And Manchester City Football Club

Posted 27 June 2017 - 11:01 AM

Hi Kay

 

Are you looking for drives that will mirror each other?

 

If so, I bought the G-SAFE system about 4-5 years ago. 2x 3TB drives in one housing which use USB to plug into my iMac and mirror each other. I use Good Sync software to sync my LR photo collection to the drives - which automatically mirror.

 

I also have a  third G-SAFE 3TB drive which I rotate in and out of the G-SAFE housing with the other two drives so having 3 copies plus the original on my iMac. The third copy is kept "off-site" in a Pelican case!

 

I have been really pleased with this system and feel very confident in its backup capabilities. I think, at the time, it cost about £400 and it has proved very reliable.

 

G-Technology is the manufacturer and there is now an updated system.


Tim
(PADI IDC Staff Instructor and former Dive Manager, KBR Lembeh Straits)
Nikon D800, Nikkors 105mm and 16-35mm, Sigma 15mmFE - Subal housing

http://www.timsimages.uk
Latest images: http://www.shutterst...lery_id=1940957


#3 sunnyboy010101

sunnyboy010101

    Lionfish

  • Member
  • PipPip
  • 60 posts
  • Gender:Male
  • Location:Vancouver Island

Posted 27 June 2017 - 12:13 PM

Just my opinion, but I would recommend RAID 5 minimum. The optimum RAID solution these days is any NAS (network attached storage). You might be able to get a package deal, but bought individually you will need to buy the bare NAS box plus 4 identical hard drives. I recommend 2TB or greater for the HDD as these things are getting pretty cheap.

 

Buy the biggest hard drives you can afford. Look for a "hot swap" NAS, and one that you can just plug drives into (or remove) without having to attach carriers.

 

The benefit of RAID 5 is that it will detect one bad drive and let you "hot swap" that drive while the system is running. It will then rebuild the "volume" (all the drives make up a volume) without losing any data.

 

So RAID 5 = single drive failure correction, more than one drive fails it will detect but you lose your data. That's pretty rare.

 

There are other levels of RAID like basic mirroring with 2 drives, or even the super-fancy RAID 10 which is RAID 5 + mirroring (so lots of drives needed).

 

I am running a LG NAS box that's discontinued, but also run a DROBO which I got cheap and love. The advantage of the DROBO if you can afford it is that mine has 8 drive slots, and you can just plug in ANY drives and it will calculate an optimal use and RAID level for them. I started with like 6 scrounged drives, but now have 4 x 4TB drives for lots and lots of RAID 5 storage.

 

Last point - if you don't have a good backup program that will back up all your important stuff EVERY DAY, you will lose data someday. Probably when you really needed it.



#4 katy-kid

katy-kid

    Lionfish

  • Member
  • PipPip
  • 64 posts
  • Gender:Female
  • Location:SWFL
  • Interests:FLORIDA!

Posted 27 June 2017 - 06:59 PM

Hey guys, very solid advice. Tim, that is pretty much what I was looking for I believe, is that considered RAID 5? Also, sunnyboy, is there a particular brand you use for the hard drives? I have used seagate and they seem ok. Also, when you say 'lose data' (*shudders), do you mean if I am not uploading images daily, or even turning on my drives daily, that they may fail? What would be considered a backup program? Thank you both, really appreciate your help.


Katy Danca Galli

Atmospheric UW Photography

http://katydancagalli.com


#5 TimG

TimG

    Sperm Whale

  • Moderator
  • 1844 posts
  • Gender:Male
  • Location:Amsterdam
  • Interests:Sunlight reefs, warm seas, fine wine, beautiful women. And Manchester City Football Club

Posted 28 June 2017 - 08:28 PM

Sorry Katy, I have no idea which RAID it is (!) and I'm not at home at the moment to check.

 

Really useful advice from sunnyboy. 


Tim
(PADI IDC Staff Instructor and former Dive Manager, KBR Lembeh Straits)
Nikon D800, Nikkors 105mm and 16-35mm, Sigma 15mmFE - Subal housing

http://www.timsimages.uk
Latest images: http://www.shutterst...lery_id=1940957


#6 bvanant

bvanant

    Sperm Whale

  • Team Wetpixel
  • 1822 posts
  • Gender:Male
  • Location:Los Angeles (more or less)
  • Interests:Science, photography, travel

Posted 29 June 2017 - 03:11 PM

You might also look at Drobo. Not Raid but it has the advantage of using drives of different sizes. We have 5 of them running continuously; 2 for video, 2 for photos and one backing up general stuff.  For drives for any NAS/Drobo/Raid box make sure you get NAS drives. I have a mix of Hitachi/WD/Seagates in my Drobos.

 

Bill


Bill
Canon 7d, Nauticam, Lots of glass, Olympus OMD-EM5, Nauticam, 60 macro, 45 macro, 8 mm fisheye, Inon, S&S, Athena Strobes plus lots of fiddly bits.
www.blueviews.net


#7 sunnyboy010101

sunnyboy010101

    Lionfish

  • Member
  • PipPip
  • 60 posts
  • Gender:Male
  • Location:Vancouver Island

Posted 30 June 2017 - 07:57 AM

Katy,

 

Yes, RAID 5 is what you want. I've used Seagate and Western Digital drives in my NAS boxes - whatever is cheapest at the time I bought them. I do not like to mix drive brands. I normally like to buy all 4 at one time.

 

As for data protection, 2 weeks ago I had one WD drive go bad in my LG box. I got the yellow light and warning that a drive had failed, so found another identical spec drive on amazon and bought it. Once arrived I just unplugged the dead drive and plugged in the new drive and about a day later the system was rebult and working. NO DATA LOSS. (it does take time to rebuild the volume when a drive is replaced).

 

With RAID 5, if one drive fails and you replace it fairly quickly, you will not lose data. If two drives fail, you WILL lose data, even with RAID5.

 

If you don't back up your computer frequently and it dies, you lose whatever was on it since the last backup. That's why I back up every day at about 1am.

 

DROBO does support RAID 5, as my main NAS these days is an 8-bay DROBO and it's running 4 drives (4TB each) and is running RAID 5.

 

 

Just to give you an idea, I've been "doing" computer stuff since 1979, and used to back up stuff on floppy, then CD, then DVD, then Blue-Ray and finally now RAID storage devices. My system has a 4TB data drive that is backed up daily to a server with the same single 4TB drive, then backed up to an LG NAS with 4x2TB drives and finally backed up to a DROBO with 4x4TB drives. Yes, I have THREE backups done every day after midnight.

 

Even with that, if my main computer were to crash at say 7pm, I would lose the day's work. Since backup programs can be run manually, if I were extra cautious, ever time I upload photos to the data drive I *could* run the backup to make sure they were saved. I don't because I'm probably getting complacent as I've not lost photos yet.

 

I have photos backed up including scanned slides from the '70s, so around 90GB total representing about 32000+ photos.



#8 bvanant

bvanant

    Sperm Whale

  • Team Wetpixel
  • 1822 posts
  • Gender:Male
  • Location:Los Angeles (more or less)
  • Interests:Science, photography, travel

Posted 03 July 2017 - 09:25 AM

It seems clear from the Drobo literature that they are not doing RAID in the ISO spec sense. You can use multiple drive sizes for example in a single box (not possible with traditional RAID).  In any case worth looking at; the 5D version is quite fast.

Bill


Bill
Canon 7d, Nauticam, Lots of glass, Olympus OMD-EM5, Nauticam, 60 macro, 45 macro, 8 mm fisheye, Inon, S&S, Athena Strobes plus lots of fiddly bits.
www.blueviews.net


#9 adamhanlon

adamhanlon

    Harbor Seal

  • Admin
  • 2122 posts
  • Gender:Male
  • Location:Lancaster, UK

Posted 04 July 2017 - 01:05 PM

The problem with Drobo is that in the event of the enclosure failing, the only way the get your data is to put the drives into another Drobo. They use a proprietary file system. RAID (at whatever level) is a standard configuration that will allow the data to be accessible in pretty much any RAID enclosure...

Adam Hanlon-underwater photographer and videographer
Editor-wetpixel
web | Flickr | twitter | Linkedin | Facebook


#10 cneal

cneal

    Lionfish

  • Member
  • PipPip
  • 53 posts
  • Gender:Male
  • Location:Charlotte, NC

Posted 30 July 2017 - 03:13 PM

As a retired IT guy I would recommend buying a spare drive for a RAID 5 array. When a drive fails a few years down the road locating an identical replacement can be a problem. If a spare is on the shelf it is a quick and painless fix.



#11 bvanant

bvanant

    Sperm Whale

  • Team Wetpixel
  • 1822 posts
  • Gender:Male
  • Location:Los Angeles (more or less)
  • Interests:Science, photography, travel

Posted 30 July 2017 - 04:04 PM

Adam: While that is true in theory, i had a couple of Wiebetech enclosures die and no other array could read the drives.  Maybe if they are from the same manufacturer with the same firmware it might work.

Bill


Bill
Canon 7d, Nauticam, Lots of glass, Olympus OMD-EM5, Nauticam, 60 macro, 45 macro, 8 mm fisheye, Inon, S&S, Athena Strobes plus lots of fiddly bits.
www.blueviews.net


#12 Barmaglot

Barmaglot

    Clownfish

  • Member
  • Pip
  • 22 posts

Posted 09 September 2017 - 09:11 AM

As a server/infrastructure admin in my day job, I have to stress a point: RAID is not a backup! RAID provides business continuance in the case of a certain class of failures, though it is by no means a guarantee by itself. Furthermore, RAID5 is the least reliable type of RAID besides RAID0 (which, by strict definition, is not RAID at all), especially as your arrays get bigger.

 

To illustrate, imagine an 8-drive RAID5 array where a drive fails - you're immediately in degraded mode, where reads have to be reconstituted from parity data, and any additional drive failure kills your array dead. Then, once you replace the failed drive, or your hot-spare (if you have one) kicks in, the array, while still degraded, is put into maximum stress mode - the rebuild process reads all the data from all the drives to build the blocks on the new drive, and that's where one of three things can happen: (1) everything is fine, the array is rebuilt and you go back into normal mode, (2) another drive fails under stress (and drives from a single batch sometimes fail close together) and your array dies, or (3) as your data is rebuilt, a bad block is discovered on one of the remaining drives, and with redundancy gone, it cannot be rebuilt, resulting in a condition known as 'punctured stripe' - your array is still online, but some data is bad, and normally you will need to destroy the array, rebuild it with good drives and restore the data from backup.

 

RAID6 avoids these two latter scenarios by computing two sets of parity data, which enables it to survive double disk failures - at the cost of higher overhead (lower performance) and lower raw space to usable space ratio.

 

RAID10 (stripe of mirrors) has the highest performance, but also the worst raw to usable space ratio of the common RAID levels (only half of your raw space can be used), and it can survive most, but not all double-disk failures. To illustrate, in the same 8-drive array configured as RAID1, you have drives 1+2, 3+4, 5+6 and 7+8 configured as mirrors of each other, and then your data is striped across these four sets of mirrors. If drive 1 dies, the array can survive the failure of any drive other than drive 2 - better than RAID5 (which can survive no additional failures) but worse than RAID6 (which can survive any one additional failure).

 

An important feature of any RAID system is proper maintenance and monitoring - drives have to be periodically scanned, and an administrator has to be alerted of any errors, in order to take immediate corrective action.

 

In a home user scenario, unless your datasets are very very very large (tens of terabytes and up), there is little use for RAID. Almost inevitably, it will provide a false sense of security, while left unmaintained and will eventually die - taking your data with it. I have seen this scenario play out more than once. If your goal is data integrity, then instead of RAID, invest in a proper backup system, with versioning and off-site storage. Backblaze, for example, will let you back up unlimited data for just $50/machine/year. Unlike RAID, this will also protect you against data corruption, user errors, ransomware infections, hardware theft, site disasters, etc.



#13 NWDiver

NWDiver

    Orca

  • Member
  • PipPipPipPipPip
  • 1369 posts
  • Gender:Male
  • Location:Seattle

Posted 20 September 2017 - 06:21 AM

Maybe same issue, maybe different.  But opinions welcome

 

Got a deal on a iMac i7 core, 4.2GH, 32GB ram, DDR4, 512GB SSD Drive. Using it mainly to edit 4k video, using FCPX.  I have older G-Drive 4TB drives with USB3 connections.  Given the "small" hard drive of the iMac I am trying to save to, and edit off an external hard drive.  When working in FCPX and try to "play" a 5min video in the timeline it stutters and stops.  You can see and hear the G-Drive working away.  But as the FCPX timeline runs the stuttering, slow play makes it very hard to sync music or see timing.  

 

Would a Raid 5 system like linked below help in SPEED OF EDITING and smoother operation when using FCPX???  Not a pro, not super worried about storage, really looking to make editing in FCPX as fluid as possible. Hope that makes sense.  Thanks again!

 

(Price is always an issue)

OWC / Other World Computing ThunderBay 4 8TB (4 x 2TB) Four-Bay Thunderbolt RAID Array

 

https://www.bhphotov...tb_4_x_2tb.html


Edited by NWDiver, 20 September 2017 - 06:23 AM.


#14 Barmaglot

Barmaglot

    Clownfish

  • Member
  • Pip
  • 22 posts

Posted 29 September 2017 - 08:57 PM

The system you linked doesn't do RAID by itself - it's a JBOD, which stands for 'Just a Bunch Of Disks'. When connected, you'll see four separate 1.86TB drives plugged into your system. You can use MacOS Disk Utility to configure it as RAID0 (all space usable, no redundancy - one drive goes and all your data is gone) or RAID10 (half the space is reserved for redundancy). Depending on how that enclosure is built and on MacOS RAID stack, you might or might not see some performance gain over a single drive. Without extra software, you won't be able to do RAID5.

 

However, for $800, you can get a single 2TB external SSD such as a Samsung T5 which is guaranteed to have much better performance than a box full of 7200rpm spinning rust.



#15 sunnyboy010101

sunnyboy010101

    Lionfish

  • Member
  • PipPip
  • 60 posts
  • Gender:Male
  • Location:Vancouver Island

Posted 30 September 2017 - 01:30 PM

Generally for video performance, a single hi-performance disk is recommended. I suppose a larger capacity SSD would be awesome, but SSD's do have drawbacks.

 

RAID is not a good fit for performance requirements, especially video editing. RAID comes in two flavors, hardware based and software based. A lot of modern RAID is software based, and it is not the best for performance, especially when writing data, as the data has to be recorded in multiple spots or spanned over drives. Some hardware RAID can be really fast, especially if the hardware includes a good processor.

 

Generally, I'd assume unless the device really blows it's horn about hardware speed and RAID performance, it's probably software RAID and may not be fast enough to really support video work.

 

I concur that a single large drive is probably a better choice.

 

Also remember that the type of drive connection can be really limiting... IDE 2 is pretty slow. IDE3 is somewhat supported (it's hit or miss even when it's supposed to be supported by a machine). E-sata external connectors are really the way to go if you want speed for video work from a plug-in drive.

 

Just my $.02



#16 Barmaglot

Barmaglot

    Clownfish

  • Member
  • Pip
  • 22 posts

Posted 30 September 2017 - 09:26 PM

Also remember that the type of drive connection can be really limiting... IDE 2 is pretty slow. IDE3 is somewhat supported (it's hit or miss even when it's supposed to be supported by a machine). E-sata external connectors are really the way to go if you want speed for video work from a plug-in drive.

 

There is no such thing as 'IDE2' or 'IDE3'. The term 'IDE' stands for 'Integrated Drive Electronics' - it goes back to 1980s, when PC hard drives had a dedicated controller card which plugged into an ISA bus slot (I'm simplifying here for the sake of brevity) - Western Digital's IDE standard, released in 1986, moved this controller into the drive enclosure, and ran a derivative of the ISA bus protocol over a flexible ribbon cable. Years later, it was standardized as AT Attachment, or ATA. The standard was iterated with higher transfer speeds, growing from 16MB/s to 33MB/s, to 66MB/s, to 100MB/s and finally 133MB/s, until it was superseded by Serial ATA (SATA), which used the same command set, but communicated over a high-speed serial link rather than ATA's 80-wire parallel link. The first iteration of SATA was capable of transfer speeds up to 1.5Gbit/s (up to 150MB/s in practice due to protocol overhead), then SATA2 (2004) increased that to 3Gbit/s, and finally SATA3 (2009) went to 6Gbit/s. Note that even on its best day, a spinning drive is hard pressed to hit 200MB/s in linear transfer rate, and in actual usage, it's generally a tiny fraction of that, so the interface is very rarely a bottleneck for performance. For SSDs, of course, the story is different - modern SSDs can and do bump into the 600MB/s limitation of SATA3, which prompted the development of NVMe.



#17 NWDiver

NWDiver

    Orca

  • Member
  • PipPipPipPipPip
  • 1369 posts
  • Gender:Male
  • Location:Seattle

Posted 01 October 2017 - 07:18 AM

Thanks for all the info!  I did download Blackmagic's Disk speed ap.  On the iMac 512GB SSD drive it shows over 1900mb read/write speeds.  On my old 4TB G-Tech drive with a Super Speed USB3.0 cable the read/write speed was about 40mb/s.  So maybe a SSD external hard drive is the best route to edit FCPX from.  Again all raw data for the projects is 1-2TB.



#18 sunnyboy010101

sunnyboy010101

    Lionfish

  • Member
  • PipPip
  • 60 posts
  • Gender:Male
  • Location:Vancouver Island

Posted 01 October 2017 - 01:43 PM

Sorry, meant USB 2 and 3. Late, tired, etc.



#19 NWDiver

NWDiver

    Orca

  • Member
  • PipPipPipPipPip
  • 1369 posts
  • Gender:Male
  • Location:Seattle

Posted 09 October 2017 - 06:36 AM

Still trying to figure out what to do.  

 

Side note; to transfer 1.95TB of 4k video from my G-Tech 4tb external drive to my Lacie Tough Drive took about 13hrs at 42mb/sec.



#20 NWDiver

NWDiver

    Orca

  • Member
  • PipPipPipPipPip
  • 1369 posts
  • Gender:Male
  • Location:Seattle

Posted 16 October 2017 - 08:25 AM

In the end decided editing 4k video was the key.  Would risk losing data if a drive fails.  So went with the following external hard drive to edit on and then will use old hard drives to store final projects on.  Thanks to Tamir at B&H to walk me through options.

 

https://www.bhphotov...id_2tb_ssd.html