Storage, Explained – How To Configure Your RAID and File System Pros & Cons

Media storage can be confusing. This guide should help you to optimize for performance, redundancy and security.

When it comes to hard drives and storage, it’s easy to be overwhelmed. Mistakes can be costly when your data is at risk. I’ve compiled this guide to help you navigate the media storage landscape armed with some basic knowledge. This article will cover direct-attached external drives, desktop RAID arrays, and internal RAID arrays. I’m not touching on NAS or networking in this post.

Duplicate Your Data!

This should go without saying. Always duplicate your data onto at least two separate physical drives. This starts when offloading camera cards on set, and continues to the storage you are using for post-production.

Project files are typically small and can easily be backed up, even to the cloud. Many other types of temporary files can be recreated, such as NLE cache files, and even transcoded media. It’s critically important that you always keep more than one copy of your original camera media.

On Set

Small bus-powered external SSD drives, or hard drives are often used on set to offload camera cards daily, and may then be used to shuttle camera media to larger storage or shared storage on or off-site. On larger productions, your DIT may also use much larger direct-attached RAID arrays on location. Whatever your particular data-wrangling workflow may be, it’s important to always maintain duplicates at every stage of the workflow. Ideally, one set of duplicated camera media should be kept off site, at a production office, your post facility, or even just at home.

Transfer Speed On Set

At this point, I will mention one often overlooked aspect of on-set data workflow when selecting external hard drives. On set time is critical, and the faster you can offload (ideally to duplicate drives simultaneously), verify and turnaround camera cards, the better. Camera media is of course very fast, but if you use cheap, slow external hard drives over a slow interface, it makes offloading camera cards a lot slower than it should be. A single spinning disk hard drive will only give you sustained read/write speeds of up to 80MB/sec. There is nothing worse than holding up an entire production waiting for data transfer from camera cards. Trying to save a bit of money on hard drives here can be very expensive for the overall production.

The Blackmagic Design MultiDock can accommodate four SSD drives which can be configured as a RAID 0 array. This is a software RAID managed by the host OS. Image credit: Jeff Loch

Post Production

It’s also important to duplicate camera media and critical project files during post-production. This is important even if you’re using enterprise-class shared storage configured with a high level of redundancy, and certainly if using a direct-attached desktop RAID, or other directly attached drives to cut from. Whatever storage you are working from, it’s important to keep a duplicate of all camera media on a second desktop RAID, or separate drive that sits on a shelf somewhere, or even off site unless needed.

Individual External Hard Drives

External hard drives come in many flavors. They aren’t all the same, and the common USB drive you find at your supermarket may be fine for backing up your photos and documents, but isn’t likely to be well suited to your video workflow. When looking at external hard drives, it’s important to consider what kind of drive is inside it, and also what kind of interface it has with the outside world.

An external hard drive that has a fast interface, such as Thunderbolt connectivity, may still have a slow hard disk inside it. What really matters is the maximum read and write speed of the internal hard drive itself. Ideally, you should consider using external SSD drives for most video work, whether you’re offloading camera cards on set or using it to edit with from your laptop. However, SSD’s are more expensive and offer far less capacity. Depending on just how much data throughput you need, even a single 7200RPM hard disk may be enough. The choice is really up to how you intend to use it, and what kind of media you plan to work with.

Hard Disk Drive (HDD)

Traditional spinning platter hard disk drives have rapidly grown in capacity in recent years, with 14TB single hard drives available. These hard drives can be anything from 2TB up to 14TB and come in both 2.5″ and 3.5″ physical sizes. The internal platters spin at either 5200RPM or 7200RPM depending on the drive, and there are enterprise drives that spin at 15,000RPM but these are not drives you will find inside a typical external hard disk drive enclosure. You can expect only around 80MB/sec sustained read/write speeds from a single spinning disk hard drive, so the bottleneck is the hard drive itself more than the interface with your computer.

A cheap and common WD Passport external HDD (left) can deliver just over 110MB/sec over USB 2.0. (130MB/sec advertised) and provides only USB-A type cable. The G-Drive Mobile (right) can deliver similar throughput over USB 3.0 and comes with adapter cables to both USB-A and USB-C type connectors. Image credit: Western Digital and G-Technology

A 5200RPM hard drive can typically maintain an average of 100-110 MB/sec read and write speed, while a 7200RPM hard drive will average 120-130MB/sec.

Solid State Drive (SSD)

A SSD, or Solid State Drive uses NAND flash memory to store data, and is capable of extremely high data throughput. A typical USB 3 external SSD can maintain well over 500MB/sec read and write, and even this is limited by the SATA3.0 interface rather than the memory itself. A typical NVMe SSD over a PCIe interface can support well over 3000MB/sec. External SSD drives with a Thunderbolt interface will support up to 2800MB/sec.

The G-Drive Mobile SSD (left) uses USB-C connector and provides up to 560MB/sec over USB 3.2 Gen 2. The G-Drive Mobile Pro SSD (right) also uses a USB-C type connector but provides up to 2800MB/sec over Thunderbolt 3. Image credit: G-Technology

SSD is the best choice for media professionals because these data rates make offloading data from camera cards much faster, and allow realtime playback of multiple streams of high-resolution media from compact, portable external drives.

Interfaces

Many external USB hard drive manufacturers misleadingly exaggerate the maximum data throughput of external hard drives. Consumers don’t understand the technicalities, and bigger numbers printed on the box must be better, right?

The interface used to connect an external hard drive with a computer is just as important as the type and speed of the drive inside. That said, most common interfaces, these being flavors of USB 3 and Thunderbolt are all fast enough to deliver a single stream of most common video codecs and formats, and even multiple streams of many of them.

Shown are the most common USB type connectors currently. The original USB Type A is for reference only and has been widely replaced by USB Type A SuperSpeed. Many portable hard drives feature the USB Type Micro B SuperSpeed connector on the enclosure, and a cable to either USB Type A SuperSpeed or USB Type C.

USB 3.x

The naming convention of USB has become really messy. Here’s how it breaks down.

  • USB 3.2 Gen 1, is USB 3.0. It has a maximum throughput of 5Gbps. This is also known as SuperSpeed USB.
  • USB 3.2 Gen 2, is USB 3.1. It has a maximum throughput of 10Gbps. This is also known as SuperSpeed USB 10Gbps.
  • USB 3.2 Gen 2×2, is USB 3.2. It has a maximum throughput of 20Gbps. This is also known as SuperSpeed USB 20Gbps.

You may also hear USB discussed in terms of the type of connector. USB types, like A, B, and C, tell you the shape of the port and connector, but not the data transfer speed.

USB-C is the latest type of connector, and is usually associated with USB 3.2 Gen 2 but not necessarily, it can also be USB 3.2 Gen 1 or it may not be USB at all.

Thunderbolt 2 & 3

To confuse things even more, the USB-C type connector is also used for Thunderbolt 3. Thunderbolt 3 offers data transfer rates up to 40GBps, which is four times that of USB 3.2 Gen 2.

  • Thunderbolt 2 has a maximum throughput of 20Gb/sec
  • Thunderbolt 3 has a maximum throughput of 40Gb/sec

Power

Both USB and Thunderbolt can carry power, and so many smaller external hard drives may be bus powered which is something to keep in mind if you need to use them with a laptop untethered from mains power.

Media Bandwidth

When considering any kind of storage, the first thing you should have in mind is the required data bandwidth to playback your media in real time. Here are just a few examples, these data rates are approximate for comparison only.

  • 4K XAVC (H.264) – 50MB/sec (MB not Mb/sec… 1MB = 8Mb)
  • 4K 24p ProRes RAW – 40-100MB/sec
  • 4K 24p ProRes RAW HQ – 80-140MB/sec
  • 6K 24p ProRes RAW HQ – 180-300MB/sec
  • 8K 24p ProRes RAW HQ – 320-530MB/sec
  • 4K 24p ProRes 422 HQ – 95MB/sec
  • 4K 24p ProRes 4444 – 140MB/sec
  • 8K 24p R3D 5:1 – 260MB/sec

It’s clear from this list that an external USB3 5400RPM hard drive isn’t going to support playback of most of these formats. One exception will be just about any H.264/AVC based video from popular mirrorless cameras and DSLR’s. The data rates required for these codecs are low enough even at 4K resolution, and your hard drive will not be a bottleneck, but decoding H.264 in real time on your computer might be.

This doesn’t mean you can’t use a slower hard drive purely for backup of your camera files, it just means you won’t be able to play the videos directly from it.

RAID

What exactly is a RAID? A RAID stands for Redundant Array of Independent Disks. It is a virtualized volume whereby data is written across an array of multiple physical hard drives to provide redundancy in case of drive failure, and/or higher performance, usually both.

The G-Drive is an example of a small, portable 2-bay RAID. Image credit: G-Technology

A small portable external RAID may contain just two physical hard disk drives, and this can be configured either for redundancy, where data is duplicated to both internal drives, or for striped for performance where data is split and written across both drives simultaneously allowing twice the data throughput. For a two-drive RAID, performance comes at the cost of redundancy and vice versa. However, the more physical hard disk drives in the RAID array, the more options exist to balance redundancy and increased performance.

The popular Promise Pegasus32 desktop RAID arrays come in different sizes from 4-bay to 8-bay. Image credit: Promise

Larger RAID arrays can come in the form of direct attached desktop or rack mounted RAID enclosures. You can also build an internal RAID array directly into a workstation chassis, and of course, there are shared NAS (Networked Attached Storage) RAID storage servers which I’m not going to get into in this article.

Any of these could have between four and eight, twelve, sixteen or more drives in an array. Rack mounted SSD RAID arrays often comprise 24 SSD’s per array and are capable of extremely high bandwidth. It is also possible to stripe data across multiple arrays in a variety of ways, however this is firmly into enterprise storage territory and beyond the scope of this article.

I have only mentioned RAID levels below that apply to the types of storage I’ve described above.

RAID 0

By en:User:CburnettOwn workThis W3C-unspecified vector image was created with Inkscape., CC BY-SA 3.0, Link

Maximum performance, no redundancy.

RAID 0 stripes data evenly across all the physical drives in the array, but has no mirroring or parity. This means that parts of a single file exist across all the drives in the set, and if one drive in the array fails, all data is permanently lost. There is no data recovery possible. RAID 0 is useful for small external two drive RAID enclosures where increased read/write bandwidth is the priority, and data is being duplicated to another physical drive as well.

I would never recommend relying on a single RAID 0 configured array for safe storage of camera media.

RAID 1

By en:User:CburnettOwn workThis W3C-unspecified vector image was created with Inkscape., CC BY-SA 3.0, Link

RAID 1 simply mirrors data between two drives, but has no parity or striping. This gives full redundancy, but in most implementations is only as fast as one of the two drives.

If one of the two drives fails, it can be replaced and the data rebuilt from the remaining drive. However, all of the data is at risk until this process is complete.

RAID 5

By en:User:CburnettOwn workThis W3C-unspecified vector image was created with Inkscape., CC BY-SA 3.0, Link

RAID 5 offers block level striping, which increases performance, and parity whereby the contents of any single disk is also distributed among the others disks. For example, in an eight drive array configured as RAID 5, the total storage capacity and throughput will be equal to seven out of the eight drives. If any drive in the array fails, it can be removed, and a new replacement drive swapped into the array. The contents of the failed drive will be rebuilt onto the new drive from the parity data existing across the other seven drives.

A hot spare can be denoted in a RAID 5 array, decreasing the useable capacity and throughput of the array by one further drive, but offering immediate rebuild in the event of a drive failure.

Data is only at risk for the duration that the replacement drive is being rebuilt.

All data in the array is permanently lost if more than one drive fails at a time, or if a second drive fails during rebuild.

RAID 6

By en:User:CburnettOwn workThis W3C-unspecified vector image was created with Inkscape., CC BY-SA 3.0, Link

RAID 6 is similar to RAID 5 but offers double distributed parity. Using the eight drive example, storage capacity and throughput of the array will be equal to six of the eight drives. Any two drives in the array can fail and their respective contents can be rebuilt from parity data existing on the remaining six drives.

The likelihood of two drives failing is considered very low. So the real advantage of RAID 6 is that the data is not at risk in the event of a single drive failure.

Which RAID Level Should You Use?

I would recommend configuring portable external two disk RAID drives as RAID 0 for higher data throughput. These should always be used in pairs to offload camera cards, and are good to shuttle a whole shoot day worth of camera media to be transferred to shared storage or larger RAID storage you’re using for post. They can be good drives to edit smaller productions directly from also, but in this case data should be duplicated somewhere else as well. I would always consider a RAID 0 drive as temporary storage, there’s no safety or redundancy built-in unless it’s mirrored.

For larger direct attached desktop RAID arrays I recommend RAID 5. It is a good balance of redundancy and performance. You should always keep at least one spare hard drive for your RAID enclosure just in case you need to swap. A RAID 5 array will still be accessible and operate with one failed drive but your data is at risk, it’s important to swap out a failed drive and let the array rebuild as soon as possible.

Duplicate Your Data Again!

Even if you’re working from a RAID 5 array and your media is protected from a drive failure, you should still have your camera media and project files backed up onto other storage. I’ve built nested RAID arrays for clients where two identical RAID sets are further mirrored as a RAID 1 but even this doesn’t count as duplication. It’s still all in one rack in one machine room, in the same building.

Camera media should always be duplicated onto separate physical storage, and kept off site if possible.

This is called disaster recovery. Disasters do happen.

A Quick Note About LTO Tape

A great way to store critical data long term is on LTO tape by using a single direct attached LTO drive. This can either be through dedicated backup software, or using LTFS for direct drag and drop data transfer.

The mLogic portable LTO drives are great for quickly backing up large amounts of data safely for long term archival. They have Thunderbolt connectivity and can be used by DIT’s on location too. Image credit: mLogic

A single LTO-8 tape can store 12TB of media and supports speeds up to 300MB/sec depending on the drive. When stored correctly, LTO tape cartridges can be expected to last up to 30 years. This is a far longer shelf life than any mechanical hard drive. Current LTO tape cartridges will remain backwards compatible in future drives by up to one generation. So a current LTO-8 cartridge will be able to be read and written in a LTO-9 drive.

The Cloud

Cloud storage isn’t yet very practical for vast amounts of high resolution camera media. However, it is good for smaller project files, certain other media assets, and even Resolve database backups.

File Systems

Everything I’ve discussed so far is about physical hardware. It’s also important to format your storage with the right file system for your needs. A file system describes how data is stored, addressed, managed and accessed on your physical storage, by a computer operating system and software.

Here is a rundown of some common file systems.

ExFAT

ExFAT is a Microsoft file system created to bridge the gap between NTFS and the older FAT32 file system. It can store single files larger than 4GB and is the only file system I’m listing here that is natively supported by both Windows and macOS.

However, ExFAT is a non-journaling file system and is at risk of corruption when write operations are interrupted. I would not recommend ExFAT for media drives, with the exception perhaps of very temporary shuttle drives used to keep data for a short period of time that needs to be shared easily between macOS and Windows machines. Even then there is a risk if these drives are not ejected safely or pulled during data transfer.

NTFS

NTFS is a journaling file system created by Microsoft that is far more secure and less susceptible to corruption even in the case of unexpected shutdowns, crashes and interruptions.

NTFS is readable by MacOS but write access on NTFS formatted volumes requires a third party software such as Paragon NTFS for Mac.

MacOS Extended

MacOS Extended is also known as HFS+ and was the standard file system used by macOS from 1998 until today for mechanical and hybrid hard drives, and up until the introduction of APFS for solid state and flash storage in High Sierra.

APFS

APFS has replaced MacOS Extended for solid state and flash storage. It will also work on mechanical hard drives, but is optimized for solid state media. APFS is not compatible with older Macs running El Capitan or earlier.

Which File System Should You Use?

Your best file system options are NTFS if you’re primarily operating in a Windows environment, and either macOS Extended or APFS is your workflow is primarily Mac-based.

If you’re formatting mechanical hard drives and RAID arrays, it’s best to choose either NTFS or MacOS Extended. These journalling file systems are robust and secure.

For SSD media you can choose NTFS or APFS depending on whether you are primarily working in a Windows or Mac environment. Third party software is available that allows Windows to mount and read/write to APFS formatted media, as well as mounting NTFS media on Mac.

Avoid using ExFAT in all cases to avoid risking your data to corruption.

Conclusion

When all of this is said and done, the most important thing to remember is to always duplicate your project files and source camera media. Transcodes can be transcoded again, cache files can be regenerated, exports and renders can be rendered again. These things may cost you time, but if you lose your source camera files you’ve lost everything.

Even if you follow all of the recommendations and best practices, the fact is, drives and even entire RAID arrays can fail. You can use your fastest and most expensive storage to work from, but having a backup copy (or even a backup of a backup) on one or more cheap consumer external hard drives can be enough to rescue a project.

I hope this helps some of you to consider or reconsider your workflow. I’d love to know what tips and tricks you have that I may have missed or overlooked. How are you managing your media and keeping your files safe? Let us know in the comments below.

The post Storage, Explained – How To Configure Your RAID and File System Pros & Cons appeared first on cinema5D.

Frame.io v3.6 – Interview with CEO Emery Wells

The latest Frame.io v3.6 update has been released earlier this month, and we had a discussion with Emery Wells, CEO of Frame.io, about all the latest features. Let’s take a closer look at the new Frame.io Transfer App, notifications, the Watermark ID system, security improvements, and much more!

 

Frameiov36_Featured

Image credit: Frame.io

Emery Wells and Frame.io v3.6

Frame.io has been released earlier this year and if you missed it, you can read our entire article about it here. During the cinema5D Virtual Show, we had the chance to discuss with Emery Wells about all the latest features, and even get a personalized tour. Here is a quick summary of what’s new in Frame.io v3.6:

  • Introduction of the Frame.io Transfer App to download large files, advanced folder structures, or entire projects with just one click.
  • Watermark ID allows you to add text blocks directly over your video file with things like name, date, location, email, and IP addresses.
  • You can now add folders to Review Links, which is easier for sharing assets.
  • For Enterprise users only: the possibility to protect all your sensitive data and review links with Login-only access.
  • A lifting of the Notifications system.
  • The new “Offline Mode” for the Frame.io iOS apps allows you to make any file available offline. Then, you can review and leave comments that will be automatically synced to your project once you’re back online.
  • A smoother After Effects integration of the Frame.io extension.
Frameiov36_05

Frame.io v3.6 introduces Frame.io Transfer. Image credit: Frame.io

Frame.io Transfer App

As Emery Wells states, the Frame.io Transfer App was scheduled for later this year. Due to the pandemic, Frame.io pushed the Beta version to the users earlier. This is a sweet and smart move from Emery Wells and a gentle reminder to all entrepreneurs that it’s better to push your product to the public as soon as possible. Even if it’s not finalized, you can get users’ feedback and make it better.

I’ve been using the Frame.io Transfer App for a couple of days now, and it’s been working well. After you complete a download, the App even makes a checksum to make sure the size of the file you just downloaded matches what’s on Frame.io’s server. This is brilliant and saves us so much time.

Did you already try the new Frame.io Transfer App? What do you think about all the new features? Don’t hesitate to let us know in the comments below!

The post Frame.io v3.6 – Interview with CEO Emery Wells appeared first on cinema5D.

Roger Deakins Podcast, Listen and Learn With A Cinematography Master

Join the acclaimed cinematographer Roger Deakins and his team in a new podcast series that discusses filmmaking, collaboration, lighting, and more.

Featured image by Stephen Vaughan.

There are seven podcast episodes available to listen to so far.

If you’ve ever wanted to know what goes through a cinematographer’s mind, now is your chance to dive in. Team Deakins, which is composed of Roger Deakins CBE, ASC, BSC, James Deakins (Roger Deakins’ wife), and Matt Wyman, have started an insightful new podcast. Deakins, known for his work on 1917, Blade Runner 2049, Skyfall, and The Shawshank Redemption (and many more films) discusses and muses on various questions on filmmaking with James and Matt.

So far the Team Deakins podcast has covered their beginnings into cinema, collaboration, practical lighting (we all know that Deakins is a true master of light), composition, and lens choice. There are seven episodes of the podcast so far, and from listening to a number of them, I sincerely hope that this is a pursuit that Team Deakins continues into the future.

Roger Deakins CBE, ASC, BSC. Photo by Jay L. Clendenin/Los Angeles Times.

“Don’t light something just ‘cos you can light it, it’s like don’t move the camera just ‘cos you can move it”.

The podcast isn’t boastful at it has just the right level of tech talk for all of us geeks out there. The dialogue is peaceful and the ‘Beginnings’ episode discusses the very humble entries into cinema for both James and Roger. Unlike some vlogs from that can be all pow and show-offy, listening to Team Deakins is very much like being on a round table with them at BSC, or talking to them over a coffee. Their musings are entertaining and have certainly given me an enlightened sense of positivity during this pandemic.

Where Can You Subscribe To The Team Deakins Podcast?

You can listen to the Team Deakins podcast here on the Apple Podcasts website, and of course, subscribe by using the Podcast app on iOS. It’s also available on other podcast sites like Podtail too.

Now is a better time than ever to spend time learning, developing skills, and expanding on knowledge. Amongst free courses from ARRI and offers from MZed and Masterclass, this podcast is a detailed insight into the Team Deakins world and certainly one to add to the playlist.

How Can You Engage With The Team Deakins Podcast?

The Team Deakins podcast is inviting listeners to submit their questions and engage in discussions through the forum on Roger Deakins’s website, which can be found here.

Roger Deakins has worked in cinema throughout the decades, so it is also interesting to learn about the history, evolution, and transformation of film and technology too.

This podcast series will be an eye-opener to seasoned professionals and inquisitive listeners alike. The content is rich and insightful, and as always it is great to hear an award-winning cinematographer about their craft. Will you be subscribing to the podcast? What kind of questions would you ask Team Deakins? Let us know in the comments.

The post Roger Deakins Podcast, Listen and Learn With A Cinematography Master appeared first on cinema5D.

A Review of the Midrange Model of the New MacBook Pro

A Review of the Midrange Model of the New MacBook Pro

The MacBook Pro is the laptop of choice for a wide range of professionals who need to be able to work on the go. The latest version of the MacBook Pro comes with a bevy of new improvements and features, and this excellent video review takes a look at it and what you can expect from it in practice.

[ Read More ]

How Does the Nikon Z 6 Hold Up After 250,000 Photos?

How Does the Nikon Z 6 Hold Up After 250,000 Photos?

The Nikon Z 6 has been on the market for a while now, and the Z system has begun to work its way into the hands of more and more professionals. This great video features a photographer discussing his thoughts on the camera after logging 250,000 images on his Z 6.

[ Read More ]

A Complete Guide to Frequency Separation, Dodging and Burning, and Color Grading in Photoshop

A Complete Guide to Frequency Separation, Dodging and Burning, and Color Grading in Photoshop

Retouching a portrait in Photoshop often takes the use of a wide variety of techniques, all of which you must have a good grasp of to create a professional image. This fantastic video will show you how to use frequency separation, dodging and burning, color grading, and more to retouch a portrait in Photoshop.

[ Read More ]

Is This the Best Camera Available for Time-Lapse and Stop Motion Photographers?

Is This the Best Camera Available for Time-Lapse and Stop Motion Photographers?

Photographer and filmmaker Matthew Vandeputte is perhaps one of the most respected time-lapse and hyperlapse creators out there. Given how often he’s asked what the best camera is for producing his style of content, he’s put together a short video explaining his choice and listing all of the features that make it so perfect.

[ Read More ]

How to Uphold (or Get Out of) Film Contracts During COVID-19

Client refusing to pay? Crew demanding cash? Here’s how you navigate contracts during a pandemic.

Here’s one: You finished a video for a client. However, the client is now saying they can’t pay because of coronavirus. What are your options?

Here’s another: you were in the middle of shooting a project, and then had to pause indefinitely because of the pandemic. Now your cast and crew are asking for their money. Do you owe them?

Courtesy of the Slamdance Film Festival, David Albert Pierce of Pierce Law Group, LLP started a Facebook Live chat titled The Producer’s Legal Guide to COVID-19 to answer exactly these kinds of questions. You can watch parts 1 and 2 below.

No Film School was on hand for Part 1 and we’ve compiled the 3 most helpful takeaways for filmmaker below.

1. Look for your contract’s mention of “Act of God” or “force majeure.”

Here’s the very first place to look if you are experiencing a contract dispute. Under the ‘boilerplate’ will be spelled out who has to do what in the face of acts of god, natural disasters, and force majeure.

Read More

Red Giant VFX Suite 1.5: Lens Distortion Plus More Tools for After Effects

Red Giant released a supercharged update to their VFX Suite which includes some very helpful new tools for editing in After Effects. Let’s check them out.