Content Insider #527 - Content Workload in the Cloud
You're in this for the Long Haul, Choose Solution(s) with Care
By Miles Weston
Whether you're a movie fan, TV binge-consumer, episodic series viewer, ad lover, stupid dude tricks connoisseur or VR thrill seeker; there's content out there for you to enjoy, thanks to the bullet train changes in the entertainment industry.
Around the world and coming from every corner of the globe; there is a growing volume of good to great content people can access to avoid work, blow-off boss/teacher presentations and educate/entertain/inform themselves.
It's all possible because of equal parts of:
- Hundreds of thousands of individuals who feel compelled to tell a story
- Technology advances that have turned the industry on its head - easier to use, more powerful, less expensive hardware and software
- Content distributors (theaters, networks, OTT services and social media folks) who need to fill their video libraries with material to keep people coming back
And in the expanding world of content development, everyone has the same question; "What do I store it on?"
The answer is simple, "It depends."
Exabytes - According to Coughlin Assoc., film today with 16,000 x 8,000 pixel resolution, 24 bits/pixel, 300 fps RAW content could result in 115 GB/s data rate, 414 TB/hr. A 360 degree, 4- camera shoot could produce 1.66 PB/hr RAW data. In 10 years, pro projects could easily generate 1 EB of data (a whole lot).
Consumers don't care if there's a valid business model or if the infrastructure will produce a positive return on investment, they want their content.
Thanks to all of the rapid-fire technology improvements the industry makes and the growing demand for video storytelling the M&E industry is producing, there's an increasing volume of content that has to be stored...somewhere.
The range of what indie filmmakers shoot/produce includes feature films, series, documentaries, company stories, ads and more.
In addition, filmmakers like Cirina Catania, of The Catania Group, capture content to tell visual stories they feel are best told with visual stories, rather than written words.
Her current project, The Kionte Storey Journey, which has been three years in production, is a documentary of a young Marine who was disabled because of an IED (improvised explosive device).
Storytelling - Cirina Catania directs Sgt. Kionte Storey on what she wants to capture from his early morning workout. She has been working on the documentary to help other returning vets, as well as those around them, to understand the costs of conflict and the way home.
She, and most filmmakers we know, use 4K cameras that capture 24-60 fps and more to deliver brilliant content.
Data Rates - High-performance, economical cameras today produce a tremendous amount of content, even at 24p; and a growing number of filmmakers are shooting at 30, 60 and more fps, resulting in increased storage requirements.
Since most cameras today use one flavor or another of SD (secure digital) cards, filmmakers are big buyers of the small flash memory-based cards.
VR Storage - Shooting a 360 VR film requires multiple cameras and a tremendous volume of flash memory storage. Despite the cost, VR filmmakers as a rule only use SD cards 10-20 times - just in case - but immediately move the content to their computer and external storage devices such as the OWC Envoy Pro to ensure the RAW content is saved in multiple locations.
While the cards aren't cheap, they are rugged and have a very good storage life.
Ordinary folks use them hundreds of times.
However, few filmmakers feel comfortable using them more than 10-20 times.
Even fewer will trust their only copy of the original content to the card.
Storage experts with a slight vested interest (Amazon, Microsoft, Google, IBM, Box, HP, Oracle, Apple and oh heck, anyone with spare storage capacity) have a great answer...the cloud.
At storage, video and cloud conferences, they constantly tell us how fast, easy and reliable it is.
Funny, it doesn't work that well in the real world--even when you're in the heart of Silicon Valley.
It's more difficult to push 10TB of content to the cloud than it is to download/stream, the same volume to your screen.
It's even tougher when you're in the middle of the Amazon or deep in the forest or in the middle of some high desert or anyplace where your connection seems a little iffy.
Conan 360 - To shoot, produce and air one of the first 360 VR projects in near realtime, members of Lewis Smithingham's (foreground right) 30ninjas VR team set up a complete post production facility - including a server and storage rack - up behind the stage with everyone focusing on delivering a complete show in less than an hour after the final shots. O'Brien, his team and the audience have used it as a benchmark for future VR shows.
Even if you're in the heart of Hollywood and the production team is working from a central server, you'll see FX, audio, colorists, editors with a strict time budget using the old faithful sneaker net (walking a drive from one workstation to another).
And even when time isn't the major concern, smart (or burned) filmmakers will immediately copy their RAW files to their computer SSD (solid state drive) and then copy it again to one or more external storage devices.
No one wants to tell the showrunner, producer/executive producer or marketing person that the only copy of probably the best stuff you ever shot went poof but is out there...somewhere.
Save, Safe - Award-winning filmmaker and documentarian Leslie Chilcott shoots and directs commercials and documentaries such as Al Gore's "An Inconvenient Truth." No matter how busy her schedule, she moves her content to her computer and storage devices like the OWC Envoy Pro Ex.
To ensure the RAW content is safe...and close at hand, that means immediately copying stuff from SDs to the computer SSD (solid state drive), then backing it up on one or more external storage devices.
Just in case.
Often, filmmakers can't wait until they get back to their disorganized desk to start work on the material--especially when they're moving from location to location or need to check the quality of the day's work (and show others the dailies).
Flying Post - To make ends meet while producing their episodic mini-series, "Please Tell Me I'm Adopted," Chris Sobchack, of Wraptastic Productions, handled sound production for the Elton John tour and did his post production work when he could (including on planes), moving content back and forth on his Mercury Elite Pro Dual Mini.
Even if there is a secure, direct connection; filmmakers tend to be a little paranoid about their content.
There's just something comforting about coming into a room (even if it's your bedroom/post facility) and seeing RAID boxes, stacks of external HDs and piles of SSDs.
Especially when reports like Deloitte's Tech Trends reassure us, "There is no such thing as hacker-proof, given how well-resourced, capable and methodical cyber criminals are these days".
So, do filmmakers avoid the cloud?
Today's workflow is complex and fast-paced because film production requires an array of freelance talent and an entire ecosystem of suppliers handling specialty work to deliver 4K/HRD or VR content that people just can't seem to get enough of today.
Virtual Production - Because of the complexity of today's 4K/HRD and VR film projects, filmmakers use the best production experts available, regardless of their location. Depending on the size of the work, segments are sent over controlled iNet connections or overnight courier services.
Even the best, most talented filmmaker works globally; which is one of the reasons the best tools are up there.
Adobe Creative Cloud, Avid Cloud Collaboration, Dolby, Harmonic and many others offer a range of solutions and capabilities for creatives around the globe.
You know; you in NYC, a VFX house in New Zealand, colorist in Berlin, audio experts in L.A., animator in Berlin, folie artist in Hollywood, producer on the plane.
It may look messy but notes, photos, video, audio, meta data, spreadsheets, social media plans/tools and homegrown databases are constantly in motion during the collaborative production process.
Can't Avoid - Because time, talent and financial budgets are always tight and work is done globally, filmmakers have to work in the cloud, just not everywhere.
Even M&E storage salespeople agree the cloud fits in the M&E workflow...just not everywhere.
It's because along the way, a film project produces a horrendous volume of data.
- Production - 3TB/day
* Camera RAW output to SSD - 960GB/day
* On site/on set de-Bayer - 2.1TB/day
* Creative editorial - 1TB per editor
* VFX - 10-20TB
* Color, mastering, versioning - 4.4TB per 4K version
- Copies and back-up of each step of the process
Most filmmakers calculate that they will need 10X the amount of storage for the original media shot.
And if you're doing VR, that storage requirements increases by a factor of 10!
Content Growth - With the burgeoning demand for quality video content for marketing, education, information and entertainment; the volume of data that needs to be stored continues to grow. Increasingly, filmmakers use a mixture of local and cloud storage to store and protect their content.
The demand for professionally produced content continues to grow; thanks to OTT services including Netflix, Amazon, Vimeo, Hulu and others as well screens everywhere. So do the number of production/post production operations.
Coughlin Associates estimates there will be more than 954,000 high-, mid- and low-end NLE (non-linear editing) post production seats next year, an increase of about five percent.
Add to that an equal number of special effects, audio and other post production workstations and suddenly the indie filmmaker and production person becomes pivotal to the success of the M&E ecosystem.
They're all focused on meeting the theatrical, network, OTT, social media pipeline demands for quality 4K/HRD and VR content.
With an industry that relies heavily on a widely dispersed workforce, embracing the ultimate cloud nirvana seems like an absolute must do!
It provides data redundancy, protection against hardware failure and reduces CAPEX (capital expenditure) costs.
That sounds compelling but...
It can be challenging to efficiently assess the risks and benefits of cloud storage to a production project, studio or filmmaker. Especially when the Internet was never designed to handle and support the type of workload film production requires -- low and fixed latency so media assets aren't lost in transit.
Of course, there is also that increasing concern about protecting the key components of the project and cybersecurity.
It sucks when an FX, audio or colorized service is hacked and segments are taken.
It can be crippling when the completed project or screeners are held ransom, posted on the dark web or simply up there and can't be found.
The cloud is an absolute necessity for the finished project archive as well as the third backup...just in case.
But nothing lets you sleep better at night than knowing there's a complete copy on the storage device in your office, in your partner's office across the country or on other side of the globe.
And, if you're really paranoid (meaning you lost your absolute greatest work once); with a copy on the bed stand, you can pet it when you go to bed at night and when you wake up.
Trust me; you'll sleep better at night. And the storage industry will thank you.