Qualstar logo
LeadershipGeneralFetched transcript

Data Management For Production

Hi, Jeff Sangpill here with Qualstar. When it comes to collaborating on content for film, television, or the internet, there's so much information available on how to write better, shoot better, and edit better. But just like you see a massive chunk of mass be...

Indexed from @Qualstar_Channel videos · 12 min read

If embedded playback is blocked on your network, use the Watch on YouTube link above.

Hi, Jeff Sangpill here with Qualstar. When it comes to collaborating on content for film, television, or the internet, there's so much information available on how to write better, shoot better, and edit better. But just like you see a massive chunk of mass below the surface of the ocean from an iceberg, data management is one of the most important parts of operating a smooth production process, but it always happens below the surface. One is the loneliest number of the 321 backup strategy. As the horrific wildfires raged in Los Angeles at the beginning of 2025, my phone kept ringing. story similar to I followed all the data backups, but they're all in the same building. So, let's say this again. Three backups of the data stored on two different types of mediums with one copy off site. The reason people were glued to their televisions, helpless as sections of roads were blocked off and internet access was down, was that all of the eggs were in one proverbial basket, a basket they couldn't even get to. They hadn't followed the one in the 321 data strategy. How big of a disaster do you need to plan for? The events then showed that it can be a pretty big one and that the one in all of this should have at least been all the way out in Palm Springs. Here's three examples of the 321 architecture in action. So you got primary, nearline, and LTO offsite. The primary is spinning disc. It contains the work the team is actively working on right now. A backup copy is stored on nearby nearline storage in case the primary storage failure happens and work can still continue at that point immediately. That data is also pushed to LTO tape. The LTO tapes are taken out of the library and taken to a

location far enough away to make sure that the primary risk that can be in your area is planned for. fires, hurricanes, terrorism. These are not pleasant things to talk about. And if your duplicated media is outside of the range of the typical problem for your geography, you're pretty well covered. It's like option two, mezzanine media plus LTO, camera originals, LTO offsite. So, you've got your primary spinning disc containing the mezzanine media. That's the stuff you've transcoded to work for editorial. The camera originals are on an LTO tape somewhere in your facility with a copy of the tapes there and then another copy possibly in another library or on a shelf on the other side of the country. Camera originals are then pulled to primary spinning discs for finishing before delivery. You've got two library options here and either one can fulfill finishing production if something unpleasant happens. Then option three, nearline plus cloud spinning mezzanine LTO original camera media. In this scenario, all camera originals are ingested to nearline spinning disc. A copy of that everything is then pushed to a cloud deep archive like AWS Glacier. After that data push is completed, a transcode process creates mezzanine materials for the primary storage for the creative team to work with. Camera originals are duplicated from that nearline disc to LTO as your disaster recovery insurance. This leaves the LTO as the quicker access to the full resolution media and the too expensive to egress without an insurance check in your hand cloud archive as your off-site location. This is similar to how Netflix operates in some of their productions. All three of these are

valid data protection concepts. However, each would recover from a disaster at different speeds. The nearline plus cloud spinning mezzanine LTODR example would have business continuity in a few hours. The primary nearline and LTO off-site example could have business continuity in a few days, while the methanine media plus LTO camera originals and LTO off-site example could have business continuity continuing in a few weeks. Why is data management important? The typical film, television, or video production will have a stack of hard drives or camera cards with media that came in from the field or the set. The bonus is if the production uses a shared storage server for video collaboration within the facility. The problems with using only separate hard drives in creative environments can be numerous. Hard drives do fail. Media can get corrupted. Ransomware attacks can lock up whole projects or even possibly a whole NAS environment or disorganized data can make it impossible to remmonetize footage for future productions. It only made sense on that first run through. Someone tries to look at it later, no idea where anything is. Data management and data protection can be as simple as having a backup hard drive or an LTO tape. These are good, better, and enterprise options for addressing ransomware protection, disaster recovery, and business continuity. So, let's break down those three terms and get into the best practices and recommendations for you. So, tiers of archive. A media archive is simply a place to store production assets for later use. Depending upon when you need the files, how the production team will use them, and the

overall cost structure is going to determine the type of archive method you're going to want to employ. This could be active archive. It's where you store media and files that you need to be used often. For example, evergreen media, establishing shots and recaps and teases from previous shows or seasons. That's act could be active second unit or B-roll footage. You need to use the media quite a bit. So, it needs to be easy to access when needed, which is maybe a few times a week, but not as expensive as storing it on your top-of-the-line shared storage environment per terabyte. Inactive archive, this is not an exact science, but let's say you need to retrieve a set of media files every six to nine months. Let's say you cannot wait 18 plus hours to retrieve the media when you need it. An inactive archive is great for this. Inactive archives typically come in the form of inexpensive spinning discs, LTO tape without a rapid retrieve setup, or your lonely one copy of LTO far away from everything but in a library and able to be easily accessed over the internet. So then there's deep archive. This is the copy to keep the compliance, accounting, and lawyer types happy. Deep archives tend to be original camera copies or fully mastered materials. Back in the day, this deep archive was physical film reels or boxes of videotapes in dry basement that looked like the warehouse from Raiders of the Lost Ark. I've been there. Smells funny. The cost is typically low per unit, and the media would eventually be shipped to far-off vaults or to old limestone quaries for safety. The other difficulty with this particular archive was it takes real estate and real estate values change. So

suddenly your very large warehouse is no longer available as it makes more sense to turn the space into condos. Always have to be looking to the future when you're thinking of your archives. Today's deep archives have tended to be in deep freeze cloud archives which is really just LTO in someone else's cloud that you pay monthly or LTO tapes that are on library shelves because the cloud isn't free. And LTO tapes do need occasional repacking to stay current on the LTO formats, but that total cost of ownership is much lower when you have it in your possession. This has resulted over time in a process called repatriation to restore your data sovereignty and get the total investment down by pulling things out of cloud archives. Backups are typically work in progress. It's a snapshot. It's your project, your data, your mezzanine or editorial media and V effects. Typically, it's a quick recovery spot where you can also quickly go and grab something to go back to in case something's accidentally deleted. Where's that bin? Some teams will transfer backups into archives depending upon their workflow. Other teams have no need for these snapshots after the delivery is done and that's okay too. Depending upon the depth of that backup and the material kept there and the location of this backup, it could also be a smallsized disaster recovery area. So business continuity in some ways disaster recovery and business continuity are like half siblings. Business continuity is essentially the rental car coverage in the disaster recovery insurance plan. Business continuity is the simple idea that delivery deadlines don't stop if there's a disaster and it allows for a level of

redundancy that keeps you able to continue to deliver despite the problems whatever situation is thrown up for you. In smaller productions, backups can be the business continuity. If the primary storage dies, someone can open their backups on the second tier spinning disc and continue to work off of them directly with very little lost time, but not necessarily in the same collaborative method you were before. So for some, a nearline storage solution may make sense. Making a copy of your primary storage and allowing staff to continue working if that something goes wrong. Sometimes that's going to be a triage process. What material do you really need to actually access to continue working toward the real deadlines that are coming possibly tonight? So, other business continuity concepts are a little more robust. They address scenarios like the LA fire issues we talked about at the beginning. Your entire building's lost, all the data, or a large-scale disaster making data or systems completely inaccessible for a good length of period. This could be a copy of all your work in progress. A large-scale backup living in a geographically separate facility, readily accessible to alternative creatives to continue the work right where it left off. The concept is simple. Your business does continue to work regardless of the circumstance. Sometimes this continues while the larger, more painful effort goes on in the background, the actual recovery from the disaster. So, disaster recovery, DR, it's really become the catchphrase for all of these other concepts we've talked about. But what really is it? It's very simple. When something terrible happens, disaster recovery is

the process and strategy to recover from the point of disaster. From a physical p point of perspective, if a building suffers a fire, flood, or an earthquake, the process that's followed is to pull out what you can keep, rebuild this building or move to a new building, and then eventually reopen and pick up where you left off. Could be months, could be years. It involves insurance companies, contractors, painters, engineers, interior decorators. It might look better. While that process is happening, you sometimes cannot use the workspace at all. Can your business afford to wait that long to continue work? There are some that can. So, to be clear, how are business continuity and disaster recovery different? Business continuity lets people continue to work. Well, disaster recovery rebuilds the ability to work as you did before. From a data perspective, just having your media and files backed up somewhere else is the first step of a disaster recovery process. Your business is going to need to decide the importance of the proximity for your backup, whether it needs to be stored in a different building or a different state. It gets into a concept that's called blast radius. How large of a disaster can your recovery from. And why that lonely one somewhere in 321 out there makes more and more business sense. Once your previously damaged facility is back to being operational, the data can all be repaired and restored and disaster recovery takes as long as it needs to take. If you need disaster recovery to take less time, then what you really need is business continuity. The important parts of disaster recovery and business continuity are how current and accurate the backup files are compared

to your primary storage. This is process driven. It could be nightly incremental data flows to keep the data up to date. it might do the trick for you. Or if your deadlines are tight and your creative talent exceedingly expensive, then smaller increments of time make sense. For some productions, losing just one bin of sequences that only had a few hours worth of work is a complete disaster. LTO is the lowest total cost of ownership way to engage in disaster recovery and in some cases business continuity. If you need help to discover what LTO can do for you and ensure your business can continue working in the event of a disaster, please give us a holler here at Kalstar. And while you're here, I'd always love a like and subscribe.