TiVo Community
TiVo Community
TiVo Community
Go Back   TiVo Community > Main TiVo Forums > TiVo Home Media Features & TiVoToGo
TiVo Community
Reply
Forum Jump
 
Thread Tools
Old 05-28-2012, 10:21 AM   #1
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
Question why bother with RAID if have backup (tivo hd recordings on server)?

Now I'm going to build a linux server for my tivo hd recordings .. after wrestling with going the blu ray route and being helped out of that choice on this forum.

I'm considering 2 servers, one solely to back up the primary server. If I do this, why bother with raid? Seems to me the only thing raid buys me is the possibility of a quicker recovery than restoring a backup; but maybe not, depending. OTOH rebuilding drives through raid thrashes the heck out of them, right?

So why bother with raid?
markmarz is offline   Reply With Quote
Old 05-28-2012, 06:51 PM   #2
wkearney99
Bill Kearney
 
wkearney99's Avatar
 
Join Date: Dec 2003
Location: Bethesda, MD USA
Posts: 1,479
RAID isn't to be used instead of backups. RAID allows you to keep things running, and rebuilding when a new drive is connected. This can be QUITE A LOT faster than having to reload everything from backups. Look at it this way, if it takes 24 hours to restore from backups, that's 24 hours of NO ACCESS. Meanwhile a RAID setup might take the same amount of time to rebuild, but STILL HAS ACCESS during that time.

But even a RAID setup can have catastrophic failures, so you still need separate backups anyway.

The drive IO for rebuilding is more than regular IO, yes. But it's hardly a matter of thrashing, especially considering how much better performing new drives are.

Last edited by wkearney99 : 05-30-2012 at 06:14 PM.
wkearney99 is offline   Reply With Quote
Old 05-28-2012, 11:36 PM   #3
heatherprotz
Registered User
 
Join Date: Apr 2012
Posts: 12
Its really great to have a backup no matter whether you are working in a computer or a TiVo. Your hard-found multimedia contents stay with you for ever and ever.
heatherprotz is offline   Reply With Quote
Old 05-29-2012, 05:15 AM   #4
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by markmarz View Post
Now I'm going to build a linux server for my tivo hd recordings
OK. Shout if you run into issues. Your first hurdle is going to be choosing a distribution. From rolling your own to the more obscure free distros to commercial distros like Red Hat, there is no wrong answer to this question, but some distros definitely facilitate certain aspects of computing and administration better than others.

Ubuntu is very popular, as is Fedora, but I much prefer Debian. Debian's packaging philosophy emphasizes rock solid stability. Just about all distros have a stable and an unstable or experimental release, but Debian has four: stable, unstable, testing, and experimental. Debian's testing version is roughly the equivalent of most distro's unstable versions, although for some, experimental is closer. Debian's unstable version is more like most distro's stable version. The Debian stable release is as rock solid as it gets. Nothing that has not been tested and scrutinized forty ways from zero is in stable. Some Debian releases have remained in unstable for more than 2 years before being shifted to stable. New features are never added into the stable release of Debian. Only fairly serious bug fixes and security updates are allowed to the packages in the stable release. Unlike other unstable releases which may countenance feature updates right until the release is deemed stable (or in some cases even after the release is converted to stable), Debian freezes feature development in the unstable release a considerable time BEFORE the release is published as stable. During that period, only bug fixes and security updates are allowed, and the systems are all scrutinized for problems before the release is deemed suitable for deployment as stable. After unstable is frozen, feature updates are only allowed in testing and experimental.

Stability and reliability are the holy grail of server deployment and for my money (well, time investment), that is Debian. If you want all the latest bells and whistles, then Ubuntu, which is a Debian derivative, is probably the way to go. Many people also prefer Ubuntu for Desktop Workstation use.

For industrial support, Red Hat / Fedora or Gentoo are good choices.

For a consumer residential server, I recommend Debian. The current Debian stable release is code named Squeeze. Squeeze contains 29,000 official packages covering a vast array of applications.

Quote:
Originally Posted by markmarz View Post
I'm considering 2 servers, one solely to back up the primary server. If I do this, why bother with raid?
First of all, some definitions:

Device - an I/O stream defined in /dev. It can represent a means of accessing a serial port, a network adapter, a hard drive, a tape drive (or other streaming media), a USB thumb drive, an optical drive, etc. Devices are either block devices, read and written in chunks, or character devices such as serial ports that are read and written one byte at a time.

Drive - a device accessed by the computer that stores data. It can be a magnetic hard drive, a read-write optical drive, an SSD, etc.

Partition - a section of a drive logically collected and separated from the bulk of the drive. Unlike Windows, under Linux it is not always necessary to create partitions on a hard drive. There can be reasons to do so, but Linux is perfectly happy to access the entire drive without partitioning.

Format - create a file system structure on a device. The Linux kernel and some low level user-space utilities such as tar, cpio, dd, and dd_rescue can access the raw data on a drive directly, but most applications are designed to work strictly with file structures. Linux supports a wide array of file systems, including the tiny handful recognized by Windows.

Member - a block retrieval random access structure formatted to be used as part of an array. It can be a physical device, a section of RAM (called a loop device), an entire drive, a partition on a drive, or an entire array.

RAID0 - Striping. All the members are stitched together into one large logical device, but the members are not simply strung together end-to-end. Instead, each device is divided into stripes, and instead of writing sequential blocks on each device, the array writes to sequential stripes on sequential drives. Thus, if D3S1 is the first stripe on the 3rd drive of a 4 member RAID0 array, the array will write data in the following order: D1S1 - D2S1 - D3S1 - D4S1 - D1S2 - D2S2, etc. What's more, the array writes all four S1 chunks simultaneously, writing D1S2 immediately after D1S1 is complete, not waiting for D2S1 - D4S1 to complete. The failure of any member probably results in the loss of all data.

RAID1 - Mirroring. The contents of the array are duplicated precisely across every member of the array. This provides redundancy only. There is no increase in storage capacity. All but one member of the array can fail without taking the array offline or losing any data. For the most part, the most suitable level if the array is to be booted is RAID1, because with the sole exception of a tiny bit of the member allocated as the superblock, a RAID1 member is identical to a non-member device. An ordinary existing bootable partition on a hard drive can be converted to a RAID1 member in a fraction of a second, and the hard drive can still be booted without RAID support.

RAID2 and 3 - Don't worry about them. No one uses them any more.

RAID4 - Striping plus redundancy via parity allocation. If the array has 5 members, one of them is reserved for parity. (If you don't know what parity is, look it up.) Thus, said array only has 4 times the storage of a single member, rather than 5 times as would be the case with a RAID0 array. OTOH, with parity calculated and stored, if one of the data drives is lost, the data on the member on that drive can be reconstructed by simply reading the other 3 data members and the parity member and XORing the four sets of data. Thus, data can continue to be read and written without the 5th drive until such time as it is replaced and the entire data set restored to it by the array management utility. The failure of 2 members takes the array offline. In that event, all the data may be lost.

RAID5 - Striping plus redundancy via distributed parity allocation. This is essentially the same as RAID4, except the parity is not stored on a single member. Instead, after each stripe is written, the parity chunk on the next stripe is written to a different member. This improves performance considerably, since the data is being read and written to all the drives in an N member array, not just N - 1. The vast majority of people used RAID5 or RAID6 instead of RAID4 these days.

RAID6 - Identical to RAID 5, except that there are 2 or more parity members, and the parity calculations on the additional parity members are a little different than on the 1st parity member. There are two advantages, here. The first and most obvious is with N data members and P parity members, the array can suffer the failure of any number up to P members without taking the array offline. Thus, a 7 member RAID6 array with 5 data members and 2 parity members can lose any 2 members without interruption of service. The second advantage comes in the form of data integrity. Suppose the computer looks at the data in the array, and the values in the data members does not match the data in the parity members? Which of the bytes is bad? The fact there are two parity bytes automatically convicts either the data bytes or the parity bytes. If the parity bytes all match (in the sense they all suggest the same data pattern, not in the sense they are equal, since they may not be), then it is the data bytes that are in error, not the parity bytes. OTOH, if one parity byte matches the data and the other does not, then the data should be OK, and it is the oddball parity byte that is bad. The thing is, a parity mismatch tells one there is an odd number of bytes in error, but not which bytes are in error. With more than one parity byte, however, and with the bytes being calculated differently, the parity bytes can do a good job of telling us which byte is actually in error, thereby allowing us to recover the data. This is something that single parity and even mirroring cannot do.

There are a number of more exotic flavors of RAID, most notably RAID10 and RAID50 / RAID60, but I'll not delve into them here. It's possible at some point you might consider RAID10, but I think for now you should stick to RAID5 or RAID6, with possibly RAID0 on your backup server, for now.

Quote:
Originally Posted by markmarz View Post
So why bother with raid?
1. Greater storage capacity. All levels of RAID other than RAID1 provide for chaining together multiple drives into a single larger logical volume. The limit for a single drive at the moment is 4 Terabytes. The limit for a RAID N array for N != 1 is several Petabytes 8 Exabytes. RAID1 is mirroring only, so no matter how many members one puts together, the limit for a RAID1 array is equal to the size of 1 drive, which at the moment is 4T. Note RAID1 allows for multiple mirrors, so for advanced redundancy, one can implement RAID1 with 3, 4, or even more members, allowing for multiple drive failures with no loss of data (or even any down-time).

2. Greater performance. Again, any RAID solution other than RAID1 distributes the data across multiple drives and reads or writes to all those drives simultaneously. As a result, a RAID array with 4 data drives is essentially four times faster than a single drive. The effective cache size (typically 32 or 64 MB in modern drives) is also effectively multiplied.

3. Flexibility and expandability. The RAID array can be grown (or shrunk) to accommodate evolving data requirements. Often this can be accomplished without taking the server or even the array offline.

4. Reliability. The redundant aspects of a RAID array and a backup solution are not designed to address the same issues. There is some overlap, to be sure, but they are not the same thing. A backup solution is designed to prevent any data loss irrespective of the cause of the potential data loss. It is not designed primarily (or at all) to keep the system running during the failure event. A RAID array is designed to allow the system to continue to function in the event of the failure of one or more of its members. Preventing the loss of data is implied, but only in the event of a member failure, and there are limits to how many failures can be tolerated. What's more, the security is strictly limited to losses in the event of member failures. Any other source of failure, including the very most common (user error) is completely unmitigated. It doesn't guard at all from data corruption.

Quote:
Originally Posted by markmarz View Post
Seems to me the only thing raid buys me is the possibility of a quicker recovery than restoring a backup
It certainly does that. For that matter, at the user level, there is no recovery at all. For any RAID level other than 0, the loss of at a minimum one member has no effect whatsoever on the data stream or server functionality. The array remains online, unless the number of member failures exceeds the redundancy level of the array. Then it is pucker time.

Quote:
Originally Posted by markmarz View Post
OTOH rebuilding drives through raid thrashes the heck out of them, right?
Well, not really. Not as much as writing all the data back from scratch, anyway, unless the data set is fairly small. Every active member must be fully read, and the replacement member(s) fully written. The active members don't get written at all. What's more, all the reads on the active members are sequential, so they are not only very fast (absolute minimum seek time), but they apply minimal stress to the members. The same is mostly true of the write. With data recovery, every member has to be written to the extent of the file allocations. The writes may not be (are probably not) sequential. Any respectable restore should include a verify, so the all the members get both read and written.

Last edited by lrhorer : 05-29-2012 at 08:24 PM.
lrhorer is offline   Reply With Quote
Old 05-29-2012, 05:32 AM   #5
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by heatherprotz View Post
Its really great to have a backup no matter whether you are working in a computer or a TiVo.
Make that:

It's really great essential to have a backup no matter whether you are working in a computer or a TiVo.

Quote:
Originally Posted by heatherprotz View Post
Your hard-found multimedia contents stay with you for ever and ever.
Well, they are fairly secure. Forever is a long time. I've had my video library intact for over six years without a major loss. This would not have been true without a solid backup strategy.

Last edited by lrhorer : 05-29-2012 at 05:51 AM.
lrhorer is offline   Reply With Quote
Old 05-29-2012, 06:40 AM   #6
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
Quote:
Originally Posted by lrhorer View Post
OK. Shout if you run into issues.
Thanks!

Quote:
Originally Posted by lrhorer View Post
I recommend Debian
Sold!

Quote:
Originally Posted by lrhorer View Post
Greater storage capacity.
This is the one RAID plus that matters to me. It's enough to push me to RAID.

I think I'm going with RAID5 (plus backup of course); I don't mind the risk vs the extra disk. The risk boils down to the system being unavailable until the data is restored from backup. With RAID6 the system is less likely to be unavailable than with RAID5, but with either RAID solution I'm going to move on a restore from backup asap when any drive fails. It's just movies, I can wait.

As far as the backup server goes, my gut tells me I want to use the same RAID level on it as on the primary, for ease of recovery. Right?

Thanks for your astoundingly helpful and thorough reply.
markmarz is offline   Reply With Quote
Old 05-29-2012, 09:10 AM   #7
MichaelK
Registered User
 
Join Date: Jan 2002
Location: NJ
Posts: 7,299
trying to figure out my next server solution. (right now I have a WHS v1 box- that i love for storing content, but there is a limit to the usefulness to run things. )

So trying to figure out RAID- my old understanding was all the drives in a RAID array had to be identical make and model- is that no longer the case?
MichaelK is offline   Reply With Quote
Old 05-29-2012, 09:18 AM   #8
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by markmarz View Post
Thanks!
You're welcome.

Quote:
Originally Posted by markmarz View Post
Sold!
http://www.debian.org

As long as you have a decent broadband connection, the easiest is to download one of the netinst images and burn it to CD or DVD as is appropriate:

http://www.debian.org/distrib/netinst

Select the one that matches the motherboard. If you get a motherboard with an Intel CPU, that would be i386. For a 64 bit AMD CPU, use amd64. You'll need some CD burning software that supports an ISO image. Load the CD on the new system with the hard drive installed and boot from the CD. You can select a desktop environment (GUI), if you like. If you are going to administer the server remotely and you don't intend to run something like kmttg on it, then you don't need a desktop environment. For a desktop environment, I prefer KDE to the default Gnome. If you want to try KDE, select Advanced Install and then Alternate Desktop Environments from the install menu. Otherwise, accept the defaults until the page comes up with Desktop Environment as one of the selections and de-select it.

Quote:
Originally Posted by markmarz View Post
I think I'm going with RAID5 (plus backup of course); I don't mind the risk vs the extra disk. The risk boils down to the system being unavailable until the data is restored from backup.
Yes, but there is also a risk of one of the drives on the backup failing before the restore is complete. It will probably take some time - maybe a day or two - for you to get a replacement hard drive after the RAID system notifies you via e-mail that it has lost a disk, and then perhaps as much as 2 or 3 days to restore the data (once you have a lot of it to transfer). That could be a week or more before all the files are done transferring. The risk that a drive could fail in that time is moderately high, when you are talking about a pool of perhaps ten drives. I suggest you think about it a bit, or at least keep it in mind as the drives age. I once had four drives out of twelve fail on an array simultaneously.

Quote:
Originally Posted by markmarz View Post
With RAID6 the system is less likely to be unavailable than with RAID5, but with either RAID solution I'm going to move on a restore from backup asap when any drive fails.
No, there's no need for that, and it poses a greater risk to your data. Just pull the dead drive, install the new drive, and go to bed (or whatever). A day or three later, the array will be clean.

Quote:
Originally Posted by markmarz View Post
It's just movies, I can wait.
Again, no need, and more importantly, replacing the drive is safer.

Quote:
Originally Posted by markmarz View Post
As far as the backup server goes, my gut tells me I want to use the same RAID level on it as on the primary, for ease of recovery. Right?
No. Recovering the files is done at the file system level, and the storage level is 100% transparent to the file system level. Restoring (or backing up) all the files can be as easy as typing

Code:
rsync  '/BackupRAIDArrayName/' UserName@MainServerName:'/MainRAIDArrayName'
from the source server and then waiting a day or so for the data to all transfer, assuming you have a gigabit link between the two machines. (Reverse the source and the target fields if you type the command from the server that is receiving the files.) I say that, but of course as soon as any one file gets transferred, it will be available, so the bulk of your videos will be available in half a day or so - less at the outset. The more drives you have in your arrays, the less time it will take. BTW, I definitely do recommend a gigabit link between the two machines. Transferring 1T of data over a 100M link will take over a day. Transferring 10T or 12T would take over a week or maybe two weeks. If you don't buy a gigabit switch, then put two LAN interfaces in one of the machines and link the two together via one of them, allowing the other to attach to your LAN. A lot of motherboards have two gigabit interfaces built in, or you can get a gig LAN adapter for under $10. The gig switch would provide the simpler setup, and a small 5 port desktop gig switch can be had for under $25. Here is a nice 8 port switch for under $30. I like the TRENDnet switches. My main switch is a TRENDnet managed 24 port Gig switch.

Last edited by lrhorer : 05-29-2012 at 09:37 AM.
lrhorer is offline   Reply With Quote
Old 05-29-2012, 09:44 AM   #9
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
[quote=lrhorer;9113985]You're welcome.

Thanks for the info on debian & desktop environment. I was planning on running kmtg & videoredo etc on a laptop which will ftp the edited movies to the primary server. I figure that'll be fast enough, and I won't have to give up videoredo, which I like. Also rather the server did little else than serve, though probably over time I'll be tempted (and will succumb) to putting more functionality on it.

Quote:
Originally Posted by lrhorer View Post
No, there's no need for that, and it poses a greater risk to your data. Just pull the dead drive, install the new drive, and go to bed (or whatever). A day or three later, the array will be clean.
That makes sense.


Quote:
Originally Posted by lrhorer View Post
No. Recovering the files is done at the file system level, and the storage level is 100% transparent to the file system level. Restoring (or backing up) all the files can be as easy as typing ..
Wait a sec. I thought I wasn't going to type anything to restore from backup, instead I was going to plop in the backup drive? Also, are you saying I won't be running RAID on the backup server? Finally one other thing nagging me: should the small boot drive be part of the RAID array? No, 2 other things: if the boot drive fails, can the backup be plopped in just as though it were a data drive. Something tells me no.

Quote:
Originally Posted by lrhorer View Post
I definitely do recommend a gigabit link between the two machines.
Currently have a Dlink 655 gigabit router, yesterday I ordered a Dlink 8-port gigabit switch. Sorry I didn't run that by you first; still I think it'll be fine, just cost more ($42 3 day shipping).
markmarz is offline   Reply With Quote
Old 05-29-2012, 06:38 PM   #10
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by markmarz View Post
Thanks for the info on debian & desktop environment. I was planning on running kmtg & videoredo etc on a laptop
If you set up a desktop environment, kmttg will run on the server just fine. Indeed, kmttg, TyTool, and Galleon are the only reasons I have a desktop environment on the server. The server is headless, so I use XDMCP to bring up the desktop from a remote workstation.

Quote:
Originally Posted by markmarz View Post
which will ftp the edited movies to the primary server.
I recommend you simply share the RAID array on the network. That way you can simply create a network drive on the laptop and have Video Redo write the finished products directly to the share from its batch manager. 'No need for an additional manual ftp operation.

Quote:
Originally Posted by markmarz View Post
I figure that'll be fast enough, and I won't have to give up videoredo, which I like.
Yeah, I've asked the developers again and again to develop a Linux port, but they have not yet done so. They haven't refused, but they haven't launched the project yet, either.

One more recommendation. If you want to use Video Redo to recode your videos to h.264, then be sure to have VRD to store the temporary file on the local hard drive. For some reason, writing the temp file to a network share really slows down the recode a lot. Reading the temp file from the local drive and writing to the network is fairly fast.

Quote:
Originally Posted by markmarz View Post
Also rather the server did little else than serve, though probably over time I'll be tempted (and will succumb) to putting more functionality on it.
That's a pretty good philosophy, especially if you get a very low end CPU. Mine is no race horse, but I do have kmttg run comskip to generate a project file to use as a starting template for editing in VRD.

Quote:
Originally Posted by markmarz View Post
Wait a sec. I thought I wasn't going to type anything to restore from backup, instead I was going to plop in the backup drive?
You mean using a set of removable drives for offline backups? That can be done, and in that case you don't need a backup server. Indeed, I do that very thing. The backup server is my main backup strategy, but I do use dar to back up the entire backup server, as well, from time to time. Dar allows one to break up the backup into even sized chunks and write to multiple targets. I bought a USB hard drive dock similar to this one, and I use it with dar to make static backups using drives I have decommissioned from other systems (mostly 1T drives). They are then taken off-site for safe storage. Since the majority of files (especially by size) are videos that never change, this works quite well.

Quote:
Originally Posted by markmarz View Post
Also, are you saying I won't be running RAID on the backup server?
No, that isn't what I was saying. I was saying RAID0 might be an OK choice for the backup server. It's not redundant, but an array failure on the backup server only disables your backup strategy (or part of it). I would not be comfortable with this strategy as the only backup solution, but you may consider it a reasonable risk, at least at first. Note a RAID0 array can be rather easily converted to a RAID5, RAID6, or RAID10 array at a later time. You could also start out by building a RAID5 or RAID6 array, but just build it with 1 or 2 drives missing, respectively. That way, upgrading is just a matter of adding an additional drive or two, respectively, at a later time.

Quote:
Originally Posted by markmarz View Post
Finally one other thing nagging me: should the small boot drive be part of the RAID array?
No. If you go with a small boot drive, it will be completely separate from the array. Its contents can be backed up to one or the other of the arrrays (or both), but it would be physically and logically separate. One can boot from the data array, but I do not recommend it, especially for novice admins.

Quote:
Originally Posted by markmarz View Post
No, 2 other things: if the boot drive fails, can the backup be plopped in just as though it were a data drive. Something tells me no.
Well, I am not entirely certain I follow. Are you asking if the boot hard drive could be removed from the backup server and simply put into the main server as a boot drive? 'Not plug and play, no. There are a number of things, including the IP address and the hostname that would have to be changed. If you are very methodical about setting up the hard drives on both systems, the changes can be fairly minimal, but I recommend instead creating a drive image to restore to a fresh drive in the case of a boot failure. There are several ways to do this. The brute force approach is to simply take a blank drive of the same size, install it in the working system, boot from CD, and dd the entire contents of the active drive to the standby drive. A better, slicker alternative is to create a RAID1 array of two small drives for the boot. The second drive can then be taken offline and stored on the shelf, or just left in place so that you are booting to a RAID1 array (3 RAID1 arrays on one drive, actually). An alternate (or better yet additional) solution is to have the system create regular backups of the boot drive to be stored (in compressed form) on one or both of the data arrays. I do both, boot from RAID1 arrays and have the system automatically create backups to the data arrays.

Specifically, each server hosts a pair of 500G drives (they could be much, much smaller, but I had these lying around) with three partitions. One tiny partition is /dev/sdX1 formatted as ext2, and it contains the boot data. One takes up over half of each drive as /dev/sdX2 formatted as ext3. The rest of the drive is partitioned as swap in /dev/sdX3. I then created three RAID1 arrays of /dev/md1, /dev/md2, and /dev/md3 consisting of the 1st, 2nd, and 3rd partitions on the two drives, respectively. This can all be done either before or after installing the OS, BTW. After may be easier. If nothing else, that way you can save some $$ up front by not buying a second boot drive for each machine at first.

Oh, BTW, the boot drives can be USB thumb drives, if you want. It takes a little more effort to set them up, but they work fairly well. 'Not the fastest things on Earth, mind you, but a 64G flash drive can be had for about $50, and once the system is booted the speed of the flash drive is not as much of an issue. Without the GUI, 32G is probably plenty.

Quote:
Originally Posted by markmarz View Post
Currently have a Dlink 655 gigabit router, yesterday I ordered a Dlink 8-port gigabit switch. Sorry I didn't run that by you first; still I think it'll be fine, just cost more ($42 3 day shipping).
That will work just fine.

Last edited by lrhorer : 05-29-2012 at 07:12 PM.
lrhorer is offline   Reply With Quote
Old 05-29-2012, 08:10 PM   #11
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
It *IS* its own thread. Mark named this thread, "why bother with RAID if have backup (tivo hd recordings on server)?" 'Sounds like a RAID related question to me. Then his very first statement was:

Quote:
Originally Posted by markmarz View Post
Now I'm going to build a linux server for my tivo hd recordings ..
'Sounds to me like a brief discussion of which distro is the best fits right in with that theme, if you ask me. Exactly how is a discussion of RAID deployment and Linux distros OT to this thread?
lrhorer is offline   Reply With Quote
Old 05-29-2012, 08:28 PM   #12
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Oh, one note on the array size limit mentioned above. The 8 Exabyte array limit only applies if the OS is 64 bit. If you run a 32 bit OS, the limit is 16 Terabytes. It will take a while, but it won't be difficult to exceed 16T, so I suggest you get a 64 bit CPU. Since all modern CPUs are 64 bit, this should not be an issue, but if you get a very old motherboard and CPU, you could wind up against a wall in a year or two.
lrhorer is offline   Reply With Quote
Old 05-30-2012, 04:35 AM   #13
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
Quote:
Originally Posted by lrhorer View Post
I suggest you get a 64 bit CPU
I wouldn't have thought of that, thanks! I think I'm cool:

http://www.newegg.com/Product/Produc...82E16813131697

I believe the AMD E-350 APU is a dual core 64 bit processor
markmarz is offline   Reply With Quote
Old 05-30-2012, 04:44 AM   #14
wkearney99
Bill Kearney
 
wkearney99's Avatar
 
Join Date: Dec 2003
Location: Bethesda, MD USA
Posts: 1,479
Quote:
Originally Posted by lrhorer View Post
It *IS* its own thread. Mark named this thread, "why bother with RAID if have backup
Um, yeah. brain fart there. Too many tabs, or something... nevermind.
wkearney99 is offline   Reply With Quote
Old 05-30-2012, 04:53 AM   #15
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
Quote:
Originally Posted by lrhorer View Post
If you set up a desktop environment, kmttg will run on the server just fine. Indeed, kmttg, TyTool, and Galleon are the only reasons I have a desktop environment on the server. The server is headless, so I use XDMCP to bring up the desktop from a remote workstation.
Excellent! Will do.

Quote:
Originally Posted by lrhorer View Post
I recommend you simply share the RAID array on the network
Also a great idea! Once I get more familiar with linux, raid & managing a network I hope to come up with these ideas myself. In the meantime thanks for the education!

Quote:
Originally Posted by lrhorer View Post
One more recommendation. If you want to use Video Redo to recode your videos to h.264
This really is something I'll probably open a new thread for; I've had some problems playing around with h.264 in the past .. something about trick play or aspect ratios .. I'll play around later and see.

Quote:
Originally Posted by lrhorer View Post
You mean using a set of removable drives for offline backups?
Apparently I did mean that ... or did I? I was assuming when you said to let the RAID rebuild with the new drive that the new drive was in fact the backup drive. But now I gather by new you meant new new (ie, formatted). If that's the case then I guess the only use for a backup is when there are too many drive failures for RAID to handle. Right?

The idea of a removable drive backup instead of a backup server is very appealing to me; I'll look into that more.

Quote:
Originally Posted by lrhorer View Post
Are you asking if the boot hard drive could be removed from the backup server and simply put into the main server as a boot drive? ...
Oh, BTW, the boot drives can be USB thumb drives, if you want.
Maybe the easiest solution is to somehow migrate to a couple of thumb drives after I've successfully got the server running off a disk boot drive; maybe then I can use another thumb drive as backup and that's that. The whole boot drive backkup discussion raises a lot of questions; perhaps another thread is best? They can wait for now.

Thanks a million!!
markmarz is offline   Reply With Quote
Old 05-30-2012, 08:26 AM   #16
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by markmarz View Post
http://www.newegg.com/Product/Produc...82E16813131697

I believe the AMD E-350 APU is a dual core 64 bit processor
Yeah, that looks like a good choice for this application.
lrhorer is offline   Reply With Quote
Old 05-30-2012, 08:36 AM   #17
lrhorer
Registered User
 
Join Date: Aug 2003
Location: San Antonio, Texas, USA
Posts: 6,869
Quote:
Originally Posted by markmarz View Post
This really is something I'll probably open a new thread for; I've had some problems playing around with h.264 in the past .. something about trick play or aspect ratios .. I'll play around later and see.
There is an issue with GOP sizes that causes the first FF to be effectively disabled. Changing the GOP size should relieve the issue. I haven't had time to play with it, and it's not a big enough deal to cause me any real concern.

Quote:
Originally Posted by markmarz View Post
Apparently I did mean that ... or did I? I was assuming when you said to let the RAID rebuild with the new drive that the new drive was in fact the backup drive. But now I gather by new you meant new new (ie, formatted). If that's the case then I guess the only use for a backup is when there are too many drive failures for RAID to handle. Right?
The main use is probably going to be when someone accidentally deletes or overwrites a file, and you need to recover it. Another real possibility is data corruption during a file system or RAID reshape. (It happened to me.) A failed array due to too many unrecoverable drive failures is a distinct possibility, although I have never had it happen, myself, at least not yet.

Quote:
Originally Posted by markmarz View Post
The idea of a removable drive backup instead of a backup server is very appealing to me; I'll look into that more.
It's a reasonable compromise for a solitary backup solution. It's downsides are speed and convenience of recovery and the fact it must be done manually, at least in part. It s a highly robust solution, though, and a very economical one.

Quote:
Originally Posted by markmarz View Post
Maybe the easiest solution is to somehow migrate to a couple of thumb drives after I've successfully got the server running off a disk boot drive; maybe then I can use another thumb drive as backup and that's that. The whole boot drive backkup discussion raises a lot of questions; perhaps another thread is best?
Or you could create a RAID1 using the hard drive as a primary boot device and a thumb drive as a backup boot device. There is a RAID1 parameter called "Write Mostly" to accommodate just such a scenario. When one sets Write Mostly for a RAID1 member, the system avoids reading from that member unless it has to. By applying this to a slower member, it speeds up the performance of the array while still allowing for low cost mirror solutions such as a thumb drive. The thumb drive only gets read if the hard drive is unavailable.

Quote:
Originally Posted by markmarz View Post
They can wait for now.
RAID1 solutions are extremely fast and easy to implement after the fact, since it involves no massaging of the data on the drive. Turning a non-RAID device into a RAID1 member involves nothing more than simply creating the superblock and writing it to the beginning or the end of the device. Adding a new member is also fairly fast, very straightforward, and very safe, since no calculations or data moves are involved. The RAID manager simply copies the entire primary member to the new member byte for byte. By comparison, adding a RAID5 or RAID6 member requires moving every single byte of the entire array (other than the very first data block of each member and the very tiny superblocks) to new locations.

Last edited by lrhorer : 05-30-2012 at 08:55 AM.
lrhorer is offline   Reply With Quote
Old 05-30-2012, 05:42 PM   #18
markmarz
Registered User
 
Join Date: Feb 2002
Location: Chicago, IL
Posts: 80
Thanks again, sir, I owe you more than a few!
markmarz is offline   Reply With Quote
Reply
Forum Jump




Thread Tools


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Advertisements

TiVo Community
Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
vBulletin Skins by: Relivo Media

(C) 2013 Magenium Solutions - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not owned or operated by TiVo Inc.
All times are GMT -5. The time now is 05:23 PM.
OUR NETWORK: MyOpenRouter | TechLore | SansaCommunity | RoboCommunity | MediaSmart Home | Explore3DTV | Dijit Community | DVR Playground |