1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Roamio 4Tb Drive Development Thread

Discussion in 'TiVo Upgrade Center' started by eboydog, Apr 25, 2014.

  1. Apr 25, 2014 #1 of 105
    eboydog

    eboydog Just TiVo'ing.....

    904
    0
    Mar 23, 2006
    Is anyone working on DIY 4tb drive upgrade and is it worth starting a new thread here ? I must say that after having 3tb of storage in my Roamio, it's somewhat overwhelming and I do see having 4tb of storage as being unmanageable. I would like to figure this out simply as a challenge to overcome but in a practical day to day operation, having that much recording storage is crazy!

    I'm not a hex editor type but with the recent thoughts of the image copy distribution of the drive that came up, I have an idea I would like to throw out to perhaps get things started in a legit manner rather than just copying someone else's work. Can't tell me that there are only one or two people on the Internet (other than Tivo themselves) that can figure this out!

    My out of the blue idea: Has anyone done a sector copy of stock Roamio drive and then applied it to a 4tb drive? I plowed though the big Roamio drive upgrade thread and saw what happens when you put a raw 4tb drive in and the box tries to format it but rejects it but, what would happen if the basic unreadable image of a stock drive was written to the beginning of a 4tb drive, could that possibly overcome that initial rejection? I'm curious what something like Clonzilla could do.... It's my thought that perhaps there is no partition on the Roamio drive and when the box boots up, it looks for the drive and reads raw data off the drive at the beginning of the drive which contains "is this good" drive and if so the OS continues the boot and initialization process until everything is loaded. This is just my thought as unless I'm wrong, all conventional attempts to read the partitions in the conventional manner has failed, there must be a completely new and different file/data structure in place which would indicate a raw, most likely encrypted, data being written to the drive in a Stream like process.

    Another issue which I mention in another thread which is a possible clue, those who have the 4tb drive solution are offering it for each Roamio model, their drive kits they offer don't appear to be a universal solution but each prepared drive only works in the model the buyer intends to install the prepared drive in. DVR Dude is very explicit in his eBay description that his kits can't be installed in a different Roamio model than specified. That implies there is some basic required identifying information on the drive that is required to start a successful identification process when the Roamio boots up. If the drive imaging process of a stock drive doesn't work, is there a utility that can read the raw sector data on a drive, save it and then rewrite such to another drive? Again I'm not describing using one of these prepared drives to accomplish this but to use a initialized stock drive to prepare a 4tb drive.

    I have a extra basic Roamio and I'm pretty sure I going to buy a 4tb drive and start experimenting, in the worse case I can use the 4tb drive for other purposes if it doesn't work out but I'm hoping others might chime in and offer practical ideas. I'm willing to start something with providing the basic hardware and time, however I need assistance from some of you more knowledgeable TiVo hard drive gurus.

    What do you think?
     
  2. Apr 25, 2014 #2 of 105
    ggieseke

    ggieseke Active Member

    4,031
    12
    May 30, 2008
    If you clone a Roamio drive to a 4TB drive it should work, but it won't use any of the extra space. Given the limitations of the 32-bit partition table the trick is to resize the first physical MFS media partition (#13) and move everything else around so that the second physical media partition (#11) starts just short of the 2TB limit. Then you resize that partition to use the rest of the available space.

    Resizing MFS partitions is new ground. In the past you just added a "blank" partition to the end but that won't work past 2TB.

    I'm working on a program to copy and expand any TiVo to its max, but it's months away.
     
  3. Apr 25, 2014 #3 of 105
    eboydog

    eboydog Just TiVo'ing.....

    904
    0
    Mar 23, 2006
    OK, I had the misunderstanding that there was some mystery "ghost like" partition on the drives that no known utility or OS could read or manipulate.

    The lack of discussion I thought was unusual but I guess most TiVo users are happy with 3tb.
     
  4. Apr 25, 2014 #4 of 105
    ggieseke

    ggieseke Active Member

    4,031
    12
    May 30, 2008
    Roamios use slightly different structures for the partition maps, MFS headers and zone maps than earlier models but they're still recognizeable. The main reason that existing programs like jmfs don't recognize it as a TiVo drive at all is that they switched the byte order.

    It's just a matter of time. I know what I want to write and (mostly) how to do it, but I'm swamped at work right now.
     
  5. Apr 25, 2014 #5 of 105
    unitron

    unitron Active Member

    16,390
    2
    Apr 28, 2006
    semi-coastal NC
    When you say "switched the byte order", are we talking about Series 1 type byte swapping to accommodate the CPU?
     
  6. Apr 26, 2014 #6 of 105
    ggieseke

    ggieseke Active Member

    4,031
    12
    May 30, 2008
    Series 1s just swapped every pair of bytes in the sector something like this:

    01 23 45 67 89 AB CD EF...
    23 01 67 45 AB 89 EF CD...

    You had to read everything in sector size chunks and 'swab' them before you could start to interpret the results.

    Roamios use Intel byte order (least significant byte first) in Block0, the APM and MFS data structures instead of the Motorola byte order (MSB first) used by all earlier models. For 2 byte values like the Block0 signature it looks the same as a Series 1 and 14 92 becomes 92 14. Four byte fields like the partition start and size in the APM are completely reversed and you read them right to left. The same goes for 8 byte fields in some of the MFS header structures. For instance, the signature field for a 64-bit MFS volume header is 0xEBBAFEED. In a hex editor it looks like this:

    EB BA FE ED (Series 2-4)
    ED FE BA EB (Roamio)
     
  7. Apr 26, 2014 #7 of 105
    unitron

    unitron Active Member

    16,390
    2
    Apr 28, 2006
    semi-coastal NC

    But is the change to accommodate the CPU, like the S1s, or for some other reason?
     
  8. Apr 27, 2014 #8 of 105
    ggieseke

    ggieseke Active Member

    4,031
    12
    May 30, 2008
    I suspect that the new Broadcom chips use LSB first, but I don't have any hard data on them.

    On Series 1s, I think that the byte swapping may have been inherent in the I/O circuitry. Even back then there were more 32-bit fields in the various headers than 16-bit fields, so matching it to the CPU doesn't really make much sense. It also made a complete hash of string values, requiring even more code for the CPU to run unless it was somehow handled on a lower level.
     
  9. May 9, 2014 #9 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    Mips chips are generally endiness flexible, but determined by the board designer. It is odd Tivo chose to make a switch at this point, but it might make sense if their hardware designs are coming from 3rd parties and the industry standard was LSB.

    I'm thinking of the Cisco, Samsung, Mini and Stream boxes here which predate the Roamio release. And the Broadcom reference platforms.

    I spent a couple nights looking into this 4TB issue.

    My initial inclination is that there's a bug in the MFS creation code, so expect this is something Tivo will fix at some point. Has someone tried switching to a 4TB drive after the box was updated to 20.4?

    Aside from MFS, the rest looks straightforward to me. One thing that would help down the road is the Roamio linux kernel. Maybe this deserves it's own thread but Tivo has fallen significantly behind in posting it's GPL and OSS code base.

    Someone want to go ask them for it?

    PS. Is there any reference / good thread on MFS data structures? I couldn't find any beyond generalities.
     
  10. May 9, 2014 #10 of 105
    unitron

    unitron Active Member

    16,390
    2
    Apr 28, 2006
    semi-coastal NC
    Yes, they come as part of the TiVo service manual package.


    Unfortunately, that gets delivered by Unicorn Express.
     
  11. May 10, 2014 #11 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    I'm getting from a hard drive + hex editor:
    Code:
    Series 5: feed ebba 
    Series 4: baeb edfe
    Edit: and that would be because I'm in word mode instead of byte mode. Nevermind.
     
  12. May 10, 2014 #12 of 105
    jmbach

    jmbach der Neuerer

    1,558
    10
    Jan 1, 2009
  13. May 10, 2014 #13 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    Yah, I looked through that and everything else I could find. I believe there are no Roamio era releases in anything on the Tivo website.

    I'd be happier to be wrong here.

    The reason I conclude that summarily is the Roamio logs indicate the kernel is Linux 3.3.8, and all the sources I could find on Tivo.com were 2.something.

    It might still be useful to someone.

    There is a program called pdisk64 for example. Has a number of interesting comments. I myself didn't see how to get it built, nor could I tell if it was bi- until it was built. Real C programmers could probably do it, cause I think I was just short 1 include for 1 function call which is probably part of the OS.
     
  14. May 10, 2014 #14 of 105
    jmbach

    jmbach der Neuerer

    1,558
    10
    Jan 1, 2009
    The Premiere OS version number on the site I do not believe corresponds to the Linux version. I am wondering though if 2.0 stands for OS 20.x.x. I haven't spent time looking through it to determine if that is the case. It just that there was an OS 19.x.x and the Web site has a version 1.9. So it leads me to think so. But I have not confirmed that in any way.
     
  15. May 11, 2014 #15 of 105
    ggieseke

    ggieseke Active Member

    4,031
    12
    May 30, 2008
    I doubt that the Linux kernel would do us much good anyway since it's all in flash memory on a Roamio and the open source code that they do release has nothing to do with MFS.
     
  16. May 11, 2014 #16 of 105
    jmbach

    jmbach der Neuerer

    1,558
    10
    Jan 1, 2009
    That is true. Would help premieres more. The previous open source releases had some MFS building and checking tools.
     
  17. May 23, 2014 #17 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    I'm at the point that my next step would be to make a test release.

    This would a 4TB whole disk image with large chunks of zero-ed data.

    I'm going to start a build tonight, and wanted to know if there's consensus on preferred image formats.

    Easiest for me would be, dd and gzip, which I know would support Linux and Mac. Which leaves Windows?

    I've looked at .vhd but the documentation I saw said it only goes up to 2TB. I think some versions of VMware (VMDK) and VirtualBox (VDI) can go to 4TB.
     
  18. May 23, 2014 #18 of 105
    eboydog

    eboydog Just TiVo'ing.....

    904
    0
    Mar 23, 2006
    So there is no way to exclude.the.zero padded part ?

    myself I envisioned.a image of.sorts.that.would contain the intial.byte image signature.instead of an entire.disk image.can you.offer any details.in what.you.did.to allow it to recognize.the 4tb drive?
     
  19. May 24, 2014 #19 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    > So there is no way to exclude.the.zero padded part ?

    I'm not saying that, but adding that functionality at the same time as the first release would negate the testing data if a bug were ever found. You wouldn't be able to tell if the bug is from Tivo OS, the original image, or an insufficient reimaging process / previous hard drive data. Limiting the initial scope restricts the number of variables to a degree that it could be debuggable if the need arises.

    > myself I envisioned.a image of.sorts.that.would contain the intial.byte image signature.instead of an entire.disk image.

    Could be possible. But my goal was certain compatibility, which directs me toward a different type of release than that. I think my goal order is Compatibility, followed by [running] Performance, followed by Speed [to image].

    My choice, doesn't diminish yours. Projects are not exclusive. The more the merrier.

    > details.

    forthcoming, after I'm done building and testing (then documenting).
     
  20. May 24, 2014 #20 of 105
    nooneuknow

    nooneuknow TiVo User Since 2007

    3,554
    0
    Feb 5, 2011
    Cox Cable...
    It's worth adding that large, contiguous, blocks of zeroed sectors compress down VERY well, with most compression utilities.

    I can compress (zip) down a virgin 2TB .VHD TiVo image file to 1.5GB, or less, and nothing is left out when it is restored, either as a quick restore, or writing all data, including the zeroes (nothing left to chance). The quick restore takes about 2 minutes, at most. The full restore takes as long as writing all zeroes to to whole drive using drive testing tools.
     

Share This Page