1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Roamio 4Tb Drive Development Thread

Discussion in 'TiVo Upgrade Center' started by eboydog, Apr 25, 2014.

  1. May 24, 2014 #21 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009

    How are you developing the image? Are you using a Roamio self built image and modding that to get a working 4TB image, building one completely from scratch, or using a 3rd party image and modding that?
     
  2. May 24, 2014 #22 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    dd and gzip should be fine. In windows 7z can read gzip files and I believe a program from HDDGuru can read the dd files and write them to a drive.
     
  3. May 24, 2014 #23 of 105
    nooneuknow

    nooneuknow TiVo User Since 2007

    3,554
    0
    Feb 5, 2011
    Cox Cable...
    HDDGuru's HDD Raw Copy Tool can do .VHD files. I just fired it up and didn't find any options besides drive to drive, VHD to drive, and drive to VHD.

    Since .VHD files have that 2TB limit, I'm not sure if it will be of much help here.

    I was quite surprised that I overlooked that raw Windows utility for my planned 3TB to 3TB Roamio drive clone (from WD Red NAS to WD AV-GP). Hopefully it won't object to >2TB on a drive-to-drive, and has better speed that GNU dd rescue (what jmfs uses).

    I'm wondering if there's a reason that utility never comes up when people ask about, or provide answers about, cloning TiVo drives. Raw is raw, so it should work (theoretically). I'll post again after I try it out.
     
  4. May 24, 2014 #24 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    The HDD Raw Copy Tool saves images with an img extension, which if I remember correctly, I had dd copy back to a drive. At least it used to. My version is several years old. It also can save an image in a compressed format it labels as a dimg.
     
  5. May 24, 2014 #25 of 105
    nooneuknow

    nooneuknow TiVo User Since 2007

    3,554
    0
    Feb 5, 2011
    Cox Cable...
    The most recent version, which is several years old, works with Windows 7, surprisingly. I haven't tried it with 8/8.1 yet.

    Upon further scrutiny, I found it does read raw drives, .vhd files as if they were drives, and can write them to .img and .imgc files. I don't have any of those .img or .imgc files to see if it can read from them to write them to a drive with the program.

    I suppose if I created a new blank VHD and had it mounted, the program should be able to write to it.

    I've never used dd for disk to file, or file to disk, so I can't comment on what currently will work and what won't, with those file types.
     
  6. May 25, 2014 #26 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    > Easiest for me would be, dd and gzip, which I know would support Linux and Mac. Which leaves Windows?

    It turns out it would have been easier for me to strip 0's, but it would have been harder to reconstitute, so it's the same situation.

    In any case, I have a Linux installer uploaded. Waiting for hash confirmations now. Feel free to PM me if you want to be the first test subject, until I get the documentation ready. Ideally with access to Linux and a Base Roamio.

    > It's worth adding that large, contiguous, blocks of zeroed sectors compress down VERY well, with most compression utilities.

    Ya, I got 1,000:1 and 640,000:1

    > How are you developing the image? Are you using a Roamio self built image and modding that to get a working 4TB image, building one completely from scratch, or using a 3rd party image and modding that?

    In-between option 1 and option 2, approximately.

    I haven't seen anyone else's images besides what my Roamio would build to avoid taint, though, having gone through this exercise, I'd be curious to hear of anything they chose differently as I spent extra effort to make what I thought were the best attributes to have.
     
  7. May 26, 2014 #27 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    How much recording space did you end up with?
     
  8. May 26, 2014 #28 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    So here's the story. If you put a 4TB drive in a Roamio and leave it to format, the logs shows a crash here:

    Formatting MFS partitions (/dev/sda10 /dev/sda11 /dev/sda12 /dev/sda13)
    /dev/sda has [sectors in 512] blocks
    Disk size >= 128 gibibytes (268435456 blocks), using big MFS extents
    Disk size >= 140 gibibytes (293601280 blocks), using 64-bit MFS format
    Creating 500000 inodes, please wait
    assert: Tmk Assertion Failure:
    assert: HPK_Result TivoDiskIOGetBlockSize(HPK_DiskIOInstance*, HPK_UInt32*), line 115 ()
    Process fsmake([PID]) killed by signal Unknown signal -2--2
    Tmk Fatal Error: Thread fsmake <[PID]> strayed!
    Paste the following into a shell to get a backtrace...
    bt -t /tvbin/tivoapp <<END_OF_BT
    [stuff]
    END_OF_BT

    And summarily reboots itself.

    What this implies is the drive is left with the MFS filesystem never formatted (and whatever else comes after that).

    Whether that's a bug vs reasonable check, and whether it will eventually be changed, I can see being debatable.

    The conclusions from this though, if you keep that value within an Unsigned Int 32bits, the format would have succeeded. Or if you properly finish the MFS format offline (and what remains of the installation), in any of variety of ways, the drive would be accepted.

    Also relevant, after the default partitioning for 4TB, the kernel recognizes / reports this partition list:
    sda: [tivo] sda1 sda2! sda3! sda4! sda5! sda6! sda7! sda8! sda9! sda10! sda11![M] sda12! sda13![M] sda14! :)

    I'm under the assumptions:
    - the '!' is used to designate a 64bit format record in the partition table.

    The kernel source would contain the exact definition of each symbol.

    Edit: This is under 20.3.6, the first OS this box came with.
     
  9. May 26, 2014 #29 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    So one cheap and quick method, is set a HPA, but I seriously hope nobody ever runs that way.

    My preferred method is:
    1) build a partition map by hand
    -everything within 32bits
    -centered, with the MFS media partitions equal
    2) MFS format offline, using whatever combination of tools / hardware
    (I know this is the interesting part, but I can not redistribute all the software, and some of the hardware is rare, so I'd rather skip over the details)
    3) Check every other partition is set sanely.
    4) Confirm the installation continues.

    > How much recording space did you end up with?
    I didn't tweak the other partition sizes, the media partitions come to:
    3900923680 * 2 * 512 bytes = 3.6TB
     
  10. May 26, 2014 #30 of 105
    ggieseke

    ggieseke Active Member

    4,029
    12
    May 30, 2008
    Yeah, it lays out the partition table on a 4TB drive using some 64-bit APM entries, but something else in the autobuild process doesn't recognize it and it's bootloop time.

    I haven't seen anyone get the 64-bit APM working on a primary drive yet, but the Weaknees 4TB external does use it.

    I haven't actually tried 20.4.1 yet but I don't expect TiVo to fix it until they start selling 4TB Roamios.
     
  11. May 26, 2014 #31 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    So we could get a little more recording space and use up more of the drive. Also ideally you want all the MFS partitions to be a multiple of 1024 sectors. Any extra sector space at the end of a MFS partition (which is usually only the media partitions that would have that property) that is over a multiple of 1024 sectors is not used by the MFS. I have not looked at the numbers yet, you probably can reduce all the placeholder partitions into one 4k block to keep the 4k alignment intact, but if it does not get you the extra sectors needed to allow any of the MFS partitions to get to the next multiple of 1024 sectors, it probably is not worth the mod.
     
  12. May 26, 2014 #32 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    You can see it would work on the primary drive too, as long as some partitions are 32. This combination comes up in certain values of HPA. Roughly, if you keep #13 MFS Media 2 and a few successors in 32, you can make #11 MFS Media 1 into 64.

    But I saw no point in putting this in until a larger driver shows up and it's actually required.

    If anything is wrong with the existing one, i'll have it rebuilt. If it's tweaks, I was thinking of putting all the tweaks together into a 2nd parallel version, if there's enough of them to warrant.

    The other thing I learned, is there's a random number (signature) in the bootsector starting at byte 304 (counting from 0). I haven't figured out which component is reading it, but it seems kinda new, as a lot of my Premiere images don't have it.
     
  13. May 27, 2014 #33 of 105
    ggieseke

    ggieseke Active Member

    4,029
    12
    May 30, 2008
    I think that's a GUID that gets assigned to the drive when it's built. I'm just guessing, but the 16 byte format is the right size and you get a different number every time you put a new drive in.
     
  14. May 27, 2014 #34 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    I agree, but since Premiere drives didn't have them back when they were built, did they eventually get them in a SW update?

    And if not, then is it important for anything?

    I was most interested in whether MFS depends on it as metadata, but I didn't immediately see it when I went looking.
     
  15. May 27, 2014 #35 of 105
    ggieseke

    ggieseke Active Member

    4,029
    12
    May 30, 2008
    I see it in Premiere images as far back as 20.3.1, but it wasn't there on 14.5.
     
  16. May 28, 2014 #36 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    Hm, I thought I have an unbooted premiere image where it's missing. You're right, it has 14.5. I'll check if it eventually shows up.

    I just noticed the jmfs source is available on github. It should be simple to patch this to be bi-endian. Am I missing something? Is this a useful tool to have available?
     
  17. May 28, 2014 #37 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    JMFS patched to work on Roamio series would be useful for people with sub 2TB images to expand larger and still keep their shows.
     
  18. May 28, 2014 #38 of 105
    ggieseke

    ggieseke Active Member

    4,029
    12
    May 30, 2008
    There are a few other changes in some of the MFS structures (fields swapped or moved slightly), but they're not that hard to spot. A lot of stuff in Block0, the APM and even the MFS headers isn't used on Roamios and zeroed out.

    If I knew much more about Java than how to spell it I'd try to patch it myself.
     
  19. Jun 4, 2014 #39 of 105
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    I'm stuck on this minor point. So I'll just think aloud, and hopefully someone will point out what I got wrong.

    Assumption - When Tivo makes media partitions, it uses 20,480 sectors as minimum chunk size.

    So shouldn't the media partitions be a multiple of 20,480 (x512byte sectors = 10MB), otherwise there will be bits at the end that are never reachable, and so never used?
     
  20. Jun 4, 2014 #40 of 105
    jmbach

    jmbach der Neuerer

    1,557
    10
    Jan 1, 2009
    This is my understanding of how this works.
    The MFS partitions are loaded end to end in clusters of 1024 physical disk sectors. So if the MFS partition on the disk larger than a multiple of 1024 sectors, any space in the MFS partition on the disk that lies outside the last cluster of 1024 sectors is never used by the TiVo.
    The TiVo sees the MFS as one large partition. Assuming the chunk size is as you stated, what ever space at the end of the MFS that is less than the chunk size is not used as you have stated unless the MFS space is expanded via an external drive or a JMFS like procedure. When the MFS is expanded, that left over space is now used as the chunk now becomes the left over space plus the extra space needed in the expanded area to complete the chunk.
    So the last MFS media partition is the one that may have space not being used. All the ones before that would be full.
    Maybe ggieseke can comment if my understanding is on par with his knowledge of the MFS structure since he has gone down deeper in the rabbit hole than I have.
     

Share This Page