1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

DIY 4TB Premiere

Discussion in 'TiVo Upgrade Center' started by jmbach, Jan 21, 2014.

  1. May 27, 2014 #41 of 207
    cykotix

    cykotix New Member

    18
    0
    May 22, 2014
    I was attempting to expand a 320GB & 1TB native to 4TB. With the help of jmbach, I wanted to share the new iso I built with a patched JMFS and a collection of tools from different individuals. This should help streamline jmbach's process he outlined.


    The current limitation of JMFS only allows a maximum partition size of 2.2TB due to 32bit tables. This patch allows you to expand beyond 1.5TB which otherwise isn't possible with rev104. The iso I built is based off the source in krbaker's github and the AppleDisk patch. krbaker's source will clean up Apple_Free partitions if they exist when doing a JMFS copy.

    Lastly I included some TivoTools scripts written by TCF member marwatk from this post.

    In the following posts, I'll explain my process and how I built the new iso.
     
  2. May 27, 2014 #42 of 207
    cykotix

    cykotix New Member

    18
    0
    May 22, 2014
    This process can be adapted for 3TB expansions but I was going to a 4TB. This process can also be adapted to go from native to 4TB, but I'm going to outline how to use an intermediate drive. The advantages of doing this put the OS and partition 10 closer to the center of the drive which helps performance. I learned all of this from jmbach in my journey.

    You can download the ISO I build here: http://bit.ly/Tivo-JMFS & select jmfs-rev105-krbaker-tivotools.iso

    https://www.dropbox.com/s/vkyd1hwd0gqltur/jmfs-rev105-krbaker-tivotools.iso - This URL will be dead soon, use the bit.ly URL above.

    I'll outline how I built the ISO in the next post.

    This process assumes a few things:
    1. Starting with a native, unexpanded drive that is <2TB
    2. Intermediary drive >=2TB
    3. Destination drive =4TB

    I was using a 3TB WD Red drive as my intermediate drive because it's what I had available. Ideally you would be using an actual 2TB drive. However, if you're using a 3TB or 4TB intermediate drive, you'll need to use `hdparm` to setup HPA (host protected area) to essential fake being a 2TB drive. jmbach mentions in his original post about using temporary HPA but I only had success doing permanent HPA and later removing it.

    If you're using =2TB, SKIP THIS. Otherwise, if you're using a 3TB or 4TB intermediate:
    Code:
    // Make >2TB look like a 2TB drive
    # hdparm -N p3907029168 --yes-i-know-what-i-am-doing /dev/IntermediateDrive
    
    // Verify HPA is set
    # hdparm -N /dev/IntermediateDrive
    
    Power cycle (NOT REBOOT) - Meaning Shutdown completely then boot
    Code:
    // Verify HPA stuck & fdisk shows =2TB, not >2TB
    # hdparm -N /dev/IntermediateDrive
    # fdisk -l /dev/IntermediateDrive
    
    If hdparm/fdisk show the correct information OR you are staring with =2TB, you can continue.
    Code:
    // JMFS Copy
    // Select [C], Source is your original drive, Destination is the 
    // IntermediateDrive.
    # ./guide.sh
    
    // When it is done copying to IntermediateDrive, JMFS will ask if 
    // you want to Expand.  Choose Yes.  Once done, select [R] to choose
    // another operation.  Select [Z] to supersize the IntermediateDrive.  
    // Then select [X] to exit `guide.sh`.
    
    Code:
    // Verify copy was successful
    # ./mfslayout.sh /dev/IntermediateDrive
    
    Switch to TivoTool scripts to rearrange and copy partitions from IntermediateDrive (2TB) to DestinationDrive (4TB)
    Code:
    # cd ~/TivoTools/coalesce_scripts
    # ./tivo_copy_rearranged.sh /dev/IntermediateDrive /dev/DestinationDrive 1 11 12 13 15 2 3 4 5 6 7 8 9 14 10
    
    Code:
    // Verify copy was successful
    # cd ~/ && ./mfslayout.sh /dev/DestinationDrive
    
    Coalesce partitions 12 & 13 on DestinationDrive (4TB)
    Code:
    # cd ~/TivoTools/coalesce_scripts
    # ./tivo_coalesce.sh /dev/DestinationDrive 12 13
    
    Remove DestinationDrive and place it back into the Premiere. Follow the instructions to divorce the missing partition. Once this completes, connect it back to the computer and boot JMFS.
    Code:
    // Select [E] to expand DestinationDrive
    # ./guide.sh
    
    Code:
    // Select [X] to exit.  Verify DestinationDrive.
    # ./mfslayout.sh /dev/DestinationDrive
    
    If all went well, put the DestinationDrive back into the Premier. You should have 640hrs HD. Congratulations!

    If you had to use hdparm, see below to remove the permanent HPA.

    Code:
    // Get full sector count.  For my drive it was 5860533168
    # hdparm -N /dev/IntermediateDrive
    
    // Remove HPA from IntermediateDrive.  Be sure to replace 
    // 5860533168 if yours is different.
    # hdparm -N p5860533168 --yes-i-know-what-i-am-doing /dev/IntermediateDrive
    
    // Verify HPA is disabled.  Power cycle (NOT REBOOT)
    // This means SHUTDOWN, then BOOT.
    # hdparm -N /dev/IntermediateDrive
    
     
    Last edited: Jun 23, 2015
  3. May 27, 2014 #43 of 207
    cykotix

    cykotix New Member

    18
    0
    May 22, 2014
    The original JMFS was built using Slax 6 and the modules use a different compression method. In this post, I'll explain how I built JMFS, compiled JMFS and TivoTools as a Slax module and then build the Slax iso.

    JMFS Source
    I was working from krbaker's github: https://github.com/krbaker/jmfs
    You can also grab the original rev104 sources from here, but keep in mind rev104 will not work with the 4TB procedure I posted.
    You'll need Apache ANT >=1.7.1 and JDK >=1.6 to compile.

    Code:
    # cd /path/to/jmfs/source
    
    // you can set raw.revision to whatever you want
    # ant -Draw.revision=105-krbaker package
    
    // Change to correct bin zip and output dir, make sure it ends with 'root'
    unzip -a jmfs-bin.zip -d /path/to/output/jmfs-rev/root
    
    // chmod *.sh
    # chmod 755 /path/to/output/jmfs-rev/root/*.sh
    
    // Compile Slax module.  File extension is .sb
    mksquashfs /path/to/output/jmfs-rev/ /path/to/module/jmfs-rev.sb -comp xz -b 512k -Xbcj x86 -noappend
    
    TivoTools
    You can grab the latest TivoTools here.
    Code:
    unzip -a master.zip
    mkdir -p TivoTools/root
    mv TivoTools-master/* TivoTools/root
    // No need for JMFS twice
    rm -rf TivoTools/root/jmfs
    
    // Compile Slax module
    mksquashfs /path/to/TivoTools/ /path/to/module/TivoTools.sb -comp xz -b 512k -Xbcj x86 -noappend
    
    Slax ISO
    Slax can be found here. More specifically, you'll need the 32bit for USB zip. My folder structure for modules was ~/slax7/slax/modules.

    Code:
    cp /path/to/module/jmfs-rev.sb ~/slax7/slax/modules
    cp /path/to/module/TivoTools.sb ~/slax7/slax/modules
    
    You'll also want to add any additional modules. TivoTools requires the xxd module. I also included nano for those not comfortable with vi and samba-client. Neither of those are necessary. You'll also want to clean up unnecessary modules in ~/slax7/slax. I removed everything except 01-core.sb.

    Once you're happy with the layout:

    Code:
    // Name it whatever you want
    ~/slax7/slax/makeiso.sh ~/slax7/ /path/to/iso/jmfs-rev-krbaker-tivotools.iso
    
    Boot it and make sure everything is where you expected it to be.
     
  4. Jun 3, 2014 #44 of 207
    jlin

    jlin New Member

    77
    0
    Nov 20, 1999
    Santa Monica
    I tried to run this CD.. it never finished loading... what I am doing wrong?
     
  5. Jun 4, 2014 #45 of 207
    telemark

    telemark New Member

    1,544
    1
    Nov 12, 2013
    What was the last thing on the screen? It's pretty common for CD's to get misburned. Have you verified the data on the disc?
     
  6. Jun 4, 2014 #46 of 207
    byeams

    byeams New Member

    12
    0
    May 31, 2014
    How long did this process take?
     
  7. Jun 5, 2014 #47 of 207
    cykotix

    cykotix New Member

    18
    0
    May 22, 2014
    Honestly, I didn't really keep track once we fleshed out all the details. Probably in the realm of 8-10hrs. I think my average speed disk to disk using eSATA was around 114 MB/s if memory serves me correctly.
     
  8. Jun 6, 2014 #48 of 207
    unitron

    unitron Active Member

    16,389
    2
    Apr 28, 2006
    semi-coastal NC
    Intel or AMD CPU?
     
  9. Jun 18, 2014 #49 of 207
    [NG]Owner

    [NG]Owner New Member

    299
    0
    Dec 19, 2006
    Kansas City
    I don't have an intermediate drive .... how do I go from native to 3TB, all on a single 3TB drive? What would need to be adapted? Can you map that out for me?

    [NG]Owner
     
  10. Jun 18, 2014 #50 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    It does make it tougher because you have to rearrange the partitions on the 3TB drive in order to coalesce one pair of partitions. If you can mount/create VHD images in your Linux box, then you can do this without a physical drive. First, in Windows create a dynamic VHD about half the size of your 3TB drive. Then mount that drive in Windows. Use DvrBARS to restore your image to that mounted VHD. Once that is done, unmount the VHD. Then in your Linux box mount the VHD you just created and use JMFS to expand and supersize it. Now connect your 3TB drive and use the tools to copy and rearrange from the mounted VHD to the 3TB drive. Once that is done, then coalesce one pair of partitions on the 3TB drive followed by using JMFS to expand the 3TB drive.
     
  11. Jun 18, 2014 #51 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    FYI, your intermediate drive just has to be between 1TB to 2TB in size for this to work and does not have to be a TiVo recommended drive.
     
  12. Jun 18, 2014 #52 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    You can use one drive only, however, it is somewhat daunting to do and will take you days to create the proper image to do the final expansion.

    It would be much easier with a higher rate of success if you can borrow a drive from someone to use for a day. It just has to be between 1TB and 2TB in size for this to work for you.
     
  13. Jun 18, 2014 #53 of 207
    [NG]Owner

    [NG]Owner New Member

    299
    0
    Dec 19, 2006
    Kansas City
    This is all way over my head. Much more difficult than anticipated. Is there anything "easy" I can do with the image I am downloading from that other thread, and my 3TB drive? I don't mind giving up a TB if I have to (though not much more than that).

    [NG]Owner
     
  14. Jun 18, 2014 #54 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    There is another way where you start with a 1TB Premiere XL image (still OS version 20.3.8 or later) . Some people frown on using an image from a different model, however, the people I know of of who have done this have not had any issues yet and they have had the images functioning for over a year. If you started with the 1TB XL image, you go straight to 3TB with one step.
     
  15. Jun 20, 2014 #55 of 207
    nooneuknow

    nooneuknow TiVo User Since 2007

    3,554
    0
    Feb 5, 2011
    Cox Cable...
    Get your base image up to a software version that supports over 2TB, so you can more easily proceed, and sometimes eliminate needing a "middle-drive" due to old software (at least for future upgrades, or for when sharing images).

    Helpful tips & explanations for all (edit of a post I've tried to standardize):

    Use my trick of using 00000 zip code, Tiny TiVo as provider, run a C&DE on the updated image, pull the plug on the reboot once the C&DE operation has completed, which self-triggers a reboot (preferably before the KS [kickstart] window of opportunity, or before the "almost there" screen at the latest), make a backup of that, and upload it for distribution, or just use for yourself, if that's how you roll.

    The 00000 zip code & Tiny TiVo provider trick only downloads one channel worth of empty guide data, keeping the new backup image small as can be, and saves time. A cablecard can be paired in this setup, but the channels can only be tested using "test channels using M-card" via DVR Diagnostics screens. This is just a FYI, for those who may use this trick for the many other uses I've found for it, like purging out corrupted data related to an existing zip and provider's guide data.

    The reason for the C&DE with timed power-pull is to stop the drive from becoming married to the TSN of the board it is updated on, eliminating image recipients from having to repeat guided setup twice, with a C&DE between the two, to remove any existing host board TSN-to-drive marriage.

    Once the software has reached a certain version number, the second screen of GS, where you select demo/home setup, has a "press enter for advanced options" option, which adds Installer Express mode to the choices, saving a big chunk of time, by only doing the absolute minimum GS needs to, leaving all the little personalized preferences to be set later through the menus.

    If you want to see what software version is on a drive that boots into GS, without running it, press Info button on the very first GS screen (Select Country). It's at the bottom of the screen, when displayed.

    EDIT/ADD: I had left out it is best that the only things connected, until done, are:

    1. Power cord/supply.
    2. Ethernet cable (or wireless, if you absolutely can't do RJ45 ethernet).

    What absolutely should NOT be connected:
    1. Coaxial cable for cable or antenna.

    Why: If a video source is available, the TiVo will continuously process and buffer all tuners, making the resulting image larger for every zeroed-sector that is changed to in the process. A C&DE does not re-zero the sectors that get changed. Even though the buffer, per tuner, is 30 minutes, it still moves forward though more sectors for as long as the TiVo is on, and has a coax connection providing cable or antenna connectivity. For the cleanest and smallest backup image, this is a big deal.

    I suppose (theoretically) somebody who insists on using MoCA, could just remove the cable/antenna input coax to the MoCA, and just use the coax for networking only.
     
  16. Jun 25, 2014 #56 of 207
    xdroccax

    xdroccax New Member

    31
    0
    Feb 26, 2006
    What steps from post #42 would need to be changed if we started with a 1tb XL image and a 4tb drive? This will be my first Premiere, and haven't been in the scene ever since upgrading our S3's to 1.5tb's years ago. thanks
     
  17. Jun 25, 2014 #57 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    The basic concepts we have to remember is that the largest single partition that can be added is about 2TB in size and that the starting sector of every partition has to start within the first 2TB of the drive.

    So to answer your question, you would have to take your 1TB image, expand it to 2TB, consolidate, then expand to 4TB.

    If you do not have an intermediate drive then you would copy and rearrange the 1TB image to the 4TB drive in the order 1,11,12,13,2,3,4,5,6,7,8,9,14,10. Then limit the size of the 4TB drive to about 2TB using hdparm. Expand the drive with JMFS to the 2TB size and supersize. Remove the hdparm limitation. Coalesce partitions 12 and 13 into 12. Place in the premiere and allow it to divorce the now non existent partition. Then finally expand out to 4TB with JMFS. Should have 640HD hours. If not then probably the hdparm step did not take for some reason.
     
  18. Jun 30, 2014 #58 of 207
    xdroccax

    xdroccax New Member

    31
    0
    Feb 26, 2006
    I managed to get ahold of a 2TB to be used as an intermediate. My question now (not sure if it belongs in this thread or the dvrBARS one) is can you back up the 'finished' 4TB drive to a VHD file using dvrBARS and use it as a template/image for future 4TB drives? This is assuming there's no recordings that need to be saved and the original image is a clean XL 1Tb. Would the partition reassignments hold; i.e. the OS being moved to the middle of the drive, the 4k alignment, etc.


    Basically, we have two 746's that is going to be upgraded to 4TB, and have about 4 other 746's belonging to family/friends that will probably go 4TB as well. Would be a whole lot faster if you could just dvrBARS to image out 4TB (or 3TB) drives.
     
  19. Jun 30, 2014 #59 of 207
    jmbach

    jmbach der Neuerer

    1,556
    10
    Jan 1, 2009
    The answer is yes sort of. You can copy the first 2TB of the drive. As long as the drive is empty of recordings it works. This is because all the important information is in the first 2TB. The last 2TB of the drive is all blank. And yes, all the partition rearrangement and 4k realignment all are maintained.

    With your 746 you can start with that image and expand it to 2TB, copy and rearrange to the 4TB drive, divorce the partition, then expand to 4TB.
     
  20. Jul 1, 2014 #60 of 207
    ggieseke

    ggieseke Active Member

    4,029
    12
    May 30, 2008
    That may take some DvrBARS tweaking on my part. I haven't tested it yet.

    The file format can handle just under 2TiB, but there's a field in the VHD header that lists the total number of sectors on the drive. Cramming the sector count for a 4TB drive into that 32-bit field will probably limit it to 1,801,763,774,464 bytes. I need to add a few lines of code so that when backing up larger drives it will use the VHD max instead of the actual drive size.
     

Share This Page