1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

MFS Tools 3.2

Discussion in 'TiVo Upgrade Center' started by jkozee, Jun 4, 2015.

  1. Jun 4, 2015 #1 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    Announcing the release of MFS Tools 3.2!

    "MFS Tools 1.0 was evolutionary in the TiVo upgrade process. MFS Tools 2.0 may prove to be revolutionary." I'll let you can choose your own adjective for MFS Tools 3.2.

    Highlights
    • Supports Series 1 through Roamio (and perhaps beyond).
    • Backwards compatible with MFS Tools v1 and v2.
    • Supports 8 TB on Premiere (perhaps more, tested with dual 4 TB drives).
    • Supports 8 TB on Roamio (perhaps more, tested with dual 4 TB drives).
    • Backup and restore (or drive-drive using mfscopy) single or dual drive setups, and preserve recordings if you choose.
    • Supersize the drive, which can be reversed even after GSOD.
    • You can include the alternate boot/kernel/root partitions (useful for customized partition layouts).
    • Automatically includes required loopset files (you no longer need to use the -f flags during backup) when not including recordings (truncated backup)
    • Automatically prunes recordings when a truncated backup is done (sets the MFS flags to cleanup, so the unit will scan objects on startup)
    • Works on drives that have been marked for cleanup (GSOD).
    • Creates a new MFS structure, so drives that have been expanded by adding pairs can be consolidated to just two pairs, just like the original.
    • Restore to drives smaller than the original.

    Overview
    MFS Tools v3.2 is a set of utilities that allow one to backup/restore/copy/supersize a TiVo drive, without losing recordings, and to expand recording space to take advantage of larger drives. Additionally, there are options that allow more advanced features and finer grain control over the TiVo drive. For example, there are options to revalidate the drive to the Linux kernel, so the partitions can be mounted or copied. The ability to increase/decrease swap, var, SQLite partition sizes on restore. Limit the amount of used space of the drive, limit media partition size, add app/media partition sets, convert between 32-bit and 64-bit MFS layouts, etc, etc, etc. Please take the time to review all of the utilities and their options to fully appreciate what the tools have to offer.

    Thanks
    A special thanks to Tiger, the original pioneer of MFS. And to all those that contribute here and on other sites that have helped me get the most out of my TiVo units.

    History (unverified)
    Tiger released the original closed source project in 2001, followed by 2.0 in 2002. In 2004, he released the source code under GPL, and made it available on sourceforge, and announced that he had no plans to continue work on it. He hen returned for a three month period in 2005 and a two month period in 2008 to work on a major update to version 3, with some ambitious new functionality and 64-bit MFS support for the TiVo HD. It has remained in this untested, almost functional, but overlooked state since then.

    I received a some anonymous patches that fixed a few bugs, added Premier handling, and got things mostly working one. Since then, I've refined the Premiere support, added Roamio support, added some new utilities, deprecated a few options and added a few more, and cleaned up and debugged the code a bit.

    Although there are several new tools, features, options, etc., available in 3.2, there was a fundamental change in the the way the MFS is handled. A simplified explanation is that in v1, the MFS is treated as a whole, and the entire structure is duplicated based on the used blocks. In v3, the inodes and data is copied individually, and during restore the MFS structure is recreated, the inodes and data blocks are allocated, and the inode and data is copied to the new locations.

    The downside is that v3 backup/restore is not quite as fast as v1. The upside is that the MFS Media Partitions can now be re-sized, which is a pretty big advantage. Because v1 copied all MFS pairs, the number of times a drive could be backed/restored and then expanded to take advantage of additional disk space was limited. With v3, the restore process creates a drive with only two MFS app/data pairs, just like the original drive, no matter how many pairs the original drive had. And yes, you can now restore to a smaller drive than the original drive (provided the smaller drive has enough space for the recordings or you truncate it).

    Notes
    Every effort has been made to keep these tools as robust as possible. I have done limited testing (confirmed boot/operation, but not filled drives to capacity) on units from Series 1 through Roamio. The tools can still backup/restore using the original v1 backup format, and should work with existing backups.

    I have deprecated several options, and will probably deprecate more to reduce the number of options for novice users. Some of the deprecated options (and additional debug options) will still be available with custom builds.

    I have only tested with drives up to 4 TB, but there's been mention of Roamios supporting larger drives with 20.4.6 or later. I was able to setup a Premiere with 8 TB (dual 4 TB drives), but had to limit the media partitions to 2 TB max. This unit passed a forced MFS fsfix scan (GSOD) without issue.

    Todo
    I still have a few things I would like to add, such as adding SW version, original swap/var/db size to the backup header (extrainfo). I would also like to do some testing with larger drives and additional units.

    Known Limitations
    Currently, the largest boot drive supported is 4TB (Premiere and newer, 2TB for THD). Backups of drives with the either of the first two app/media partitions is not supported.

    Attachment
    The iso image has been removed from sourceforge temporarily to address some of the issues reported in this thread regarding large drives.
    Please go to sourceforge for the latest release.
     
    Last edited: Jun 24, 2015
    dante99 and Zack411 like this.
  2. Jun 4, 2015 #2 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    mfstool
    Code:
    Usage: ./mfstool <function> <args> or <function> <args>
    Available functions:
      backup    Backup TiVo drive fast and small.
      restore   Restore mfstool backups to TiVo drive.
      copy      Copy TiVo drive to a new drive.
      add       Add partitions to your TiVo MFS volume.
      ck        Perform a simple consistency check on MFS.
      mls       List files in the MFS volume.
      supersize Supersize a TiVo drive.
      d         Dump raw data from MFS volume.
      info      Display information about MFS volume.
    For help on a particular function: ./mfstool <function> -h
    
    backup
    Code:
    Usage: ./backup [options] Adrive [Bdrive]
    Options:
     -h        Display this help message
     -o file   Output to file, - for stdout
     -1 .. -9  Compress backup, quick (-1) through best (-9)
     -v        Do not include /var in backup
     -d        Do not include /db (SQLite) in backup (Premiere and newer)
     -s        Shrink MFS in backup (implied for v3 backups without -a flag)
     -F format Backup using a specific backup format (v1, v3, winmfs)
     -q        Do not display progress
     -qq       Do not display anything but error messages
     -t        Use total length of stream in calculations
     -T        Backup total length of stream instead of used length
     -a        Backup all streams
     -i        Include all non-mfs partitions from Adrive (alternate, custom, etc.) 
    Available in deprecated/debug build only
     -f max    Backup only fsids below max
     -L max    Backup only streams less than max MiB
     -D        Do not force loopset and demo files to be added
    
    restore
    Code:
    Usage: ./restore [options] Adrive [Bdrive]
    Options:
     -h        Display this help message
     -i file   Input from file, - for stdin
     -k        Optimize partition layout with kernels first
     -r scale  Override v3 media blocksize of 20480 with 2048<<scale (scale=0 to 4)
     -q        Do not display progress
     -qq       Do not display anything but error messages
     -v size   Recreate /var as size MiB (Only if not in backup)
     -d size   Recreate /db (SQLite in source) as size MiB (if not in backup)
     -S size   Recreate swap as size MiB
     -l        Leave at least 2 partitions free
     -b        Force no byte swapping on restore
     -B        Force byte swapping on restore
     -w 32/64  Write MFS structures as 32 or 64 bit
     -c size   Carve (leave free) in blocks on drive A
     -C size   Carve (leave free) in blocks on Drive B
     -m size   Maximum media partition size in GiB for v3 restore
     -M size   Maximum drive size in GiB (ie lba28 would be 128)
    Available in deprecated/debug build only
     -P        Do NOT optimize the partition layout
     -z        Zero out partitions not backed up
    
    mfscopy
    Code:
    Usage: ./mfscopy [options] SourceA[:SourceB] DestA[:DestB]
    General options:
     -h        Display this help message
     -q        Do not display progress
     -qq       Do not display anything but error messages
    Source options:
     -t        Use total length of stream in calculations
     -T        Copy total length of stream instead of used length
     -a        Copy all streams
     -i        Include all non-mfs partitions from Adrive (alternate, custom, etc.)
    Available in deprecated/debug build only
     -f max    Copy only fsids below max
     -L max    Copy only streams less than max MiB
     -D        Do not force loopset and demo files to be added
    Target options:
     -s        Shrink MFS whily copying (implied for v3 copies)
     -k        Optimize partition layout with kernels first
     -r scale  Override v3 media blocksize of 20480 with 2048<<scale (scale=0 to 4)
     -v size   Recreate /var as size MiB and don't copy /var
     -d size   Recreate /db (SQLite in source) as size MiB and don't copy /db
     -S size   Recreate swap as size MiB
     -l        Leave at least 2 partitions free
     -b        Force no byte swapping on target
     -B        Force byte swapping on target
     -R        Just copy raw blocks (v1) instead of rebuilding data structures (v3)
     -w 32/64  Write MFS structures as 32 or 64 bit
     -c size   Carve (leave free) in blocks on drive A
     -C size   Carve (leave free) in blocks on Drive B
     -m size   Maximum media partition size in GiB for v3 restore
     -M size   Maximum drive size in GiB (ie lba28 would be 128)
    Available in deprecated/debug build only
     -P        Do NOT optimize the partition layout
     -z        Zero out partitions not copied
    
    supersize
    Code:
    Usage: ./supersize [Adrive [Bdrive]] [options values]
    Options:
     -h        Display this help message
     -m        MaxDiskSize in KB (Default is 2147483647)
     -u        User SizeInKb in KB (Default is -1)
     -c        TivoClips SizeInKb in KB (Default is -1)
    
    mfsadd
    Code:
    Usage: ./mfsadd [options] Adrive [Bdrive] [NewApp NewMedia]
    Options:
     -h        Display this help message
     -r scale  Override media blocksize of 20480 with 2048<<scale (scale=0 to 4)
     -x        Create partition(s) on all drives
     -X drive  Create partition(s) on a specific drive
     -m size   Maximum media partition size in GiB
     -M size   Maximum drive size in GiB (ie lba28 would be 128)
     -f        Use with -m to fill the drive multiple media partitions
    NewApp / NewMedia
      Existing partitions (Such as /dev/hda13 /dev/hda14) to add to
      the MFS volume set
    
    mfsinfo
    Code:
    Usage: ./mfsinfo Adrive [Bdrive]
    Options:
     -h        Display this help message
    
    mfsck
    Code:
    Usage: ./mfsck [options] Adrive [Bdrive]
    Options:
     -h        Display this help message
     -r        Revalidate TiVo partitions on Adrive [Bdrive]
    Available in deprecated/debug build only
     -m [1-5]  Set volume header magic to OK, FS_CHK, LOG_CHK, DB_CHK, or CLEAN
     -e [1-3]  Set vol_hdr.v64.off0c to 0x00000010, TiVo, or Dish
    
    mfsd
    Code:
    Usage: ./mfsd [options] Adrive [Bdrive]
    Options:
     -h        Display this help message
     -f FSID  Dump a single FSID
     -F        Dump ALL FSIDs
     -i indoe  Dump a single inode
     -l log    Dump a single transaction log
     -s sector Read from sector, or from offset into file
     -c count  Read count sectors, where applicable
     -C        Perform consistency checkpoint before displaying data
     -H        Display in hex, no matter the format
     -b        Display in binary, no matter the format
     -x        Display formatted values in hex
     -z zone   Read from a single zonemap
     -Z        Dump ALL zonemap info
    
    mls
    Code:
    Usage: ./mls [options] Adrive [Bdrive] <path|fsid>
    Options:
     -h        Display this help message
     -l        long list (with size)
     -R        recurse
    
     
    Last edited: Jun 9, 2015
  3. Jun 4, 2015 #3 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    (reserved for usage examples)
     
  4. Jun 4, 2015 #4 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    (reserved for no good reason, pictures perhaps?)
     

    Attached Files:

    Last edited: Jun 4, 2015
  5. Jun 4, 2015 #5 of 865
    Keen

    Keen Member

    262
    0
    Aug 2, 2009
    Minneapolis, MN
    Holy crap, the day is upon us! A tool to upgrade a Roamio drive without losing recordings?! Huzzah!

    I look forward to upgrading my Roamio to a 6TB drive down the road.
     
  6. Jun 4, 2015 #6 of 865
    unitron

    unitron Active Member

    16,491
    23
    Apr 28, 2006
    semi-coastal NC
    Will this work with .bak files created by spike's MFS Live cd?

    Will this create .bak files that can be used by spike's MFS Live cd?
     
  7. Jun 4, 2015 #7 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    Yes. The mfslive cd is a MFS Tools v2 variant. It will NOT work on WinMFS backups, as those backups use a different file format.
     
  8. Jun 4, 2015 #8 of 865
    jmbach

    jmbach der Neuerer

    1,904
    75
    Jan 1, 2009
    Cool beans.


    EDIT: 9/29/2015

    Since the developer's ISO has been pulled, I created an ISO and other images that can be used. They can be found here until such time someone with better Linux skills creates a better ISO.
     
    Last edited: Sep 29, 2015
  9. Jun 5, 2015 #9 of 865
    jmbach

    jmbach der Neuerer

    1,904
    75
    Jan 1, 2009
    Does this 4k align the zones and inodes.
     
  10. Jun 5, 2015 #10 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    No, or least probably not. The partitions are aligned on 4k sectors, but I haven't had time to investigate alignment of the internal mfs structures.

    My initial impression was that 4k alignment of zones more involved than just aligning the mfs partition to 4k boundaries. There are other tools that offer this feature, so porting it to MFS Tools v3.2 is a matter of collaboration or reverse engineering.

    I made a quick attempt at investigating other tools, but I was not able to use them as my test unit is not running 20.4.6 yet.
     
  11. Jun 5, 2015 #11 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    FYI, I have had some PM's that I cannot reply to directly, because I have not met the minimum number of posts yet. I will address them in this thread, provided they are not personal and seem appropriate to the general discussion of MFS Tools. Otherwise, I will respond to PM's when I am "allowed" to. I will not release any names, and if I offend anyone, please let me know and I will remove the post(s).
     
  12. Jun 5, 2015 #12 of 865
    jmbach

    jmbach der Neuerer

    1,904
    75
    Jan 1, 2009
    Sourceforge seems to only have the original MFSTools documentation. Is there supplemental documentation on your iso for the new changes?
     
  13. Jun 5, 2015 #13 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    A few things to address, and it will be from memory, so don't be surprised if I get it wrong and need to update this post later, or add some clarifications.

    A coalesced partition is a single partition that masquerades itself as a app/media pair. It's been my experience that although this appears to work during the expansion, there is no guarantee that fsfix (GSOD) will complete, and from experience it will fail. So, from my experience, adding a coalesced partition is asking for trouble.

    Additionally, adding a partition (or app/media pair) is not required for the initial restore. The entire drive will be used, and contain only 2 pairs, just like the original.

    However, an unknown at this point is the 6TB drive. Indications are that the latest SW does not have limitations, but previous versions would not access a drive with partitions > 2TB, or at least did not begin above the 2TB mark.

    MFS Tools 3.2 is aware of this issue, and will not create a media partition that begins above the 2 TB mark.

    So, for example, with a 6TB drive will contain two pairs of app/media partions, but the first media partition will be < 2 TB to make sure that that second partition does not start above 2TB.
     
    Zack411 likes this.
  14. Jun 5, 2015 #14 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    Nope, hence my request for help. However, all of the tools will report an accurate list of options, with at least some indication of what they do.
     
  15. Jun 5, 2015 #15 of 865
    ThAbtO

    ThAbtO TiVoholic by the bay

    7,674
    113
    Apr 6, 2000
    SF Bay Area
    Space! The final frontier! These are the voyages of MFSTools. To go boldly where no Tivo has gone before! :D
     
  16. Jun 5, 2015 #16 of 865
    telemark

    telemark New Member

    1,544
    2
    Nov 12, 2013
    Last edited: Jun 5, 2015
  17. Jun 5, 2015 #17 of 865
    jmbach

    jmbach der Neuerer

    1,904
    75
    Jan 1, 2009
    Having an issue with the iso reading my 4TB correctly. It is seeing it as a 2TB. The drive is sitting in a USB dock connected to my laptop. My Linux Mint installation on the same computer reads it just fine. And it will not detect my 6TB drive at all whereas my Linux Mint installation does as well.
     
  18. Jun 5, 2015 #18 of 865
    telemark

    telemark New Member

    1,544
    2
    Nov 12, 2013
    I'm hitting some bugs too, mostly around Roamio or large disk support.
    Is there somewhere you want them reported, or somehow patched?
     
    Last edited: Jun 5, 2015
  19. Jun 5, 2015 #19 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    The boot cd is based on this distro. Perhaps there is an issue with th 32-bit build. Please try th 64-bit version there to see if it handles your drives correctly.

    Also, you can copy the binaries off of the cd and try them on your Mint installation. Let me know the outcome.
     
  20. Jun 5, 2015 #20 of 865
    jkozee

    jkozee New Member

    46
    3
    Jan 12, 2006
    What issues are you seeing? Accessing the drive from the boot cd or the Roamio using the restored image?

    What size drive are you using?
     

Share This Page