TiVo Community
TiVo Community
TiVo Community
Go Back   TiVo Community > Underground Playground > TiVo Upgrade Center
TiVo Community
Reply
Forum Jump
 
Thread Tools
Old 06-24-2015, 10:23 PM   #1
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
MFS Tools 3.2 Development

This thread is intended to spur further development of mfstools. Feel free to ask questions and offer patches.

jkozee is offline   Reply With Quote
Old 06-26-2015, 09:52 AM   #2
telemark
Registered User
 
Join Date: Nov 2013
Posts: 1,483
What's the biggest bug that needs to be fixed on Premiere?
What's the biggest bug that needs to be fixed on Roamio?

telemark is offline   Reply With Quote
Old 06-27-2015, 09:16 AM   #3
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Currently I am trying to take a Premiere image that has been expanded via the coalesce and expand method and trying to copy it over to another 4TB drive to put it back in a native TiVo format. It currently fails. What I see is that it seems to copy partitions 10 and 12 intact over to the new drive before modifying them for the larger media structure. Since partition 12 is a coalesce of partition 12 and 13 of the original image, it copies that whole coalesced partition over. It needs a way to detect coalesced partitions and separate out the media and app portions when reconstructing the image.
This maybe only more important when one of the original app partitions are coalesced with a media partition. If we assume only one of the original app partitions are coalesced, then a quick test would be to compare the sizes of partitions 10 and 12 and use the smaller of the two sizes to copy from the beginning of the source partitions 10 and 12 with that size to the target drive. Then hopefully the program can create the new media partitions accordingly and copy the streams over.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-27-2015, 10:41 AM   #4
telemark
Registered User
 
Join Date: Nov 2013
Posts: 1,483
Aren't all Premiere MFS app partitions the same size?

Wasn't it possible to put 10+11+12 -> 10 before? Though I agree that might be an odd one.

telemark is offline   Reply With Quote
Old 06-27-2015, 01:43 PM   #5
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Yes they all are the same size and yes you can technically, as long as you follow the rules, coalesce any combination of MFS partitions.

I doubt that TiVo will change the app partition size on the Premiere with any update as that would require a major adjustment of the partition layout of drive.

The best solution would be to create new MFS app partitions denovo and then transfer information from the old to the new. However, that may take some major rewriting.

An alternative would be to tease out the MFS app partitions out of the coalesced partitions, use them, and proceed with the normal routine from there. Hopefully just a small patch and not a major rewrite.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-27-2015, 01:56 PM   #6
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,330
One caveat to watch for when coalescing app partitions is that TiVo expects to find the primary "superheader" in the first sector of the first app partition, and the backup copy in the last sector of that partition. It might run anyway if it can't find the backup, but it won't be happy about it.

ggieseke is online now   Reply With Quote
Old 06-27-2015, 03:26 PM   #7
telemark
Registered User
 
Join Date: Nov 2013
Posts: 1,483
I don't think it's too hard to find the original 10 & 12, but doing so forces the 11 to be the original size 11 too?

What is the target layout you're going to want?

telemark is offline   Reply With Quote
Old 06-27-2015, 07:27 PM   #8
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
The program will take a single expanded Premiere image and create a two app and two media MFS layout. The issue is when it is a coalesced multi expanded Premiere image. I would like to be able to take a multi expanded image and create a 2 app and 2 media partition standard TiVo image.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-29-2015, 10:17 PM   #9
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
I think mfstools should be able to read the coalesced drive and zones ok. The problem would be that assumes that 10 and 12 are both app partitions and it will count both of them as required space to restore the drive, in addition to the media space also in partition 12. It shouldn't be too hard to force partition 12 to match the size of partition 10 in the case of a coalesced 12/13->12. Creating coalesced partitions and handling a single App partition would require more work though.

jkozee is offline   Reply With Quote
Old 06-29-2015, 10:55 PM   #10
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
telemark,

There's really only two issues that come to mind. The first is 64-bit APM creation. TiVo's pdsik64 code allowed for creating both 32-bit and 64-bit APM entries (as does mfstools). Mixing 32-bit and 64-bit doesn't appear to cause an issue, but it's probably better to sort it out now. Another question is when should 64-bit APM's be applied to Premiere restores. So, really it comes down to use cases: when should we use 64-bit APM's (platform, drive size, os version).

The seconds issues is that the small linux distro I was using was not handling large drives correctly (I think the kernel was ok, but not busybox).

jkozee is offline   Reply With Quote
Old 06-29-2015, 10:59 PM   #11
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
There are also some wish list items, such as adding additional metadata to the v3 backups (OS version, original drive/swap/var/db size, etc.)

jkozee is offline   Reply With Quote
Old 06-29-2015, 11:46 PM   #12
telemark
Registered User
 
Join Date: Nov 2013
Posts: 1,483
Remaking a TiVo oriented Linux boot-CD is on my todo list, so I should be able handle that last one.

telemark is offline   Reply With Quote
Old 06-30-2015, 12:17 AM   #13
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
This link seems like a good place to start. I used the pre-built iso, but perhaps the build scripts would allow for better large drive support.

jkozee is offline   Reply With Quote
Old 06-30-2015, 05:56 AM   #14
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Quote:
Originally Posted by jkozee View Post
telemark,

There's really only two issues that come to mind. The first is 64-bit APM creation. TiVo's pdsik64 code allowed for creating both 32-bit and 64-bit APM entries (as does mfstools). Mixing 32-bit and 64-bit doesn't appear to cause an issue, but it's probably better to sort it out now. Another question is when should 64-bit APM's be applied to Premiere restores. So, really it comes down to use cases: when should we use 64-bit APM's (platform, drive size, os version).

The seconds issues is that the small linux distro I was using was not handling large drives correctly (I think the kernel was ok, but not busybox).
Some testing done on Premieres with OS 20.3.8 indicated that the TiVo did not like mixing 32/64 bit APM entries and that all partitions needed to be 32bit in size and start within a 32bit address space. The APM entries needed to be either all 32 bit entries or all 64bit entries except for partition 1 which needed to be 32bit.

With OS 20.4.6, partitions could now start outside the 32bit address space but the other limitations still existed.

At this point, the only time 64bit partitions need to be used is when restoring requires placement of a partition beyond the 32bit address space. When it does that, all APM entries except the first one need to be converted to a 64bit APM entry. Might need to consider OS testing the backup before restoring because a boot loop issue would occur if OS version 20.3.8 were to be restored using 20.4.6 criteria. (Having a partition that starts outside the 32 bit address space)

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-30-2015, 07:03 AM   #15
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Quote:
Originally Posted by jkozee View Post
I think mfstools should be able to read the coalesced drive and zones ok. The problem would be that assumes that 10 and 12 are both app partitions and it will count both of them as required space to restore the drive, in addition to the media space also in partition 12. It shouldn't be too hard to force partition 12 to match the size of partition 10 in the case of a coalesced 12/13->12. Creating coalesced partitions and handling a single App partition would require more work though.
That's good. Since the standard TiVo layout is always 2 app and 2 media partitions, the quick and dirty method would be to assume partition 10 is not coalesced and use that size to create partition 12 on the target drive and copy that portion over to the target drive.

A more elegant way would be to obtain the original partition 10 and 12 size from the MFS and the original partition 12 location from the MFS (Since partition 10 is always the first partition of the MFS we just need its size if it happens to be coalesced as well) and the use those to create the target drive.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-30-2015, 02:07 PM   #16
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
It's probably safe to assume that anything 4TB and below can be formatted with all 32-bit

The 64-bit APM issue really only matters on drives larger than 4TB. For 4TB and below, mfstools will size the media partitions to keep them below the 32-bit barrier for start and size. And it's safe to assume that mixing 32-bit with 64-bit was never required (except the first partition which is always 32-bit). So, I'll update the code to use 64-bit for all entries (except the first) if the drive is greater than 4TB.

Currently, mfstools does not work correctly with drives larger than 4TB. I have some local patches that will allow it to create MFS partitions larger than 2TiB, but the TiVo OS does not support them. There are workarounds to creating larger drives, but nothing is implemented in mfstools yet. I'm hoping that a future TiVo SW release will correct the issue and allow us to create a standard two pair (app/media) partition.

It might be possible to use a partition larger than 2TiB by splitting it multiple zones, however I was not successful in my initial attempt.

jkozee is offline   Reply With Quote
Old 06-30-2015, 03:09 PM   #17
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
How about adding a switch that can be set to force 64bit APM entries independent of drive size.

Also, if TiVo does fix adding a pair of partitions to a premiere, then MFSAdd will need to be modified to convert the APM when adding a pair of partitions on large drives. As an aside, they would probably fix the 32bit limit on the media partitions first as that would obviate the need for adding a pair of partitions. When that will happen, who knows. All we can do is work with the cards we are dealt with at this time.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-30-2015, 11:53 PM   #18
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
We could add a switch, but that seems premature to me. Ideally, we should handle this based on use cases, from observed behavior. Seems more appropriate to sort this out in development thread...

Also, I'm not sure I follow the rest of your post. I don't see anything wrong with the native Premiere drive that needs to be fixed. Adding an additional app/media pair to an existing (SINGLE) drive seems unlikely to occur. Adding a second drive may need some work, as MFSadd has only been tested by adding an additional drive.

jkozee is offline   Reply With Quote
Old 07-01-2015, 06:02 AM   #19
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
What I was getting at is that there is no guarantee that TiVo will fix anything on how 64bit APMs are currently handled on the Premiere. So if we are going to attempt to increase the recording size to 6TB on a SINGLE drive, if it going to work, then we have to either add a coalesced partition to a standard 4TB Premiere image (2 app and 2 media partitions) as we cannot just add a standard app/media pair or create an image similar to what mfsr does for the Roamio that is 1 app and 3 media partitions.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-01-2015, 06:11 AM   #20
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Quote:
Originally Posted by jkozee View Post
We could add a switch, but that seems premature to me. Ideally, we should handle this based on use cases, from observed behavior. Seems more appropriate to sort this out in development thread...
Isn't this the development thread?

Quote:
Originally Posted by jkozee View Post
Also, I'm not sure I follow the rest of your post. I don't see anything wrong with the native Premiere drive that needs to be fixed. Adding an additional app/media pair to an existing (SINGLE) drive seems unlikely to occur. Adding a second drive may need some work, as MFSadd has only been tested by adding an additional drive.
What I was getting at is that there is no guarantee that TiVo will fix anything on how 64bit APMs are currently handled on the Premiere. So if we are going to attempt to increase the recording size to 6TB on a SINGLE drive, if it going to work, then we have to either add a coalesced partition to a standard 4TB Premiere image (2 app and 2 media partitions) as we cannot just add a standard app/media pair or create an image similar to what mfsr does for the Roamio that is 1 app and 3 media partitions.

If we are able to create a 6TB image by adding space to a 4TB drive via MFSAdd, MFSAdd will have to add that extra space as a 64bit APM entry to the APM and convert all the current APM entries (except the 1st partition entry) to 64bit APM entries.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-01-2015, 09:12 PM   #21
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
Quote:
Originally Posted by jmbach View Post
Isn't this the development thread?
Snarkyness aside, I thought that was obvious, but perhaps my response was not...

All I meant is that as a developer, it is probably a waste of your time to add a feature to mfstools that is just a way to test use cases for 64-bit APM entries. Adding a flag means a bunch more code changes, usage changes, and would be required in restore, copy, mfsadd, macpart, etc. to yield the results you are after. A simpler way for you to test is to change a single line of code in macpart.c to force everything to 64-bit for any testing you would like to do. I see no reason why we would need to force a drive to 64-bit that does not require it outside of development testing. Do you see a need to have a flag in the general release?

At this point, I cannot see any reason we would need to force a drive to 64-bit APM. As long as all partition start and lengths are less than 32-bits, then a standard APM shuold suffice. If any exceed 32-bits, then all entries should be 64-bit or else the partition could not be accessed correctly. If any testing proves otherwise, we should build that logic into mfstools.

And by sorting out in *this* thread, I just meant lets discuss if further to see if there really is a need for a new feature before doing in work that may prove counterproductive.

Quote:
Originally Posted by jmbach View Post
If we are able to create a 6TB image by adding space to a 4TB drive via MFSAdd, MFSAdd will have to add that extra space as a 64bit APM entry to the APM and convert all the current APM entries (except the 1st partition entry) to 64bit APM entries.
Not sure I see the issue here. If mfstools is changed to create 64-bit partitions when the drive is over 4TB (regardless of -c -m -M flags), then there would be no reason to convert any entries, as they would already be 64-bit entries when the restore/copy was done to the 6TB drive. Sound right?

jkozee is offline   Reply With Quote
Old 07-01-2015, 09:18 PM   #22
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
To force all entries (except the first) to 64-bit change this code in macpart.c:
Code:
			if (table->partitions[loop].start <= UINT_MAX &&
					table->partitions[loop].sectors <= UINT_MAX )
To this:
Code:
			if (table->partitions[loop].start <= UINT_MAX &&
					table->partitions[loop].sectors <= UINT_MAX && loop == 0)

jkozee is offline   Reply With Quote
Old 07-01-2015, 09:52 PM   #23
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Quote:
Originally Posted by jkozee View Post
Snarkyness aside, I thought that was obvious, but perhaps my response was not...
I took your response as there existed another thread in which to discuss this.

Quote:
Originally Posted by jkozee View Post
All I meant is that as a developer, it is probably a waste of your time to add a feature to mfstools that is just a way to test use cases for 64-bit APM entries. Adding a flag means a bunch more code changes, usage changes, and would be required in restore, copy, mfsadd, macpart, etc. to yield the results you are after. A simpler way for you to test is to change a single line of code in macpart.c to force everything to 64-bit for any testing you would like to do. I see no reason why we would need to force a drive to 64-bit that does not require it outside of development testing. Do you see a need to have a flag in the general release?

At this point, I cannot see any reason we would need to force a drive to 64-bit APM. As long as all partition start and lengths are less than 32-bits, then a standard APM should suffice. If any exceed 32-bits, then all entries should be 64-bit or else the partition could not be accessed correctly. If any testing proves otherwise, we should build that logic into mfstools.

And by sorting out in *this* thread, I just meant lets discuss if further to see if there really is a need for a new feature before doing in work that may prove counterproductive.
Agreed


Quote:
Originally Posted by jkozee View Post
Not sure I see the issue here. If mfstools is changed to create 64-bit partitions when the drive is over 4TB (regardless of -c -m -M flags), then there would be no reason to convert any entries, as they would already be 64-bit entries when the restore/copy was done to the 6TB drive. Sound right?
It does sound correct if TiVo corrects the 2TB partition limit. If not then my thought process was that if we create a 6TB drive by copying over a 4TB image to a 6TB drive, followed by adding a pair of partitions, then the current 32bit APM will need to be converted to 64bit when MFSAdd is used to add the pair of partitions. However with current limitations, that pair of partitions would have to coalesced into a single partition.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-01-2015, 10:17 PM   #24
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
Quote:
Originally Posted by jmbach View Post
It does sound correct if TiVo corrects the 2TB partition limit. If not then my thought process was that if we create a 6TB drive by copying over a 4TB image to a 6TB drive, followed by adding a pair of partitions, then the current 32bit APM will need to be converted to 64bit when MFSAdd is used to add the pair of partitions. However with current limitations, that pair of partitions would have to coalesced into a single partition.
Agreed about the need for coalescing for a 6TB drive if greater than 2TiB partitions are not supported, but I still don't see the converting issue.

Once the patch is made, when you copy over the 4TB image to a 6TB drive, the 6TB drive will have 64-bit entries. Remember, mfstools can create the 6TB drive with a standard two pair setup and limit the resulting image size to 4TB. Perhaps you were thinking you would need to dd the 4TB to the 6TB drive instead of using mfstools?

jkozee is offline   Reply With Quote
Old 07-01-2015, 10:48 PM   #25
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
I guess if copying the image to a drive larger than 4TB, the program will automatically create a 64bit APM even if the image is not expanded at the same time, then nothing else needs to be adjusted.

But yes, I was thinking of a dd or a drive duplicator option for the initial copy as I thought it would be faster.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-01-2015, 11:02 PM   #26
tivoROCKSme
Registered User
 
Join Date: Jun 2003
Posts: 97
just want to say that it's awesome that you guys are working on this new technology. Good job collaborating and thanks for working on the next gen

__________________
Tivo HD 1TB (WD OEM drive) Motorola M card Comcast
tivoROCKSme is offline   Reply With Quote
Old 07-01-2015, 11:32 PM   #27
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
dd would probably be the slowest option, as it is a block level copy of the entire drive, but would produce an exact replica of the original. Using mfstools has the advantage of being able to skip unused space (especially if you omit the -a and skip recordings). Using v1 format is faster that v3, but only light testing has been done on v1 beyond series 3 units and v1 does not offer the ability to resize MFS media partitions.

jkozee is offline   Reply With Quote
Old 07-02-2015, 12:18 AM   #28
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
You can speed up dd by adjusting the block size when copying. Not sure how fast it would be as compared to MFSTools.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-03-2015, 08:25 AM   #29
jmbach
der Neuerer
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 1,110
Testing a modified version of MFSTools to work with a DIY 4TB Premiere image. Looks like about 50 hours to copy the drive. Will see if it works when it is done.

__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 07-03-2015, 08:09 PM   #30
jkozee
Registered User
 
Join Date: Jan 2006
Posts: 33
Great. Post the results.

The issue with the DIY 4TB drive is that is has a coalesced 12/13 pair. This was unexpected by mfstools. It can read the drive just fine, but when it lays out a new drive it blindly bases the two app partitions on the partition size of the original ones, so it assumes that 10 and 12 are app partitions.

A quick hack was done to workaround this, but we will need to refine it before we put it into the release. There will be similar, but more complex issues if we add support for mfsr created drives. Those drives only contain one app partition, so mfstools would double the app size with the way this workaround was implemented. Additionally, I suspect mfstools will have issues, as it will expect a second app zone, which will be missing.

jkozee is offline   Reply With Quote
Reply
Forum Jump







Thread Tools


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Advertisements





TiVo Community
Powered by vBulletin® Version 3.8.5
Copyright ©2000 - 2015, Jelsoft Enterprises Ltd.
vBulletin Skins by: Relivo Media


(C) 2015 DBNet - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not owned or operated by TiVo Inc.
All times are GMT -5. The time now is 05:47 AM.
Page generated in 0.17396402 seconds (72.31% PHP - 27.69% MySQL) with 18 queries