TiVo Community
TiVo Community
TiVo Community
Go Back   TiVo Community > Underground Playground > TiVo Upgrade Center
TiVo Community
Reply
Forum Jump
 
Thread Tools
Old 06-10-2014, 07:04 PM   #61
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
The data ID change does not affect the recordings on the drive (premium or not) just my premium channels I receive.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-11-2014, 12:51 AM   #62
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by jmbach View Post
The data ID change does not affect the recordings on the drive (premium or not) just my premium channels I receive.
It's possibly a YMMV situation, like I described in this post:

http://www.tivocommunity.com/tivo-vb...1#post10137631

While my Cox market makes little use of the CCI-bit, the cablecard pairing authentication is tightly locked-down.

Cablecard pairing is lost on sector-by-sector drive clones (all the way back to the TiVo HD), even if the drive model and size is identical.

The CCI-bit protected content is only able to play back if the cablecard that was used to record it is still installed, and authenticated.

None of this was the case a few cablecard firmware revisions back.

I'm very likely in a rare market condition here (just like me being in a full 1GHz RF spectrum market, that still carries analog, which uses SDV). If I can do anything to help, being in this market situation, please don't hesitate to ask me (anybody).

I really want to help. I have an uncle who could probably do what everybody is trying to, and make it work in both Linux and Windows, if he only knew anything, at all, about TiVo. He's willing to tackle any challenge I present to him, provided I can get him to understand the TiVoization factors involved...

He is an actual programmer, at a very large corporation, who often gets told "That can't be done", only to write the program to do exactly that, and make it work company-wide, and cross-platform.
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 04:51 AM   #63
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
Porting JMFS should a straightforward task, which is why I suggested it. I downloaded it just now, and it builds. It should take like a day or weekend to patch it. I honestly never used it before though, so I might have some questions re what it it's intending to do.

> I really want to help. I have an uncle who could probably do what everybody is trying to, and make it work in both Linux and Windows, if he only knew anything, at all, about TiVo. He's willing to tackle any challenge I present to him, provided I can get him to understand the TiVoization factors involved...

You make it sound like he doesn't have any Tivos in his house? There are useful tasks for any skill set, you don't have to be programmer.

> He is an actual programmer, at a very large corporation, who often gets told "That can't be done", only to write the program to do exactly that, and make it work company-wide, and cross-platform.

Ya, I like those stories. Can you say what city?

This probably sounds surprising, but I'm not an actual programmer. Because of my skill set, I get tasked with other things, but I do get to technically manage junior developers.

I have a pretty long list of TODOs, more than I will ever finish, but at the same time I know how to do almost everything I added to the list. Those I don't know how to do, are really hard, unless someone has some specialized skills in some of the tech.

Maybe we should be having the conversation, what larger projects are worthwhile.
__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA

Last edited by telemark : 06-11-2014 at 09:33 AM.
telemark is offline   Reply With Quote
Old 06-11-2014, 10:49 AM   #64
lessd
Registered User
 
Join Date: Jan 2005
Location: CT
Posts: 6,149
Quote:
Originally Posted by nooneuknow View Post
It's possibly a YMMV situation, like I described in this post:

http://www.tivocommunity.com/tivo-vb...1#post10137631

The CCI-bit protected content is only able to play back if the cablecard that was used to record it is still installed, and authenticated.

None of this was the case a few cablecard firmware revisions back.

.
You can play any recorded program on any TiVo that it was recorded on even, if the TiVo now has no service or has no cable card installed. the CCI-Bit has nothing with the play-back, except for some PPV movies that have built in time outs for playback.
__________________
Les Daniels

_____________________________________________
3 Roamio Plus upgraded to 2Tb & 3Tb, and 2 Minis,
lessd is offline   Reply With Quote
Old 06-11-2014, 02:14 PM   #65
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
Quote:
Originally Posted by lessd View Post
You can play any recorded program on any TiVo that it was recorded on even, if the TiVo now has no service or has no cable card installed. the CCI-Bit has nothing with the play-back, except for some PPV movies that have built in time outs for playback.
This has been my experience as well. As such I believe the recordings are attached to the TiVo serial number rather than the cableCARD.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-11-2014, 03:07 PM   #66
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by jmbach View Post
This has been my experience as well. As such I believe the recordings are attached to the TiVo serial number rather than the cableCARD.
Quote:
Originally Posted by lessd View Post
You can play any recorded program on any TiVo that it was recorded on even, if the TiVo now has no service or has no cable card installed. the CCI-Bit has nothing with the play-back, except for some PPV movies that have built in time outs for playback.
I respect both of you. I'd rather have you loosely "in my corner", than being firmly in the opposite one.

I've gone out of my way to make sure what I say is as accurate a depiction I can make, using the best possible reasons I can come up with, for what I experience, and can replicate, in my posts.

Perhaps what is being done in my market is (possibly) illegal, and Cox is only trying to verify it works, they can get away with it, and/or plans to take it wide scale, if no legal hurdles come up. Perhaps it's a TiVo bug, manifesting in my "less than usual" Cox Fibre/Coax 1GHz network w/analog, digital, and SDV, all in use, situation.

Unless a person is in my market, using the same infrastructure, same cablecards, same firmware, and replicating the ecosystem of the network, as well as on-the-premise equipment, who are they to say it's not happening, or is not possible to happen?

I've stayed silent on the matter, until telemark came along. I'm looking for fresh eyes, fresh thoughts, and not the Same-Old-Sh*t, I got rained on me the last time I brought this specific matter up, when it first reared its ugly head.

If I have to take all the data points I can provide to PM, or other private back-channel, I can do that. I'm trying to do "the community thing".

I can replicate the results without having to make special effort to make them happen. If I still had my older-than-Roamio platforms, I could replicate the results all the way back to the TiVo HD.

I have never failed to replicate the results I spoke of. I've never had to make everything "exactly just-so", to get/replicate the results.

Perhaps it's just the choices of descriptions I use, the way I say them, or the best-educated-guess reasons I share, for why I can get what I get, and replicate it.

The bottom line is that the few CCI-bit protected recordings I end up with, will playback for years without issue. If I remove the cablecard that was authorized when the recordings were made, they won't playback (TiVo error claims no signal was present at the time of the pre-existing recording). Re-inserting and re-pairing the same card, restores the playback (of a recording that has played-back before, without issue). Inserting another card, and pairing it up (or not pairing), results in the aforementioned error, on a recording that had been playing-back for a year, or more.

I see this as a reason to fully understand the suspected GUID, any linkages to cablecards, any linkages to pairing, and/or any linkages to imaging/cloning.

I feel being able to get to the bottom of this requires my input (no matter how impossible some claim my experiences are), and/or more participation from others in my market, with the right factors present.

A perfect example recording is "The Hitchhikers Guide To The Galaxy" (the movie), always recorded from IFCP (considered a "premium", included in a channel "pak"), which does set the cci-bit. I can count on losing access to playing it back, under the circumstances I describe, and can count on restoring playback of it, if I still have the same drive, same cablecard, all in-place (and I haven't since authorized any other card, than the one used during recording).

So, how I proceed is up to the community (options) :

1. I retreat again, and work with nobody on this.
2. I retreat again, and work in private on this.
3. I continue to work with the community, in the open, and hopefully find some others to join-in, from my market.
__________________
...

Last edited by nooneuknow : 06-11-2014 at 06:31 PM.
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 05:48 PM   #67
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
noneuknow, I have not said that you were wrong, lying, misleading, etc. I have not accused you of or blamed you for anything. I opened my remark with "my experience" because I know that conditions vary with cable companies, locations, etc. So as not to clutter up this thread I will PM you with a response to your post.

Let us go back to the subject at hand.

Perhaps telemark, you can list your TODO list and there might something in there that either I or another TCF member can help with. TODO lists go much quicker with multiple people working on it and perhaps can get it done.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-11-2014, 06:22 PM   #68
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by jmbach View Post
noneuknow, I have not said that you were wrong, lying, misleading, etc. I have not accused you of or blamed you for anything. I opened my remark with "my experience" because I know that conditions vary with cable companies, locations, etc. So as not to clutter up this thread I will PM you with a response to your post.

Let us go back to the subject at hand.

Perhaps telemark, you can list your TODO list and there might something in there that either I or another TCF member can help with. TODO lists go much quicker with multiple people working on it and perhaps can get it done.
Yeah, another nooneuknow "nuke", that I pretty much dropped indiscriminately. Sorry about that...

I had brought this unusual "lockdown" of content up in the past, when I first observed it, after a cablecard flash update went out.

It got very ugly, very fast. I was accused of "incorrectly cloning" as I tried to gather data points. I was accused of making things up. It seemed that every comment was pegging me (or my actions) as somehow creating the results, not any part of any system, and so on...

I had to just back-out, and say to myself "It's not worth it, if this is what I'm going to get as a result".

The party that really just wouldn't let up on claiming what I observed, and replicated "was impossible", isn't even present (yet).

I've seen new talent come in, and a new opportunity to possibly get to the bottom of this nagging matter, after having to force myself to not even bring it up all this time.

Then I go and lash-out, at the first inkling that I'm going to just get a repeat of last time. Realistically, I sabotaged myself.

I really think this suspected GUID on the drives that has been found, the cablecard pairing data anomaly I have found, and the future of drive changes and upgrades are all part of a design, that I may be able to provide some necessary data on. I just need some help finding what is relevant, and filtering out the rest.

I'd guess that if the way my market locks-down content wasn't isolated, but was instead the "norm", or became the norm, things would be much different from where I sit, with my POV.

Let's see if I can fix that post, and get back on track...
__________________
...

Last edited by nooneuknow : 06-11-2014 at 06:32 PM.
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 06:47 PM   #69
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by telemark View Post
Porting JMFS should a straightforward task, which is why I suggested it. I downloaded it just now, and it builds. It should take like a day or weekend to patch it. I honestly never used it before though, so I might have some questions re what it it's intending to do.

> I really want to help. I have an uncle who could probably do what everybody is trying to, and make it work in both Linux and Windows, if he only knew anything, at all, about TiVo. He's willing to tackle any challenge I present to him, provided I can get him to understand the TiVoization factors involved...

You make it sound like he doesn't have any Tivos in his house? There are useful tasks for any skill set, you don't have to be programmer.

> He is an actual programmer, at a very large corporation, who often gets told "That can't be done", only to write the program to do exactly that, and make it work company-wide, and cross-platform.

Ya, I like those stories. Can you say what city?

This probably sounds surprising, but I'm not an actual programmer. Because of my skill set, I get tasked with other things, but I do get to technically manage junior developers.

I have a pretty long list of TODOs, more than I will ever finish, but at the same time I know how to do almost everything I added to the list. Those I don't know how to do, are really hard, unless someone has some specialized skills in some of the tech.

Maybe we should be having the conversation, what larger projects are worthwhile.
Yeah, unfortunately, my uncle does not have any TiVos. I'm in the process of parting-out the last unsubbed TiVo HDs I have, which is the last of any "extra" TiVos I have around...

I'll decline on stating the company he works for, the exact position, etc. It's pretty much nationwide and multinational scale-wise.

I think of him as ggieseke, except linux is his preferred playground, and Windows is just something he has to make his projects work on.

At one point, before you came around, I'd asked ggieseke if it would make sense for a two-pronged approach, leaving ggieseke to update DVRBARS, and I'd have my uncle bring JMFS up to date. That conversation kind of died.

I also wondered how I could expect my uncle to remake JMFS, with just source-code, and no TiVo Roamio (or other model) to work with. I'm not well-off enough to buy him the hardware platforms and drives, and it's not on his to-do list, either...

I really want to dig into finding any GUIDs that exist on TiVos, and find a way to backup and restore just the data needed to keep the cablecard pairing from being lost, or "zapped". You've already read enough to know the "why" behind this, and my motivations.
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 07:16 PM   #70
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,071
FWIW, I think CableCARD pairing data will always be a black hole that's dependant on your provider. Unless it gets buried somewhere in the tyDb files on the MFS partitions we're screwed there since nobody has come up with a way to read the flash memory on a Roamio yet.
ggieseke is offline   Reply With Quote
Old 06-11-2014, 07:35 PM   #71
lessd
Registered User
 
Join Date: Jan 2005
Location: CT
Posts: 6,149
Quote:
Originally Posted by nooneuknow View Post
I respect both of you. I'd rather have you loosely "in my corner", than being firmly in the opposite one.

I've gone out of my way to make sure what I say is as accurate a depiction I can make, using the best possible reasons I can come up with, for what I experience, and can replicate, in my posts.

Perhaps what is being done in my market is (possibly) illegal, and Cox is only trying to verify it works, they can get away with it, and/or plans to take it wide scale, if no legal hurdles come up. Perhaps it's a TiVo bug, manifesting in my "less than usual" Cox Fibre/Coax 1GHz network w/analog, digital, and SDV, all in use, situation.

Unless a person is in my market, using the same infrastructure, same cablecards, same firmware, and replicating the ecosystem of the network, as well as on-the-premise equipment, who are they to say it's not happening, or is not possible to happen?

I've stayed silent on the matter, until telemark came along. I'm looking for fresh eyes, fresh thoughts, and not the Same-Old-Sh*t, I got rained on me the last time I brought this specific matter up, when it first reared its ugly head.

If I have to take all the data points I can provide to PM, or other private back-channel, I can do that. I'm trying to do "the community thing".

I can replicate the results without having to make special effort to make them happen. If I still had my older-than-Roamio platforms, I could replicate the results all the way back to the TiVo HD.

I have never failed to replicate the results I spoke of. I've never had to make everything "exactly just-so", to get/replicate the results.

Perhaps it's just the choices of descriptions I use, the way I say them, or the best-educated-guess reasons I share, for why I can get what I get, and replicate it.

The bottom line is that the few CCI-bit protected recordings I end up with, will playback for years without issue. If I remove the cablecard that was authorized when the recordings were made, they won't playback (TiVo error claims no signal was present at the time of the pre-existing recording). Re-inserting and re-pairing the same card, restores the playback (of a recording that has played-back before, without issue). Inserting another card, and pairing it up (or not pairing), results in the aforementioned error, on a recording that had been playing-back for a year, or more.

I see this as a reason to fully understand the suspected GUID, any linkages to cablecards, any linkages to pairing, and/or any linkages to imaging/cloning.

I feel being able to get to the bottom of this requires my input (no matter how impossible some claim my experiences are), and/or more participation from others in my market, with the right factors present.

A perfect example recording is "The Hitchhikers Guide To The Galaxy" (the movie), always recorded from IFCP (considered a "premium", included in a channel "pak"), which does set the cci-bit. I can count on losing access to playing it back, under the circumstances I describe, and can count on restoring playback of it, if I still have the same drive, same cablecard, all in-place (and I haven't since authorized any other card, than the one used during recording).

So, how I proceed is up to the community (options) :

1. I retreat again, and work with nobody on this.
2. I retreat again, and work in private on this.
3. I continue to work with the community, in the open, and hopefully find some others to join-in, from my market.
You are the first person on this Forum (that I have read) that has said that a TiVo recorded playback depends on having the cable card in the TiVo, people have moved from one place to another taking their TiVo(s) with them, then putting a new cable card for their new MSO in the TiVo, they have never reported any loss of playback recordings already on the TiVo, I have seen that I can play any recording on a non activated TiVo without its cable card, no problem, but I can't answer for your area or MSO, it just that if this was the way things worked in parts of the country I just assumed (maybe incorrectly) this Forum would have been talking about this playback issue sometime after sept. 2006 when the first cable cards TiVos went on sale, that almost 8 years ago.
__________________
Les Daniels

_____________________________________________
3 Roamio Plus upgraded to 2Tb & 3Tb, and 2 Minis,
lessd is offline   Reply With Quote
Old 06-11-2014, 07:42 PM   #72
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by ggieseke View Post
FWIW, I think CableCARD pairing data will always be a black hole that's dependant on your provider. Unless it gets buried somewhere in the tyDb files on the MFS partitions we're screwed there since nobody has come up with a way to read the flash memory on a Roamio yet.
I'm not sure if I "doubt" it's in the Roamio's flash, or just really "don't want it to be" there... Even if it is, if no GUID change is detected, shouldn't it stay intact? Basically, trick the Roamio into preserving it, thus negating a need to see the flash contents?

Maybe the guy wanting to know the smallest possible hard drive that a Roamio could prepare (or run, pre-made) was onto something...

Smaller drives make raw disk editors/hex editors less likely to take years off your life as you scan the data on them, looking for crumbs/clues/changes...

I swear, if Apple made these things, they'd have been "jailbroken" the day before they went on sale...
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 08:00 PM   #73
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by lessd View Post
You are the first person on this Forum (that I have read) that has said that a TiVo recorded playback depends on having the cable card in the TiVo, people have moved from one place to another taking their TiVo(s) with them, then putting a new cable card for their new MSO in the TiVo, they have never reported any loss of playback recordings already on the TiVo, I have seen that I can play any recording on a non activated TiVo without its cable card, no problem, but I can't answer for your area or MSO, it just that if this was the way things worked in parts of the country I just assumed (maybe incorrectly) this Forum would have been talking about this playback issue sometime after sept. 2006 when the first cable cards TiVos went on sale, that almost 8 years ago.
Nothing about my market reflects the rest of the country, except that we use coax and cablecards (this is the short version of my point).

I often feel like I'm the only TiVo owner in my market. I get treated like that's the case when I deal with them (Cox).

Who else, except me, is going to be OCD enough to investigate why "The Hitchhiker's Guide to the Galaxy" (and maybe six other recordings) consistently fail to be watchable every time I change the cablecards?

Who else, in my market had eight TiVos and eight cablecards at one time, and would move the cablecards around, or replace them every so often?

Most probably would shrug the loss of a few movies they wanted to hang onto, delete them, and never even know TCF exists, right?

I'm not trying to be argumentative in this. I see this as a logical answer to the question you pose: "Why only hear this from me?"

Who messes around with their TiVos, changes drives, whole configurations, and all these other things never meant to be done, just to replicate the results and come here to report them? That seems to be limited to me, when it comes to my market.

I don't see that as making what I report any less true, or "impossible" and some will flat-out say...

The content and spirit of this whole thread, and all the DIY threads are not what "most people" do when they buy a DVR...

While I feel all this has a part in "best practices" for future drive expansions, I also don't want to hijack the thread, over what is just part of a bigger picture...
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-11-2014, 08:07 PM   #74
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,071
Quote:
Originally Posted by nooneuknow View Post
I'm not sure if I "doubt" it's in the Roamio's flash, or just really "don't want it to be" there... Even if it is, if no GUID change is detected, shouldn't it stay intact? Basically, trick the Roamio into preserving it, thus negating a need to see the flash contents?

Maybe the guy wanting to know the smallest possible hard drive that a Roamio could prepare (or run, pre-made) was onto something...

Smaller drives make raw disk editors/hex editors less likely to take years off your life as you scan the data on them, looking for crumbs/clues/changes...

I swear, if Apple made these things, they'd have been "jailbroken" the day before they went on sale...
YMMV says it all.
ggieseke is offline   Reply With Quote
Old 06-11-2014, 09:10 PM   #75
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
LE jmfs, what it does, atm.

So I posted 4 LittleEndian object implementations. #3 had an inconvenient web page. #2 was implemented as a subclass, which is slightly different than the others.

The Google one, #1, looks complete, but it didn't have an ant build file, and was large. The OReilly one, #4, was simple, but incomplete. I frankensteined them and added 2 missing functions myself.

I don't know if it's 100% correct but this does build.

On another attempt I'll try a different combination. The Google library should work out of the box, and the subclass might be fine too.

Bad news:
Spoiler:

Giving it a 500GB hard drive the ROAMIO freshly produced,
./mfslayout.sh /dev/disk1

java.lang.Exception: No root MFS found
at tivo.Mfs.addDisks(Mfs.java:311)
at tivo.Mfs.<init>(Mfs.java:75)
at tivo.Mfs.<init>(Mfs.java:69)
at jmfs.MfsLayout.main(MfsLayout.java:42)
MfsLayout: done


Good News:
Spoiler:

As baseline, I compiled the krbaker and jmbach github jmfs without modification, and ran them against my PREMIERE backups.

About half work, but half of them give me:
./mfslayout.sh /dev/disk1

java.lang.Exception: No root MFS found
at tivo.Mfs.addDisks(Mfs.java:309)
at tivo.Mfs.<init>(Mfs.java:74)
at tivo.Mfs.<init>(Mfs.java:68)
at jmfs.MfsLayout.main(MfsLayout.java:42)
MfsLayout: done



I could have corrupt images, or a bad build. Or has this been seen before?
32bit vs 64bit formats maybe?
__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA

Last edited by telemark : 06-11-2014 at 10:29 PM.
telemark is offline   Reply With Quote
Old 06-12-2014, 12:31 AM   #76
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
I have had that error in the past. If I recall correctly, it is looking for a valid Partition 10 and cannot find it. There is an environment variable you can set to tell it where its at but it never worked for me. Sometimes when this happens, I found that there was an issue reading the APM. Other times there was some corruption in partition 10. The rest I did not have a clue.

Now JMFS does not recognize a 32bit MFS layout. If you used it on all Premiere images, they are all 64bit MFS, so that should not have been an issue.

One test would be to see if pdisk from MFSLive CD can read the APM or if dd can recognize the partition by trying to copy it.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-12-2014, 02:18 AM   #77
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
I'm trying the slax live CD now and it is more consistent. Very odd, I can move the same build from a desktop OS to the CD environment and it'll start working.
__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA
telemark is offline   Reply With Quote
Old 06-12-2014, 09:12 AM   #78
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
You have brought back painful memories. I had no clue that dd would understand a TiVo APM for a long time until reading the forums and seeing that members were using it to copy individual partitions. So when I was working on figuring out how to expand a premiere to 4TB, I thought I would just use dd. I had so many issues using dd to copy partitions that I used iBored instead. In retrospect, it might have been related to me using VirtualBox to run the ISOs and it can only access the drive I was working on via USB2 and not eSata. I figured out that was hdparm's problem but never thought it was dd's issue as well.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-12-2014, 05:35 PM   #79
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by jmbach View Post
I had no clue that dd would understand a TiVo APM for a long time until reading the forums and seeing that members were using it to copy individual partitions.
I found out by accident, due to sleep-deprivation (a state that often contributes to some of my meandering/long posts).

When using GNU dd_rescue intending a full clone, I typed "dev/sda1 dev/sda2" instead of "/dev/sda /dev/sdb". Big "oops!" then "aha!" moment. Luckily, I'd practiced what I preached and was working with a backup copy of sda.

I don't know the long-term effects of copying the boot sector over the beginning of partition 2 of a Premiere, but it effectively stopped software from installing to that partition via KS52, and didn't kill the unit, even after a KS57&58 failed to fix it (and didn't create a boot-loop). I'd hoped to test if it would "block" a newly downloaded software from installing itself, but never remembered to try it. Another thought was it was a way to take a partition with bad sectors "out of play", letting you limp along on the good alternate.

Not Roamio-related, AFAIK, but maybe it will give somebody an idea on how to do something they hadn't thought of a way to.
__________________
...

Last edited by nooneuknow : 06-12-2014 at 06:00 PM.
nooneuknow is offline   Reply With Quote
Old 06-12-2014, 05:59 PM   #80
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by telemark View Post
I'm trying the slax live CD now and it is more consistent. Very odd, I can move the same build from a desktop OS to the CD environment and it'll start working.
I seem to recall slax being used by Comer for at least few things during the making of JMFS, and it sometimes being the choice of others, over the many distros out there.

I've yet to learn enough about linux to know what distro is best. I'm mostly looking for:

1. Most drivers already included (especially USB3 onboard and add-on card)
2. Up to date device drivers (not necessarily for the newest HW)
3. Best raw copy rate between two USB3 drive docks
4. I like GNU dd_rescue, and like to run the newest release
5. I want it to not be shy about using the RAM and resources available, like for caching disk I/O

I like Live DVDs, or alternately Live USB sticks. Most of my dedicated test systems (guts on a bench) are mid Core2Duo era, with 4-6GB RAM.

What's suggested for my preferences, and also works well with 3 & 4TB drives, if I'm going to start doing more than talk about wanting to try to help find ways to build large images, and find ways to preserve Roamio content when moving to larger drives?

I have to start somewhere... Programming/coding is something I know so little about, that I guess all that is left is working with partitions and layouts, and things that can be done with disk/hex editors.
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-12-2014, 06:18 PM   #81
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
Linux dd doesn't understand APM, it only understands block devices and offsets within them. Kernels (or tivopart) with APM-Tivo support will create sub-devices at the partition boundaries. Once/if those exist then you can use those sub-devices with dd to copy partitions.

jmfs was still rejecting my drive signature, and in tracking it down I discovered another class doing reads, called RandomAccessFile. I'm going to ask a Java friend look at this.

Other tasks, hmm, it's pretty long. Well, one thing that I'd like to see more of but is not sexy, is tivo monitoring, and first off the logs. When the Roamio came around, and a few people have consistent reboots, the logs have more answers and we had to do a lot of guessing. And I understand the lack of desire, when my Tivo is working I don't care to turn it off. But when it's acting wonky, it still feels like too much work unless I have the data collection completely automated.

I was going to write something for Windows/Mac users since Linux can already read ext, but I found one already made called testdisk.

This might be good for nooneuknow's uncle, unless he's into hardware, as it is gpl, is self-contained, doesn't require a Tivo, and it has the description of the basic Tivo drive format to start from.

Re CableCards, sure it's possible your Tivo's doing that and no-one else's is. Through diligent work, you should be able to document and perhaps understand why, but at then end, you might not be able to do anything about it. It's a locked down platform, the only entities that can change things is Tivo (often uninterested), and maybe few things by your Cable Company (generally inept/unaware). That's also why I would assume any variances you're experiencing is more likely a technical oversight than something maliciously planned.

Re Flash, at this point, I don't think anything operational is written to Flash, well aside I guess, not any more than some old systems had things written to prom. That of course might change at any time.

If you want to create a CableCard or Cox-your-market thread, I'll add what I can, but you should probably stick to Series 3 or 4 since the drives are fully readable.
__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA
telemark is offline   Reply With Quote
Old 06-12-2014, 07:33 PM   #82
nooneuknow
TiVo User Since 2007
 
Join Date: Feb 2011
Location: Cox Cable Market, NV
Posts: 3,235
Quote:
Originally Posted by telemark View Post
jmfs was still rejecting my drive signature, and in tracking it down I discovered another class doing reads, called RandomAccessFile. I'm going to ask a Java friend look at this.
Since KMTTG is written for Java, perhaps the author may be of some assistance on Java matters?

Quote:
Originally Posted by telemark View Post
Other tasks, hmm, it's pretty long. Well, one thing that I'd like to see more of but is not sexy, is tivo monitoring, and first off the logs. When the Roamio came around, and a few people have consistent reboots, the logs have more answers and we had to do a lot of guessing. And I understand the lack of desire, when my Tivo is working I don't care to turn it off. But when it's acting wonky, it still feels like too much work unless I have the data collection completely automated.
Almost anything is better than having to pull the disk and use a disk editor to view the logs, which is the only method I know aside from TiVo Backdoor. The raw-disk method seems to expose logging not available via TB. The biggest issues I have with scanning the logs with the TiVo operational using TB is having no way to search, other than with my eyes, and no way to capture what I find (to copy, print, share), and the rate at which the logs rotate out, usually before I can scan all the different log sections.

Quote:
Originally Posted by telemark View Post
I was going to write something for Windows/Mac users since Linux can already read ext, but I found one already made called testdisk.

This might be good for nooneuknow's uncle, unless he's into hardware, as it is gpl, is self-contained, doesn't require a Tivo, and it has the description of the basic Tivo drive format to start from.
Very true. He's the software guy. I'm the hardware guy. You'll need to PM me a crash-course on setting up that JTAG-serial link you told me about (if that's possible with a Roamio).

Quote:
Originally Posted by telemark View Post
Re CableCards, sure it's possible your Tivo's doing that and no-one else's is. Through diligent work, you should be able to document and perhaps understand why, but at then end, you might not be able to do anything about it. It's a locked down platform, the only entities that can change things is Tivo (often uninterested), and maybe few things by your Cable Company (generally inept/unaware). That's also why I would assume any variances you're experiencing is more likely a technical oversight than something maliciously planned.

Re Flash, at this point, I don't think anything operational is written to Flash, well aside I guess, not any more than some old systems had things written to prom. That of course might change at any time.
I'm dropping my specific market parameters from the discussion. I tried to just make a reference to another thread I detailed it in, then wound up infesting the thread when I could feel the doubt about what I said creeping in.

I'm beginning to think TiVo might have moved the cc pairing data into the flash (at least some/most of it). Changes to most of that data happen at about the same rate as TiVo software updates, less if you never change your cablecard or drive. I'd suspect any data that tends to stay static could be moved into flash, and anything that updates when the cc renews authorization every xx days, or x months, or otherwise is fluid, could be buried/hidden on the hard drive. It makes sense. It doesn't make a goal of being able to find, capture, and restore it easier (which makes such a move make more sense). I've both read about, and experienced, where a Roamio will refuse to detect the cableco has unpaired the card, and retains data that makes re-pairing the same card, or pairing a new one fail. If given time (20 minutes to 3 hours), of just being left without a card present, it finally forgets the data and you can move on. This feels/seems like something one might expect when that "leftover" data is on flash. I never saw this, or heard about this exact scenario, pre-Roamio.

I'd drop the whole cablecard discussion, if it weren't for the possibility that those working on these projects might need to know what's what, to avoid issues down the road with DIY Roamio drive preparation, and for the goal of being able to upgrade drive sizes, without losing what is on the existing drive...

Quote:
Originally Posted by telemark View Post
If you want to create a CableCard or Cox-your-market thread, I'll add what I can, but you should probably stick to Series 3 or 4 since the drives are fully readable.
When I volunteered my help to ggieseke, I still had TiVo HDs and Premieres. I had enough that I could have easily swapped some out for testing. I've since migrated to three base-Roamios (one is a "hot spare"), and have sold off all the older equipment (nobody was taking me up on my offers to assist). I'm left with just one unsubbed TiVoHD, until somebody buys it, or buys the power supply inside it.

Any reason I should take that last HD off the market, so I have something older to work with, or should I take the $35 going-rate for a rebuilt power supply (if/when the opportunity presents)? I could always just build a makeshift power supply to use...

I'm very good at improvising and hardware matters, and have TONS of hardware around, going back decades, if anybody needs anything for building test-rigs or needs to make special-use devices out of parts I have.
__________________
...
nooneuknow is offline   Reply With Quote
Old 06-13-2014, 08:17 AM   #83
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
I don't mind scanning the drive and capturing the logs. I have a non dock USB to SATA adapter that allows me to connect the drive without removing it. All I have to do is leave the cover off if you are wanting frequent logs.
Now if there was a way to get the logs off the unit via ethernet, then we are more likely to get more logs. TiVo does it.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-13-2014, 04:45 PM   #84
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
After an absurd amount of typing to do something basic (Java interfaces), this is what I get after adding some debug lines when pointed at a 500GB, Base Roamio HD:
Code:
Debug:  5266 ? 5266
Debug:  Block0 isValid true
Debug:  Block0 isValid true
Debug:  0 
Debug:  1 
Debug:  2 
Debug:  3 
Debug:  4 
Debug:  5 
Debug:  6 
Debug:  7 
Debug:  8 
Debug:  9 
Debug:  10 
Debug:  11 
Debug:  12 
Debug:  13 
Debug:  storageSize=500107862016, nextFreeBlock=976773168, free=0
Debug: AppleDisk constructor #3
Debug: TivoDisk loadHeader
Debug:  TD isValidRoot MfsHeader64 { MfsHeaderHeader { state=0xEBBAFEED, magic=0x00000000, CRC=0x572822DA }, validCRC=FALSE, sectors=967,859,200, partitions='/dev/sda10 /dev/sda11 /dev/sda12 /dev/sda13', logStart=1, logStamp=0, logSectors=1,000}
Debug:  TD isValidRoot MfsHeader64 { MfsHeaderHeader { state=0xA790A79, magic=0x0A790A79, CRC=0x0A790A79 }, validCRC=FALSE, sectors=754,645,927,544,294,009, partitions='', logStart=754,645,927,544,294,009, logStamp=175,704,697, logSectors=175,704,697}
Debug:  TD isValidRoot MfsHeader64 { MfsHeaderHeader { state=0x1991F800, magic=0x00000000, CRC=0x19AAF7FF }, validCRC=FALSE, sectors=-6,148,914,691,236,517,206, partitions='��', logStart=-6,148,914,691,236,517,206, logStamp=-1,431,655,766, logSectors=-1,431,655,766}
Debug:  TD isValidRoot MfsHeader64 { MfsHeaderHeader { state=0x0000, magic=0x00000000, CRC=0x00000000 }, validCRC=FALSE, sectors=0, partitions='', logStart=0, logStamp=0, logSectors=0}
Debug:  Added TivoDisk from '/dev/sda'
Roamio:
Code:
512 bytes (512 B) copied, 0.000880768 s, 581 kB/s
00000000  ed fe ba eb 00 00 00 00  da 22 28 57 10 00 00 00  |........."(W....|
00000010  01 00 00 00 40 00 00 00  40 06 00 00 00 c0 28 2f  |....@...@.....(/|
00000020  60 c1 28 2f 2f 64 65 76  2f 73 64 61 31 30 20 2f  |`.(//dev/sda10 /|
00000030  64 65 76 2f 73 64 61 31  31 20 2f 64 65 76 2f 73  |dev/sda11 /dev/s|
00000040  64 61 31 32 20 2f 64 65  76 2f 73 64 61 31 33 00  |da12 /dev/sda13.|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000a0  00 00 00 00 00 00 00 00  00 5c b0 39 00 00 00 00  |.........\.9....|
000000b0  01 00 00 00 00 00 00 00  9d 00 00 00 00 00 00 00  |................|
000000c0  e9 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  61 04 00 00 00 00 00 00  fe ff 18 00 00 00 00 00  |a...............|
000000e0  01 00 00 00 00 00 00 00  00 00 08 00 00 00 00 00  |................|
000000f0  00 00 08 00 00 00 00 00  78 00 00 00 e8 03 00 00  |........x.......|
00000100  20 a1 07 00 65 00 00 00  02 00 00 00 cf 02 00 00  | ...e...........|
00000110  80 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200
...
31fffe00  ed fe ba eb 00 00 00 00  da 22 28 57 10 00 00 00  |........."(W....|
...
I can see it's broken. Do we know what it is suppose to say, before I attempt to fix it?
__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA

Last edited by telemark : 06-13-2014 at 07:17 PM.
telemark is offline   Reply With Quote
Old 06-13-2014, 06:12 PM   #85
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
What is the debug output on a premiere that we know it runs on.

This probably something more ggieseke can address/answer. The checksum calculations to verify the checksum might have to be adjust for the different endian order.

Which version of JMFS are you using for your debugging.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-13-2014, 06:47 PM   #86
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,071
At first glance, the "validCRC=FALSE" lines stands out. TiVo uses an ANSI X366 algorithm to validate several of the MFS headers.

Where are these logs coming from?

I've had several beers right now, but if you post at least the hex dump of that entire sector I may be able to spot what's up tomorrow. I have to work in the morning but my afternoon is free.
ggieseke is offline   Reply With Quote
Old 06-13-2014, 07:05 PM   #87
telemark
Registered User
 
Join Date: Nov 2013
Posts: 859
Quote:
Originally Posted by jmbach View Post
What is the debug output on a premiere that we know it runs on.

This probably something more ggieseke can address/answer. The checksum calculations to verify the checksum might have to be adjust for the different endian order.

Which version of JMFS are you using for your debugging.
Uh, I guess that's the absurd part. It didn't run on my Premiere backups consistently, unless I make some patches first, so I guess I have to track that down first. Maybe this needs to start from a different codebase.

The version I'm using, I forked what was on github krbaker, and then added Litle Endian reading modes in 3 ways / reading pathways. Then added some debug lines, and forced it into 64-bit mode since it defaults to 32bit on misreads.

What looks wrong to me, is why is State EBBAFEED, Magic should be EBBAFEED. But it does that on some Premiere too.

Edit: moved Roamio first sector dump to prior post. Will add Premiere version here for comparison.
I found an drive that works with mediafire and krbarker, but not mine, so I have something to track down.

Looking at them side-by-side. Looks to me the magic number is 64bits long, which makes me question if there's a value called "state".

Premiere:
Code:
Fine:  storageSize=320072933376, nextFreeBlock=625142448, free=0
Fine:  TD isValidRoot MfsHeader64 { MfsHeaderHeader { state=0x0000, magic=0xEBBAFEED, CRC=0x9A6F5744 }, validCRC=TRUE, sectors=616,457,216, partitions='/dev/sda10 /dev/sda11 /dev/sda12 /dev/sda13', logStart=1, logStamp=6,883,168, logSectors=1,000}
Fine:  Found boot TivoDisk at '/dev/sda'. Boot device is '/dev/sda'

# dd if=/dev/disk1s10 count=1 bs=512 | hexdump -C  
1+0 records in
1+0 records out
512 bytes transferred in 0.022048 secs (23222 bytes/sec)
00000000  00 00 00 00 eb ba fe ed  9a 6f 57 44 00 00 00 10  |.........oWD....|
00000010  00 00 00 01 00 00 00 40  00 00 06 40 00 00 00 01  |.......@...@....|
00000020  00 00 00 01 2f 64 65 76  2f 73 64 61 31 30 20 2f  |..../dev/sda10 /|
00000030  64 65 76 2f 73 64 61 31  31 20 2f 64 65 76 2f 73  |dev/sda11 /dev/s|
00000040  64 61 31 32 20 2f 64 65  76 2f 73 64 61 31 33 00  |da12 /dev/sda13.|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 24 be 64 00  |............$.d.|
000000b0  00 00 00 00 00 00 00 01  00 00 00 00 00 69 07 60  |.............i.`|
000000c0  00 00 00 00 00 00 03 e9  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 04 61  00 00 00 00 00 18 ff fe  |.......a........|
000000e0  00 00 00 00 00 00 00 01  00 00 00 00 00 08 00 00  |................|
000000f0  00 00 00 00 00 08 00 00  00 00 00 78 00 00 03 e8  |...........x....|
00000100  00 07 a1 20 00 0a 2b d4  00 00 00 e8 00 00 00 02  |... ..+.........|
00000110  00 00 00 80 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

__________________
Premiere 2 tuner & SiliconDust
on Comcast CableCard + OTA

Last edited by telemark : 06-13-2014 at 08:46 PM.
telemark is offline   Reply With Quote
Old 06-13-2014, 07:43 PM   #88
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
If you use krbarker version it needs to be patched to handle images larger than 0xEFFFFFFF sectors. I patched mine which is a fork of krbarker but only a few days ago.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-13-2014, 10:04 PM   #89
jmbach
Registered User
 
jmbach's Avatar
 
Join Date: Jan 2009
Posts: 850
If "magic" was an 8 byte entry and not a 4 byte entry, then it makes more sense and I agree there is no "state" value. That is assuming the "state" value of 4 bytes near the traditional "magic" value is what JMFS is considering as the "state" value.

The challenge will be to figure out which entries are 2, 4, and 8 byte entries to correct for the endianness in JMFS. Some of it can be figured out by comparing Roamio and Premiere blocks. If the checksum comes out correctly, then we probably have guessed correctly.
__________________
"Delay is preferable to error" - Thomas Jefferson
"If I have seen further it is by standing on the shoulders of Giants" - Sir Isaac Newton
jmbach is offline   Reply With Quote
Old 06-14-2014, 08:04 AM   #90
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,071
What you're seeing is one of the changes in the superheader structure that I mentioned before. On a Roamio state and magic are still 32 bit fields, but they are swapped (magic comes first). The logstamp and zonemap_type fields are also swapped.
ggieseke is offline   Reply With Quote
Reply
Forum Jump




Thread Tools


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Advertisements

TiVo Community
Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
vBulletin Skins by: Relivo Media

(C) 2013 Magenium Solutions - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not owned or operated by TiVo Inc.
All times are GMT -5. The time now is 11:20 PM.
OUR NETWORK: MyOpenRouter | TechLore | SansaCommunity | RoboCommunity | MediaSmart Home | Explore3DTV | Dijit Community | DVR Playground |