pyTivo - Transcoding server

Discussion in 'TiVo Home Media Features & TiVoToGo' started by armooo, Nov 25, 2006.

  1. n2lovell

    n2lovell New Member

    21
    0
    May 17, 2014

    Advertisements

    If the "programId" (which I think is a tms formatted id if I remember correctly) matches a program in existing guide data on the box. It will reuse that until it gets GC'd out. The only way to get rid of that quickly is to delete the recording(s) referencing them completely then running clear pgd... Otherwise it takes several days.

    So if you transfer a recording with different metadata once and then transfer the same one again but with the same programId it reuses the old metadata. Its possible that in the cases where people are getting incorrect data in their transfers its because something "fun" is going on with coming up with unique fake ids.

    If you set the programId in the above case to EP012599310002. It probably would have come across correctly (if it was a pull). Pushes screw with both the seriesId and programId fields, and the search it does with just the title + epTitle returns two results (so it won't have matched any data).

    The metadata searching isn't based on the programId at the moment because it was too reliant on padding. Its only used for disambiguation after getting results.
    So the minimum set for a search that returns multiple results to bring in metadata for a tv series is title, episode title, and programId (or seriesId if you've previously inserted/found the correct series from another metadata match).
     
  2. jcthorne

    jcthorne Well-Known Member

    3,116
    196
    Jan 28, 2002
    Houston
    Reducing the metadata.txt to the following still results in a push with metadata from the first episode of the series pushed weeks ago, not the data for the current episide being pushed. Exactly the same as with complete metadata file.

    title : Scorpion
    seriesTitle : Scorpion
    episodeTitle : Revenge
    programId : EP019223320011
    seriesId : SH01922332




    BUT, I then tried with the same metadata.txt file but deleted the last line, seriesId, and pushed again.

    This time the file pushed, got the correct episode title and other data. MUCH more metadata. Actors, director, rating etc, just like a tivo recording. Not only that, it grouped on the tivo with other episodes of this series that the tivo recorded and episodes that I pulled last week. Pushes have never grouped with pulls and tivo recordings before.

    This begins to look like a needed change in the pytivo code for pushes, needs to send a very limited set of data and let the mind server look the data up.

    Can someone else verify what I have seen this morning? This is a major step forward. I think.
     
  3. lpwcomp

    lpwcomp Well-Known Member

    9,461
    298
    May 6, 2002
    John's...
    And you know this because...?
     
  4. jcthorne

    jcthorne Well-Known Member

    3,116
    196
    Jan 28, 2002
    Houston
    Perhaps he cannot say.....

    But the info turned out to be correct. Needs more testing but its working for me and its much appreciated. Still to figure out where this leads. Its a shift in how metadata is applied for sure.

    I'll take info from a little birdie if it works. Would not be the first time someone 'inside' dropped in here at TCF and left a nugget of info to get a development issue sorted.
     
  5. bantar

    bantar Member

    130
    9
    Apr 27, 2004

    Advertisements

    This note is a bit long, but it has potentially useful findings. Maybe someone will be able to take this a step further.

    The randomness of which metadata is used appears to be the metadata of the 1st file transferred. In cases where my Tivo is picking up metadata from, let's say, E07, it just so happens that the metadata for E01-E06 had already been cleared/aged out of its memory. With my testing so far, everytime I start a new grouping, the system uses the metadata from the first transfer. From hereafter, I will just call the error "mangled metadata".

    I took another series that was previously transferred with the mangled metadata and decided to pull a few episodes to see what happened. For this test, the metadata files of both the pushes and pulls were the original as generated by metagenereator with the exception that I add [S02Exx] to the title. Two interesting observations from this exercise. 1) Grouping is DIFFERENT. Pulls are in a different group from pushes. 2) Pulls correctly pulled the metadata. 3) "Correctly" is not quite right. Pulls have valid metadata, but it is different from what is stored in my metadata file. The description is different. Also, by accident, I left episode : 101 in 2 different files, but it was ignored. Each episode was correct after the transfer. So, I decided to again push these same 2 files that were successfully pulled. As expected, it went in the previous group and had mangled metadata.

    Using "Tivo remembers some data" as a hypothesis, I set out to see if new series would behave differently.

    I kept my two original test video files, but rewrote the abbreviated metadata file to use older series (Benson) that I know have never been recorded on this device. Then these were pushed from pyTivo. It worked correctly. Pushed the same 2 again. It worked correctly again. Pushed 2 new episode metadatas using full metadata file as generated by metagenerator. Interesting. Abbreviated vs Full Metadata results in DIFFERENT Grouping. Those 2 files with full metadata ended up with the mangled metadata.

    Example metadata file that works for pushes (that Tivo doesn't already remember):
    Code:
    title : Benson
    seriesTitle : Benson
    episodeTitle : Benson in Love
    programId : EP000005270104
    seriesId : SH000527
    
    Example metadata file that fails for pushes:
    However, this works for pulls:
    Code:
    title : Benson
    seriesTitle : Benson
    episodeTitle : Benson in Love
    episodeNumber : 104
    programId : EP000005270104
    originalAirDate : 1979-10-04T00:00:00Z
    description : Benson meets an attrative woman in the mansion and asks her out, not knowing she is a senator.
    isEpisode : true
    seriesId : SH000527
    vSeriesGenre : Comedy
    vPrimaryGenre : Comedies
    vWriter : Colleary|Bob
    vActor : Bird|Billie
    vActor : Fraser|Bob
    vActor : McWilliams|Caroline
    vActor : Conn|Didi
    vActor : Phillips|Ethan
    vActor : Swenson|Inga
    vActor : Noble|James
    vActor : Stadlen|Lewis J.
    vActor : Gold|Missy
    vActor : Auberjonois|Rene
    vActor : Guillaume|Robert
    
    Continuing with the memory theory, I cleared the program data to see if it would help. The steps for this were: Deleted existing subject TV series recordings. Permanently deleted the files from Recently Deleted. Cleared Program Data. Pushed files using abbreviated metadata file format. Transferred 4 episodes each of 3 series. Of the 12, 10 failed and 2 in one series were correct. I can't explain this inconsistent behavior. It would appear that this is not a solution.


    Possible facts:
    1) The same series can cause at least 3 different groupings. 1) Pushes with abbreviated Metadata, 2) Pushes with full (metagenerator) metadata files and 3) Pulls with either metadata (abbreviated or full)

    2) Pulls have valid metadata, but replace the metadata with its own version. (Not sure this is universally true). It does keep Title changes, such as the addition of [S0XE0Y].

    3) This new metadata mangling problem is tied to Tivo's memory of program data. Clearing the program data does not appear to fix this issue.

    4) The memory issue is attached to a particular grouping. It's not related to the episode in anyway. Transferring new episode for a remembered group fails the same as transferring a previously transferred episode.

    5) Pulls seem to ignore some extra data in episode titles and still work. For instance, both formats of title are fine: "Birthdates [S01E03]" or "Birthdates". It replaces the title with its own version.

    6) Push works today with the following caveats: 1) Must be a series not already remembered by Tivo, 2) Must use abbreviated metadata file containing: title, seriesTitle, episodeTitle, programId, seriesId. You can push as many as you want using this format.

    7) For existing series that are now mangled, you must now use pulls if you want correct metadata.
     
  6. lpwcomp

    lpwcomp Well-Known Member

    9,461
    298
    May 6, 2002
    John's...
    Ok, I did some testing and n2lovel appears to be correct as to how it works now. I got correct metadata even when I had no programId as long as the seriesTitle and episodeTitle were correct. He is also correct in saying that once bad data gets associated with a programId, there is no easy way to fix it.

    This has revealed an issue with both Metagenerator and pyTivoMetaThis (which is what I usually use). They both get episodeTtitle from theTVDB. The problem is that this is not always the same as the one that TMS has, and it must be in order for the metadata to be correct.
     
  7. n2lovell

    n2lovell New Member

    21
    0
    May 17, 2014
    Hrmm, if the seriesId is set then it isn't properly discarding the mangled series information in all cases. This accounts for why its not linking the found metadata back to the same folder as a normal "finding" in all cases (i.e. its only working the first time).

    Ah, and that actually accounts for the reason for episodeTitles not changing.

    Note: Dropping the seriesId normally is a bad idea for a push though because even if you don't set them, they get sent down. Thus only matches that return a single result work (i.e. shows with Spanish dubs would have issues foldering).

    This is only a problem for push and not pull.
     
  8. HerronScott

    HerronScott Well-Known Member

    7,535
    1,102
    Jan 1, 2002
    Staunton, VA
    Sorry I've been skimming these (plus never really looked very hard at the metadata before), but could this change have anything to do with the issue people have been reporting when transferring folders that all the shows get the same information as the first show transferred?

    Scott
     
  9. n2lovell

    n2lovell New Member

    21
    0
    May 17, 2014
    For push - yes. Only way around it for now is to drop the seriesId (you can keep the programId). Although it might not folder properly that way.
     
  10. lpwcomp

    lpwcomp Well-Known Member

    9,461
    298
    May 6, 2002
    John's...
    Or not queuing up multiple pushes of the same series. IOW, wait for the transfer of one episode to complete before queuing up the next one. All things considered, it would probably be easier to do pulls until this problem is fixed.

    Not sure what happens if you alternate series so that you are never have two episodes of the same series queued up back to back.
     
  11. bantar

    bantar Member

    130
    9
    Apr 27, 2004
    While you might be able to drop the series id for new series that the tivo doesn't have info cached, this will not fix a series that you have already broken. For broken ones, you MUST use pulls to resolve.

    I had my Tivo pulling for hours today after deleting the corrupted files. Every pull worked just fine with no need to alter the metadata.
     
  12. jcthorne

    jcthorne Well-Known Member

    3,116
    196
    Jan 28, 2002
    Houston
    I was able to push a number of series that previously resulted in mangled metadata by using the abbreviated metadata dropping seriesId but keeping programId. They pushed correctly but grouped with other episodes that had been pulled. Pushes grouping with pulls and tivo recorded items has never worked before.

    Seems there is a bug with the seriesId though as that should not cause the metadata to be incorrect as long as the seriesId is correct.

    This could also be why TivoDesktop is not pushing correctly.

    Seems a bug on the tivo side, perhaps on the mind server.
     
  13. email68

    email68 New Member

    25
    0
    Dec 8, 2014
    New user trying to get pyTivo running on my WHS 1.0 box.

    I followed the instructions from the pyTivo wiki. My pyTivo.conf file looks like this:

    So far I can pull shows from the WHS via the MyShows->Devices menu on Tivo. But when I try and pull a show from Tivo, I get the Unable to Connect error.

    The correct IP address is shown, and the correct name of my Tivo is shown. I have a Tivo Roamio Plus.

    I don't know how to create or save the log file, let me know and I'll post it.
     
  14. HerronScott

    HerronScott Well-Known Member

    7,535
    1,102
    Jan 1, 2002
    Staunton, VA
    I have had this working on my WHS 1.0 server for a couple of years now and it works great. I do recall having to make exceptions in the Windows Firewall to support it.

    pyTivo
    Port 9032
    TCP

    Zeroconf for pyTivo
    Port 5353
    UDP

    Scott
     
  15. caddyroger

    caddyroger New Member

    1,730
    1
    Mar 14, 2005
    Some where...
    In the tivo box it has Caddy and caddy 2 listed but I only have 1 tivo named Caddy. When I install kmttg it will have 2 tivos listed No 1 is caddy with Ip address of 192.168.1.15 which is correct. I am guessing that Caddy2 is using the 192.168.1.99 as the streaming ip.
    Auto push is showing Caddy and caddy 2 and defaulting to Caddy 2. Auto push does work not unless I go to the tivo and click on Caddy.
    How can get auto push to only show Caddy in the tivo box?
    I have a screen shot of router connection ip.
     

    Attached Files:

  16. moyekj

    moyekj Well-Known Member

    12,213
    862
    Jan 23, 2006
    Mission...
    If you update your kmttg installation it won't see the 2nd bogus Caddy (2) which is the internal Stream.
     
  17. caddyroger

    caddyroger New Member

    1,730
    1
    Mar 14, 2005
    Some where...
    I am very sorry but this is the pytivo auto push software.
     
  18. bantar

    bantar Member

    130
    9
    Apr 27, 2004
    To be sure, I repeated my test by removing just the seriesId from the metadata file and pushed 4 test files. They failed as before all taking the metadata of the first one pushed. Not sure why we might be seeing different results. However, another tell is that mine did not group with the pulled ones.
     
  19. jcthorne

    jcthorne Well-Known Member

    3,116
    196
    Jan 28, 2002
    Houston
    Mine that failed at first and then worked with minimal metadata only had 4 lines in the metadata.txt file:

    title : Scorpion
    seriesTitle : Scorpion
    episodeTitle : Revenge
    programId : EP019223320011

    I pushed several like this, they got metadata from the mind server and grouped with others in the Scorpion series. Before trying with the minimal meatadata, they did just as yours, all taking the metadata of the oldest push on the tivo for that series. I also have a couple episodes in that group that were pulled. The only difference I can see between the pulls and pushes now is that the pushes are copy protected.
     
  20. overFEDEXed

    overFEDEXed Member

    282
    7
    Nov 11, 2002
    Centerville,...
    I posted in that thread and have an odd fix FOR TIVO DESKTOP ONLY.

    I know this is a pyTivo thread with this problem, but wanted to ask n2lovell or others a question. Why if I drop my video files in an older folder on my PC, they push with the correct title? With my new folders, I get all of the same names of course. These older, working folders were created before the Nov. updates, but I don't know if it matters or not.

    This works for what I need, but I'm just curious.

    Thanks
     

Share This Page

spam firewall

Advertisements