1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pyTivo auto pushes

Discussion in 'TiVo Home Media Features & TiVoToGo' started by moyekj, Oct 18, 2009.

  1. Oct 23, 2009 #21 of 216
    reneg

    reneg Member

    753
    17
    Jun 19, 2002
    There is a clue for in progress transfers in the NPL xml:
    <Details>
    ...
    <InProgress>Yes</InProgress>
    ...
    </Details>

    Queued and completed transfer status are more challenging.

    As an aside, when I checked the NPL XML, there is what appears to be Tivo generated ids for Program ID & Series ID with a "BS" prefix, example:
    <ProgramId>BS15000541</ProgramId>
    <SeriesId>BS271055170</SeriesId>
    It doesn't help anything, but it was interesting to me.

    From what I've seen on the pyTivo side, the push requests are sent to the Tivo as they come in from the script. It's up to the Tivo then to actually initiate the pull of the file. The Tivo is in control.

    As far as incorporating this script into pyTivo, one way could be to run the auto_push script as a thread within pyTivo, however I think this script needs more soak time and maturity before introducing it, and unless you want to start a new fork, you'd want wmcbrine on board.
     
  2. Oct 23, 2009 #22 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    I think building something like this from scratch within pyTivo would be more robust/flexible and I think very easy for someone like wmcbrine to do, however it doesn't seem like there has been enough interest by pyTivo users/and or developers or this probably would have happened shortly after the push capability was implemented. I have no interest in starting another pyTivo fork and trying to implement this in pyTivo itself as there are much better qualified people to do that than me. Perhaps this thread may give a little motivation to the pyTivo developers which I think is mostly wmcbrine these days.
     
  3. Oct 23, 2009 #23 of 216
    spocko

    spocko Member

    452
    2
    Feb 4, 2009
    That makes a lot of sense. It might be good to see if there is any interest from the pyTivo devs before you sink much more work into your implementation. Thanks for your effort.
     
  4. Oct 23, 2009 #24 of 216
    orangeboy

    orangeboy yes, I AM orangeboy!

    4,087
    0
    Apr 19, 2004
    East Moline, IL
    Same sentiments here: thanks!
     
  5. Oct 24, 2009 #25 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    One idea struck me related to auto pushes and tracking. Instead of the clunky way now where all video files under a share are attempted to be pushed unless marked with an empty .processed file, how about instead the program only watches for new video files deposited in the share hierarchy after the auto push program is started?
    i.e. The program takes an inventory of files when it first starts and only tries to push new files not in initial inventory? After a successful push a video file is then added to that inventory. If you then shut down the program and start it up again then of course only new files deposited will be pushed.
    Of course the down side to that approach is if you add files to the hierarchy while program is not running then of course they won't be pushed, but in some cases that actually might be desirable.
    Seems like a cleaner way of handling things to me. Thoughts?
     
  6. Oct 24, 2009 #26 of 216
    reneg

    reneg Member

    753
    17
    Jun 19, 2002
    The current method is simple and works. I'm used to metadata files so the .processed files don't bother me. It could look cleaner with a .processed directory and keep the .processed files there, like you can do with metadata.

    If you do implement a new method, how about only adding files to the inventory after they have been pushed? That would take care of files that were added while the script is not running. The inventory in effect becomes an exclusion list for pushes.

    Totally unrelated, but here is some code I added to the main loop to auto-prune files after a some time. Free free to use/change this or not.
    Code:
                   if not os.path.exists(entry_processed) and isVideo(entry):
                      process(tivoName, share, path, entry)
    
                   # Delete old files
                   if os.path.exists(entry)and not os.path.isdir(entry) and not re.search(".txt$",entry) : 
                     ftime = os.path.getmtime(entry)
                     curtime = time.time()
                     difftime = curtime - ftime
                     if difftime > (86400*6):  # 86400 is 1 day in seconds, 604800 is 7 days in seconds
                       log("removing file: " + entry)
                       os.remove(entry)
    
    
     
  7. Oct 25, 2009 #27 of 216
    ggieseke

    ggieseke Well-Known Member

    4,180
    56
    May 30, 2008
    If you write one file to each share that contains the date/time stamp of the last file that was pushed successfully I think that would work even better. You could do away with the .processed files and it wouldn't really matter if the program was running or not. Just have it check the time file on startup and push everything (updating as you go of course) with a later date/time.
     
  8. Oct 25, 2009 #28 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    That is cleaner but makes it harder to purposefully avoid pushing files especially the first time you setup the auto push. Perhaps if there is a utility to go mark all video files in a share as "pushed" initially that would suffice. In any case the syntax of the tracking file should be simple enough to add exclusions manually, so I think perhaps just the full path name of the file should be enough with an optional 2nd column having a push time stamp just for information purposes?
     
  9. Oct 25, 2009 #29 of 216
    ggieseke

    ggieseke Well-Known Member

    4,180
    56
    May 30, 2008
    I would think that the whole point of setting up an auto-push share is to send everything there to the DVR even if it's being set up for the first time, but that's just how I'd probably use it - YMMV.

    Keeping the entire path and a timestamp in a log file would be ideal but it's probably a lot of work to write the code to read it back and make informed decisions on what to push every time the program runs.
     
  10. Oct 25, 2009 #30 of 216
    reneg

    reneg Member

    753
    17
    Jun 19, 2002
    I concur.
     
  11. Oct 25, 2009 #31 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    No I think it's pretty easy. Only need to read file once when starting up and then store the entries in memory. Any successful pushes simply append to the file. And I would probably make the file paths relative to the top share directory so in case the top share path changes it won't affect anything. Only problem with this type of approach is if the tracking file gets corrupted in some way due to full disk or similar problem there's a (small) chance all entries are lost and then next time everything gets pushed again, but that is probably pretty rare.

    One could get fancier and check entries in the file versus whether they still exist on disk and remove entries whose files no longer exist.
     
  12. Oct 25, 2009 #32 of 216
    tlphipps

    tlphipps Member

    41
    0
    Jul 3, 2002
    Dallas, TX
    I haven't had time to set this up just yet, but I fully intend to. I've wanted this functionality for quite some time. So thanks for all the work so far!

    But even without using it yet, I have a feature request: Would it be possible to add a pre-processing step to allow for auto-running something like videoredo's quickstreamfix? When I pull videos in from my DVDs I generally have to run the QSF to make everything kosher with the Tivo, so automating that as well would be awesome!
     
  13. Oct 26, 2009 #33 of 216
    spocko

    spocko Member

    452
    2
    Feb 4, 2009
    I don't think that would work well. Files with old timestamps could be copied/moved to the auto push folder. I think the tracking method needs to remain per-file, whether that is implemented using .processed files or some other kind of index file. It should be simple and user-editable so the user can easily add/remove exclusions.
     
  14. Oct 26, 2009 #34 of 216
    spocko

    spocko Member

    452
    2
    Feb 4, 2009
    Here's an idea for what I think would be a useful addition to the auto push tool.

    The existing logic can be described as:
    Push all files that are not marked for exclusion.

    I think the inverse logic would also be interesting:
    Push only files that are marked for inclusion.

    This would allow individual files to be explicitly marked for push. In this mode, there would be a list of inclusions rather than a list of exclusions. The mode could potentially be set on a per-share basis, so one share could use an exclusion list, while another could use an inclusion list.

    For example, a "mode" could be added to the watchlist definition:

    Code:
    watchList = [
       {"tivo": "LivingRoom", "path": "c:\\videos\\podcasts", "share": push_all", "mode": "exclude"}
       {"tivo": "LivingRoom", "path": "c:\\videos\\movies", "share": selective_push", "mode": "include"}
    ]
    
    The script currently uses .processed files to track exclusions, so something similar could be used for inclusions. For example, an empty file with a .push extension could mark an inclusion. After the corresponding file is pushed, the .push file could just be deleted.

    The main use that I see for this is with batch processing. Files could selectively be processed and marked for push.

    What do you guys think?
     
  15. Oct 28, 2009 #35 of 216
    reneg

    reneg Member

    753
    17
    Jun 19, 2002
    Fixed a minor bug in the script for Windows in the isVideo routine to enclose the file name in quotes.
    changed
    Code:
          command = "" + ffmpeg + " -i " + file
    
    to
    Code:
          command = "" + ffmpeg + " -i \"" + file + "\""
    
     
  16. Oct 28, 2009 #36 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    Probably should wrap ffmpeg in quotes as well in case it has spaces as well...
     
  17. Oct 28, 2009 #37 of 216
    reneg

    reneg Member

    753
    17
    Jun 19, 2002
    Good point.
     
  18. Oct 28, 2009 #38 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    Added the above fixes with the extra wrapper quotes for the ffmpeg call. Also added checking of push call so that unsuccessful pushes are not marked with .processed file.
     
  19. Oct 30, 2009 #39 of 216
    moyekj

    moyekj Well-Known Member

    11,377
    126
    Jan 23, 2006
    Mission...
    Java version of the program with GUI and ability to run in service mode is now available.

    A big change compared to python script is the tracking of pushed files is now done using a tracking file called auto_push.txt at the top of each video share being monitored with entries for files already pushed (as discussed a little earlier in this thread).

    Documentation, download and source code are now located in a Google project:
    http://code.google.com/p/pytivo-auto-push/
     
  20. Dec 8, 2009 #40 of 216
    Zaphod

    Zaphod Member

    133
    0
    Feb 17, 2003
    Des Moines,...
    If I can get this utility to work I think I'll love it!

    It won't pick up my pyTivo.conf file. I launch the .jar file, I browse to my pyTivo.conf file and it displays that full path in the appropriate field on the screen (my path is D:\Apps\pyTivo\pyTivo.conf). But I don't see any of the shares show up in the lower half (I have 3 shares defined). I click "save" and don't get any errors, but when I close and reopen the auto-push gui, the path to pyTivo.conf file is missing. When I look at the config.ini file, I see the <pyTivo.conf> section but there's nothing in it. I also notice in the log file after I close and reopen the GUI, I see the following error:

    2009_12_07_22:59:57 ERROR: File does not exist:

    Any ideas? Oh, this is on Windows Vista, and I do have pyTivo all working for both pulls and manual pushes.

    Thank you.
     

Share This Page