PyTivo Video Manager HME App for pyhme

[lrhorer]

Oh, ho! 'Most illuminating. There are a number of metafiles that have, for example, a vProgramGenre tag, but it is blank! These passed the grep test, but your code caught the error.

Thanks, Jeff.

[lrhorer]

Hey, Jeff, I have another request. For various reasons, I could make use of a parameter in the metafile of each video that details when vidmgr last pushed the video in question to a TiVo. If you decide to implement it, the feature should probably be optional, since many people may not want vidmgr to monkey with their metafiles, but when enabled, it could simply update the metafield with a timestamp that could later be readily sorted in a virtual share in vidmgr.

[jbernardis]

Originally Posted by lrhorer

Hey, Jeff, I have another request. For various reasons, I could make use of a parameter in the metafile of each video that details when vidmgr last pushed the video in question to a TiVo...

Let me think about this. Up until this point I never write out any metafiles. I'm away from home right now with no access to a PC. I'm not even sure if the metadata code I "borrowed" from Bill McBrine has logic to write out a file. If it does, it might not be too bad. The thing I need to guard against is that I already add items to the metadata in memory, and I'm not sure I want to write them out to disk. I'd need a method to prevent this. I'll let you know in a few days after I get back.

[lrhorer]

Surely. As always, I'm available for alpha testing, if you decide to move ahead with the idea.

[jbernardis]

Leslie - I have it working, but I wanted to confer with you on some of the details.

1) There are multiple possibilities for the location of the meta file, including default.txt, and possibly a .meta subdirectory. Here is what I decided on. I NEVER overwrite a default.txt file. If a file named <path>/.meta/<title>.txt exists then I over-write it otherwise if <path>/<title>.txt exists, then I overwrite THAT. If neither exists, then I create a new file - if the .meta directory exists, I create it there, otherwise I create it in the same directory as the video file itself.

2) The format of the timestamp in the meta file. Initially I just had epochtime, but I decided on using the format that other timestamps are using: 2012-08-23T23:42:00Z. Right now I am using GM time (hence the Z) but I could easily use local time.

There is also a new config option - savepushdate - that defaults to false. It needs to be set to true for this logic to be activated.

Let me know what you think. I can finalize it and get it out in a few days.

[lrhorer]

Originally Posted by jbernardis

Leslie - I have it working, but I wanted to confer with you on some of the details.

What a guy!

Originally Posted by jbernardis

1) There are multiple possibilities for the location of the meta file, including default.txt, and possibly a .meta subdirectory. Here is what I decided on. I NEVER overwrite a default.txt file. If a file named <path>/.meta/<title>.txt exists then I over-write it otherwise if <path>/<title>.txt exists, then I overwrite THAT.

'Sounds 100% to me.

Originally Posted by jbernardis

If neither exists, then I create a new file

Well, that works great for me, but will it create an issue for anyone who has a default.txt file and no specific metafiles? Will you then copy the default.txt file to the new file and add the new field? Thinking about it briefly, it seems that would work. Personally, I'm OK with it no matter what, since all my videos have metafiles, but maybe someone else has objections I don't.

Originally Posted by jbernardis

- if the .meta directory exists, I create it there, otherwise I create it in the same directory as the video file itself.

Once again, it sounds perfect, to me.

Originally Posted by jbernardis

2) The format of the timestamp in the meta file. Initially I just had epochtime,

That's what I figured you might do.

Originally Posted by jbernardis

but I decided on using the format that other timestamps are using: 2012-08-23T23:42:00Z.

Yeah, that works, too. As long as sorting by the field produes uniform results, which this does, it's fine. I suppose it doesn't hurt that it is human readable, either.

Originally Posted by jbernardis

Right now I am using GM time (hence the Z) but I could easily use local time.

To my mind, it doesn't really matter.

Originally Posted by jbernardis

There is also a new config option - savepushdate - that defaults to false. It needs to be set to true for this logic to be activated.

Of course. What will be the name of the tag?

Originally Posted by jbernardis

Let me know what you think. I can finalize it and get it out in a few days.

I think it's super.

[jbernardis]

Originally Posted by lrhorer

..., but will it create an issue for anyone who has a default.txt file and no specific metafiles? Will you then copy the default.txt file to the new file and add the new field?

This is exactly what I do - the contents of the default.txt file will be written to the new file along, of course, with the new pushDate tag. The original default.txt file will be unaffected and will remain in force for the other files in that directory.

The problem is that the meta data dictionary that I build is actually a concatenation of default.txt, .meta/default.txt, <title>.txt and .meta/<title>.txt, and I have no representation for which data item came from which file. This creates an issue for this request of yours. With my "solution", the "New" meta file will actually overlay the data that was previously read in from the default file. It produces what you want and I don't think it's much of a compromise

Originally Posted by lrhorer

Of course. What will be the name of the tag?

As mentioned above, the tag name is pushDate.


I am going to email you a zip containing the new files. If you could load it up and see if it works for you before I put it up on git, I'd appreciate it.

[lrhorer]

Originally Posted by jbernardis

I am going to email you a zip containing the new files. If you could load it up and see if it works for you before I put it up on git, I'd appreciate it.

I'm sure it bounced. I have a new email address. You should have received a message from me on the new account.

[lrhorer]

Originally Posted by jbernardis

The problem is that the meta data dictionary that I build is actually a concatenation of default.txt, .meta/default.txt, <title>.txt and .meta/<title>.txt, and I have no representation for which data item came from which file. This creates an issue for this request of yours. With my "solution", the "New" meta file will actually overlay the data that was previously read in from the default file. It produces what you want and I don't think it's much of a compromise

Um, well, hold on. Perhaps I am mis-understanding. Vidmgr does not display the contents of a specific metafile along with the default.txt file when the video is selected, so why would the output be a concatenation? I think I understand what you mean by saying the dictionary does not know the file name whence the data came, but I don't see why the output (or input for that matter) would be a concatenation of default.txt and the other files, if they exist. Creating a new file with whatever text was available from an unspecified source (probably default.txt) is just fine, but outputting the union of default.txt and <video name>.<codec>.txt back to <video name>.<codec>.txt would cause real issues. Surely I misunderstand?

[jbernardis]

It IS a concatenation of sorts. If two or more files have the same tags, then the tags from the latter files will totally replace the tags from the earlier files. If, however, the first file has a unique tag, then that will survive the merge from the successive files. If I then write out a video specific meta file, then that unique tag will show up in the result because I don't record its ultimate source.

[lrhorer]

Jeff or William,

Is there a way for an external process on the server to determine if any HME app is in use at the present time? Over in the Linux thread on the Home Media forum I demonstrated a method to implement logrotate for pyTivo. It is fairly easy to check for an active pyTivo transfer, since such a transfer opens a file for reading or writing. Most of the pyHME apps don't seem to keep any particular file open when active, however, and it doesn't look like HME for Python spawns any easily recognizable threads when one of its apps is active. Of course, one can merely restart HME for python without worrying about whether one of its apps is active or not, but that would rather rudely dump the user out of whatever application he is running without warning.

[jbernardis]

I can't think of anything I'm doing in either of my apps that you could see from the outside. Perhaps from pyhme itself?

[cweb]

I have two shares (Tivo Share & Tivo Share2) on two different hard disks. Everthing shows up the way you'd think in pytivo and vidmgr.

I want to create a virtual share that combines the two shares. I've created a virtual share using the parameters:
[All Shows]
values=all
groupby=seriesTitle
display=file
sort=file

It almost does what I want, but not quite. I like to see empty folders displayed, the real shares show the folder and a zero count. I'd like to see the jpeg images for the folders in the list. Lastly, I would like my mpgs with no metadata.txt file to be grouped in the actual folders they reside in.

I get why I get the behavior I do. What I would like to achieve is a virtual share that displays everything the way the real shares do, but with the list of folders combined. Is that possible?

Thanks in advance.

[jbernardis]

That's a tough one. When I am building the virtual shares, I am working from a list of existing videos. The directory they're in just goes along for the ride - it doesn't drive the process. Even if I organized them by their directory (which I probably could relatively easily) there'd be nothing to point me to empty directories.

I'm not saying no - I just need to think about how it could be done. I'll let you know.

[cweb]

Thanks for thinking about. I was hoping there was something already implemented that I was missing or not understanding.

[lrhorer]

Why do you have shares on two different drives? The answer to that may provide the best means of handling your situation.

In the mean time, there are a number of possibilities, depending to some extent on the OS. For example, one solution might be to create a parent directory, share that directory, and then mount both hard drives in the parent directory.

[cweb]

I have maxed out the first raid. So I added another. The OS is OSX.

Thanks.

[lrhorer]

Originally Posted by cweb

I have maxed out the first raid.

How can you "max out" a RAID system? What level of RAID?

Originally Posted by cweb

So I added another. The OS is OSX.

Hmm, yeah. I don't know anything about RAID under OSX, but I did just a bit of reading, and it sounds like OSX is a rather poor choice for a RAID based server. I did see mention of an OSX version of ZFS, though. That might be a long term solution for you. In the mean time, mounting both drives under a single share, as I mentioned before, might work for you.

[cweb]

Originally Posted by lrhorer

How can you "max out" a RAID system? What level of RAID?

I misspoke. I actually have two Drobos (kind of raid-ish).

Originally Posted by lrhorer

Hmm, yeah. I don't know anything about RAID under OSX, but I did just a bit of reading, and it sounds like OSX is a rather poor choice for a RAID based server. I did see mention of an OSX version of ZFS, though. That might be a long term solution for you. In the mean time, mounting both drives under a single share, as I mentioned before, might work for you.

I'm not sure the single share will work for me. I'll investigate.

I don't need all of the items on my Vidmgr share list (e.g., empty folders showing up). If any were possible, it would be just that much better.

[cweb]

When pushing a show to my Tivo using vidmgr, that I previously downloaded via kmttg, it appears on the Tivo as if it was copyrighted (can only be streamed, not transfer again).

Is this due to vidmgr (or pytivo, kmttg, something else)? Thanks.

[wmcbrine]

Originally Posted by cweb

When pushing a show to my Tivo using vidmgr, that I previously downloaded via kmttg, it appears on the Tivo as if it was copyrighted (can only be streamed, not transfer again).

Is this due to vidmgr (or pytivo, kmttg, something else)? Thanks.

It's due to the push process. TiVo Inc. flags everything transferred this way. It doesn't happen if you pull.

[lrhorer]

Hey Jeff,

How difficult would it be to allow the user to specify cover art for virtual shares? It would be rather nice if the following had cover art:

[jbernardis]

Let me look at it. It should be doable.

[jbernardis]

Originally Posted by cweb

Thanks for thinking about. I was hoping there was something already implemented that I was missing or not understanding.

Cweb - sorry it took so long to get back with an answer to this, but I don't think this will be too easy to do - I think I'm going to have to decline.

The problem is that for physical shares, the root directory is an integral part of the data structure, and for virtual shares, they know nothing about the physical structure of the disk - they're passed one video at a time for consideration for inclusion or exclusion. I just don't see how your request could be incorporated without some major design changes.

[jbernardis]

Originally Posted by lrhorer

Hey Jeff,

How difficult would it be to allow the user to specify cover art for virtual shares? It would be rather nice if the following had cover art:

Leslie,

Adding the logic to display the images is not a great task. However, the difficulty arises when trying to decide WHERE the images need to be located. I don't think it's practical for you to specify a full path name in the ini file - that would be OK for the share itself but a potential nightmare if you needed to add an entry for all the different genre or actors. So I am thinking of the following:

Let's create a directory beneath the vidmgr directory -call it "artwork" or something like that. In there there'd be a jpg file whose basename is the exact name of the share (for example "Browse by Genre.jpg"). If there is no filename, no artwork is displayed. Potentially in there is also a directory named "Browse by Genre", and inside THAT directory would (potentially) be a file named "Action.jpg" or "Comedy.jpg". And a similar thing for the other virtual share types. A structure like this would be at most 3 layers deep with the current virtual share model I have.

The ONLY concern I have with this is that I can't always guarantee that the value of the meta data will form a legal file name. I guess I could add some logic to strip offending characters, but I'm not sure there's an easy OS independent way of doing that. Also, it would have to be a very well defined algorithm so that users would be able to predict what they need to name their files.

I'm still working on it.

[lrhorer]

Originally Posted by jbernardis

Leslie,

Adding the logic to display the images is not a great task. However, the difficulty arises when trying to decide WHERE the images need to be located. I don't think it's practical for you to specify a full path name in the ini file - that would be OK for the share itself but a potential nightmare if you needed to add an entry for all the different genre or actors.

Yeah, that's clear.

Originally Posted by jbernardis

So I am thinking of the following:

Let's create a directory beneath the vidmgr directory -call it "artwork" or something like that. In there there'd be a jpg file whose basename is the exact name of the share (for example "Browse by Genre.jpg"). If there is no filename, no artwork is displayed.

'Works for me.

Originally Posted by jbernardis

Potentially in there is also a directory named "Browse by Genre", and inside THAT directory would (potentially) be a file named "Action.jpg" or "Comedy.jpg". And a similar thing for the other virtual share types. A structure like this would be at most 3 layers deep with the current virtual share model I have.

You lost me, there. I'm certainly not complaining, but why would there be, in effect, "sub-types"?

Originally Posted by jbernardis

The ONLY concern I have with this is that I can't always guarantee that the value of the meta data will form a legal file name. I guess I could add some logic to strip offending characters, but I'm not sure there's an easy OS independent way of doing that.

It seems to me the user should be responsible for not creating illegal file names.

Originally Posted by jbernardis

Also, it would have to be a very well defined algorithm so that users would be able to predict what they need to name their files.

I'm inclined to say you should just put the requirement in the docs and let the users worry about it. It's entirely up to you, of course.

Originally Posted by jbernardis

I'm still working on it.

Many, many thanks!

[jbernardis]

Originally Posted by lrhorer

You lost me, there. I'm certainly not complaining, but why would there be, in effect, "sub-types"?

Ok - here's the idea. Suppose you had a "Browse by Genre" virtual share and you had genre of "Comedy" and "Action". In the main directory you'd have "Browse by Genre.jpg" which would be the artwork for that virtual share. You'd also have a "Browse by Genre" directory, and it would/could contain "Comedy.jpg" and "Action.jpg" and these would be the artwork for those genre. Because some of the virtual shares ALSO allow for a grouping metadata item, this essentially creates another layer of directories. I wasn't complaining about this - I was pointing out that because of this small limit, the job wouldn't be too daunting.

Originally Posted by lrhorer

It seems to me the user should be responsible for not creating illegal file names.

But the issue is that the filenames are NOT created by the user - at least not directly - they are created from the metadata values. For genre, I don't see an issue, but, for example, Actor names could have special characters - umlaut's tildes, etc - that might result in illegal (or at least difficult to deal with) file names.

Maybe you're right - I could just put it on the user. If they have garbage metadata, they might end up with a screwy filename.



I'll proceed on that assumption. I don't think I'm too far away from having something. I've just been distracted lately trying to dig out from the hurricane. No major damage, but we have a house on Long Beach Island - yeah THAT Long Beach Island - that needs some attention. Compared to other people, we were very very lucky, and I'm grateful for that, but it still diverts my focus.

[lrhorer]

Originally Posted by jbernardis

Ok - here's the idea. Suppose you had a "Browse by Genre" virtual share and you had genre of "Comedy" and "Action". In the main directory you'd have "Browse by Genre.jpg" which would be the artwork for that virtual share. You'd also have a "Browse by Genre" directory, and it would/could contain "Comedy.jpg" and "Action.jpg" and these would be the artwork for those genre. Because some of the virtual shares ALSO allow for a grouping metadata item, this essentially creates another layer of directories.

Ah, I see. 'Very thorough of you.

Originally Posted by jbernardis

I wasn't complaining about this - I was pointing out that because of this small limit, the job wouldn't be too daunting.

Oh, I absolutely knew you weren't. I just wanted everyone to know I wasn't, either.

Originally Posted by jbernardis

But the issue is that the filenames are NOT created by the user - at least not directly - they are created from the metadata values.

Oh, I see what you mean, and sometimes those values are created by automated processes, such as Metagenerator or the feed from Tribune Media.

Originally Posted by jbernardis

For genre, I don't see an issue, but, for example, Actor names could have special characters - umlaut's tildes, etc - that might result in illegal (or at least difficult to deal with) file names.

Yeah, I see. perhaps it would be good to have Buildcache complain if it comes across something your code does not handle well, and so skips. Like many people, I am sure, I don't run Buildcache manually on a regular basis, but I certainly would run it if artwork on one of my shares was not showing up, or indeed right after adding any artwork.

Originally Posted by jbernardis

Maybe you're right - I could just put it on the user. If they have garbage metadata, they might end up with a screwy filename.

Such is life. Yours is not devoted to making sure we don't ever make a mistake.

Originally Posted by jbernardis

I'll proceed on that assumption. I don't think I'm too far away from having something. I've just been distracted lately trying to dig out from the hurricane. No major damage, but we have a house on Long Beach Island - yeah THAT Long Beach Island - that needs some attention. Compared to other people, we were very very lucky, and I'm grateful for that, but it still diverts my focus.

Dude, take care of your needs. You're doing this for fun. Anyone who doesn't like it can take a long walk on what's left of a short pier near that house.

[jbernardis]

Thanks for the support, but believe me that when dealing with these kinds of issues, having a nice distraction like working on this software is therapeutic. It takes my mind off of things.

[lrhorer]

Oh, well, there's that, I'm sure. If it keeps your bubble at the top of the glass in the level AND gets us great software, then I'm all for it!