Storage requirements seem suddenly high--anyone else?

1778 Views 16 Replies 6 Participants Last post by moyekj, Oct 24, 2007

mikeyts

Discussion Starter · Oct 13, 2007

I'm seeing some marginally high numbers for size of recordings on my S3 lately. CBS programming in particular seems high, in some cases higher than theoretically possible (the highest bitrate that can theoretically be acheived in 6 MHz 8 VSB ATSC channel is 19.39 Mbps, or 8.12 GB; I'm seeing as high as 8.19 GB for an hour (19.54 Mbps).

None of the other channels seems unreasonable, but the aggregate of KPBS-HD and V-Me (multicast by KBPS) has been up to 20% higher than possible, as recorded from North San Diego County Cox Cable.

Can anyone think of an explanation for this? I'm running 9.1 for the past week and I hate to think that they did something to make the unit waste storage.

1 - 17 of 17 Posts

bkdtv

· #2 · Oct 13, 2007

mikeyts,

You've got to be careful about the assumptions you make. It you calculate Mbps using the method you suggested in the bitrate result thread, that will not yield correct results.

As you noted in that thread, data transfer and data storage rates are defined differently, with the former based on 1000 and the latter 1024. However, we don't know what definition Tivo is using to report file sizes. Your numbers above assume they are actually reporting file sizes using 2^10.

It sounds like Cox cable is now passing the full bitstream from your CBS affiliate. Evidently, they were not before. KPIX is a CBS O&O affiliate, and like other CBS owned affiliates, they distribute their feeds at 18.9 Mbps.

8.12Gb * 8000 / (60 min * 60 s) = 18.04 Mbps ABR
8.19Gb * 8000 / (60 min * 60 s) = 18.20 Mbps ABR

moyekj

· #3 · Oct 13, 2007

Mike, if you look back at the bitrate thread you will notice that my 720p Cox file sizes were higher than their OTA counterparts which also didn't make much sense to me... So I don't think it's apples to apples comparing cable feed to OTA - if nothing else certainly PSIP information is likely different between the 2 feeds. Also remember that the cable companies are muxing in with other channels on the 256 QAM and doing some bitrate shaping, and 256 QAM can fit up to around 38Mbps. So after the cable company manipulates the stream I don't think we can assume the OTA feed is unaltered in any way by Cox.

I suppose another assumption we are making is that Tivo is reporting the correct file size in GB to begin with and Tivo reports total time rounded off to the nearest minute - so more margin of error.

Finally one other thought - the network OTA transmissions usually have sub-channels other than the primary HD so they are not devoting the full 19Mbps to the primary HD channel. The LA KCBSDT feed is accompanied by other sub-channels, don't know about the San Diego feed.

See less See more

bkdtv

· #4 · Oct 13, 2007

moyekj said:
Finally one other thought - the network OTA transmissions usually have sub-channels other than the primary HD so they are not devoting the full 19Mbps to the primary HD channel. The LA KCBSDT feed is accompanied by other sub-channels, don't know about the San Diego feed.

Most CBS owned and operated affiliates don't run subchannels. They reserve a few tenths of a megabit for datacasting and the rest goes to their HD feed. Your KCBS affiliate may be one of the exceptions.

I use an off-air antenna in part to access the CBS O&O affiliate from an adjacent market. Picture quality is a noticeable improvement over my local affiliate.

mikeyts

Discussion Starter · #5 · Oct 13, 2007

This is a sudden change and a very large one--I've been spot checking average bit rate estimates since I bought the TiVo in January and the reading that I've been getting on CBS have gone up 18% since then (from 16+ Mbps assuming size reported in 2-to-the-30th-byte units to 19+ Mbps calculated with the same assumptions). The difference between Mbps calculated on the same file size assuming 10-to-the-9th-byte units and the the number calculated using 2-to-the-30th-byte units should be that the latter will be 7.3% of the former higher.

Even assuming that TiVo is reporting in 10-to-the-9th-byte units, I got an aggregate 4.93GB used to store a particular half hour of KPBS and V-Me which works out to 21.91 Mbps (23.52 Mbps assuming 2-to-the-30th-byte units). 21.91 Mbps is 113% of the maximum 19.39 Mbps.

I asked here yesterday whether anyone knew if TiVo was reporting 10-to-the-9th or 2-to-the-20th GBs on the Program Details dialog (in this thread). morac responded that, using a hack method for extracting file sizes from his Series2 he got sizes in single byte units which, when divided by 2-to-the-30th, corresponded to the numbers reported on the Program Details dialog. I strongly doubt that they'd rewrite this little bit of code for the Series3, but who knows? In any case, it seems as though at least Series2 reports files sizes in power-of-2 GBs, like every other piece of computing equipment I've ever encountered.

See less See more

Hyrax

· #6 · Oct 13, 2007

Mickyts-
Perhaps your cable company decided to stop throttling the CBS broadcast. I can record both cable and OTA broadcasts on my computer and directly compare filesize and bitrates. It is shocking how much the bitrates are for my cable CBS vs the OTA CBS. For example, when I edit out commercials, remove NULL packets from a football game my MPEG editing software tells me that the bitrate from cable is 14.4 Mbps and is 18.2 Mbps from OTA. I've got a 100" screen and the difference between the two sources is painfully obvious.

When I saw that the OTA file sizes were so much bigger than the cable sizes I assumed that the OTA source included subchannels or other streams, but they do not. I've done a frame-by-frame comparison and the cable version has obviously worse picture quality. Typically you'll frequently see players faces become blurred when they move too quickly, or solid colors take on a blocky appearance.

bizzy

· #7 · Oct 13, 2007

Theres no such thing as "throttling". There's no variable faucet on the transport stream. Either you pass it unmolested, or you transcode it.

moyekj

· #8 · Oct 13, 2007

Obsessed at getting this right once and for all I did following experiment:
30 min manual recording (11:30-12:00pm) of KTTVDT from Cox cable (FOX HD).
I setup above recording both on my PC QAM tuner and 1 of my S3 units. Here are the resulting file sizes:
PC: 3523057664 Bytes = 3,440,486 KB = 3,360 MB = 3.28 GB
S3 Info: 3.50 GB

Now that's a large discrepancy indeed. But guess what:
3.50 * 1e9 / 2^30 = 3.26 GB

CONCLUSION: As bkdtv suspected, Tivo reports GB as 1e9 not 2^30

So the correct equation for bitrate calculations based on Tivo numbers is:
Rate (Mbps) = SIZE * 8000 / (MINS*60)
where:
SIZE = GB reported by Tivo Info
MINS = Program length in mins reported by Tivo

See less See more

mikeyts

Discussion Starter · #9 · Oct 13, 2007

If this is true, I think that its a recent modification, possibly in the 9.1 release, which would explain why the increase seems so sudden. I stated before, in another thread morac proved that Series2 is reporting in 2-to-30th-byte units.

HDTiVo

· #10 · Oct 13, 2007

mikeyts said:
This is a sudden change and a very large one--I've been spot checking average bit rate estimates since I bought the TiVo in January and the reading that I've been getting on CBS have gone up 18% since then (from 16+ Mbps assuming size reported in 2-to-the-30th-byte units to 19+ Mbps calculated with the same assumptions). The difference between Mbps calculated on the same file size assuming 10-to-the-9th-byte units and the the number calculated using 2-to-the-30th-byte units should be that the latter will be 7.3% of the former higher.

Right, so 2^30 vs 10^9 is not the whole story, if at all.

I've gotten higher bitrates on CBS lately with noticeable picture improvement. Maybe the cable co is just sending you a better quality product.

moyekj

· #11 · Oct 14, 2007

HDTiVo said:
Right, so 2^30 vs 10^9 is not the whole story, if at all.

Well it means the Tivo numbers are 7.37% inflated in terms of disk size which is not negligible at all.

bkdtv

· #12 · Oct 14, 2007

moyekj said:
Well it means the Tivo numbers are 7.37% inflated in terms of disk size which is not negligible at all.

Of course, you could say much the same thing about the quoted 160Gb and 250Gb capacities. But that's just the way manufacturers market their disk drives.

Hyrax

· #13 · Oct 15, 2007

bizzy said:
Theres no such thing as "throttling". There's no variable faucet on the transport stream. Either you pass it unmolested, or you transcode it.

Picky, picky. Comcast is sending out a low bitrate version of broadcasts. Call it transcoding instead of throttling, if you wish.

A 4 hour recording of Sunday's Cowboys Vs. Patriots game was 29,367,740 KB via Comcast and 38,885,334 KB via OTA. This is the file size as measured on my computer (Windows XP). You can easily see the difference in the picture quality.

I've no idea where the lower bitrate video originates - for all I know Comcast could be getting it directly from the network. It may be possible that networks create these versions specifically at the request of the cable companies. All I know is that they (Comcast) are providing a much lower bitrate version of the football game than I get from my antenna.

I was proposing that the original poster may now be getting the higher bitrate signal from his cable company that I'm getting from my antenna.

Edit: I just noticed that I padded both recordings with an extra 33 minutes (3 at the start, 30 at the end). In case anyone wants to determine the bitrate based on the file size/recording time.

See less See more

mikeyts

Discussion Starter · #14 · Oct 15, 2007

Hyrax said:
I've no idea where the lower bitrate video originates - for all I know Comcast could be getting it directly from the network. It may be possible that networks create these versions specifically at the request of the cable companies. All I know is that they (Comcast) are providing a much lower bitrate version of the football game than I get from my antenna.[/I]

You know, you can probably go after Comcast for this. FCC regs forbid rebroadcast of OTA stations in degraded form. Code of Federal Regulations, Title 47 §76.62(b) states:

(b) Each such television broadcast signal carried shall be carried without material degradation, and, for analog signals, in compliance with technical standards set forth in subpart K of this part.

EDIT: Sorry--didn't get the link to the online posting of the CFR in there before. Fixed.

Hyrax

· #15 · Oct 15, 2007

mikeyts said:
You know, you can probably go after Comcast for this. FCC regs forbid rebroadcast of OTA stations in degraded form. Code of Federal Regulations, Title 47 §76.62(b) states:

Well, that is good news. However, I suspect that Comcast (The Wascally Wabbits), is not the one actually degrading the signal. I cannot imagine what it would take to transcode a high def signal apparently in real time. It could be that they are getting the signal from the network in that form (perhaps there is a loop-hole in the regulation?).

Anyway, I will try getting in touch with my State's Attorney General and try to get this figured out. It really upsets me that HD broadcasts are such low bitrates - seems like false advertising to me.

mikeyts

Discussion Starter · #16 · Oct 24, 2007

Well, at least a part of this puzzle is solved. Getting TiVoToGo today let me transfer a file onto my PC that was marked 7.46GB on its Program Details dialog in TiVo. On disk, the .tivo file was 7,382,529,547 bytes long about .1% less than 7.46 billion. So, S3 would seem to be reporting file sizes in 10^9 byte units and not 2^30. (And for some reason, content takes slightly more space on TiVo than it does when transferred off).

moyekj

· #17 · Oct 24, 2007

mikeyts said:
Well, at least a part of this puzzle is solved. Getting TiVoToGo today let me transfer a file onto my PC that was marked 7.46GB on its Program Details dialog in TiVo. On disk, the .tivo file was 7,382,529,547 bytes long about .1% less than 7.46 billion. So, S3 would seem to be reporting file sizes in 10^9 byte units and not 2^30. (And for some reason, content takes slightly more space on TiVo than it does when transferred off).

Right, this confirms for sure my previous experiment above where bkdtv and I had reached that conclusion. The .1% difference could be a combination of roundoff error and the fact that the .TiVo file on the PC is encrypted in probably a different format than what resides on the Tivo.

1 - 17 of 17 Posts

This is an older thread, you may not receive a response, and could be reviving an old thread. Please consider creating a new thread.