TiVo Community
TiVo Community
TiVo Community
Go Back   TiVo Community > Main TiVo Forums > TiVo Home Media Features & TiVoToGo
TiVo Community
Reply
Forum Jump
 
Thread Tools
Old 05-01-2013, 01:20 PM   #1
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
C++/Windows TiVoServer and TiVoClient software.

I decided I wanted to write my own transfer programs in C++ to transfer files from the TiVo to my PC and from the PC back to the TiVo.

Transferring from the TiVo to the PC is pretty straightforward.
1. Listen for Beacons to find the tivo servers.
2. Visit the /TiVoConnect?Command=QueryContainer&Container=/ url to get a list of containers.
3. Using the TiVo MAK for http authentication, Parse the appropriate containers.
4. select and transfer a file from the URL in the container, store it as a .TiVo file.
5. optionally use tivodecode and the MAK to convert the .tivo file to a .mpg file.
6. optionally use ffmpeg to convert/transcode the .mpg to a different container format.

I'm having a problem letting the TiVo pull files back from the PC.

I broadcast beacons.
The Tivo connects and gets the "/" container.
I can browse the container under the "/" from my tivo.
I can transfer .tivo files back tot he tivo just fine. (just sending the file via http)
I am using the latest version of ffmpeg for container muxing.
I can transfer files that I'm not transcoding just fine. eg "-vcodec copy -acodec copy"

My problem happens on any file that ffmpeg is transcoding. The tivo resets the TCP connection anywhere from 2 to 5 minutes into the transfer.

I've compared ffmpeg command lines with what pyTiVo is using, and gone so far as to exactly duplicate the command, but I'm still getting the issue.

because the transfer is happening fast than realtime, the video is failing anywhere between 5 and 15 minutes in. On the same video, it does not happen at the same frame in multiple tests.

Can anyone tell me why the tcp connection is getting reset?
wcbonner is offline   Reply With Quote
Old 05-01-2013, 01:44 PM   #2
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,015
I don't think there's quite enough here to diagnose the problem, although I have a couple of ideas. Any chance of seeing the code?
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 05-01-2013, 03:07 PM   #3
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I'm still not happy with the way my code looks, as it has a ton of branches where I'm testing things, plus hardcoded hacks and copied ideas from pyTiVo and elsewhere, but right now, figuring anything out is useful. The problem is caught in the line before where I issue the TerminateProcess() call, which I added to clean up my debugging slightly..

Here's a large chunk of code related to sending the file to the tivo.

Code:
		std::stringstream HttpResponse;
		//HttpResponse << "HTTP/1.1 200 OK\r\n";
		HttpResponse << "HTTP/1.1 206 Partial Content\r\n";
		HttpResponse << "Server: Wims TiVo Server/1.0.0.1\r\n";
		HttpResponse << "Date: " << getTimeRFC1123() << "\r\n";
		HttpResponse << "Transfer-Encoding: chunked\r\n";
		HttpResponse << "Content-Type: video/x-tivo-mpeg\r\n";
		HttpResponse << "Connection: close\r\n";
		HttpResponse << "\r\n";
		send(DataSocket, HttpResponse.str().c_str(), HttpResponse.str().length(),0);

		if (!TiVoFileToSend.GetPathName().Right(5).CompareNoCase(_T(".tivo")))
		{
			int nRet;
			std::ifstream FileToTransfer;
			FileToTransfer.open(CStringA(TiVoFileToSend.GetPathName()).GetString(), ios_base::in | ios_base::binary);
			if (FileToTransfer.good())
			{
				std::cout << "[                   ] Sending File: " << CStringA(TiVoFileToSend.GetPathName()).GetString() << endl;
				std::cout << HttpResponse.str() << endl;
				bool bSoFarSoGood = true;
				long long bytessent = 0;
				char * RAWDataBuff = new char[0x400000];
				std::stringstream ssChunkHeader;
				while (!FileToTransfer.eof() && (bSoFarSoGood))
				{
					FileToTransfer.read(RAWDataBuff, 0x400000);
					int BytesToSendInBuffer = FileToTransfer.gcount();
					int offset = 0;
					ssChunkHeader.str("");
					ssChunkHeader << hex << (BytesToSendInBuffer-offset) << "\r\n";
					send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
					nRet = send(DataSocket, RAWDataBuff+offset, BytesToSendInBuffer-offset, 0);
					bytessent += nRet;
					bSoFarSoGood = nRet == BytesToSendInBuffer;
					ssChunkHeader.str("");
					ssChunkHeader << "\r\n";
					send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
				}
				delete[] RAWDataBuff;
				std::cout << "[                   ] Finished Sending File, bSoFarSoGood=" << boolalpha << bSoFarSoGood << " BytesSent(" << bytessent << ")" << endl;
				ssChunkHeader.str("");
				ssChunkHeader << hex << 0 << "\r\n\r\n";	// 0\r\n is last chunk ending, and \r\n is the trailer.
				send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
			}
		}
		else
		{
			char XMLDataBuff[1024*11] = {0};
			CComPtr<IXmlWriter> pWriter;
			CreateXmlWriter(__uuidof(IXmlWriter), reinterpret_cast<void**>(&pWriter), NULL);
			CComPtr<IStream> spMemoryStream(::SHCreateMemStream(NULL, 0));
			if ((pWriter != NULL) && (spMemoryStream != NULL))
			{
				pWriter->SetProperty(XmlWriterProperty_ConformanceLevel, XmlConformanceLevel_Fragment);
				pWriter->SetOutput(spMemoryStream);
				pWriter->SetProperty(XmlWriterProperty_Indent, FALSE);
				pWriter->WriteStartDocument(XmlStandalone_Omit);
				TiVoFileToSend.GetTvBusEnvelope(pWriter);
				pWriter->Flush();

				// Allocates enough memeory for the xml content.
				STATSTG ssStreamData = {0};
				spMemoryStream->Stat(&ssStreamData, STATFLAG_NONAME);
				SIZE_T cbSize = ssStreamData.cbSize.LowPart;
				if (cbSize >= sizeof(XMLDataBuff))
					cbSize = sizeof(XMLDataBuff)-1;
				// Copies the content from the stream to the buffer.
				LARGE_INTEGER position;
				position.QuadPart = 0;
				spMemoryStream->Seek(position, STREAM_SEEK_SET, NULL);
				ULONG cbRead;
				spMemoryStream->Read(XMLDataBuff, cbSize, &cbRead);
				XMLDataBuff[cbSize] = '\0';
			}
#pragma pack(show)
#pragma pack(2)
#pragma pack(show)
			auto ld = strlen(XMLDataBuff);
			auto chunklen = ld * 2 + 44;
			auto padding = 2048 - chunklen % 1024;
			#define SIZEOF_STREAM_HEADER 16
			struct {
				char           filetype[4];       /* the string 'TiVo' */
				/* all fields are in network byte order */
				unsigned short dummy_0004;
				unsigned short dummy_0006;
				unsigned short dummy_0008;
				unsigned int   mpeg_offset;   /* 0-based offset of MPEG stream */
				unsigned short chunks;        /* Number of metadata chunks */
			} tivo_stream_header;
			ASSERT(sizeof(tivo_stream_header) == SIZEOF_STREAM_HEADER);
			std::string("TiVo").copy(tivo_stream_header.filetype, 4);
			tivo_stream_header.dummy_0004 = htons(4);
			tivo_stream_header.dummy_0006 = htons(13); // mime = video/x-tivo-mpeg so flag is 13. If mime = video/x-tivo-mpeg-ts, flag would be 45
			tivo_stream_header.dummy_0008 = htons(0);
			tivo_stream_header.mpeg_offset = htonl(padding + chunklen);
			tivo_stream_header.chunks = htons(2);
			#define SIZEOF_STREAM_CHUNK 12
			struct {
				unsigned int   chunk_size;    /* Size of chunk */
				unsigned int   data_size;     /* Length of the payload */
				unsigned short id;            /* Chunk ID */
				unsigned short type;          /* Subtype */
				//unsigned char  data[1];       /* Variable length data */
			} tivo_stream_chunk;
			ASSERT(sizeof(tivo_stream_chunk) == SIZEOF_STREAM_CHUNK);
			tivo_stream_chunk.chunk_size = htonl(ld + sizeof(tivo_stream_chunk) + 4);
			tivo_stream_chunk.data_size = htonl(ld);
			tivo_stream_chunk.id = htons(1);
			tivo_stream_chunk.type = htons(0);
			struct {
				unsigned int   chunk_size;    /* Size of chunk */
				unsigned int   data_size;     /* Length of the payload */
				unsigned short id;            /* Chunk ID */
				unsigned short type;          /* Subtype */
				//unsigned char  data[1];       /* Variable length data */
			} tivo_stream_chunk2;
			ASSERT(sizeof(tivo_stream_chunk) == SIZEOF_STREAM_CHUNK);
			tivo_stream_chunk2.chunk_size = htonl(ld + sizeof(tivo_stream_chunk2) + 7);
			tivo_stream_chunk2.data_size = htonl(ld);
			tivo_stream_chunk2.id = htons(2);
			tivo_stream_chunk2.type = htons(0);
			auto TiVoChunkBufferSize = sizeof(tivo_stream_header) + sizeof(tivo_stream_chunk) + strlen(XMLDataBuff) + 4 + sizeof(tivo_stream_chunk2) + strlen(XMLDataBuff) + padding;
			char * TiVoChunkBuffer = new char[TiVoChunkBufferSize];
			char * pTiVoChunkBuffer = TiVoChunkBuffer;
			memcpy(pTiVoChunkBuffer, &tivo_stream_header, sizeof(tivo_stream_header));
			pTiVoChunkBuffer += sizeof(tivo_stream_header);
			memcpy(pTiVoChunkBuffer, &tivo_stream_chunk, sizeof(tivo_stream_chunk));
			pTiVoChunkBuffer += sizeof(tivo_stream_chunk);
			memcpy(pTiVoChunkBuffer, XMLDataBuff, strlen(XMLDataBuff));
			pTiVoChunkBuffer += strlen(XMLDataBuff);
			for (auto index = 0; index < 4; index++)
				*(++pTiVoChunkBuffer) = '\0';
			memcpy(pTiVoChunkBuffer, &tivo_stream_chunk2, sizeof(tivo_stream_chunk2));
			pTiVoChunkBuffer += sizeof(tivo_stream_chunk2);
			memcpy(pTiVoChunkBuffer, XMLDataBuff, strlen(XMLDataBuff));
			pTiVoChunkBuffer += strlen(XMLDataBuff);
			for (auto index = 1; index < padding; index++)
				*(++pTiVoChunkBuffer) = '\0';
			std::stringstream ssChunkHeader;
			ssChunkHeader << hex << TiVoChunkBufferSize << "\r\n";
			send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
			send(DataSocket, TiVoChunkBuffer, TiVoChunkBufferSize, 0);
			delete[] TiVoChunkBuffer;
#pragma pack()
#pragma pack(show)

			// Set the bInheritHandle flag so pipe handles are inherited. 
			SECURITY_ATTRIBUTES saAttr;  
			saAttr.nLength = sizeof(SECURITY_ATTRIBUTES); 
			saAttr.bInheritHandle = TRUE; 
			saAttr.lpSecurityDescriptor = NULL; 

			// Create a pipe for the child process's STDOUT. 
			HANDLE g_hChildStd_OUT_Rd = NULL;
			HANDLE g_hChildStd_OUT_Wr = NULL;
			if ( ! CreatePipe(&g_hChildStd_OUT_Rd, &g_hChildStd_OUT_Wr, &saAttr, 0x80000) ) 
				std::cout << "[" << getTimeISO8601() << "] "  << __FUNCTION__ << "\t ERROR: StdoutRd CreatePipe" << endl;
			else
			{
				// Ensure the read handle to the pipe for STDOUT is not inherited.
				if ( ! SetHandleInformation(g_hChildStd_OUT_Rd, HANDLE_FLAG_INHERIT, 0) )
					std::cout << "[" << getTimeISO8601() << "] "  << __FUNCTION__ << "\t ERROR: Stdout SetHandleInformation" << endl;
				else
				{
					PROCESS_INFORMATION piProcInfo; 
					ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION) );
 
					// Set up members of the STARTUPINFO structure. 
					// This structure specifies the STDIN and STDOUT handles for redirection.
					STARTUPINFO siStartInfo;
					ZeroMemory( &siStartInfo, sizeof(STARTUPINFO) );
					siStartInfo.cb = sizeof(STARTUPINFO); 
					siStartInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
					siStartInfo.hStdInput = GetStdHandle(STD_INPUT_HANDLE);
					siStartInfo.hStdOutput = g_hChildStd_OUT_Wr;
					siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
 
					std::wstringstream ss;
					if (!TiVoFileToSend.GetSourceFormat().Compare(_T("video/mpeg2video")))
						ss << L"ffmpeg.exe -i " << QuoteFileName(TiVoFileToSend.GetPathName()).GetString() << L" -vcodec copy -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec copy -report -f vob -";
					else
						ss << L"ffmpeg.exe -i " << QuoteFileName(TiVoFileToSend.GetPathName()).GetString() << L" -vcodec mpeg2video -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec ac3 -copyts -report -f vob -";
						//ss << L"ffmpeg.exe -i " << QuoteFileName(TiVoFileToSend.GetPathName()).GetString() << L" -vcodec mpeg2video -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec ac3 -copyts -report -f vob -";
					TCHAR szCmdline[1024];
					szCmdline[ss.str().copy(szCmdline, (sizeof(szCmdline)/sizeof(TCHAR))-sizeof(TCHAR))] = _T('\0');
					std::wcout << L"[                   ] CreateProcess: " << szCmdline << std::endl;

					// Create the child process.     
					//TCHAR szCmdline[]=TEXT("child");
					BOOL bSuccess = CreateProcess(NULL, 
						szCmdline,     // command line 
						NULL,          // process security attributes 
						NULL,          // primary thread security attributes 
						TRUE,          // handles are inherited 
						0,             // creation flags 
						NULL,          // use parent's environment 
						NULL,          // use parent's current directory 
						&siStartInfo,  // STARTUPINFO pointer 
						&piProcInfo);  // receives PROCESS_INFORMATION 
   
					// If an error occurs, exit the application. 
					if ( bSuccess ) 
					{
						CloseHandle(g_hChildStd_OUT_Wr);	// If I don't do this, then the parent will never exit!
						long long bytessent = 0;
						char * RAWDataBuff = new char[0x80000];
						CTime ctStart(CTime::GetCurrentTime());
						CTimeSpan ctsTotal = CTime::GetCurrentTime() - ctStart;
						unsigned long long CurrentFileSize = 0;
						for (;;) 
						{ 
							DWORD dwRead = 0;
							BOOL bSuccess = ReadFile(g_hChildStd_OUT_Rd, RAWDataBuff, 0x80000, &dwRead, NULL);
							if (dwRead < 100)
								std::cout << "\n\r[                   ] Small Read Data, Still it should be valid: " << dwRead << std::endl;
							if( (!bSuccess) || (dwRead == 0)) break; 
							if (DataSocket != INVALID_SOCKET) 
							{
								ssChunkHeader.str("");
								ssChunkHeader << hex << "\r\n" << dwRead << "\r\n";
								send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
								int nRet = send(DataSocket, RAWDataBuff, dwRead, 0);
								if (SOCKET_ERROR == nRet)
								{
									TerminateProcess(piProcInfo.hProcess, 0);
									int errCode = WSAGetLastError();
									//std::cout << "[                   ] WSAGetLastError(): " << errCode << std::endl;
									// ..and the human readable error string!!
									// Interesting:  Also retrievable by net helpmsg 10060
									LPTSTR errString = NULL;  // will be allocated and filled by FormatMessage  
									int size = FormatMessage( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, // use windows internal message table
										0,			// 0 since source is internal message table
										errCode,	// this is the error code returned by WSAGetLastError()
													// Could just as well have been an error code from generic
													// Windows errors from GetLastError()
										MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),	// auto-determine language to use
										(LPTSTR)&errString, 
										0,			// min size for buffer
										NULL );		// 0, since getting message from system tables
									std::cout << "\n\r[                   ] Error code: " << errCode << " " << CStringA(errString, size).Trim().GetString() << std::endl;
												// WSAECONNRESET is 10054L
												//[                   ] Error code: 10054 An existing connection was forcibly closed by the remote host.
									LocalFree(errString);										WSASetLastError(0);		// Reset this so that subsequent calls may be accurate
									break;
								}
								bytessent += nRet;
								if (nRet != dwRead)
								{
									std::cout << "\n\r[                   ] Not all Read Data was Sent. Read: " << dwRead << " Send: " << nRet << std::endl;
									char * ptrData = RAWDataBuff + nRet;
									int DataToSend = dwRead - nRet;
									while ((DataSocket != INVALID_SOCKET) && (DataToSend > 0) && (SOCKET_ERROR != nRet))
									{
										nRet = send(DataSocket, ptrData, DataToSend, 0);
										ptrData += nRet;
										DataToSend -= nRet;
									}
								}
								CurrentFileSize += nRet;
								ctsTotal = CTime::GetCurrentTime() - ctStart;
								// This is another experiment trying to find out why I'm failing to send files to the TiVo
								// I was initially going to use select() on the socket, but later decided that the ioctlsocket() call mould be simpler.
								// 								u_long iMode = 0;
								if (SOCKET_ERROR != ioctlsocket(DataSocket, FIONREAD, &iMode))
									if (iMode > 0)
									{
										char *JunkBuffer = new char[iMode+1];
										recv(DataSocket, JunkBuffer, iMode, 0);
										JunkBuffer[iMode] = '\0';
										std::cout << "\n\r[                   ] Unexpected Stuff came from TiVo: " << JunkBuffer << std::endl;
										delete[] JunkBuffer;
									}
							}
						} 
						ssChunkHeader.str("");
						ssChunkHeader << hex << "\r\n" << 0 << "\r\n\r\n";	// \r\n ends previous chunk, 0\r\n is last chunk ending, and \r\n is the trailer.
						send(DataSocket, ssChunkHeader.str().c_str(), ssChunkHeader.str().length(), 0);
						delete[] RAWDataBuff;
						// Close handles to the child process and its primary thread.
						// Some applications might keep these handles to monitor the status
						// of the child process, for example. 
						CloseHandle(piProcInfo.hProcess);
						CloseHandle(piProcInfo.hThread);
						auto TotalSeconds = ctsTotal.GetTotalSeconds();
						if (TotalSeconds > 0)
							std::cout << "[" << getTimeISO8601() << "] Finished Sending File, BytesSent(" << bytessent << ")" << " Speed: " << (CurrentFileSize / TotalSeconds) << " B/s, " << CStringA(ctsTotal.Format(_T("%H:%M:%S"))).GetString() << std::endl;
						else
							std::cout << "[" << getTimeISO8601() << "] Finished Sending File, BytesSent(" << bytessent << ")" << std::endl;
					}
				}
				CloseHandle(g_hChildStd_OUT_Rd);
			}

wcbonner is offline   Reply With Quote
Old 05-01-2013, 11:11 PM   #4
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
Here's an example output to the console of the previous set of code running. I've got the socket error 10054 with the other end getting reset, but I haven't figured out why.

Code:
[                   ] CreateProcess: ffmpeg.exe -i \\Acid\TiVo\Hannibal.S01E04.720p.HDTV.x264-MoTv.mkv -vcodec mpeg2video -b:v 16384k -maxrate 30000k
-bufsize 4096k -ab 448k -ar 48000 -acodec ac3 -copyts -report -f vob -
ffmpeg started on 2013-05-01 at 20:52:27
Report written to "ffmpeg-20130501-205227.log"
ffmpeg version N-52458-gaa96439 Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr 24 2013 22:24:12 with gcc 4.8.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnu
tls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --en
able-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264
 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 27.101 / 52. 27.101
  libavcodec     55.  6.100 / 55.  6.100
  libavformat    55.  3.100 / 55.  3.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 60.101 /  3. 60.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Input #0, matroska,webm, from '\\Acid\TiVo\Hannibal.S01E04.720p.HDTV.x264-MoTv.mkv':
  Metadata:
    creation_time   : 2013-05-01 21:31:40
  Duration: 00:40:48.12, start: 0.000000, bitrate: 2315 kb/s
    Stream #0:0(eng): Video: h264 (High), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Stream #0:1(eng): Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s (default)
[graph 0 input from stream 0:0 @ 00000000023494c0] w:1280 h:720 pixfmt:yuv420p tb:1/1000 fr:25/1 sar:1/1 sws_param:flags=2
[graph 1 input from stream 0:1 @ 00000000042ed6c0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
Output #0, vob, to 'pipe:':
  Metadata:
    encoder         : Lavf55.3.100
    Stream #0:0(eng): Video: mpeg2video, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 16384 kb/s, 90k tbn, 25 tbc (default)
    Stream #0:1(eng): Audio: ac3, 48000 Hz, stereo, fltp, 448 kb/s (default)
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg2video)
  Stream #0:1 -> #0:1 (ac3 -> ac3)
Press [q] to stop, [?] for help
frame=33643 fps= 53 q=2.0 size=  820454kB time=00:22:25.95 bitrate=4993.6kbits/s
[                   ] Error code: 10054 An existing connection was forcibly closed by the remote host.
[2013-05-02T04:03:07] Finished Sending File, BytesSent(839516160) Speed: 1315856 B/s, 00:10:38

wcbonner is offline   Reply With Quote
Old 05-02-2013, 09:28 AM   #5
Iluvatar
Registered User
 
Iluvatar's Avatar
 
Join Date: Jul 2006
Posts: 377
have you taken the TiVo out of the test and just transcoded with FFmpeg to a local file? That could help you narrow it down. The TiVo may be resetting the connection because it did not agree with the data being sent to it or the TiVo is expecting a different file size (among others). There are lots of things that could be going on.

Have you run it under GDB or some debugger?
__________________
My
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
- Read link for changes

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
and
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
for OS X
Iluvatar is offline   Reply With Quote
Old 05-02-2013, 12:29 PM   #6
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
@Iluvatar: Thanks for the suggestion. It provided some interesting results.

I manually converted a file with ffmpeg from it's h264/mp4 file format to an mpeg2/mkv format that ffmpeg would still process but with the codec-copy settings. That mkv file transferred properly.

Creating the mkv file was done with the same options as I was using for failed transfers EXCEPT that instead of writing to stdout from ffmpeg, I was writing to the mkv file.

I'm sending the files to the TiVo using http chunked encoding. I'm not enforcing any particular rules related to the chunk sizing. I think it's time that I introduce some debug information on the number of chunks sent, and perhaps the average chunk size.

Standalone ffmpeg command:
Code:
ffmpeg.exe -i "\\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mp4" -vcodec mpeg2video -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec ac3 -copyts -report -f vob \\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mkv
Successful Transfer:
Code:
[                   ] CreateProcess: ffmpeg.exe -i \\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mkv -vcodec copy -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec copy -report -f vob -
ffmpeg started on 2013-05-02 at 10:06:41
Report written to "ffmpeg-20130502-100641.log"
ffmpeg version N-52458-gaa96439 Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr 24 2013 22:24:12 with gcc 4.8.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnu
tls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --en
able-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264
 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 27.101 / 52. 27.101
  libavcodec     55.  6.100 / 55.  6.100
  libavformat    55.  3.100 / 55.  3.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 60.101 /  3. 60.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
[mpeg @ 00000000023bb480] max_analyze_duration 5000000 reached at 5005000 microseconds
Input #0, mpeg, from '\\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mkv':
  Duration: 00:21:29.95, start: 0.528033, bitrate: 3975 kb/s
    Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p, 720x404 [SAR 1:1 DAR 180:101], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x80]: Audio: ac3, 48000 Hz, stereo, fltp, 448 kb/s
Output #0, vob, to 'pipe:':
  Metadata:
    encoder         : Lavf55.3.100
    Stream #0:0: Video: mpeg2video, yuv420p, 720x404 [SAR 1:1 DAR 180:101], q=2-31, 16384 kb/s, 29.97 fps, 90k tbn, 29.97 tbc
    Stream #0:1: Audio: ac3, 48000 Hz, stereo, 448 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
No more output streams to write to, finishing.e=00:21:28.55 bitrate=3974.0kbits/s
frame=38658 fps= 99 q=-1.0 Lsize=  626022kB time=00:21:29.98 bitrate=3975.5kbits/s
video:547315kB audio:70546kB subtitle:0 global headers:0kB muxing overhead 1.320893%

[                   ] Small Read Data, Still it should be valid: 0
[2013-05-02T17:13:14] Finished Sending File, BytesSent(641046528) Speed: 1627021 B/s, 00:06:34
Unsuccessful Transfer:
Code:
[                   ] CreateProcess: ffmpeg.exe -i \\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mp4 -vcodec mpeg2video -b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -acodec ac3 -copyts -report -f vob -
ffmpeg started on 2013-05-02 at 10:21:23
Report written to "ffmpeg-20130502-102123.log"
ffmpeg version N-52458-gaa96439 Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr 24 2013 22:24:12 with gcc 4.8.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnu
tls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --en
able-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264
 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 27.101 / 52. 27.101
  libavcodec     55.  6.100 / 55.  6.100
  libavformat    55.  3.100 / 55.  3.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 60.101 /  3. 60.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '\\Acid\TiVo\The.Daily.Show.2013.04.29.Jon.Hamm.HDTV.x264-2HD.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isom
    creation_time   : 2013-04-30 04:44:32
  Duration: 00:21:29.98, start: 0.000000, bitrate: 987 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 720x404, 855 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc
    Metadata:
      creation_time   : 2013-04-30 04:37:34
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s
    Metadata:
      creation_time   : 2013-04-30 04:44:33
      handler_name    : GPAC ISO Audio Handler
[graph 0 input from stream 0:0 @ 0000000000339720] w:720 h:404 pixfmt:yuv420p tb:1/30000 fr:30000/1001 sar:0/1 sws_param:flags=2
[graph 1 input from stream 0:1 @ 000000000223ec00] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
Output #0, vob, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isom
    encoder         : Lavf55.3.100
    Stream #0:0(und): Video: mpeg2video, yuv420p, 720x404, q=2-31, 16384 kb/s, 90k tbn, 29.97 tbc
    Metadata:
      creation_time   : 2013-04-30 04:37:34
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: ac3, 48000 Hz, stereo, fltp, 448 kb/s
    Metadata:
      creation_time   : 2013-04-30 04:44:33
      handler_name    : GPAC ISO Audio Handler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg2video)
  Stream #0:1 -> #0:1 (aac -> ac3)
Press [q] to stop, [?] for help
frame=10815 fps= 92 q=2.0 size=  186622kB time=00:06:01.14 bitrate=4233.2kbits/s
[                   ] Error code: 10054 An existing connection was forcibly closed by the remote host.
[2013-05-02T17:23:22] Finished Sending File, BytesSent(191043584) Speed: 1619013 B/s, 00:01:58

wcbonner is offline   Reply With Quote
Old 05-02-2013, 01:28 PM   #7
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
Now I'm looking for ideas on how to further diagnose the issue. I added a chunk counter, and variables to monitor the minimum and maximum size of the chunks. None of the values I got jumped out as suspicious.

Unsuccessful Transfer:
Code:
[                   ] Error code: 10054 An existing connection was forcibly closed by the remote host.
[2013-05-02T18:08:53] Finished Sending File, BytesSent(190560256) Speed: 1549270 B/s, 00:02:03
[                   ] ChunkCount: 383 AvgChunkSize: 497546
[                   ] MaxChunkSize: 524288 MinChunkSize: 2048
Successful Transfer:
Code:
[2013-05-02T18:16:59] Finished Sending File, BytesSent(641046528) Speed: 1387546 B/s, 00:07:42
[                   ] ChunkCount: 1231 AvgChunkSize: 520752
[                   ] MaxChunkSize: 524288 MinChunkSize: 2048

wcbonner is offline   Reply With Quote
Old 05-02-2013, 02:25 PM   #8
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I realized that I was limiting my pipe size with ffmpeg to 0x80000 and that was the upper limit of my chunk size because of that. I increased my pipe buffer size, and also increased the read/write buffer. While that has significantly changed the maximum chunk sizes, it has not otherwise helped my underlying problem. It did provide interesting results on the CPU meter though, in that the larger buffers cause the local CPU meter to spike much more than the smaller buffer.
wcbonner is offline   Reply With Quote
Old 05-02-2013, 07:45 PM   #9
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,015
One thing I notice is that you're not sending the closing "\r\n" for a block until you're ready to send the length for the next one. I'd send it immediately after the data for the block. The TiVo has some... unusual sensitivities to socket timing issues, which gave me quite a headache once.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 05-02-2013, 07:53 PM   #10
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I'll try moving the end-of-chunk \r\n later on tonight. Do you have any idea if seperate socket send() calls cause problems, or may get consolidated by the tcp stack? I know it shouldn't matter, but I also have spent enough time staring at network packets in the past to know it might matter. I just don't want to have to build a consolidated buffer if I don't have to.
wcbonner is offline   Reply With Quote
Old 05-03-2013, 01:10 PM   #11
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I've changed the normal chunk handling to send the header, payload, and footer in three consecutive send() calls, but it has not changed the behavior.

I even went so far as to change the chunk handling in an ugly hack that consolidates each header/payload/footer into a single buffer with a single call for send() and it didn't fix my problem.
wcbonner is offline   Reply With Quote
Old 05-03-2013, 05:48 PM   #12
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
As an experiment, I made all chunks 2048 bytes long and set the socket option to TCP_NODELAY in an attempt to make the underlying IP packets as consistent as possible. This was all after I had consolidated the buffer so that the chunk header/payload/footer were all in a single send() function call.

I've been looking at the packet stream with "Microsoft Network Monitor 3.4" http://www.microsoft.com/en-us/downl...s.aspx?id=4865 which may not be the best packet monitor but I've used in the past.

Most of the outgoing IP packets have a length of 2107, which works out to be a single packet per http chunk, but there are some packets that were obviously consolidated with sizes such as 1500 or 2948.

I've seen some "Dup Ack" messages scattered through the stream, but in general the data stream seems to be operating as it should.

At the very end of the transmission, it seems I get two packets from the tivo, the first looks like a normal ACK, while the second has the same SequenceNumber and AcknowledgementNumber, but the Checksum, TimeStamp, are different, and it includes the reset flag.

This is way too deep into the internals of ethernet for my mind to keep straight. I'm just looking for possible settings that I should or should not use at the high level socket interface.
wcbonner is offline   Reply With Quote
Old 05-05-2013, 06:03 PM   #13
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
A question related to the fake tivo file wrapper that pyTivo includes may point me in the right direction or it may be totally useless.

I used my tivo client program to download the same file from both my server and from pytivo. (I couldn't got it to download from tivo desktop because my program doesn't deal with the tivo https encryption properly.)

When I looked at the raw file in a binary editor, the transferred pytivo file started with 3000\r\nTiVo while my server file simply started with TiVo.

I notice further things that look like chunk encoding after the first chunk, and I'm wondering if there are extra chunk entries I'm supposed to be adding that the tivo itself expects.

I'm using the exact same code in my client to transfer the files, the Microsoft Foundation Class CHttpFile, which I assume takes care of coalescing the chunks correctly.

I dumped the http headers from both pytivo and my server, and they were identical, except that pytivo declared it was transferring using http protocol 1.0 and I was declaring protocol 1.1. When I switched my server to declare 1.0, I got similar chunk information inside my retrieved file. It made no difference in transferring to the actual TiVo. Still, I was wondering if declaring the http version as 1.0 was required by some versions of tivo?
wcbonner is offline   Reply With Quote
Old 05-05-2013, 06:34 PM   #14
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,015
Chunked encoding is an HTTP 1.1 feature. Theoretically, pyTivo should be using HTTP 1.1, but the TiVo doesn't care. Apparently CHttpFile does.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 05-06-2013, 07:35 PM   #15
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I de-chunked the file I transferred from pyTiVo and compared it to the file I got from My Server. The tivo header with the XML was different, owing to the fact that I generate my XML differently, and the MPEG stream started at a different offset because of that. pyTivo MPEG started at 0x3000 whle my server MPEG started at 0x1C00.

The MPEG portion of the files was the same, byte for byte.

If I was doing something wrong in the TiVo wrapper, I'd have expected the file to fail transfer to the TiVo early, and not play at all. Since I'm getting a significant amount transferred and playable, I'm at a loss as to what to look at next.

The announcement today that TiVo is dropping support for free TiVo desktop just makes me want to have this working that much more.
wcbonner is offline   Reply With Quote
Old 05-08-2013, 08:38 PM   #16
Dan203
Super Moderator
 
Dan203's Avatar
 
Join Date: Apr 2000
Location: Nevada
Posts: 24,823
You realize you still need the DirectShow filter included with TiVo Desktop to actually play the downloaded files right? They're encrypted by default. And wile tivodecode can be used if you download the old PS format it does not work with the new TS format, which is faster and the only option available when the show is H.264 encoded.
__________________
Dan Haddix
Super Moderator
Developer for VideoReDo
Dan203 is offline   Reply With Quote
Old 05-08-2013, 10:00 PM   #17
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,015
Actually yes, there's a version of tivodecode that works with TS files, not perfectly but acceptably. Anyway, for now, 99% of programming is still MPEG-2.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 05-08-2013, 11:04 PM   #18
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
Quote:
Originally Posted by Dan203 View Post
You realize you still need the DirectShow filter included with TiVo Desktop to actually play the downloaded files right? They're encrypted by default. And wile tivodecode can be used if you download the old PS format it does not work with the new TS format, which is faster and the only option available when the show is H.264 encoded.
My understanding is that Directshow is the legacy playback path from Windows XP , and getting the right filters installed for a Win7 x64 system is questionable to begin with. VLC seems to work with my files without any problems, and it doesn't rely on underlying windows codecs, which is especially nice on Win8.

My TiVo is a TiVo HD, with a TSN that starts with 652, and I've not yet run into any file that TivoDecode doesn't handle. The latest version of ffmpeg does a great job with so many formats that it didn't handle last year, so I use it to clean up the output of TiVoDecode into a container that I prefer.

My primary project that I'm working on is to get videos from the PC to the TiVo, but I'm stuck diagnosing why the TiVo is resetting the TCP connection between 2 and 4 minutes into the transfer. The TiVo is generally playing what's been streamed up to that point, and this is usually happening with SD content that transfers faster than realtime.
wcbonner is offline   Reply With Quote
Old 05-12-2013, 10:05 PM   #19
wcbonner
Registered User
 
wcbonner's Avatar
 
Join Date: Nov 2007
Posts: 15
I finally figured out how to completely transfer a transcoded file. It turns out that in my XML for the TiVoItem in response to the GetTivoContainer command, I was returning the actual file size in the SourceSize element. When I changed so that I report 1000*duration, the transcoded files seem to get to the tivo.

I'm guessing the underlying problem is that the tivo allocates approximately enough space when the transfer starts, and if it doesn't allocate a big enough block it aborts when the allocated block gets filled.

Right now I'm just happy to have it working, so I can focus on the other features I was working on.
wcbonner is offline   Reply With Quote
Reply
Forum Jump




Thread Tools


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Advertisements

TiVo Community
Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
vBulletin Skins by: Relivo Media

(C) 2013 Magenium Solutions - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not owned or operated by TiVo Inc.
All times are GMT -5. The time now is 06:24 PM.
OUR NETWORK: MyOpenRouter | TechLore | SansaCommunity | RoboCommunity | MediaSmart Home | Explore3DTV | Dijit Community | DVR Playground |