63526

Google API BatchRequest: An established connection was aborted by the software in your host machine

Question:

I'm testing Google BatchRequest (C#) of InsertAllRequest. once a batch reaches more than 60 requests (~30,000 bigquery table rows / > 10,880,366 Http ContentLength in total), Exceptions happens as below. It's the same when I turned off my firewall. Solutions I found online such as turn http keep-alive off doesn't work in this case because I don't have controls on how the API uses the HttpClient.

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state) at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state) --- End of inner exception stack trace --- at System.Net.GZipWrapperStream.EndRead(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.EndRead(IAsyncResult asyncResult) at System.Net.Http.StreamToStreamCopy.StartRead() --- End of inner exception stack trace --- at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Google.Apis.Requests.BatchRequest.<ExecuteAsync>d__3.MoveNext() in c:\ApiaryDotnet\default\Src\GoogleApis\Apis\Requests\BatchRequest.cs:line 175 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at BigData.BigQuery.API.BigQueryHelper.<InsertBatchAsync>d__1f.MoveNext() in c:\Users\fionazhao\Documents\BigDataCode\Framework\BigData\BigQuery\API\BigQueryHelper.cs:line 197 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

Here's the Google's API code that throwed this exception:

public async Task ExecuteAsync(CancellationToken cancellationToken) { if (Count < 1) return; ConfigurableHttpClient httpClient = service.HttpClient; var requests = from r in allRequests select r.ClientRequest; HttpContent outerContent = await CreateOuterRequestContent(requests).ConfigureAwait(false); var result = await httpClient.PostAsync(new Uri(batchUrl), outerContent, cancellationToken) .ConfigureAwait(false); result.EnsureSuccessStatusCode(); // Get the boundary separator. const string boundaryKey = "boundary="; var fullContent = await result.Content.ReadAsStringAsync().ConfigureAwait(false); var contentType = result.Content.Headers.GetValues("Content-Type").First(); var boundary = contentType.Substring(contentType.IndexOf(boundaryKey) + boundaryKey.Length); int requestIndex = 0; // While there is still content to read, parse the current HTTP response. while (true) { cancellationToken.ThrowIfCancellationRequested(); var startIndex = fullContent.IndexOf("--" + boundary); if (startIndex == -1) { break; } fullContent = fullContent.Substring(startIndex + boundary.Length + 2); var endIndex = fullContent.IndexOf("--" + boundary); if (endIndex == -1) { break; } HttpResponseMessage responseMessage = ParseAsHttpResponse(fullContent.Substring(0, endIndex)); if (responseMessage.IsSuccessStatusCode) { // Parse the current content object. var responseContent = await responseMessage.Content.ReadAsStringAsync().ConfigureAwait(false); var content = service.Serializer.Deserialize(responseContent, allRequests[requestIndex].ResponseType); allRequests[requestIndex].OnResponse(content, null, requestIndex, responseMessage); } else { // Parse the error from the current response. var error = await service.DeserializeError(responseMessage).ConfigureAwait(false); allRequests[requestIndex].OnResponse(null, error, requestIndex, responseMessage); } requestIndex++; fullContent = fullContent.Substring(endIndex); } }

Answer1:

That's just above 10MB which is the request size limit for streaming inserts. Adjust your code for these limits:

Maximum row size: 1 MB<br /> HTTP request size limit: 10 MB<br /> Maximum rows per second: 100,000 rows per second, per table. Exceeding this amount will cause quota_exceeded errors.<br /> Maximum rows per request: 500<br /> Maximum bytes per second: 100 MB per second, per table. Exceeding this amount will cause quota_exceeded errors.

<a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery" rel="nofollow">https://cloud.google.com/bigquery/streaming-data-into-bigquery</a>

Recommend

  • How to connect to modems with tcp clients, in multiple port or other way?
  • Invoke or call more than one method at same time?
  • Is it possible to detect if a Stream has been closed by the client?
  • How to update gui in c# wpf from asynchronous method callback
  • Read asynchronously data from NetworkStream with huge amount of packets
  • GridBag Layout How to push components north
  • Add an entry to the legend for a manually added line
  • Coerce raster time series (rts) object back to raster* object
  • Swiftmailer email message returns error when attaching file
  • How to debug 'value of ESP was not saved across function call' error?
  • How to use CompletableFuture without risking a StackOverflowError?
  • Function arguments VBA
  • Tiny Images, No Rotation, but still get OutOfMemoryError: bitmap size exceeds VM budget
  • double precision error when converting to scientific notation
  • Creating multiple dialogs in an MFC app with no main Window, they become children of each other
  • segfault from C stack overflow
  • Stack size of secondary threads, significant differences between DEBUG and RELEASE versions
  • What are “Java EE 7 API Library” and “Java EE Web 7 API Library” and when to use them?
  • apache camel how to get header attributes from sql-component
  • Data binding in Polmyer's
  • Create recursive binary tree?
  • Exception handling as per java coding standards
  • .NET Core 2.0 RSA PlatformNotSupportedException
  • Flot returns incorrect x value (mm/dd/yy - date) after zoom for stack bar chart
  • numpy 64bit support in PTVS and numpy System.Int64 casting
  • How to run bash commands like “npm install” on complie
  • unrecognized selector isPitched called
  • Python: Split a String Field into 3 Separate Fields using Lambda
  • maven jboss-as:start A required class was missing … org/sonaty…/ArtifactResolutionException
  • Is there a way to call library thread-local init/cleanup on thread creation/destruction?
  • How do I add a File Type Association in a Windows Phone 8.1 app manifest?
  • Where these are stored?
  • Graphics.CopyFromScreen [Web application] + The handle is invalid
  • Display images in Django
  • Problem deserializing objects from cache on MyBatis 3/Java
  • javascript inside java/jsp code
  • Android Studio and gradle
  • Acquiring multiple attributes from .xml file in c#
  • IndexOutOfRangeException on multidimensional array despite using GetLength check
  • How can i traverse a binary tree from right to left in java?