45378

C# - Is there a limit to the size of an httpWebRequest stream?

I am trying to build an application that downloads a small binary file (20-25 KB) from a custom webserver using httpwebrequests.

This is the server-side code:

Stream UpdateRequest = context.Request.InputStream; byte[] UpdateContent = new byte[context.Request.ContentLength64]; UpdateRequest.Read(UpdateContent, 0, UpdateContent.Length); String remoteVersion = ""; for (int i = 0;i < UpdateContent.Length;i++) { //check if update is necessary remoteVersion += (char)UpdateContent[i]; } byte[] UpdateRequestResponse; if (remoteVersion == remotePluginVersion) { UpdateRequestResponse = new byte[1]; UpdateRequestResponse[0] = 0; //respond with a single byte set to 0 if no update is required } else { FileInfo info = new FileInfo(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll")); UpdateRequestResponse = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll")); //respond with the updated file otherwise } //this byte is past the threshold and will not be the same in the version the client recieves Console.WriteLine("5000th byte: " + UpdateRequestResponse[5000]); //send the response context.Response.ContentLength64 = UpdateRequestResponse.Length; context.Response.OutputStream.Write(UpdateRequestResponse, 0, UpdateRequestResponse.Length); context.Response.Close();

After this the array UpdateRequestResponse contains the entire file and has been sent to the client.

The client runs this code:

//create the request WebRequest request = WebRequest.Create(url + "pluginUpdate"); request.Method = "POST"; //create a byte array of the current version byte[] requestContentTemp = version.ToByteArray(); int count = 0; for (int i = 0; i < requestContentTemp.Length; i++) { if (requestContentTemp[i] != 0) { count++; } } byte[] requestContent = new byte[count]; for (int i = 0, j = 0; i < requestContentTemp.Length; i++) { if (requestContentTemp[i] != 0) { requestContent[j] = requestContentTemp[i]; j++; } } //send the current version request.ContentLength = requestContent.Length; Stream dataStream = request.GetRequestStream(); dataStream.Write(requestContent, 0, requestContent.Length); dataStream.Close(); //get and read the response WebResponse response = request.GetResponse(); Stream responseStream = response.GetResponseStream(); byte[] responseBytes = new byte[response.ContentLength]; responseStream.Read(responseBytes, 0, (int)response.ContentLength); responseStream.Close(); response.Close(); //if the response containd a single 0 we are up-to-date, otherwise write the content of the response to file if (responseBytes[0] != 0 || response.ContentLength > 1) { BinaryWriter writer = new BinaryWriter(File.Open(Path.Combine(Directory.GetCurrentDirectory(), "ServerPlugins", "PointAwarder.dll"), FileMode.Create)); writer.BaseStream.Write(responseBytes, 0, responseBytes.Length); writer.Close(); TShockAPI.Commands.HandleCommand(TSPlayer.Server, "/reload"); }

The byte array responseBytes on the client should be identical to the array UpdateRequestResponse on the server, but it isn't. after about 4000 bytes every byte after that is set to 0 rather than what it should be (responseBytes[3985] is the last non-zero byte).

Does this happen because httpWebRequest has a size limit? I can't see any bug in my code that could be causing it and the same code works in other instances where I only have to pass around short sequences of data (less than 100 bytes).

The MSDN pages don't mention any size limit like this.

Answer1:

It's not that it has any artificial limit, this is a byproduct of the Streaming nature of what you're attempting to do. I have a feeling the following line is the offender:

responseStream.Read(responseBytes, 0, (int)response.ContentLength);

I've had this issue in the past (with TCP streams), it doesn't read all of the contents of the array, because they haven't all been sent over the wire yet. This is what I would try instead.

for (int i = 0; i < response.ContentLength; i++) { responseBytes[i] = responseStream.ReadByte(); }

That way, it will make sure to read all the way until the end of the stream.

<strong>EDIT</strong>

usr's BinaryReader based solution is much more efficient. Here is the relevant solution:

BinaryReader binReader = new BinaryReader(responseStream); const int bufferSize = 4096; byte[] responseBytes; using (MemoryStream ms = new MemoryStream()) { byte[] buffer = new byte[bufferSize]; int count; while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0) ms.Write(buffer, 0, count); responseBytes = ms.ToArray(); }

Answer2:

You are assuming that Read is reading as many bytes as you request. But the requested count is just an upper limit. You must tolerate reading small chunks.

You can use var bytes = new BinaryReader(myStream).ReadBytes(count); to read an exact number. Don't call ReadByte too often because that is very CPU intensive.

The best solution would be to step away from the fairly manual HttpWebRequest and use HttpClient or WebClient. All of this is automated for you and you get back a byte[].

Recommend

  • ASP.NET Capture and replace output in Global.asax
  • How to send image as base64 string in JSON using HTTP POST in Android?
  • Where does the file get saved using “File file = new file(filename)” in Android
  • quiver not drawing arrows just lots of blue, matlab
  • Suppressing passwd when calling sqlplus from shell script
  • Convert Type Decimal to Hex (string) in .NET 3.5
  • ViewController With Transparent Background Entering Current ViewController With Push Transition
  • Appending Character to Character Array In C
  • PHP CURL timing out but CLI CURL works
  • Declaring variable dynamically in VB.net
  • CakePHP 2.0.4 - findBy magic methods with conditions
  • Jackson Parser: ignore deserializing for type mismatch
  • copying resource to sdcard gives a damaged file in android
  • Setting up SourceTree to merge unity3d scenes with UnityYAMLMerge
  • Initializer list vs. initialization method
  • Using variable in a value field in jMeter
  • Email format validation in mvc3 view
  • Is my CUDA kernel really runs on device or is being mistekenly executed by host in emulation?
  • Date difference with leap year
  • How to add date and time under each post in guestbook in google app engine
  • TFS: Get latest causes slow project reloading
  • Updating server-side rendering client-side
  • Running a C# exe file
  • Perl system calls when running as another user using sudo
  • vba code to select only visible cells in specific column except heading
  • Rearranging Cells in UITableView Bug & Saving Changes
  • How to pass list parameters for each object using Spring MVC?
  • Is there a mandatory requirement to switch app.yaml?
  • File upload with ng-file-upload throwing error
  • ExecuteAsync RestSharp to allow backgroundWorker CancellationPending c#
  • AngularJs get employee from factory
  • Benchmarking RAM performance - UWP and C#
  • Error creating VM instance in Google Compute Engine
  • Hits per day in Google Big Query
  • Angular 2 constructor injection vs direct access
  • coudnt use logback because of log4j
  • how does django model after text[] in postgresql [duplicate]
  • IndexOutOfRangeException on multidimensional array despite using GetLength check
  • sending mail using smtp is too slow
  • Reading document lines to the user (python)