74285

Cross-Platform Protobuf Serialization

I have three applications that are communicating via ZeroMQ all performing different operations. The different applications are as follows:

<ol> <li>

The first is a C++ application which is an engine for "hard work", this takes a Protobuf message as a request from a client, does some work, and returns a Protobuf message back to that client (or whoever is connected, Request/Reply Pattern). This uses 0MQ version 4.0.4 and using protobuf-2.6.0 where we have built the required header files ourselves, the Protobuf classes were created by protoc.

</li> <li>

Second I have a Java code and is a data provider, this uses jeromq-0.3.4.jar for the ZeroMQ messaging and protobuf-java-2.6.1.jar for the Protobuf serialization etc.

</li> <li>

Third I have a C# code which performs some analysis and has a nice UI etc. This uses Marc Gravell's protobuf-net (https://www.nuget.org/packages/protobuf-net/) as a NuGet package and NetMQ (native C# port of ZeroMQ) for the messaging.

</li> </ol>

Now, C++ <-> Java works great and with out problems, however, C++ <-> C# does not work correctly. When I send a basic request from C# to the C++ "server" via

using (NetMQContext context = NetMQContext.Create()) using (var requestSocket = context.CreateRequestSocket()) { requestSocket.Connect(_requestAddress); // "tcp://127.0.0.1:6500" requestSocket.Send(mux.ToByteArray<Taurus.FeedMux>()); }

with

public static byte[] ToByteArray<T>(this T o) where T : ProtoBuf.IExtensible { if (o == null) return null; using (MemoryStream ms = new MemoryStream()) { ProtoBuf.Serializer.Serialize(ms, o); return ms.ToArray(); } }

The C++ code receives the message but despite setting a mandatory

Taurus.FeedMux mux = new Taurus.FeedMux(); mux.type = Taurus.FeedMux.Type.OXX; mux.oxx = Oxx.GetOxx();

I get an error in the C++ application

[libprotobuff ERROR ..\\message_lite.cc:123] Can't parse message of type "Taurus.FeedMux" because it is missing required fields: type

But I am clearly setting type and in the C++ code, type seems to be set (using the debugger to inspect the deserialized object). I have tried two different Protobuf libraries (one I built and Mark Gravell's library via NuGet) as I thought this was a serialization issue , but this does not fix this problem.

I have also tried the clrZMQ wrapper library as well as the C# native NetMQ library, again this does not help, message is received using any combination of the above, but the received seems corrupt in some way.

What could be going wrong here and is there anything I should be doing that I have not mentioned?

Answer1:

My guess is that type is being treated as optional, where-as in the C++ it is marked as required. I can't see how your model is defined in the C#, but if you're using protobuf-net's attributes, you can force it to serialize via the IsRequired attribute member:

[ProtoMember(42, IsRequired = true)] // change the number! public Taurus.FeedMux.Type type {get;set;}

Recommend

  • R Markdown Not Rendering Citation
  • is DirectorySearcher.SizeLimit = 1 for FindAll() equal to FindOne() [DirectoryServices/.net]
  • Find all tables with a field containing xml string values
  • Trying to bind configuration: System.ComponentModel.TypeConverter can't be loaded
  • NuGet: automatically migrate references to packages
  • How to dynamically generate javascript using ScriptSharp?
  • Azure PDF Sharp not using Unicode font
  • Could not install package 'Microsoft.Owin.Security 2.0.1'
  • Install different versions of nuget packages inside one solution file with two projects
  • Prevent Tomcat from caching request during starup
  • pillow imaging ImportError
  • Problems installing Yesod for Haskell
  • Access variable of ScriptContext using Nashorn JavaScript Engine (Java 8)
  • error importing numpy
  • Marklogic : Query response time is very high
  • Parsing a CSV string while ignoring commas inside the individual columns
  • Jetty 9 HashLoginService
  • Problem deserializing objects from cache on MyBatis 3/Java
  • MongoDb aggregation
  • azure media services - The request body is too large and exceeds the maximum permissible limit
  • Unity3D & Android: Difference between “UnityMain” and “main” threads?
  • How to create a file in java without a extension
  • OpenGL ES texture problem, 4 duplicate columns and horizontal lines (Android)
  • ImageMagick, replace semi-transparent white with opaque white
  • does jqgrid support a multiple checkbox list for editing
  • Cannot connect to cassandra from Spark
  • Installing Hadoop, Java Exception about illegal characters at index 7?
  • Why ng-show works with ng-repeat but ng-if doesn't? [duplicate]
  • Optimizing database types to compact database (SQLite)
  • RectangularRangeIndicator format like triangular using dojo
  • Get object from AWS S3 as a stream
  • Validaiting emails with Net.Mail MailAddress
  • Alternatives to the OPTIONAL fallback SPARQL pattern?
  • Do I've to free mysql result after storing it?
  • How to format a variable of double type
  • Revoking OAuth Access Token Results in 404 Not Found
  • coudnt use logback because of log4j
  • Turn off referential integrity in Derby? is it possible?
  • XCode 8, some methods disappeared ? ex: layoutAttributesClass() -> AnyClass
  • JaxB to read class hierarchy