Home » Android » java – Limitations of proto buffers – loading partial data and sharing strings

java – Limitations of proto buffers – loading partial data and sharing strings

Posted by: admin May 14, 2020 Leave a comment

Questions:

I want to check if proto buffer is the best serializer for my use, my research found nothing else coming close.
I’m working on java backend and android (java) mobile app, however it’s possible that other client will be created in not so distant future, so I want something cross platform.
Rough draft of data structure:

message All {
    repeated Line lines = 1;
    Common common = 2;
}

There are a couple hundreds of Line object, each Line is quite complex and takes ~100 kB on its own.

Two problems I see with proto buffer
– on application startup I need just fraction of available data – just “Common” and basic information from “Line”. Is it possible to load partial data?
– each Line object contains hundreds of strings, but the same string occurs in several Line objects so I want to try hard to share them between those objects. Is it possible on proto buf level, or does it need to be part of application level?

Thanks!

How to&Answers:

From the limited specification you gave, it is pretty hard to give a proper feedback. You stated that for your problem the best solution seems to be protobuf, but we cannot reassess that from the information given.

Based on what you wrote I would even say that a simple GZIPped byte array (or JSON as most of it is printable (you stated they are Strings)) may be better for you (“reusing” a lot of stuff across Line objects => GZIP will rock).

And as others stated: with protobuf it’s not possible to load “partial data structures” (actually yours would not be a partial one, as long as the “repeated” part is e.g., a Collection, because than protobuf will take care of segmenting your data in the structure itself).

I would put my two cents on GZIPped JSON streams (e.g., with Jackson), of course in this case you would reduce bandwidth with the cost of CPU cycles.

Answer:

Answering my own question regarding loading partial data. I haven’t tried it in practice yet, but I wonder if the following wouldn’t work:

message All {
    repeated Line lines = 1;
    Common common = 2;
}

message All_partial {
    Common common = 2;
}

As all fields in proto3 are optional, we can have second definition of our structure, with “partial” fields. If we keep the same field number, I hope we will be fine.