wrapping flatbuffers inside flatbuffers

Michael C.

unread,

Nov 23, 2015, 2:38:04 AM11/23/15

to FlatBuffers

Hi,

I want to use flatbuffers which contain flatbuffers inside. The use case is a messaging library, which wraps any datatype inside a flatbuffer as a payload:

// message.fbs

namespace Lib;

enum Type : byte { Request, Reply }

table Header {
   type:Type = Request;
   timestamp:uint = 0;
   datatype:uint;
}

table Message {
   header:Header;
   payload:[ubyte];
}

root_type Message;



The messaging library does not no at compile time which datatypes will be contained in its payload (so I can't use a union).

I tried the following code with some simple demo datatypes:


// app1.fbs
namespace App1;

table SomeDataType {
   value:uint = 123;
}

root_type SomeDataType;


// app2.fbs
namespace App2;

table Data {
   text:string;
}

root_type Data;

// demo.cpp

#include <iostream>
#include "flatbuffers/flatbuffers.h"

#include "app1_generated.h"
#include "app2_generated.h"
#include "message_generated.h"

int main(int argc, char* argv[])
{
   flatbuffers::FlatBufferBuilder fbb;

   // create app data
   int sentDataType = std::stoi(argv[1]);
   if (sentDataType == 1)
   {

     auto sdt = App1::CreateSomeDataType(fbb, 555);
     fbb.Finish(sdt);

   }  
   else if(sentDataType == 2)
   {
     auto text = fbb.CreateString("foo");
     auto data = App2::CreateData(fbb, text);
     fbb.Finish(data);
   }

   // create wrapper message
   auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());
   auto header = Lib::CreateHeader(fbb, Lib::Type_Request, 456, sentDataType);
   fbb.Finish(header);

   Lib::MessageBuilder msg(fbb);
   msg.add_header(header);
   msg.add_payload(payload);
   auto msgLoc = msg.Finish();
   fbb.Finish(msgLoc);

   // ... transmit to receiver ...

   auto recvMsg = Lib::GetMessage(fbb.GetBufferPointer());

   // dynamically determine the datatype inside payload
   auto recvDataType = recvMsg->header()->datatype();
   if (recvDataType == 1)
   {
     auto recvSdt = App1::GetSomeDataType(recvMsg->payload()->Data());
     std::cout << recvSdt->value() << std::endl;
   }
   else if (recvDataType == 2)
   {
      auto recvData = App2::GetData(recvMsg->payload()->Data());
      std::cout << recvData->text()->str() << std::endl;
   }

   return 0;
}

This seems to work:

$ ./demo 1
555

$ ./demo 2
foo

I have several questions about my approach:

Is my approach valid?
Is there a more "native flatbuffers-style" way to solve this?
"flatbuffers.h" says "Use with care." about Vector::Data(). What could go wrong?
Is my approach platform and language indendent?
Can the copy happening at fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize()); be avoided?

Thanks!

mikkelfj

unread,

Nov 23, 2015, 3:17:24 AM11/23/15

to FlatBuffers

What you do is similar to using nested flatbuffers which use a [ubyte] vector instead of a string to carry the payload. Except for trailing zero it is the same.

Nested buffers do have a type in the attributes so it may not work for you.

Strings are aligned only to 4 bytes because the header field is 4 bytes. If you have data that require higher alignment, you should hide your payload in a vector [ulong] for example.

I am not sure about the status of nested buffers in C++, but I don't think they currently ensure alignment beyond 4 bytes.

As what could go wrong with vectors - I'm not sure what the specific warning refers to, but perhaps the consideration is that if you access vectors raw on a big endian platform, the data will be garbage. For the same reason, you should not cast data hidden in a vector directly to raw structs / classes, but to flatbuffer and read it as such. Unless you don't care about big endian at all. The flatbuffer you cast to need not be in the same schema as the message transport, but obviously the reader must somehow know the type.

mikkelfj

unread,

Nov 23, 2015, 3:24:01 AM11/23/15

to FlatBuffers

Oh, and the final thing that can go wrong if you don't cast to a flatbuffer type is that each compiler may use a different layout in memory - in praxis rarely an issue, but flatbuffers guarantee an exactly memory layout. So if you only use flatbuffers for storing the message type and not the real payload, you might as well not use flatbuffers at all, in effect.

Michael C.

unread,

Nov 23, 2015, 3:33:00 AM11/23/15

to FlatBuffers

Thanks for your reply.

I am not sure I understand you correctly. Do you think I use a string for storing the payload? I thought payload:[ubyte]; meant a vector of bytes.

Why would the buffer I get through fbb.GetBufferPointer() and store inside the payload need to have any special alignment treatment inside Message?

What do you mean by "cast to flatbuffer type"? I pass the payload buffer to flatbuffers using e.g. App1::GetSomeDataType(recvMsg->payload()->Data());
Is this the correct approach?

mikkelfj

unread,

Nov 23, 2015, 3:45:05 AM11/23/15

to FlatBuffers

Den mandag den 23. november 2015 kl. 09.33.00 UTC+1 skrev Michael C.:

Thanks for your reply.

I am not sure I understand you correctly. Do you think I use a string for storing the payload? I thought payload:[ubyte]; meant a vector of bytes.

Sorry, I misread your schema - the string was in the payload schema. So it is exactly as for nested flatbuffers.

Why would the buffer I get through fbb.GetBufferPointer() and store inside the payload need to have any special alignment treatment inside Message?

Because to container buffer is typically aligned to 8 bytes when allocated in memory. So the pointer you get to any embedded buffer will only be 8-byte aligned iff you ensure it has that alignment relative to the buffer start. If you copy the data to a new aligned memory buffer it doesn't matter. On x64 it does't matter at all, but this isn't portable.

What do you mean by "cast to flatbuffer type"? I pass the payload buffer to flatbuffers using e.g. App1::GetSomeDataType(recvMsg->payload()->Data());
Is this the correct approach?

Yes - I was unclear - you do it right - I was referring to what could happen if you don't.

Wouter van Oortmerssen

unread,

Nov 25, 2015, 4:24:28 PM11/25/15

to mikkelfj, FlatBuffers

Also, please note that using the same FlatBufferBuilder for different buffers like in your example won't work, you'll end up with all these buffers concatenated (and GetBufferPointer/GetSize won't get you the right values). Use an individual FlatBufferBuilder for each.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wouter van Oortmerssen

unread,

Nov 30, 2015, 2:56:25 PM11/30/15

to mikkelfj, FlatBuffers

Note that you can now use this function when using nested flatbuffers to fix any alignment issues mentioned above:

https://github.com/google/flatbuffers/commit/07da3fc216c62b18eb13a8bcb9afa95d7c325418

Reply all

Reply to author

Forward