path: root/doc/builder.md

# Builder Interface Reference

<!-- vim-markdown-toc GFM -->

* [Introduction](#introduction)
* [Size Prefixed Buffers](#size-prefixed-buffers)
* [Namespaces](#namespaces)
* [Error Codes](#error-codes)
* [Endianess](#endianess)
  * [Deprecated](#deprecated)
* [Buffers](#buffers)
* [Tables](#tables)
  * [Adding Fields](#adding-fields)
  * [Nested Tables](#nested-tables)
* [Packing tables](#packing-tables)
* [Strings](#strings)
* [Structs](#structs)
  * [Fixed Length Arrays in Structs](#fixed-length-arrays-in-structs)
* [Nested Buffers](#nested-buffers)
* [Scalars and Enums](#scalars-and-enums)
* [Vectors](#vectors)
* [Unions](#unions)
  * [Union Vectors](#union-vectors)
  * [Unions of Strings and Structs](#unions-of-strings-and-structs)
* [Error Handling](#error-handling)
* [Type System Overview](#type-system-overview)
* [Cloning](#cloning)
* [Picking](#picking)
* [Sorting Vectors](#sorting-vectors)
  * [Dangers of Sorting](#dangers-of-sorting)
  * [Scanning](#scanning)
* [Example of different interface type users](#example-of-different-interface-type-users)
* [Special Emitters](#special-emitters)

<!-- vim-markdown-toc -->


## Introduction

We assume a separate read-only file and add extensions to this with
support from a builder library and a builder object.

The underlying builder library supports two modes of operation that mix
together: `create` which sends data directly to the target buffer
(emitter object) and a stack driven `start/end` approach which allocates
objects and vectors on the stack. The code generator chooses the most
efficient approach given the circumstances.

Unlike most FlatBuffer language interfaces, tables and vectors are not
created back to front: They are either created completely in one
operation, or they are constructed on a stack front to back until they
can be emitted.  The final buffer is still constructed back to front.
For big-endian platforms this may require temporary stack allocation of
complete vectors where little endian platforms can emit directly.

Tables and vectors stored in other tables or vectors must be completed
before the can be stored, but unlike must language interfaces they can
be constructed while a parent is also being constructed as long as
nesting remains balanced. While this occasionally may require more
stack, it may also avoid external temporary allocation.

A builder object is required to start buffer construction. The builder
must be initialized first and can be reset and reused between buffers,
reusing stack allocation. The builder can have a customized emitter
object but here we use the default. Finalizing the buffer depends
the emitter and we can use a default finalizer only because we use the
default emitter - it allocates and populates a linear buffer from a
paged emitter ring buffer.

Note that in most cases `flatcc_builder_finalize_buffer` is sufficient,
but to be strictly portable, use
`flatcc_builder_finalize_aligned_buffer` and `aligned_free`.
`aligned_free` is often implemented as `free` in `flatcc/portable` but
not on all platforms. As of flatcc version 0.5.0
`flatcc_builder_aligned_free` is provided to add robustness in case the
applications `aligned_free` implementation might differ from the library
version due to changes in compile time flags.

Generally we use the monster example with various extensions, but to
show a simple complete example we use a very simple schema (`myschema.fbs`):

    table mytable { myfield1: int; myfield2: int; }

    #include "myschema_builder.h"

    void testfun() {

        void *buffer;
        size_t size;
        flatcc_builder_t builder, *B;
        mytable_table_t mt;
        B = &builder;
        flatcc_builder_init(B);

        /* Construct a buffer specific to schema. */
        mytable_create_as_root(B, 1, 2);

        /* Retrieve buffer - see also `flatcc_builder_get_direct_buffer`. */
        /* buffer = flatcc_builder_finalize_buffer(B, &size); */
        buffer = flatcc_builder_finalize_aligned_buffer(B, &size);

        /* This is read-only buffer access. */
        mt = mytable_as_root(buffer);
        assert(mytable_myfield1(mt) == 1);
        assert(mytable_myfield2(mt) == 2);

        /* free(buffer); */
        flatcc_builder_aligned_free(buffer);

        /*
         * Reset, but keep allocated stack etc.,
         * or optionally reduce memory using `flatcc_builder_custom_reset`.
         */
        flatcc_builder_reset(B);

        /* ... construct another a buffer */

        /* Reclaim all memory. */
        flatcc_builder_clear(B);
    }

Note that a compiled schema generates a `myschema_reader.h` file and
optionally a `myschema_builder.h` and some common support files. When
building a buffer the `myschema_builder.h` must be used but when only
reading then the `myschema_reader.h` file should be used instead. Here
we are only concerned with building. When building, it is necessary to
link with `libflatccrt.a` runtime library but when reading, all
nesessary code is contained in the generated header files.

The builder object only manages a stack of currently active objects and
does not store an object that is complete. Instead it calls an emitter
object with the partial data ready for emission, similar to a write
function. A default emitter is provided which implements a ring buffer
and the result may be written to a file, copied to a buffer or a
finalized to an allocated buffer. The builder supports these methods
directly for default emitter, and only the default emitter because
emitters are otherwise defined by only one simple emit function - see
`emit_test.c` for a simple example of a custom emitter.
A custom allocator may be useful when working with small buffers in a
constrained environment - the allocator handles temporary stacks,
virtual table caches etc. but not the emitter.

The allocator and emitter interface is documented in the builder library
header pflatcc_builder.h] and the default implementation in
[flatcc_emitter.h]. The default allocator is implemented as part of the
flatcc_builder source.

The builder can be reused between buffers using the `reset` operation.
The default emitter can also be reused and will automaticallhy reset
when the buffer is. For custom emitters, any reset operation must be
called manually. The same applies to clear. The reset operations
maintain allocated memory by also reduce memory consumption across
multiple resets heuristically.


## Size Prefixed Buffers

Buffers can be created with a size prefix of type `uoffset_t`. When
doing this, the buffer is aligned relative to the size prefix such that
buffers can be stacked in a file and for example be accessed via memory
mapping.

The usual `create_as_root` and `start_as_root` has a variant called
`create_as_root_with_size` and `start_as_root_with_size`.

To read a buffer with a size prefix use:

    size_t size;
    buffer = flatbuffers_read_size_prefix(rawbuffer, &size);

The size the size of the buffer excluding the size prefix. When
verifying buffers the buffer and size arguments should be used. See also
[monster_test.c] for an example.

Note that the size prefix ensures internal alignment but does not
guarantee that the next buffer in a file can be appended directly
because the next buffers alignment is unknown and because it potentially
wastes padding bytes.  The buffer size at offset 0 can increased to the
needed alignment as long as endianness is handled and the size of the
size field is subtracted, and zeroes are appended as necesary.

## Namespaces

The generated code is typically wrapped in a custom namespace and
functions and definitions that are library specific are usually mapped
into the namespace. We often use an empty namespace for custom types and
`flatbuffers_` for library names, but usually a `foo_` prefix could also
be used on both cases, where `foo` is a custom namespace.

Note that the name `flatcc_emitter` is only used with the default emitter
and the name [flatcc_builder] is only used for buffer management but not
for constructing content. Once a valid buffer is ready the common and
namespace (`flatbuffers`) and schema specific (or empty) namespace is used
with schema specific operations.

All schema specific content is prefixed with a namespace to avoid
conflicts - although the namespace is empty if the schema doesn't
specify any. Note that the same schema can have multiple
namespaces. An example of a namespace prefixed operation:

    MyGame_Example_Monster_create_as_root(B, ... lots of args);

To simplify this we can use a macro to prefix a namespace. The use
of the name `ns` is arbitrary and we can choose different names for
different namespaces.

    #undef ns
    #define ns(x) MyGame_Example_ ## x

But the above doesn't work with nested calls to ns such as

    ns(Monster_color_add(B, ns(Color_Green));

it would have to be:

    ns(Monster_color_add)(B, ns(Color_Green);

Therefore we have a helper macro the does allow nesting:

    #undef ns
    #define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)

The common namespace can also be wrapped for a more consistent
appearance:

    #undef nsc
    #define nsc(x) FLATBUFFERS_WRAP_NAMESPACE(flatbuffers, x)

    nsc(string_ref_t) s;
    s = nsc(string_create_str(B, "hello, world!"));

instead of

    flatbuffers_string_ref_t s;
    s = flatbuffers_string_create_str(B, "hellow, world!);


## Error Codes

Functions return values can be grouped roughly into 4 groups: functions
returning pointer, references, `size_t` lengths, and `int` status codes.
Pointers and references return 0 on error. Sizes do not return error.
Status codes return 0 on success or an error code that is usually -1.
Status codes may be checked with `flatbuffers_failed(...)`.


## Endianess

The function `flatbuffers_is_native_pe()` provide an efficient runtime
check for endianness. Since FlatBuffers are little endian, the function
returns true when the native endianness matches the protocol endianness
which for FlatBuffers is little endian. We do not hardcode little endian
because it enables us to support other protocols in the future - for
example the struct conversions may be very useful for big endian network
protocols.

> As of flatcc 0.4.0 it is possible to compile flatcc with native
> big-endian support which has been tested on AIX. More details in
> [README Endianness](https://github.com/dvidelabs/flatcc#endianness)


By testing `is_native_pe` dependencies on speficic compile time flags
can be avoided, and these are fragile:

During build, vectors and structs behave differently from tables: A
table updates one field at a time, doing endian conversion along the
way. A struct is either placed in a table, and is converted by the table
specific operation, or it is placed in a vector. A vector only does the
endian conversion when the vector is finished, so when a vector is not
created atomically with a single `create` call, the elements are placed on a
stack. By default this is in native format, but the user may choose to
place buffer encoded structs or scalars in the vector and call
`vec_end_pe`. The same `push` operation can be used to place a
natively encoded struct and a buffer encoded struct in the vector
because it does no conversion at that point. Therefore there is also no
`push_pe` method that would mean to push an unconverted element unto
the stack. Only for tables and entire vectors does the pe command make
sense. If a vector wishes to push a buffer encoded struct when the
vector is otherwise constructed in native encoding or vice versa, the
vector may be extended empty and then assigned using any of the
`assign`, `assign_from_pe` or `assign_to_pe` calls.

We did not mention that a struct can also be a standalone object
as a buffer root, and for that it has a `end_pe` call that essentially
works like a single element vector without a length prefix.

The `clone` operation is a more userfriendly `pe` operation which takes
an object or a vector from an existing buffer and places it in a new
buffer without endian conversion.

### Deprecated

__NOTE: `FLATBUFFERS_LITTLEENDIAN` is deprecated and will be removed in
a future version. It just complicates endina handling.__

The header files tries to define `FLATBUFFERS_LITTLEENDIAN` to 0 or 1
based on system definitions but otherwise leaves the flag undefined.
Simply testing for

    #if FLATBUFFERS_LITTLEENDIAN
    ...
    #endif

will not fail if the endianness is undetected but rather give the
impression that the system is big endian, which is not necessarily true.
The `flatbuffers_is_native_pe()` relates to the detected or system
provided conversion functions if a suitable `endian.h` file after the
header file gave up on its own detection (e.g. `le16toh(1) == 1`).
Therefore, it is better to use `flatbuffers_is_native_pe()` in most
cases. It also avoids making assumptions on whether the protocol is
little or big endian.

## Buffers

A buffer can most simply be created with the `create_as_root` call for
a table or a struct as seen ealier. The `as_root` part is just a thin
wrapper around buffer start and stop calls and using these allows for
more flexibility. the `as_root` also automatically uses the defined file
identifier if any.

The build process begins with starting a buffer. The buffer may contain
a struct or table, so one of these should be constructed subsequently.
Structs are generally created inline in tables, only at the buffer level
is a struct created independently. The api actually permits other
formats, but it will not be valid flatbuffers then.

    flatcc_builder_ref_t root;
    flatcc_builder_init(B);
    /* 0 indicates no file identifier. */
    flatcc_builder_buffer_start(B, 0);
    root = /* ... construct a table or a struct */
    flatcc_builder_buffer_end(B, root);

`buffer_start` takes a file identifier as second argument. If null or a
string with null characters, the identifier is not stored in the buffer.

Regardless of whether a struct or table is declared as root in the schema or
not, there are methods to automatically start both the buffer and struct or buffer
and table such as `Monster_start/end_as_root`. This is also valid for
nested buffers. If the schema has a file identifier, it is used as
identifier for the created object. The alternative
`create_as_root_with_identifier` allows for explicitly setting an id or
explicitly dropping an id by providing a null argument. The
corresponding reader function `Monster_as_root(buffer)` also has a
`Monster_as_root_with_identifier(buffer, id)`. Here the id is ignored if the id
is null, and otherwise the operation returns null if the id does not match.
For the most part ids are handled transparently by these defaults.

The buffer can be started with block alignment and/or a custom
identifier using the `flatcc_builder_buffer_start_aligned`:

    flatcc_builder_buffer_start_aligned(B, "myid", 16);
    ...
    flatcc_builder_buffer_end(B, root);

The alignment can be 0 using the minimum required alignment, which is
derived from the operations between `start/end`. The alignment argument
is called `block_align` and is useful if the emitter operates on blocks
such as encryption, cache line isolation, or compression blocks where
the final buffer should align with the blocks used during construction.
This can lead to significant zero padding just after the block header,
depending on block size.

The schema specified identifier is given as:

    flatbuffers_identifier

and defaults to null. The schema specified extension is given as:

    flatbuffers_extension

and defaults to null. Note that `flatbuffers_` is replaced by whatever
namespace is chosen. Each specific schema type also has a named file
exntension reflection the extension active when the type was defined,
for example:

    MyGame_Example_Monster_file_identifier

This define is used when `create_as_root` automatically sets a file
identifier.

NOTE: before flatcc 0.6.1, the identifier was named

    MyGame_Example_Monster_identifier (DEPRECATED)

but that would conflict with a table field named `identifier` which
happened often enough to be a problem. This naming is now removed on
conflict and will be completely removed in a future version.

When the buffer is ended, nothing special happens but only at this point
does it really makes sense to access the resulting buffer. The default
emitter provides a copy method and a direct buffer access method. These
are made available in the builder interface and will return null for
other emitters. See also [flatcc_builder.h] and the default emitter in
`flatcc_emitter.h`.


## Tables

### Adding Fields

If `Monster` is a table, we can create a Monster buffer (after
builder init) as follows:

    Monster_start(B);
    Monster_Hp_add(B, 80);
    ...
    flatcc_builder_buffer_create(B, Monster_end(B));

All scalar and enums are added similar to the `Monster_add_Hp` call. We
will subsequently see how to deal with other types.

A table can also be created in a single operation using `create`:

    Monster_ref_t m;
    m = Monster_create(B, 80, ...);

The create arguments are those taken by the individual fields `add`
operations which is either an scalar, enum, or a reference returned by
another create or end call. Note that unlike the C++ interface, unions
only take a single argument that is also accepted by the `add` operation
of a union field. Deprecated fields are not included in the argument
list.

As of v0.5.3 the arguments are given in field id order which is usually
the same as the schema listed order, except with id attributes are
given explicitly. Using id order ensures version stability. Note that
since deprecated fields are omitted, deprecated fields can still break
existing code.

BREAKING: Prior to flatcc v0.5.3 the create call would use the schema order
also when fields have id attributes specifying a different order. This
could break code across versions and did not match the C++ behavior.
It was also document that the `original_order` attribute affected create
argument order, but that was incorrect.

NOTE: If the `original_order` attribute is set on a table, the `create`
implementation adds fields to the table in schema listed order,
otherwise it adds fields in order of decreasing size to reduce alignment
overhead. Generally there should be no need to use the `original_order`
attribute. This doesn't affect the call argument order although that
was incorrectly document prior to v 0.5.3.

NOTE: the `create` and `create_as_root` operations are not guaranteed to
be available when the number of fields is sufficiently large because it
might break some compilers. Currently there are no such restrictions.

Scalars and enums do not store the value if it it matches the default
value which is by default 0 and otherwise defined in the schema. To
override this behavior, use `force_add`. In the monster example, health
points default to 100 (percent), so if we wish to force store it in the
buffer we could use:

    Monster_hp_force_add(B, 100);

Only scalar fields and enums have a `force_add` operation since only these
types have a default value, and other types have a meaningful
interpretation of null. (It is not quite clear if empty tables separate
from null/absent are valid in all implementations).

`force_add` may be useful when roundtripping data from a database where it is
relevant to distinguish between any valid value and null. Most readers will not
be able to tell the difference, but it is possible to inspect a flatbuffer to
see if a table field is present, present and default, or absent, meaning null.

NOTE: As of mid 2020, FlatBuffers added optional scalar table fields with support in flatcc 0.6.1. These fields automatically imply `force_add` to represent null values when a field is absent and therefore these fields do not have a `force_add` method and these fields also do not have a default value other than `null`, i.e. null if not added.

If Monster is declared as root, the above may also be called as:

    Monster_start_as_root(B);
    Monster_add_hp(B, 80);
    ...
    Monster_end_as_root(B);

(Calling `Monster_end` instead would require `buffer_end` call
subsequently, and is basically a violation of nesting).

### Nested Tables

Tables can be nested, for example the Mini field may have type
Monster table again (a recursive type):

    buffer_start(B);
    Monster_start(B);
    Monster_add_Hp(B, 80);
    Monster_start(B);
    Monster_hp_add(B, 81);
    ...
    Monster_mini_add(Monster_end(B));
    ...
    flatcc_builder_buffer_end(B, Monster_end(B));

The child Monster table may be created before the parent or as above
between the tables start and end. If created before, reference must be
stored until it can be added. The only requirement is that start and
end are balanced, that the sub-table is ended before the parent, and
that both are created in the same buffer (nested buffers can be created
while the parent buffer is still being created, similar to sub-tables,
so it is possible to mess this up):

    Monster_ref_t root, mini;

    buffer_start(B);
    Monster_start(B);
    Monster_hp_add(B, 81);
    mini = Monster_end(B);

    Monster_start(B);
    Monster_hp_add(B, 80);
    Monster_mini_add(B, mini);
    root = Monster_end(B);

    flatcc_builder_buffer_end(B, root)


Rather than adding a child table explicitly, it can be started and ended
as an operation on the field name, here with `Monster_Mini_start/end`:

    Monster_ref_t root;

    Monster_start(B);
    Monster_add_Hp(B, 80);
    Monster_mini_start(B);
    Monster_hp_add(B, 81);
    Monster_mini_end(B);
    root = Monster_end(B);

    flatcc_builder_buffer_end(B, root);

We can repeat the the table nesting as deep as we like, provided our
builder is willing to allocate enough stack space.

**Warning**: It is possible to use the wrong table type operations
between `start/end` - don't do that. It is a tradeoff between usability
and type safety.

Note that vectors, strings and structs map several standard operations
to a field name, for example `mytable_myfield_push(B, x)`. This is not the
case with table fields which only map `start/end/create` in part because it
would never terminate for recursive types and in part because each table
is different making a generic mapping rather complex and with very long
names.

A table may be created with a constructor, but it requires all
non-scalar objects to be references or pointers. Struct fields must be
pointers to zero padded structs, and strings, vectors and tables must be
references. The constructors are probably most useful for simple tables
with mostly scalar values (here we use the original Monster fields and
leaves out any we have invented for the sake of illustration):

IMPORTANT: objects can generally only be created within a buffer
context, i.e. after `buffer_start`. For example calling
`flatbuffers_uint8_vec_create` before `Monster_create_as_root`
technically violates this rule because the create call also starts the
buffer. It is, however, allowed at the top level. For nested buffers
(see later) this must be avoided because the vector would end up in the
wrong buffer.

    Monster_ref_t m;
    uint8_t invdata[4] = { 1, 2, 3, 4 };
    Vec3_t vec;

    flatbuffers_uint8_vec_ref_t inventory =
        flatbuffers_uint8_vec_create(B, invdata, 4);
    m = Monster_create(B, &vec, 150, 80, name, inventory,
        Color_Red, Any_as_NONE());
    flatcc_builder_buffer_create(m);

or

    Monster_create_as_root(B, &vec, 150, 80, name, inventory,
        Color_Red, Any_as_NONE());

## Packing tables

By reordering the fields, the table may be packed better, or be better
able to reuse an existing vtable. The `create` call already does this
unless the attribute `original_order` has been set. Unions present a
special problem since it is two fields treated as one and the type field
will generally waste padding space if stored in order:

To help pack unions better these can be added with the type
seperate from the value reference using `add_type(B, test.type)`,
`add_value(B, test)` where the value is only added if the type is
not `NONE`. The `add_type` should be called last since it is the
smallest type.

The same field should not be added more than at most once. Internal
reservations that track offset fields may overflow otherwise. An
assertion will fail in debug builds.

Required table fields will be asserted in debug builds as part of the
`end/create` call.  Only offset fields can have a required attribute.

The generated `monster_test_reader.h` from [monster_test.fbs] shows how
the default packing takes place in generated `create` calls, see for
example the typealias test. Note that for example vectors are stored
together with integers like `uint32` because references to vectors have
the same size as `uint32`.


## Strings

Strings can be added to tables with zero terminated strings as source

    Monster_start(B);
    ...
    Monster_name_create_str(B, "Mega Monster");
    Monster_end(B);

or strings potententially containing zeroes:

    #define MONSTER "Mega\0Monster"
    Monster_start(B);
    ...
    /* Includes embedded zero. */
    Monster_name_create(B, MONSTER, sizeof(MONSTER));
    Monster_end(B);

or zero terminated source up to at most `max_len` characters.

    #define MONSTER "Mega\0Monster"
    Monster_start(B);
    ...
    /* "Mega" */
    Monster_name_create_strn(B, MONSTER, 12);
    Monster_end(B);

The `create_str` and `create_strn` versions finds the string length via strlen
and strnlen respectively. `append_string` also has `_str/_strn` versions.

A string can also be created from an existing flatbuffer string in which
case the length is expected to be stored 4 bytes before the pointer in
little endian format, and aligned properly:

    Monster_name_clone(B, mybufferstring);

or, create a string at most 4 characters long starting at 0-based index
10, if present:

    Monster_name_slice(B, mybufferstring, 10, 4);

If index or index + len goes beyond the source, the result is truncated
accordingly, possibly resulting in an empty string.

A string can also be create independently. The above is just shortcuts
for that:

    flatbuffers_string_ref_t monster_name;
    monster_name = flatbuffers_string_create_str("Mega Monster");
    Monster_name_add(B, monster_name);

Strings are generally expected to be utf-8, but any binary data will be
stored. Zero termination or embedded control codes are includes as is.
The string gets a final zero temination regardless, not counted in the
string length (in compliance with the FlatBuffers format).

A string can also be constructed from a more elaborate sequence of
operations. A string can be extended, appended to, or truncated and
reappended to, but it cannot be edited after other calls including calls
to update the same string. This may be useful if stripping escape codes
or parsed delimiters, etc., but here we just create the same "Mega
Monster" string in a more convoluted way:

    flatbuffers_string_ref_t name;
    char *s;
    #define N 20
    Monster_start(B);
    ...
    flatbuffers_string_start(B);
    flatbuffers_string_append(B, "Mega", 4);
    flatbuffers_string_append(B, " ", 1);
    s = flatbuffers_string_extend(B, N);
    strncpy(s, "Monster", N);
    flatbuffers_string_truncate(B, N - strlen(s));
    name = flatbuffers_string_end(B);
    Monster_name_add(B, name);
    ...
    Monster_end(B);

`flatbuffers_string_create...` calls are also available when creating
the string separate from adding it to a table, for example:

    flatbuffers_string_h name;
    name = flatbuffers_string_create_str(B, "Mini Monster");

It is guaranteed that any returned the string buffer is zero filled and
has an extra zero after the requested length such that strlen can be
called on the content, but only the requested bytes may be updated.

Every call only returns the substring being added to the string in that
operation. It is also possible to call `flatbuffers_string_edit` to get a
modifiable pointer to the start of the string.

`flatbuffers_string_reserved_len(B)` returns the current string length
including any embedded zeroes, but excluding final zero termination. It
is only valid until `string_end` is called.

See [flatcc_builder.h] for detailed documentation. Essentially `extend`
reserves zeroed space on the stack and returns a buffer to the new
space, and truncate reduces the overall size again, and the string is
then given the final length and a zero termination at the end.

There is no endian conversion (except internally for the string length),
because UTF-8 strings are not sensitive to endianness.

Like tables, the string may be created while a parent container is being
constructed, or before.

Strings can also be used as vector elements, but we will get that when
discussing vectors.

## Structs

Structs in tables can be added as:

    Monster_pos_create(B, 1, 2, 3);

The above essentially does the following:

    Vec3_t *v;
    v = Monster_pos_start(B);
    Vec3_assign(v, 1, 2, -3.2);
    Monster_pos_end(B);

Some versions of the monster schema has extra test fields - these would
break the assign approach above because there would be extra arguments.
Instead we can rely on the zero intialization and assign known fields.

    Vec3_t *v;
    v = Monster_pos_start(B);
    v->x = 1, v->y = 2, v->z = -3.2;
    Monster_pos_end(B);

`Monster_pos_end_pe(B)` can be used when the struct is known to be
little endian (pe for protocol endian, meaning no conversion is necessary),
for example copied from an existing buffer, but then `clone` is a better
choice:

    Monster_pos_clone(B, &v);

When the struct is created alone for use as root:

    Vec3_ref_t root;
    root = Vec3_create(B, 1, 2, 3)
    flatcc_builder_buffer_create(B, root);

An existing struct can be added as:

    Vec3_t v;
    Vec3_assign(&v, 1, 2, 3);
    /* v does not have to be zero padded. */
    Monster_pos_add(B, &v);

When adding a struct that is already little endian, presumably from an
existing buffer, it can be cloned using:

    Monster_pos_clone(B, &v);

Clone assumes the source struct is both little endian and that padding
is already zeroed (example ignores error handling), and `end_pe`
does nothing.

    *Monster_pos_start(B) = v;
    Monster_pos_end_pe(B);

There are several assignment types that convert between host (native)
endianness and buffer endiannes. We use `pe` to indicate
`protocol_endian` rather than just `le` for `little endian` because it
allows us to change endianness to big endian in the the future and it
more clearly states the intention. While big endian is not allowed in
FlatBuffers, big endian structs may be useful in other network
protocols - but it is not currently supported because it would force
little endian platforms to support byte-swapping. The operations are:

`assign_from_pe`, `assign_to_pe`, `copy`, `copy_from_pe`,
`copy_to_pe`, `to_pe` and `from_pe`.

All the copy operations takes a const pointer as source, and
`to/from_pe` is just copy with same source and destination:

    Vec3_t v, v2;
    Vec3_assign_to_pe(&v2, 1, 2, 3);
    Vec3_copy_from_pe(Vec3_clear(&v), &v2);
    Vec3_to_pe(&v);

`from_pe` means from little endian to native endian, end `to_pe`
is the opposite. On little endian platforms all copy operations behave
the same and only move fields, not padding. `to/from_pe` conversion
will leave deprecated fields either as they were, or zero them because
the operation may be skipped entirely on protocol endian native platforms.

While struct fields cannot be deprecated officially, they are supported
if the schema compiler is flagged to accept then. The struct fields are
renamed and assigned 0 when using assign or copy, and assign / create has
no argument for them.

Because padding can carry noise and unintended information, structs
should be cleared before assignment - but if used as a source to copy
the padding is not copied so only the destation need to be zeroed.

If a struct is nested, the assign operation includes all fields as if
the struct was flattened:

    typedef struct Plane Plane_t;
    struct Plane {
        Vec3_t direction;
        Vec3_t normal;
    };
    Plane_t plane;
    Plane_clear(&plane);
    Plane_assign(&plane, 1, 2, 3, 7, 8, 9);

Structs can also be created standalone, similar to tables and vectors,
but FlatBuffers only support this when the struct is used as root.

Assuming Vec3 is declared as root, a buffer only holding a Vec3 struct
can be created using:

    Vec3_create_as_root(B, 1, 2, 3);

Important: do not store the above as a nested buffer - it would be
missing the vector size field. If `Monster_playground` is a ubyte vector
with `nested_flatbuffer` attribute, then
`Monster_playground_start/end_as_root` may be used.

Structs also support `start/end_as_root`. In this case `start` returns
the struct pointer, and `end_pe_as_root` is supported:

    Vec3_t *v;
    v = Vec3_start_as_root(B);
    v->x = 1, v->y = 2, v->z = 3;
    Vec3_end_as_root(B);

(Be careful with the different result codes since a tables `start_as_root`
returns an integer result code where 0 is success while a struct returns
a pointer that is null on failure.)

The following also creates a buffer at top-level, but it may also be
added as a nested buffer because the stack frame detects the nesting:

    Vec3_t *v;
    flatcc_builder_buffer_start(B);
    v = Vec3_start(B);
    v->x = 1, v->y = 2, v->z = 3;
    flatcc_builder_buffer_end(B, Vec3_end(B));

or
    flatcc_builder_buffer_start(B);
    ...
    Monster_start(B);
    flatcc_builder_buffer_start(B);
    v = Vec3_start(B);
    v->x = 1, v->y = 2, v->z = 3;
    Monster_playground_add(B,
        flatcc_builder_buffer_end(B, Vec3_end(B)));
    flatcc_builder_buffer_end(B, Monster_end(B));

or

    flatcc_builder_buffer_ref_t nested_root;
    flatcc_builder_buffer_start(B);
    nested_root = Vec3_create_as_root(B, 1, 2, 3);
    Monster_start(B);
    Monster_playground_add(B, nested_root);
    flatcc_builder_buffer_end(B, Monster_end(B));

A `buffer_ref_t` can be used as `uint8_vec_ref_t` when the
buffer is nested, and otherwise the reference cannot be used
for anything other than testing for failure. The buffer content
should match the type declared in a `nested_flatbuffers` attribute
but it isn't enforced, and a root can be stored in any field of
[ubyte] type.

When `Monster_playground` is declared as nested:

    ...
    Monster_start(B);
    Monster_playground_create_as_root(B, 1, 2, 3);
    flatcc_builder_buffer_end(B, Monster_end(B));
    ...

Be aware that `Vec3_t` is for native updates while `Vec3_struct_t` is a const
pointer to an endian encoded struct used in the reader interface, and actually
also as source type in the clone operation.

### Fixed Length Arrays in Structs

As of flatcc 0.6.0 it is possible to have fixed length arrays as structs
members. A fixed length array is equivalent to having a struct field repeated
one or more times. The schema syntax is `name : [type:count];` similar to an
ordinary struct field `name : type;`. The type is any type that can ba valid
struct field type including enums and nested structs. The size cannot be 0 and
the overall size is limited by the maximum struct size the array is contained
within which is typically 65535 (2^16-1).

For example, given the schema:

    struct MyStruct {
      counters:[int:3];
      // char is only valid as a fixed length array type
      name:[char:6];
    }
    table MyTable {
      mystruct:MyStruct;
    }

The table can be created with:

    ns(MyStruct_t) *x;
    ns(MyTable_start_as_root(B));
    x = ns(MyTable_mystruct_start(B));
    x->counters[0] = 1;
    x->counters[1] = 2;
    x->counters[2] = 3;
    strncpy(x->name, "Kermit", sizeof(x->name));
    ns(MyTable_mystruct_end(B));
    ns(MyTable_end_as_root(B));

Note that char arrays are not zero terminated but they are zero padded, so
strncpy is exactly the right operation to use when assigning to char arrays,
at least when they do not contain embedded nulls which is valid.
Char arrays are expected to be ASCII or UTF-8, but an application may use
other encodings if this is clear to all users.

With assignment:

    int data[3] = { 1, 2, 3 };
    ns(MyStruct_t) *x;
    ns(MyTable_start_as_root(B));
    x = ns(MyTable_mystruct_start(B));
    // Careful: the name argument does not use strncpy internally
    // so the source must be at least the expected length
    // like other array arguments. Strings can have embedded nulls.
    ns(MyStruct_assign(x, data, "Kermit");
    ns(MyTable_mystruct_end(B));
    ns(MyTable_end_as_root(B));

To read a struct the pointer to the struct is retrieved first

    int sum;
    int i;
    const char *name;
    size_t name_len;
    ns(MyTable_table_t) t;
    ns(MyStruct_struct_t) x;

    t = ns(MyTable_as_root(buf));
    x = ns(MyTable_mystruct_get(t));
    for (sum = 0, i = 0; i < ns(MyStruct_counters_get_len()); ++i) {
      sum += ns(MyStruct_counters_get(x, i)) +
      // char arrays are endian neutral, so we can use pointer access.
      name = ns(MyStruct_name_get_ptr(x);
      name_len = strnlen(name, ns(MyStruct_name_get_len()));
      printf("Added counters from %.*s", name_len, name);
      // char arrays can be accessed like other arrays:
      // ns(MyStruct_name_get(x, i);
    }

An alternative to `strnlen` is strip trailing zeroes which will allow for
char arrays embedded zeroes, but there is no direct support for this. The JSON
printer uses this approach to shorten the printed char array string.

The `_get` suffix can be ommitted in the above if the flatcc `-g` has not
supplied to reduce the risk of name conflicts, but not for `_get_len` and
`_get_ptr`.

Note that it is not possible to have fixed length arrays as part of a table but
it is possible to wrap such data in a struct, and it is also possible to have
vectors of structs that contain fixed length arrays.


## Nested Buffers

These are discussed under Structs and Table sections but it is worth
noting that a nested buffers can also be added as pe ubyte vectors
which is probably the original intention with nested buffers. However,
when doing so it can be difficult to ensure the buffer is correctly
aligned. The untyped `flatcc_builder` has various options to deal with
this, but with generated code it is better to create a nested buffer
inline when suitable (with nested `buffer_start/end` or
`mytable_myfield_create_as_root`) - for example a message wrapper with
a union of tables holding buffer for a specific message type. In other
cases the buffer may truly be created independently of the current
buffer and then it can be added with controlled alignment using either
the `flatcc_builder` api for full control, or the `nest` operation on
nested table and struct fields:

To create and add a ubyte vector with a higher alignment than ubytes
single byte alignment, the following operation is available as an
operation on a nested buffer field:

    Monster_playground_nest(B, void *data, size_t size, uint16_t align);

If alignment is unknown, it can be set to 0, and it will default to 8
for nested table types, and to the struct alignment for struct buffers.

Block alignment is inherited from the parent buffer so the child buffer
ends up in its own set of blocks, if block alignment is being used. If
the nested buffer needs a different block alignment, the `flatcc_builder`
api must be used.

All structs and tables have an `start/end/create_as_root` even if they
are not referenced by any `nested_flatbuffers` field and they will
create [ubyte] vectors containing a nested buffer but only [ubyte]
fields with `nested_flatbuffers` attribute will dedicated
`start/end/create_as_root` on the field name. Structs also have
`end_pe_as_root`.


## Scalars and Enums

Scalars keep their original type names `uint8_t`, `double`, etc, but
they get some operations similar to structs. These are contained in a
namespace which by default is `flatbuffers_`, for example:

    uint16_t *flatbuffers_uint16_to_pe(uint16_t *p);
    uint16_t *flatbuffers_uint16_from_pe(uint16_t *p);
    flatbuffers_bool_t *flatbuffers_bool_to_pe(flatbuffers_bool_t *p);
    flatbuffers_bool_t *flatbuffers_bool_from_pe(flatbuffers_bool_t *p);

These may be used freely, but are primarily present as an interface to
the vector operations also defined for structs.

Enums have similar definitions which may be used to convert endianness
without being concerned with the underlying integer type, for example:

    Color_enum_t *Color_to_pe(Color_enum_t *p);

## Vectors

Vectors can be created independently, or directly when updating a table - the
end result is the same. Builder vector operations always reference element
values by pointer, or by reference for offset types like tables and strings.

    uint8_t v;
    Monster_inventory_start(B);
    v = 1;
    flatbuffers_uint8_vec_push(B, &v);
    v = 2;
    flatbuffers_uint8_vec_push(B, &v);
    v = 3;
    flatbuffers_uint8_vec_push(B, &v);
    Monster_inventory_end(B);

or

    flatbuffers_uint8_vec_ref_t inv;
    uint8_t v;
    flatbuffers_uint8_vec_start(B);
    v = 1;
    flatbuffers_uint8_vec_push(B, &v);
    v = 2;
    flatbuffers_uint8_vec_push(B, &v);
    v = 3;
    flatbuffers_uint8_vec_push(B, &v);
    inv = flatbuffers_uint8_vec_end(B);
    Monster_inventory_add(B, inv);

Because it can be tedious and error-prone to recall the exact field
type, and because the operations are not type safe (any kind of push
would be accepted), some vector operations are also mapped to the field
name:

    uint8_t v;
    Monster_inventory_start(B);
    v = 1;
    Monster_inventory_push(B, &v);
    v = 2;
    Monster_inventory_push(B, &v);
    v = 3;
    Monster_inventory_push(B, &v);
    Monster_inventory_end(B);

Note: vector operations on a type uses the `_vec_<operation>` syntax, for
example `uint8_vec_push` or `Monster_vec_push` while operations that are mapped
onto table field names of vector type do not use the `_vec` infix because it is
not a type name, for example `Monster_inventory_push`.

A slightly faster operation preallocates the vector:

    uint8_t *v;
    Monster_inventory_start(B);
    v = Monster_inventory_extend(B, 3);
    v[0] = 1, v[1] = 2, v[2] = 3;
    v = Monster_inventory_extend(B, 2);
    v[0] = 4, v[1] = 5;
    Monster_inventory_end(B);

Push just extends one element at time.  Note that `extend` returns the
pointer to the extended vector segment. The full vector can be accessed
with `edit` and `reserved_len` between `start/end` (recalling that pointers
cannot be reused across buffer calls):

    uint8_t *v, i;
    uint8_t data[] = { 1, 2 };
    Monster_inventory_start(B);
    Monster_inventory_push(B, &data[0]);
    Monster_inventory_push(B, &data[1]);
    v = Monster_inventory_edit(B);
    for (i = 1; i < Monster_inventory_reserved_len(B); ++i) {
        v[i] = v[i - 1] + v[i];
    }
    Monster_inventory_end(B);

Note that the name `reserved_len` is to avoid confusion with
`_vec_len` read operation. It also indicates that it is not the final
size since it may change with `truncate/extend`.

A vector can also contain structs. Let us extend the Monster example
with a vector of positions, so we can have a breadcrumb trail:

    Monster_breadcrumbs_start(B);
    Vec3_vec_push_create(B, 1, 2, 3);
    Vec3_vec_push_create(B, 3, 4, 5);
    Monster_breadcrumbs_end(B);

or

    Monster_breadcrumbs_start(B);
    Monster_breadcrumbs_push_create(B, 1, 2, 3);
    Monster_breadcrumbs_push_create(B, 3, 4, 5);
    Monster_breadcrumbs_end(B);

or

    Vec3_t *trails[2];
    Monster_breadcrumbs_start(B);
    trails = Monster_breadcrumbs_extend(B, 2);
    Vec3_create(&trails[0], 1, 2, 3);
    Vec3_create(&trails[1], 4, 5, 6);
    Monster_breadcrumbs_end(B);

The `vec_start/exttend/end/end_pe/create/create_pe/clone/slice` are
translated into similar calls prefixed with the field name instead of
`vector` and except for `start`, the calls also add the vector to the
table if successful, for example:

    uint8_t data[] = { 1, 2, 3 };
    Monster_inventory_create(B, data, 3);
    Monster_breadcrumbs_slice(B, some_other_breadcrumbs, 0, 10);

Vector operations that are allowed between `vec_start` and
`vec_end(_pe)` are also mapped. These are
`vec_extend/append/truncate/edit/reserved_len`, and `push/push_create/push_copy`.
`push_copy` ensures only valid fields are copied, not zero padding (or
the unofficial deprecated fields).

A struct `push_clone` is the same as a `push_copy` operation
because structs are stored inline in vectors - with the
exception of union vectors which have `push_clone` that does the
right thing.

The `add` call adds a vector created independently from the table field,
and this is what is going on under the surface in the other calls:

    Vec3_t x;
    Vec3_vec_ref_t inv;

    /* Clear any padding in `x` because it is not allocated by builder. */
    Vec3_assign(Vec3_clear(&x), 3, 4, 5);
    Vec3_vec_start(B);
    Vec3_vec_push_create(B, 1, 2, 3);
    Vec3_vec_push(B, &v);
    inv = Vec3_vec_end(B);

    Monster_breadcrumbs_add(B, inv);

As always, a reference such as `inv` may only be used at most once, and
should be used once to avoid garbage.

Note that `Vec3_vec_start` would create an independent struct instead of a
vector of structs. Also note that `vec_ref_t` is a builder specific
temporary type while `vec_t` is intended as a const pointer to the first
element in an existing buffer in little endian encoding with a size
prefix (to be used with clone, for example).

An existing Vec3 struct can also be pushed with `Vec3_push(B, &v)`. The
argument must be zero padded. Because vectors are converted at the end,
there is no `push_pe`, but a struct may be in little endian using push
on all platforms if `vec_end_pe` is used at the end.

A vector may also be created from an existing array:

    uint8_t data[] = { 1, 2, 3 };
    Monster_inventory_add(B, flatbuffers_uint8_vec_create(B, data, 3));

This also applies to arrays of structs as long as they are properly zero
padded. `create_pe` is similar but does not do any endian conversion,
and is similar to `clone` except there are no header prefix.

Likewise an existing vector with proper zero padding may be appended
using the `extend` operation. The format must be native or little endian
depending on whether `vec_end` or `vec_end_pe` is called at the end.

All vectors are converted to little endian when the `end` command is
called. `end_pe` prevents this from happening.

`clone` and `slice` and can be used to copy an entire, or a partial
array from an existing buffer. The pointer must be to the first vector
element in little endian format, and it must have a size prefix and be
aligned (like any flatbuffer vector). `slice` takes a base-0 index and
a vector length where the result is truncated if the source is not
large enough.

    Monster_inventory_clone(B, v);

or

    Monster_inventory_add(flatbuffers_int8_clone(B, v);

or

    Monster_inventory_add(flatbuffers_int8_slice(B, v, 2, 4);

or

    Monster_inventory_slice(B, v, 2, 4);

A vector of strings an be constructed as (`friends` is a string
vector field that we just invented for the occasion):

    flatbuffers_string_ref_t friend, *p;
    Monster_friends_start(B);
      friend = flatbuffer_string_create_str(B, "Peter Pan");
      Monster_friends_push_create_str(B, "Shrek");
      Monster_friends_push_create_str(B, "Pinnochio");
      Monster_friends_push_create_str(B, "Pinnochio");
      Monster_friends_push_create(B, "Hector", 6);
      Monster_friends_push(friend);
      p = Monster_friends_extend(B, 1);
      *p = flatbuffers_string_create_str("Cindarella");
      Monster_friends_push_start(B);
        flatbuffers_string_append("The Little");
        flatbuffers_string_append("Mermaid");
      Monster_friends_push_end(B);
    Monster_friends_end(B);

Vectors and strings have a second argument to start, see also the `spawn` example
below.

Finally, vectors can contain tables. Table vectors are offset
vectors just like string vectors. `push_start` pushes a new table and
allows for updates until `push_end`. If we have a spawn vector of monsters in
the Monster table, we can populate it like this:

    Monster_spawn_start(B);
      Monster_vec_push_start(B);
        Monster_Hp_add(B, 27);
      Monster_vec_push_end(B);
      Monster_vec_push_create(B,
        /* Approximate argument list for illustration only. */
        &vec, 150, 80, name, inventory, Color_Red, Any_as_None());
    Monster_spawn_end(B);

The push operation has constructors `push_start/end/create` for both tables
struct, and string elements. String elements also have
`push_create_str/create_strn/clone/slice`. Structs also have
`push_copy`. Between `push_start` and
`push_end` the operations valid for the given table or string element can be
used (typically `add` for tables, and `append` for strings).

Instead of `Monster_vec_push_start` we can also uses
`Monster_spawn_push_start` etc. - in this case the child type is the
same as the parent, but using the field specific `push_start` ensures we
get the right table element type.

`Monster_spawn_push_start(B)` takes no length argument because it is a
table element, while `Monster_friends_push_start(B)` because it is a
string element (similar to a vector).

`Monster_spawn_start(B)` should just be followed by push operations
rather than following up with `Monster_spawn_extend(B, n)` because we
risk loose references that can lead to crashes. But handled carefully
it is possible:

    Monster_vec_ref_t mvec;
    Monster_spawn_start(B);
    mvec = Monster_spawn_extend(B, 2);
    mvec[0] = Monster_create(B, ...);
    mvec[1] = Monster_create(B, ...);
    Monster_spawn_end(B);

We can also push a reference to an independently create monster table,
all as seen before with strings.

As of flatcc version 0.5.2 it is also possible to clone tables.
Therefore we also have `push_clone` on vectors of tables.

While the use of `extend` and `truncate` is possible with vectors of
strings and tables, they should be used with care because the elements
are references and will just end up as garbage if truncated. On the
other hand, unused elements should be truncated as 0 elements in an
offset vector is not valid.

A vector of tables or strings can be created using an externally built
array of references, for example:

    Monster_ref_t monsters[20];
    Monster_vec_ref_t mvec;
    monsters[0] = Monster_create(B, ...);
    ...
    mvec = Monster_vec_create(B, monsters, 20);

By convention, create calls bypass the internal stack when the endian
format is otherwise compatible, and thus feed the emitter directly.
This is not possible with table and string vectors because the
references in the source vectors must be translated into offsets.
Therefore these create calls are similar to start, append, end calls.
There is an internal, but unexposed `flatcc_builder` version
`create_offset_vector_direct` which destroys the source vector instead
of allocating a stack copy.

## Unions

Unlike the C++ Flatbuffers library, we do not expose a separate union
type field except via a small struct with a union of typed references
and a type field. This struct is given to the create argument, and above
it is zero initialized meaning default None.

Unions can be created with value specific `start/end/create` calls. The add
call is not specialized since it takes a union reference:


    Monster_test_Weapon_start(B);
    Weapon_rounds_add(B, 50);
    Monster_test_Weapon_end(B);

or

    Monster_test_Weapon_create(B, 50);

or

    Monster_test_Weapon_add(B, Weapon_create(B, 50));

or

    Monster_test_Pickup_start(B);
    Pickup_location_create(B, 0, 0, 17);
    Pickup_hint_create_str(B, "Jump High!");
    Monster_test_Pickup_end(B);

or

    Pickup_ref_t test;
    Pickup_start(B);
    Pickup_location_create(B, 0, 0, 17);
    test = Pickup_end(B);
    Monster_test_add(B, Any_as_Pickup(test));

or

    Any_union_ref_t test;
    Pickup_start(B);
    Pickup_location_create(B, 0, 0, 17);
    /* test.Pickup = Pickup_end(B); no longer possible as of v0.5.0 */
    test.value = Pickup_end(B); /* As of v0.5.1. */
    test.type = Any_Pickup;
    Monster_test_add(B, test);

The following is valid and will not return an error, but also has no effect:

    Monster_test_add(B, Any_as_NONE());


_Note: the union structure has been changed for v0.5.0, and v0.5.1.
Both unions and union vectors are now represented by a struct with the
fields { type, value } in the low level interfaces. Before 0.5.0 only
unions of tables were supported._


### Union Vectors

The `monster_test.fbs` schema has a field named manyany in the Monster
table. It is vector of unions of type Any.

We can create a vector using

    Any_union_vec_ref_t anyvec_ref;

    Any_vec_start(B);
    Any_vec_push(TestSimpleTableWithEnum_create(B));
    anyvec_ref = Any_vec_end(B);
    Monster_manyany_add(anyvec_ref);

A union can be constructed with type specific `_push` or `_push_create` operations:

    Monster_manyany_start(B);
    Monster_manyany_push(B, Any_as_TestSimpleTableWithEnum(ref));
    Monster_manyany_end(B);

    Monster_manyany_start(B);
    Monster_manyany_TestSimpleTableWithEnum_push(B, ref);
    Monster_manyany_end(B);

    Monster_manyany_start(B);
    Monster_manyany_TestSimpleTableWithEnum_push_create(B, args);
    Monster_manyany_end(B);

and other similar operations, much like other vectors.

Note that internally `anyvec_ref` is really two references, one to type
vector and one to a table vector. The vector is constructed a single
vector of unions and later split into two before final storage. If it is
necessary to create a union vector from a vector of tables and types,
the low level builder interface has a `direct` call to do this.

Union vectos generally use more temporary stack space because during
construction because each element as a struct of type and reference
which don't back as densely as a two separate tables. In addition the
separated type and table vectors must be constructed temporarily. The
finaly buffer result is resonably compatct since the type vector does
not use much space. Unions will also be somewhat slower to construct,
but not unreasonably so.


### Unions of Strings and Structs

_Note: as of v0.5.0 unions can also contain strings and structs in
addition to tables. Support for these types in other languages may vary,
but C++ does support them too._

All union values are stored by reference. Structs that are not unions
are stored inline in tables and cannot be shared but unions of struct
type are stored by reference and can be shared. A union value is
therefore always a reference. This is mostly transparent because the
generated table field methods has `create/start/end` calls for each union
value type and addition to `add`.

To illustrate the use of these variation we use the Movie table from
[monster_test.fbs]:

    namespace Fantasy;

    table Attacker {
        sword_attack_damage: int;
    }

    struct Rapunzel {
        hair_length: uint16;
    }

    struct BookReader {
        books_read: int;
    }

    union Character {
        MuLan: Attacker = 2,  // Can have name be different from type.
        Rapunzel = 8,         // Or just both the same, as before.
        Belle: Fantasy.BookReader,
        BookFan: BookReader,
        Other: string,
        Unused: string = 255
    }

    table Movie {
        main_character: Character;
        antagonist: Character;
        side_kick: Character;
        cameo: Character;
        characters: [Character];
    }


and the mixed type test case from [monster_test.c]:


    nsf(Character_union_ref_t) ut;
    nsf(Rapunzel_ref_t) cameo_ref;
    nsf(Attacker_ref_t) attacker_ref;
    nsf(BookReader_ref_t) br_ref;
    nsf(BookReader_t *) pbr;
    nsf(Movie_table_t) mov;

    nsf(Movie_start_as_root(B));
    br_ref = nsf(BookReader_create(B, 10));
    cameo_ref = nsf(Rapunzel_create(B, 22));
    ut = nsf(Character_as_Rapunzel(cameo_ref));
    nsf(Movie_main_character_Rapunzel_create(B, 19));
    nsf(Movie_cameo_Rapunzel_add(B, cameo_ref));
    attacker_ref = nsf(Attacker_create(B, 42));
    nsf(Movie_antagonist_MuLan_add(B, attacker_ref));
    nsf(Movie_side_kick_Other_create_str(B, "Nemo"));
    nsf(Movie_characters_start(B));
    nsf(Movie_characters_push(B, ut));
    nsf(Movie_characters_MuLan_push(B, attacker_ref));
    nsf(Movie_characters_MuLan_push_create(B, 1));
    nsf(Character_vec_push(B, nsf(Character_as_Other(nsc(string_create_str(B, "other"))))));
    nsf(Movie_characters_Belle_push(B, br_ref));
    pbr = nsf(Movie_characters_Belle_push_start(B));
    pbr->books_read = 3;
    nsf(Movie_characters_Belle_push_end(B));
    nsf(Movie_characters_Belle_push(B, nsf(BookReader_create(B, 1))));
    nsf(Movie_characters_Belle_push_create(B, 2));
    nsf(Movie_characters_Other_push(B, nsc(string_create_str(B, "another"))));
    nsf(Movie_characters_Other_push_create_str(B, "yet another"));
    nsf(Movie_characters_end(B));
    nsf(Movie_end_as_root(B));

Note that reading a union of string type requires a cast which can be
seen in the full test case in [monster_test.c].
## Error Handling

The API generally expects all error codes to be checked but the
following table and vector operations will accept and return an error:

- `add` null reference to table, vector, or string.
- `push` null reference to table or string.
- `buffer_end/create` null reference to root.

This can simplify pushing or adding atomically created objects, for
example by adding a cloned vector to table field.

It is especially important to check start operations because the builder
will not be in the expected stack frame context after failure and will
not have reserved necessary internal memory, for example when adding a
table field.

On a server with reasonable amount of memory using the default
allocator, and with an emitter that will not return errors, and when it
can be expected that inputs will not exceed the size contraints of the
flatbuffer data types, and if the api is being used correctly, then there
are no reason for failure and error handling may be skipped. However,
it is sometimes desireable for servers to restrict a single clients
memory usage, and then errors are very likely unless the source data is
already limited. As an opposite example, an embedded device sending
small network packages using a fixed but large enough allocation pool,
would be in total control and need not be concerned with any errors.



## Type System Overview

The generated methods for building buffers may look the same but
have different semantics. For example `_clone` on a table field
such as `Monster_enemy_clone` will actually create a table based
on the content of a table in a another buffer, then add that
table to the currently open table. But `Monster_clone` will
create clone and just return a reference without adding the
reference to any table. There is also `push_clone` which adds
an element to an open vector. The same applies to many other
operations.

Basically there are
the following different types of methods:

- Methods on native flatbuffer types, such as
  `flatbuffer_string_start`.
- Methods on generated types types such as `Monster_start`
- Methods on field members such as as `Monster_emeny_start`
- Methods on vectors on vectors of the above such as
  `flatbuffers_string_vec_start`, `Monster_vec_start`.
  `Monster_inventory_vec_start`.
- Slight adaptions for buffer roots and nested buffer roots.

For unions and union vectors the story is more complex - and the
api might need to be cleaned up further, but generally there are
both union type fields, union value fields, and union fields
representing both, and vectors of the same. In additional there
are pseudo fields for each union member because `create` on a
union does not make sense, but
`Monster_myvariant_MyTable_create` does create and `MyTable`
table and assigns it with the correct type to the field
`Monster_myvariant_type` and `Monster_myvariant.


## Cloning

As of flatcc v0.5.2 it is also possible to clone tables, unions,
vectors of tables, vectors of strings, and vectors of unions.
Previously many operations did have a clone or a `push_clone`
operator, but these were all raw byte copies. Table cloning and
union cloning is signficantly more involved as it a simple copy
will not work due to stored references, possible sharing of
references and because the required alignment of table is hard
to reason about without building a new table. Unions and union
vectors are even more difficult.

That said, cloning is now implemented for all relevant data
types.

All clone operations expect the content to originate from
another finalized buffer. For scalars and structs there are
copy operations that are almost the same as clone - they both
avoid endian conversion.

Structs have a special case with clone and copy: Whenever a
struct is stored inline in the desitination buffer, it behaves
like copy. Whenever the destination is a buffer root, or a union
member, the result is a reference to an independent memory
block. When calling clone on a struct type the destination is
unknown and a indendpendent reference is created. If this is not
the intention a `copy` operation can be used. When used field
methods the destination type is known at the right thing will
happen.

Cloning a table will, by default, expand any shared references
in the source into separate copies. This is also true when
cloning string vectors, or any other data that holds references.
Worst case this can blow up memory (which is also true when
printing JSON from a buffer).

It is possible to preserve the exact DAG structure when cloning.
It may not worthwhile for simple use cases but it goes as
follows:

The builder has a pointer to a `flatcc_refmap_t` object. This is
a fairly small stack allocated object that implements a
hashtable. By default this pointer is null, and we have the
above mentioned expansion. If it is not null, each newly cloned
object will have its reference stored in the refmap. The next
time the same object is cloned, the existing reference will be
taken from the refmap instead. See source comments in
`flatcc_refmap.h` and `flatcc_builder.h`, and `monster_test.c`
clone tests.

Note that, for example, it might be relevant to preserve DAG
structure when cloning one object with all its sub-objects, but
if it is cloned a second time, a new copy is desired still while
preseving the inner DAG structure. This can be done by working
with multiple refmaps and simple swapping them out via
`flatcc_builder_set_refmap`. It is also possible to add
references manually to a refmap before cloning.

Warning: the refmap MUST not hold any foreign references when
starting a nested root clone or when cloning inside a nested
buffer that has been started but not ended because it is
invalid to share references between buffers and there are no
safety checks for this.


## Picking

Picking is a method that is related to clone and also introduced
with flatcc 0.5.2. A pick method is only defined on a table
field or a struct field. Instead of taking an a read reference
of same type as the field, it takes a reference to to the same
container type (table or struct). Essentially pick means: find
myself in the other table, clone me, and and me to the new table
which is currently open. So clone takes an entire table where
pick takes a single field. Table cloning is implemented as a
sequence of pick method, one for each field as can be seen in
the generated builder source. A pick operation does nothting if
the field is not set. Pick also works with refmaps because it
does an internal clone operation.  In the generated code, only
clone on types will use the refmap but other clone and pick
operations do depend on these type clone methods.


## Sorting Vectors

Vectors can be sorted, but not by the primary builder interface because:

String and table elements cannot be accessed after they have been
emitted. The emitter can do all sorts of async operations other than
actually building a buffer, for example encrypting blocks and / or send
partial buffers over the network. Scalars could be sorted, but the most
efficient way of emitting vectors does not create a temporary vector but
emits the source directly when endianess allows for it. Less
significant, the buffer producer is likely busy processing content and /
or on a resource constrained device. Altogether, it is much simpler to
not support sorting at this interface level.

To understand how sorting is implemented, lets first look at how an
already sorted vector can be searched:

Every vector of string, scalar and enum element types have a `find`
operation in the reader interface that performs a binary seach. Every
vector of table and struct elements have a `find_by_<field_name>` iff
there is a key attribute on at least one top-level scalar, enum or
string field type. FlatBuffers do not officially allow for multiple key
attributes, but if enabled, there will by a `find_by` operation for
every keyed element field. In addition there is a `find` operation that
maps to the first keyed field.

The read interface returns a vector type, which is a const pointer, when
accessing a table field of vector type. The find operation takes such a
vector as first argument, and a key as second. Strings have variations
to allow for keys with a given length (similar to strcmp vs strncmp).

This leads us to the sort interface:

Every `find` and `find_by` operation has a matching `sort` and `sort_by`
operation table and struct vectors maps `sort` to the first keyed
`sort_by` operation. The sort operation takes a non-const vector which
has the type name suffix `_mutable_vec_t`. These
vectors are not available via the reader interface and must be cast
explicitly from `_vec_t` to `_mutable_vec_t`. When this is done, the
vector can be sorted in-place in the buffer without any memory
allocation and without any recursion.



If the namespace is
`flatbuffers`, a string vector is sorted by:

    flatbuffers_string_vec_t vec;
    vec = ...;
    `flatbuffers_string_vec_sort((flatbuffers_string_mutable_vec_t)vec)`

Scalar and enum vectors have similar inline sort operations, for
example:

    flatbuffers_uint8_vec_sort(flatbuffer_uint8_mutable_vec_t vec);

For vectors of tables or structs the sort function is named by the key
field. Assuming the Monster table has a key attribute on the `Hp` field,
the following sort operation is available:

    MyGame_Example_Monster_vec_t monsters;
    monsters = ...;
    MyGame_Example_Monster_vec_sort_by_Hp(
        (MyGame_Example_Monster_mutable_vec_t)monsters);

Note: this is the reader interface. Any kind of `ref_t` type used by the
builder do not apply here. (Advanced: if an emitter builds a buffer, the
ref type can be used to find the actual vector pointer and then it can
be sorted by casting the pointer to a vector, even if the buffer isn't
finished).

Multiple keys per table or struct is an optional feature. Each key will
have its own sort and find function similar to the above. The first key
also has the shortcut:

    MyGame_Example_Monster_vec_sort(m);

The current implementation uses heap sort which is nearly as fast as
quicksort and has a compact implementation that does not require
recursion or external memory and is robust against DOS attacks by having
worst case O(n log n). It is, however, not a stable sort. The sort
assumes struct have a reasonable size so swap operations can be done
efficiently. For large structs a decicated sort operation building an
external index vector would be better, but this is not supported.

Note that a DAG is valid so there can be multiple vectors referring to
the same table elements, and each can be sorted by a different key.

The find operations are stable meaning they always return the lowest
index of any matching key or `flatbuffers_not_found` which is larger
than any other index.

### Dangers of Sorting

If a buffer was received over, say, an untrusted network the buffer
should be verified before being accessed. But verification only makes it
safe to read a buffer, not to modify a buffer because for example two
vectors can be crafted to overlap each other without breaking any
verification rules.

Thus, sorting is intended to be done shortly after the buffer is
constructed while it can still be trusted.

Using find on a buffer that is supposed to be sorted, but isn't, can
yield unexpected search results, but the result will always be a one
element in the vector being searched, not a buffer overrun.

### Scanning

Some vectors can be sorted by different keys depending on which version
version of `_sort_by` is being used. Obviously `_find_by` must match the
sorted key.

If we need to search for a key that is not sorted, or if we simply do
not want to sort the vector, it is possible to use scanning operations
instead by using `_scan` or `_scan_by`. Scanning is similar to find
except that it does a linear search and it supports scanning from a
given position.

More information on scanning in the
[README](https://github.com/dvidelabs/flatcc#searching-and-sorting)
file, and in the [monster_test.c] test file.


## Example of different interface type users

A resource constrained microcontroller is building flatbuffers from
sensor data using an emitter that sends UDP packages of the flatbuffer
as soon as enough data is ready. A server reassembles the packages or
discards them if any UDP package was lost. One the package is assembled,
the server sorts specific vectors such as temparture levels in the buffer
before it sends the buffer upstream to a storage service through a
TCP/IP connection. The analyzers perform taks such as detecting
abnormal temparature readings based on the sorted vector data.

In the above example, the original sensor devices are not interested in
the reader interface nor the sort interface. While the sort and find
operations may be available, it is dead inline code that does not
inflate the binary codes image size, but the libflatccrt library is
linked in. The collecting server is not interested in the builder
interface and does not link with the `libflatccrt` library but uses
both the inline functions of the reader intrface and the sort interface.
The upstream data storage service uses no interface at all since it
treats the buffers as binary blobs in a database indexed by device and
time. The end users only use the read only interface to visualize and
analyze and has no need for the builder or the sort interface.


## Special Emitters

An emitter only need to implement one function to replace or wrap the
default emitter. See [flatcc_builder.h] on `flatcc_builder_emit_fun` for
details, and also `emit_test.c` for a very simple custom emitter that
just prints debug messages, and [flatcc_emitter.h].

When adding padding `flatcc_builder_padding_base` is used as base in iov
entries and an emitter may detect this pointer and assume the entire
content is just nulls. Usually padding is of limited size by its very
nature so the benefit of handling this is also limited, but it, or a
similar user provided constants can be used for similar purposes:

When creating a vector in a single operation from an external C-array,
no copying takes place on the internal builder stack. Therefore it is
valid to provide a null pointer or a valid array such as
`flatcc_builder_padding_base` that is is too small for the given length,
provided that the emitter is aware of it. This in turn can be used to
allocate space in the emitters internal datastructure so the vector can
be filled after the fact if so desired. Pointer tagging may be another
way to communicate special intent. Be aware that only `create` calls
support this - any `append`, `start/end` or other dynamic operation will
require valid inpout and will stack allocate temporary space.

Emitters always receive a small table of iov entries that together form
a single object including necessary headers and padding, for example a
vector, a string, a nested buffer header, or a vtable. This is
guaranteed by the api, but there is no coordination to provide details
about which call is in order to keep the interface simple and fast. If
this is desired the user must hint the emitter out of band before
calling the relevant build operation. This can also be one indirectly by
setting `user_state` in the emitter and have the emitter inspect this
setting.

When adding vectors piecemeal using `append` or similar as opposed to
zero or less than zero copy approach above, the memory cost is obviously
higher, but unless the individual objects grow large, the stack will
operate in hot cpu cache so the bandwidth from main memory to cpu and
back will not necessarily double. If the stack grows large it may also
be worthwhile trimming the stack with a custom allocator and custom
builder reset between buffers to reduce stack size and initialization
overhead.

[monster_test.c]: https://github.com/dvidelabs/flatcc/blob/master/test/monster_test/monster_test.c
[flatcc_builder.h]: https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/flatcc_builder.h
[flatcc_emitter.h]: https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/flatcc_emitter.h
[monster_test.fbs]: https://github.com/dvidelabs/flatcc/blob/master/test/monster_test/monster_test.fbs