Notes

Google protocol buffers

protobuf is developed to serialize and retrieve structured data with high efficiency.

Basics

// addressbook.proto
syntax = "proto3";

package tutorial;

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}
protobuf

  • message

    like a class, contains fields and subclasses.

  • field modifiers:

    • required: must be provided.
    • optional: if not set, use the default, if default is not set, use the system default.
    • repeated: this field may be repeated for 0 to any times. (like dynamic arrays)

  • compile

    protoc -I=$src_dir --python_out=$src_dir $src_dir/addressook.proto

    -I is short for --proto_path

    protoc --python_out=./ *.proto

    this generate *_pb2.py files in the python_out directory.

  • API

    import addressbook_pb2
person = addressbook_pb2.Person()
person.name = "John Doe" # req
person.id = 1234 # req
person.email = "jdoe@example.com" # opt

phone = person.phones.add() # add a repeated field
phone.number = "555-4321"
phone.type = addressbook_pb2.Person.HOME # or simply `phone.type = 1`
    python

Proto3

Message

can be viewed as class in python.

Field

  • field type

    • bool, string, double, float, int32, int64, uint32, uint64
    • sint32, sint64 (signed int, more efficient in coding negative integers than regular int32 and int64)
    • enumeration
    • other self-defined message type

  • field number

    each field has a unique number in the message.

    best in [1, 15] for only one byte to encode. (Not use 0)

    the exact range is [1,229−1][1, 2^{29}-1] \ [19000, 19999]

  • field modifier

    • singular (by default)

      proto3 removed required, so there is only optional singulars.

    • repeated

  • enum types

    the first enum should start from 0.

    exact range is uint32

    enum Corpus {
    UNIVERSAL = 0; // which is the default value too.
    WEB = 1;
}

enum EnumAllowingAlias {
  option allow_alias = true; // allow same enum for a number.
  UNKNOWN = 0;
  STARTED = 1;
  RUNNING = 1;
}
enum EnumNotAllowingAlias {
  UNKNOWN = 0;
  STARTED = 1;
  // RUNNING = 1;  // Uncommenting this line will cause a compile error inside Google and a warning message outside.
}
    protobuf

comments

same as C/C++.

reserved field

message Foo {
  reserved 2, 15, 9 to 11;
  reserved "foo", "bar";
}
protobuf

you should reserve the field number or name for your deleted fields, in case of later modifications reuse these fields and cause bugs.

import

// old.proto
import public "new.proto"
import "other.proto";
protobuf

when old.proto is imported by another proto file saying client.proto,

client.proto also import new.proto, but doesn't import other.proto.

​

package

python will ignore package declarations, since python modules are organized according to locations in file system.

oneof

oneof is a type that means the message will only record one of its fields or subclasses.

message SampleMessage {
  oneof test_oneof {
    string name = 4;
    SubMessage sub_message = 9;
  }
}
protobuf
SampleMessage message;
message.set_name("name");
CHECK(message.has_name());
message.mutable_sub_message();   // Will clear name field.
CHECK(!message.has_name());
java

Type to search.