Battling with Go's Generics

A while back, I needed a small Go utility to fetch and unmarshal protobuf files.

Basically, I needed to read various types serialized protobuf files from S3. We had a half-dozen types of protobufs, and a bunch of boilerplate to read the different file types.

I wanted a simple utility, which could be used like:

fooFetcher := NewProtobufFetcher[FooProtobufRecord](bucket) 
protos, err := fooFetcher.fetch(filePaths) // returns list of type *FooProtobufRecord

The caller would specify the proto type, and the tool would unmarshal each file to that record type. The specifics aren’t terribly important, but I wanted it to be really fast.

And to make it fast, I wanted to use generics instead of reflection. Sounds easy, right?

Not so fast!

Surprisingly, it was… not so easy.

In my library, I wanted to convert raw bytes into a Go object using the Go protobuf library’s Unmarshal method:

func Unmarshal(b []byte, m Message) error

The argument m is what we’re writing into, and must be a pointer to a protobuf type. (It’s a type alias to the protoiface.MessageV1 interface.)

So I had some requirements for the library:

• The library user should specify the type of protobuf record using generics

• We library needs to instantiate the generic type in order to unmarshal a file into it

Naive Attempt: Use a Generic value type

type ProtoFetcher[T any] struct {
    ...
}

func NewProtobufQuerier[T any] () {
    return &ProtoFetcher[T]{
        ...
    }
}

func (q *ProtobufFetcher[T]) unmarshal(b []byte) (*T, error) {
    protoMsg := new(T) 
    err := proto.Unmarshal(b, &protoMsg) // invalid: Can't use 't' (type *T) as the type Message
    ...

    return msg, nil
}

This code fails on the call to unmarshal, because the Unmarshal function expects a type which implements protobuf’s Message interface. The type [T any] won’t work, we have to be restrict it somehow.

Worse, the new function expects a concrete type, not a pointer type.

So now we need both a generic that’s both:

• a concrete type

• able to be converted into a pointer type implements the Message interface.

Solution: Type Constraints

How can define such a weird constraint? This leads us down the dark rabbit hole of type constraints.

Type constraints are a deep subject, but the key point is that interfaces can be to restrict the set of types we allow. (I recommend this article for more info.)

To setup our type constraint, we define a new interface:

type ProtoMessageType[T any] interface {
    *T
    proto.Message
}

This interface says: The type T can be anything whose pointer type implements proto.Message.

So far, not too crazy.

Our struct, on the other hand, becomes pretty crazy:

type ProtobufFetcher[T any, PT ProtoMessageType[T]] struct {
    ...
}

Here T is the value type, while PT is the “pointer type” to the same value.

Now the unmarshal function looks like:

func (q *ProtobufFetcher[T, PT]) unmarshal(b []byte) (PT, error) {
    var msg PT
    msg = new(T)

    err := proto.Unmarshal(b, msg)
    if err != nil {
        return nil, err
    }
    return msg, nil
}

Aside from the bizarre type signature, not too bad! But let’s discuss some oddities.

The new function returns a pointer type of our concrete type T. We have to pre-declare the destination type msg as PT, because otherwise Go will just treat it as *T, which is sort of the same thing but without our type constraint.

At last, we can unmarshal the bytes into the proto object, and return the correct pointer type PT:

err := proto.Unmarshal(b, msg)
if err != nil {
    return nil, err
}
return msg, nil

Result

We now have a utility that produces protobuf records of a generic type.

It’s slick to use:

fetcher := NewProtobufFetcher[FooProtoRecord]()
records, err := fetch.FetchProtos(ctx, paths)

Note there’s a bit of extra type inference magic here, which let’s the user just specify the value type.

Was it worth the complexity?

Honestly, I’m not sure. The library is nice to use, but the bizarre [T, PT] generic types are confusing. I struggled to remember how it worked even as I wrote this.

This level of complexity counter to the spirit of Go. If I did it all over again, I might just create a type registry and do type conversions at runtime.

But it was an interesting learning experience, and I learned a bit about the sharp-edges of Go’s generics.