Skip to content

Question: How does this work with BigQuery Managed Writer #340

@newtonnthiga

Description

@newtonnthiga

Migrated recently to use Bigquery Storage Write API(https://cloud.google.com/bigquery/docs/write-api-batch#go), and was wondering how this package works with the new API. Infer schema works perfect but it conflicts with the Proto Descriptor which is now needed when writing to pending streams

        // We need to communicate the descriptor of the protocol buffer message we're using, which
        // is analagous to the "schema" for the message.  Both SampleData and SampleStruct are
        // two distinct messages in the compiled proto file, so we'll use adapt.NormalizeDescriptor
        // to unify them into a single self-contained descriptor representation.
        m := &exampleproto.SampleData{}
        descriptorProto, err := adapt.NormalizeDescriptor(m.ProtoReflect().Descriptor())
        if err != nil {
                return fmt.Errorf("NormalizeDescriptor: %w", err)
        }

        // Instantiate a ManagedStream, which manages low level details like connection state and provides
        // additional features like a future-like callback for appends, etc.  NewManagedStream can also create
        // the stream on your behalf, but in this example we're being explicit about stream creation.
        managedStream, err := client.NewManagedStream(ctx, managedwriter.WithStreamName(pendingStream.GetName()),
                managedwriter.WithSchemaDescriptor(descriptorProto))
        if err != nil {
                return fmt.Errorf("NewManagedStream: %w", err)
        }
        defer managedStream.Close()

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions