Kafka

Overview

The Kafka connector joins a consumer group for the topics you choose and commits offsets only after a run has accepted each message. Messages are decoded with your Schema Registry, so each one arrives as a typed event with its key, headers and partition in the trace.

Teams use it to start fraud checks from a payments topic, to land order events in Snowflake through a workflow and to publish Postgres changes to a topic that other services consume.

Supported clusters

• Apache Kafka 2.8 to 4.0, self-hosted

• Confluent Cloud, Amazon MSK, Aiven and Redpanda

• Avro, Protobuf and JSON Schema through a Schema Registry

How consumption works

Tessel runs a consumer per connection in your region and scales it to the number of partitions. It reads with read_committed isolation, so aborted transactions never start a run, and it writes with an idempotent producer, so retries do not duplicate messages.

bootstrap.servers=broker-1.kafka.internal:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
group.id=tessel-orders
isolation.level=read_committed
auto.offset.reset=earliest
enable.idempotence=true

bootstrap.servers=broker-1.kafka.internal:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
group.id=tessel-orders
isolation.level=read_committed
auto.offset.reset=earliest
enable.idempotence=true

bootstrap.servers=broker-1.kafka.internal:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
group.id=tessel-orders
isolation.level=read_committed
auto.offset.reset=earliest
enable.idempotence=true

Ordering

Messages with the same key land on the same partition and are processed in order. Different keys run in parallel, up to the concurrency limit you set on the trigger.

Permissions needed

Create a dedicated principal for Tessel with ACLs scoped to its topics and group. It needs read access to the topics it consumes. Write access is only required if a workflow produces messages.

• READ and DESCRIBE on each consumed topic

• READ on the consumer group, such as tessel-orders

• WRITE on topics you produce to, plus IDEMPOTENT_WRITE on older clusters