Frequently Asked Questions

What is Kafka?

According to the Apache Kafka web site, “Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.”

In recent years, Apache Kafka has seen wide adoption by the astronomy community, notably by the Vera C. Rubin Observatory. GCN operates a highly available cluster of Kafka brokers in the cloud on Amazon Web Services that distributes GCN Notices to users.

The Kafka platform itself does not care about the format of the records. However, Avro and JSON are very common formats for Kafka records because of a rich ecosystem of open-source and commercial tools for them built on Kafka (for example: Kafka Streams, ksqldb, Kafka Connect). The Vera C. Rubin Observatory's alert system uses Avro over the Confluent Wire Format. For these reaons, GCN is developing a unified schema for communicating Notices in Avro or JSON records over the Confluent Wire Format (see roadmap).

How do I start receving GCN notices via Kafka?

See Client Configuration.

What ports do I need to open in order to receive or send GCN notices with Kafka?

Clients connecting to GCN only need to be able to make outbound (egress) TCP connections. The client uses the following ports.

Protocol	Direction	Port	Purpose
TCP	outbound	443	HTTPS
TCP	outbound	9092	Kafka

What does the warning `Subscribed topic not available: gcn.classic.text.AGILE_GRB_GROUND: Broker: Unknown topic or partition'` mean?

This warning means that there have not been any recent alerts on that topic.

How do I receive GCN Notices via email from GCN Classic over Kafka?