According to the Apache Kafka web site, “Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.”
In recent years, Apache Kafka has seen wide adoption by the astronomy community, notably by the Vera C. Rubin Observatory. GCN operates a highly available cluster of Kafka brokers in the cloud on Amazon Web Services that distributes GCN Notices to users.
The Kafka platform itself does not care about the format of the records. However, Avro and JSON are very common formats for Kafka records because of a rich ecosystem of open-source and commercial tools for them built on Kafka (for example: Kafka Streams, ksqldb, Kafka Connect). The Vera C. Rubin Observatory's alert system uses Avro over the Confluent Wire Format. For these reaons, GCN is developing a unified schema for communicating Notices in Avro or JSON records over the Confluent Wire Format (see roadmap).
See Client Configuration.
Clients connecting to GCN only need to be able to make outbound (egress) TCP connections. The client uses the following ports.
What does the warning
Subscribed topic not available: gcn.classic.text.AGILE_GRB_GROUND: Broker: Unknown topic or partition' mean?
This warning means that there have not been any recent alerts on that topic.
This feature is under development and will be released in the near future. Currently GCN Classic over Kafka serves GCN notices only via the Kafka broker, which you can setup via the Start Streaming GCN Notices quick start guide.