Frequently Asked Questions

Kafka

What is Kafka?

According to the Apache Kafka web site, “Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.”

In recent years, Apache Kafka has seen wide adoption by the astronomy community, notably by the Vera C. Rubin Observatory. GCN operates a highly available cluster of Kafka brokers in the cloud on Amazon Web Services that distributes GCN Notices to users.

The Kafka platform itself does not care about the format of the records. However, Avro and JSON are very common formats for Kafka records because of a rich ecosystem of open-source and commercial tools for them built on Kafka (for example: Kafka Streams, ksqldb, Kafka Connect). The Vera C. Rubin Observatory's alert system uses Avro over the Confluent Wire Format. GCN has developed a unified schema for communicating Notices as JSON records.

How do I start receiving GCN notices via Kafka?

See Kafka Client Setup.

What ports do I need to open in order to receive or send GCN notices with Kafka?

Clients connecting to GCN only need to be able to make outbound (egress) TCP connections. The client connects to the following hosts and ports.

How can I tell that my Kafka client is working?

Subscribe to the topic gcn.heartbeat, which broadcasts a test message approximately once a second. The contents of the heartbeat messages looks like this:

{
  "$schema": "https://gcn.nasa.gov/schema/v4.3.2/gcn/notices/core/Alert.schema.json",
  "alert_datetime": "2024-07-25T15:50:35.792451Z"
}

I have an intermittent or flaky Internet connection. How do I ensure that my Kafka client script won't miss messages?

A common mistake that we see is that users stop and restart their Kafka client script to try to resolve connectivity issues. We do not recommend this.

Instead, simply leave your client script running. As long as you leave your client running, the GCN Kafka broker will remember the last record that it sent to you by saving your consumer offset. Shortly after your Internet connection stabilizes, the broker will send your client any messages that were queued while your connection was down.

What does the warning Subscribed topic not available: gcn.classic.text.AGILE_GRB_GROUND: Broker: Unknown topic or partition' mean?

This warning means that there have not been any recent alerts on that topic.

As a GCN Notice producer, what is the largest message that I can send over Kafka?

The GCN Kafka brokers will accept messages up to about 4 MiB (4,194,304 bytes) in size. Please contact us if your mission requires larger messages.

Note that if your messages are 1 MB (1,000,000 bytes) or more in size, then you will need to set the message.max.bytes client configuration property when you create your Kafka producer. For example, in Python you would set that property as follows:

from gcn_kafka import Producer
producer = Producer(
    client_id='fill me in',
    client_secret='fill me in',
    **{'message.max.bytes': 4194304})

See GitHub issue confluentinc/librdkafka#3125.

After an hour, my Python Kafka producer script starts printing the error message %3|1746446717.457|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://kafka3.test.gcn.nasa.gov:9092/3]: sasl_ssl://kafka3.test.gcn.nasa.gov:9092/3: SASL authentication error: {"status":"invalid_token"} (after 419ms in state AUTH_REQ, 3 identical error(s) suppressed).

This is a known issue related to the Kafka client library's background thread that periodically renews access tokens. It will not affect your ability to produce Kafka messages because the access token will be automatically renewed the instant that your script produces a message. You can safely ignore it.

Suppressing the error message entirely requires an upstream patch for confluent-kafka-python.

How do I receive GCN Notices via email from GCN Classic over Kafka?

To get started, sign in or sign up and then select 'Email Notifications' from account dropdown menu. See also GCN Circular 32517.

Circulars

Why do GCN Circulars that I submit by email appear to be double spaced?

Some email clients in some configurations are known to add extra line breaks to emails. When you are preparing to submit a GCN Circular by email, you should always make sure that you are composing a plain text message. See the following instructions for various mail clients:

• Apple Mail
• Microsoft Outlook for Mac
• Microsoft Outlook for Windows

Note that it used to be common practice for Circulars submitters to add line breaks to manually wrap long paragraphs in GCN Circulars. This practice is no longer recommended in the GCN Circulars style guide.

Why do I receive duplicates of old GCN Circulars over Kafka on the gcn.circulars topic?

We publish a record to the gcn.circulars topic whenever a Circular is created or modified. A Circular can be modified in one of two situations:

• when a correction is made to an existing Circular, resulting a new version of it, or
• when bibliographic data is added to a Circular from the SAO/NASA Astrophysics Data System (ADS). GCN synchronizes with ADS every Monday at 08:00 UTC, resulting in a bulk update of many of the GCN Circulars from the preceding one-week period.

Accounts

How do I sign in as a legacy GCN Classic Circulars user?

If you had a GCN Circulars account prior to April 17, 2023, your account was migrated to the new GCN. Your receiving preferences, submitting permission, name and affiliation will be associated with your account as soon as you sign in for the first time. If you choose to sign in via email address and password, tap "Sign up" not "Forgot your password?".

Operations

Does GCN keep working during a U.S. federal government shutdown?

Yes. During a U.S. federal government shutdown, all GCN services (GCN Circulars, GCN Notices, Kafka cluster, the web site) remain fully operational. However, there may be the following minor impacts to GCN users:

• Responses to questions and requests through the GCN help desk may be delayed.
• Issues and pull requests on https://github.com/nasa-gcn repositories may not be promptly reviewed.
• There will be no redeployments of the GCN web site except to address emergencies.

Does GCN block IP addresses or connections from certain countries?

No. The GCN web site and Kafka brokers accept connections from anywhere in the world. GCN does not block any IP addresses based on geographic locations or countries of origin.