Data Streams

Bitly NSQ Data Streams allow services or customers to receive real-time traffic for a large number of links via NSQ. Access is restricted to whitelisted accounts. For access contact api@bit.ly.

See also documentation of Public Bitly Data

/v3/nsq/lookup

This is an API endpoint for connecting to Bitly provided NSQ datastreams.

go-nsq usage

cfg := nsq.NewConfig()
cfg.TlsV1 = true
cfg.AuthSecret = "$ACCESS_TOKEN"
cfg.MaxInFlight = 1000
c := nsq.NewConsumer(topic, channel, cfg)
c.SetHandler(....)
lookup := "https://api-ssl.bitly.com/v3/nsq/lookup?access_token=$ACCESS_TOKEN"
c.ConnectToNSQLookupd(lookup)
<- c.StopChan

pynsq usage

lookup = "https://api-ssl.bitly.com/v3/nsq/lookup?access_token=$ACCESS_TOKEN"
r = nsq.Reader(message_handler=handler,
        lookupd_http_addresses=[lookup],
        auth_secret="$ACCESS_TOKEN",
        tls_v1=True,
        max_in_flight=1000,
        topic=..., 
        channel=....
    )
nsq.run()

Built in NSQ Conmmand Line Utilities usage (nsq_to_file, nsq_tail, etc)

./nsq_tail --lookupd-http-address="https://api-ssl.bitly.com/v3/nsq/lookup?access_token=$ACCESS_TOKEN" \
   --reader-opt="auth_secret,$ACCESS_TOKEN" \
   --reader-opt="tls_v1,true" \
   --max-in-flight=1000 \
   --topic=.... --channel=....

ERRORS

/v3/nsq/stats

This lists NSQ Topic and Channel Message Information and Connection State for a Topic.

To calculate size of message processing queue, sum() channel depth and channel deferred_count across all producers.

Parameters

  • topic - NSQ Data Stream Topic

Return Values

  • topics - A list of authorized NSQ topics.
  • channels - A list of channels that each get a copy of messages.
    • producer - which producer this producer (nsqd instance) this is for.
    • depth - Current number of messages that have not yet been sent to a Consumer for processing
    • in_flight_count - Current number of messages sent to a consumer that have not yet finished processing. This number is not counted towards depth.
    • deferred_count - Current number of messages requeued after a consumer attempted processing which have not yet been re-processed.
    • timeout_count - Lifetime total messages that were requeued due to timeout waiting for consumer response.
    • message_count - Lifetime total messages seen since producer (nsqd instance) start
    • paused - If message flow on this channel has been administratively paused.
  • clients - The individual connections to a channel that a consumer recieves messages over. Each
    • requeue_count - Connection lifetime total messages requeued
    • message_count - Connection lifetime total messages processed
    • connect_ts - timestamp for when this connection was initiated
    • remote_address - where this connection was initiated from.

Example Request

API Address: https://api-ssl.bitly.com
GET /v3/nsq/stats?topic=data_stream&access_token=ACCESS_TOKEN

Example Response

{
  "data": {
    "topics": [
      {
        "channels": [
          {
            "backend_depth": 140910, 
            "channel_name": "bitlyapioauthdemo", 
            "clients": [], 
            "deferred_count": 0, 
            "depth": 165911, 
            "in_flight_count": 0, 
            "message_count": 214456483, 
            "paused": false, 
            "producer": "publicnsq04.ec2.bitly.net:4151", 
            "requeue_count": 7, 
            "timeout_count": 7
          }, 
          {
            "backend_depth": 768709, 
            "channel_name": "bitlyapioauthdemo", 
            "clients": [
              {
                "auth_identity": "bitlyapioauthdemo", 
                "client_id": "hostname", 
                "connect_ts": 1404395711, 
                "deflate": false, 
                "finish_count": 1143, 
                "hostname": "hostname.local", 
                "in_flight_count": 1, 
                "message_count": 1144, 
                "name": "hostname", 
                "ready_count": 0, 
                "remote_address": "127.0.0.1:63890", 
                "requeue_count": 0, 
                "sample_rate": 0, 
                "snappy": true, 
                "state": 3, 
                "tls": true, 
                "user_agent": "nsq_tail/0.2.29-alpha go-nsq/1.0.0-alpha", 
                "version": "V2"
              }
            ], 
            "deferred_count": 0, 
            "depth": 793710, 
            "in_flight_count": 1, 
            "message_count": 251043535, 
            "paused": false, 
            "producer": "publicnsq05.ec2.bitly.net:4151", 
            "requeue_count": 54, 
            "timeout_count": 54
          }
        ], 
        "producers": [
          "publicnsq04.ec2.bitly.net:4151", 
          "publicnsq05.ec2.bitly.net:4151"
        ], 
        "topic_name": "data_stream"
      }
    ]
  }, 
  "status_code": 200, 
  "status_txt": "OK"
}

ERRORS