Lumen help

Log Streaming and Management

Log Delivery Formats

Message Format: JSON

Either JSON messages separated by newline characters or a JSON formatted array messages are used for Log Streaming.

A simple JSON example is as follows (new lines and beginning of line spacing added for clarity, not in final message):

                  {
    "cached":1,
    "cs(Cookie)":"-",
    "cs(Referer)":"http://www.test.com/some/referrer/path",
    "cs(User-Agent)":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)
     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
    "cs-ip":"127.75.23.245",
    "cs-method":"get",
    "cs-uri":"/test/test.gif?auth=ouAoARQAFoKCAAQABgAIAAoAGAAaR0b24xLnNq",
    "cs-version":"HTTP/1.1",
    "date":"2017-07-12",
    "sc(Content-Type)":"-",
    "sc-bytes":383,
    "status":200,
    "time":"22:27:10",
    "time-taken":0.01,
    "x-disid":2124407,
    "x-headersize":340,
    "cs(Range)":"bytes=0-123456”,
    "x-tcpinfo_rcv_space": 28960,
    "x-tcpinfo_rtt": 14000,
    "x-tcpinfo_rttvar": 7000,
    "x-tcpinfo_snd_cwnd": 32,
    "x-tcwait":245,
    "x-tdwait":0.003
  }
            

There is no set number of messages in a batch sent for each POST.  Messages will be sent when either the Max Bytes/Message (in bytes) or when the Max Post Interval (in seconds) has been reached since the last HTTP POST.  If there are no messages to be sent in a batch, then the system will skip sending on that interval and wait until there is at least one message in the batch. This is done for each process, currently one per machine. Both configuration parameters have defaults defined by Lumen for optimal performance.

The log streaming JSON encoding escapes common HTML characters by default. The following characters are mapped to provided escaped Unicode:

  • ‘&’ : ‘\u0026’

  • ‘<’ : ‘\u003c’

  • ‘>’ : ‘\u003e’

  • U+2028 (Unicode line separator) : ‘\u2028’

  • U+2029 (Unicode paragraph separator) : ‘\u2029’
     

These can commonly appear inside URL query strings or HTML parts sent through the log streaming system:

        --   “/example?key=value&hello=world” becomes “/example?key=value\u0026hello=world”

        --   “<b>hello world!<\b>” becomes “\u003cb\u003ehello world! \u003c\b\u003e”

It should be noted that customers with large traffic volumes can create a significant volume of logs and must provide a log endpoint capable of ingesting at a high rate. Failure to provide such an endpoint may result in logs being dropped as the CTL log processing system only queues logs for a short period of time if the endpoint is unavailable or unable to keep up.

the log file format follows the W3C Extended Log File Format defined by the World Wide Web Consortium. By default, messages will be available once the MaxBytesPerFile threshold is met (default set to 1 Gig) or when the MaxWaitSecForFinalWrite is met (default to 15-minute duration) (whichever occurs first).

A simple W3C log example is as follows:

                #Version: 1.0
#Software: 4.0
#Fields: date time cs-ip cs-method cs-uri status sc-bytes time-taken cs(Referer)
cs(User-Agent) cs(Cookie) x-Custom x-LogGroup x-disid x-extstatus x-headersize cache
cs(Range) x-tcwait x-tcpinfo_rtt x-tcpinfo_rttvar x-tcpinfo_snd_cwnd x-tcpinfo_rcv_space
x-tdwait sc(Content-Type) cs-version
2017-07-23 12:10:02.319 10.10.10.10 GET
http://www.example.com/index.html 200 25574 0.003 “-”
“user-agent/1.1” “-” “-” y 153001 0 803 MISS
“-” 0 29058 3486 162 28960 3 “text/html” “HTTP/1.1”
            

Note: the use of the #Software: 4.0 log line. This indicates what version of logging is enabled for the customer. Lumen currently offers multiple W3C log formats – V1, V3 and V4 (default).

Log Delivery

Default Log Fields

Optional Log Fields

Deprecated log fields

Activating CDN Log Collection and Delivery

Limitations