Follow

How To: Ingest Data with Non-ISO 8601 Timestamps

By default, we expect all date values in data to be ingested to match the ISO 8601 standard (http://www.iso.org/iso/home/standards/iso8601.htm). If you have data that does not match this, it may still be possible to ingest it, following the guidelines below to modify your Flume configuration.

 

We expect ISO-8601 out of the box. (<flume config path>.timeFormat=iso8601)

  • ISO 8601 is at the minimum '2016-11-19T14:45:15' or 'yyyy-mm-dd'T'hh:mm:ss'.
  • We can handle the 'T' part being a space ('2016-11-19 14:45:15' or 'yyyy-mm-dd hh:mm:ss').
  • We also handle there being milliseconds after the seconds ('2016-11-19T14:45:15.789' or 'yyyy-mm-dd'T'hh:mm:ss.SSS' as well as '2016-11-19 14:45:15.789' or 'yyyy-mm-dd hh:mm:ss.SSS').
  • We also can handle there being a timezone at the end (such as -05:00, -0500, or 'Z', for UTC, such as '2016-11-19T14:45:15-05:00' or '2016-11-19T14:45:15.789-05:00' or '2016-11-19 14:45:15.789-0500' or '2016-11-19T14:45:15Z').
    • If there is no timezone info, we assume the timezone of the machine we're running on

We can also handle epoch (1479584954) and epoch milli (1479584954789) via '<flume config path>.timeFormat=epoch' and'<flume config path>.timeFormat=epoch_milli' respectively.

We also allow definition of completely custom formats, by specifying them at the same config key. The formats are defined here by Java, under the 'Patterns for Formatting and Parsing' section.

For example, we can parse "19 Nov 2016 2:43 PM EST" with the config '<flume config path>.timeFormat=dd MMM yyyy hh:mm a zzz'.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk