Analytics in "Windowed" mode has a notion of a "peek-back" window. This window exists as a best-effort buffer to ensure that delayed or out-of-order data isn't missed entirely between windowed runs.
The default value for the peekback window is 24 hours.
What this means is that when a windowed analytics job runs, it will load data starting from the end of the last run, minus 24H. For example, if the previous windowed run started at midnight and completed successfully at noon today, the next run will load data beginning from noon yesterday. The intention being to include data that may have been ingested after midnight, but with event timestamps between noon and midnight of the previous day.
The peek-back window can be adjusted as follows. Note that values specified are in milliseconds, with a default of "86400000.0" (24 hours):
- As the interset user, log in to the Master node where Interset Analytics is installed.
- Run /opt/interset/analytics/bin/sql.sh --action console
Note: If using Kerberos, you will need to run kinit and specify a keytab/principal with appropriate permissions in HBase before this command will function.
- In the Phoenix console enter the following commands, replacing VALUE_IN_MS_HERE with the desired value of the peekback window, specified in milliseconds (e.g. 86400000.0):
UPSERT INTO PARAMETERS(TID, NAME, VAL) VALUES('0', 'SCORE WINDOW BUFFER', VALUE_IN_MS_HERE);
UPSERT INTO PARAMETERS(TID, NAME, VAL) VALUES('0', 'AGGREGATE WINDOW BUFFER', VALUE_IN_MS_HERE);
- Type !quit to exit the console