Follow

Analytics failed and following error is outputted in the analytics.log "System times on machines may be out of sync. Check system time and time zones."

 

Issue

Analytics has failed, and the following error message is outputted in the analytics.log:

  • Application application_1516727049801_0114 failed 2 times due to Error launching appattempt_1516727049801_0114_000002. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
  • This token is expired. current time is 1516964105592 found 1516943446969
  • Note: System times on machines may be out of sync. Check system time and time zones.
  • at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  • at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  • at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  • at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  • at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
  • at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
  • at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122)
  • at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250)
  • at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  • at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  • at java.lang.Thread.run(Thread.java:748)
  • . Failing the application.

Cause

The cause for the failed Analytics run is due to a time drift that is greater than 10 minutes between the Resource and Node Managers, where each node may not be synchronized to time server.

Resolution Steps

In order to restore run Analytics again, time must be validated and synchronized accordingly on the MASTER and COMPUTE NODE(s).

Validate time on MASTER NODE(s)

  1. SSH to the MASTER NODE(s) as the Interset User
  2. Type in the following to check the date/time that is currently set:
    • date
  3. If the time is off, please correct the time by performing one of the following:
    • Set correct time manually (this is a temporarily fix as time drift will occur in the future)
      • EXAMPLE:
        • date -s "19 APR 2012 11:14:00"
    • Re-synchronize node time with NTP
      • EXAMPLE:
        • For EL6:
          • sudo service ntpd stop
          • sudo service ntpd start
        • For EL7:
          • sudo systemctl stop ntpd
          • sudo systemctl start ntpd
      • NOTE If NTP is not installed, please install NTP
        • EXAMPLE:
          • sudo yum install ntp
    • If it is VM, and VMTools is used, please ensure time synchronization is enabled
      • EXAMPLE:
        • cd /usr/sbin
        • vmware-toolbox-cmd timesync status 

Validate time on COMPUTE NODE(s)

  1. SSH to the COMPUTE NODE(s) as the Interset User
  2. Type in the following to check the date/time that is currently set:
    • date
  3. If the time is off, please correct the time by performing one of the following:
    • Set correct time manually (this is a temporarily fix as time drift will occur in the future)
      • EXAMPLE:
        • date -s "19 APR 2012 11:14:00"
    • Re-synchronize node time with NTP
      • EXAMPLE:
        • For EL6:
          • sudo service ntpd stop
          • sudo service ntpd start
        • For EL7:
          • sudo systemctl stop ntpd
          • sudo systemctl start ntpd
      • NOTE If NTP is not installed, please install NTP
        • EXAMPLE:
          • sudo yum install ntp
    • If it is VM, and VMTools is used, please ensure time synchronization is enabled
      • EXAMPLE:
        • cd /usr/sbin
        • vmware-toolbox-cmd timesync status

Kick off Analytics

Once the time issue has been resolved, please attempt to kick off an Analytics run to ensure analytics is successful in submitting the jobs, and running properly. 

  1. SSH to the MASTER NODE (where ANALYTICS is installed) as the Interset User
  2. Type in the following to kick off an Analytics run:
    • sudo -H -u spark -i /opt/interset/analytics/bin/analytics.sh /opt/interset/analytics/conf/interset.conf 

Applies To

  • Interset 5.4.x or higher 
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk