How to Enable Tez History/UI for Hadoop?

We've found the following setup working

Hadoop Cluster

• Since your Hadoop cluster needs a running Timeline Server, check that the following properties are enabled in yarn-site.xml

  ...
  <property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
  </property>
  ...
  <property>
  <name>yarn.timeline-service.hostname</name>
  <value>localhost</value><!-- configure a proper hostname that is available for all nodes -->
  </property>
  ...
  <property>
  <name>yarn.timeline-service.http-cross-origin.enabled</name>
  <value>true</value>
  </property>
  ...
  <property>
  <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
  <value>true</value>
  </property>
  ...

• It runs per default on port 8188
• If it is not running, start it with ./sbin/yarn-daemon.sh start timelineserver

On Datameer Application Server

• Download and unpack Apache Tomcat 8.5.20
• Configure the HTTP port in conf/server.xml e.g. 8280
• Download and unzip Apache Tez 0.7.1 binaries
• Delete everything under TOMCAT_HOME/webapps/*
• Unzip the tez-ui-0.7.1.war into TOMCAT_HOME/webapps/ROOT
• Edit TOMCAT_HOME/webapps/ROOT/scripts/configs.js
  • timelineBaseUrl must point to the Timeline Server (ATS)
  • RMWebUrl must point to the Resource Manager (RM)
• Double check that JAVA_HOME is set, (It should be set as it's a pre-requirement for Datameer)
• Start the server with TOMCAT_HOME/bin/catalina.sh start

• Configure a test job with Hadoop Custom Properties and Debug Logging Implemented

  das.execution-framework=Tez
  das.debug.tasks.logs.collect.force=true
  tez.task.log.level=DEBUG
  tez.am.log.level=DEBUG
  tez.allow.disabled.timeline-domains=true
  tez.am.history.logging.enabled=true
  tez.dag.history.logging.enabled=true tez.history.logging.service.class=org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
  tez.tez-ui.history-url.base=http://<thisHost>:8080/#/main/view/TEZ/tez_cluster_instance
  yarn.timeline-service.enabled=true
  yarn.timeline-service.hostname=<ATS> 

• If you have a Kerberos Secured cluster you should configure the cluster in the yarn-site.xml as described in section Security Configuration

  yarn.timeline-service.principal=datameer@EC2.INTERNAL
  yarn.timeline-service.keytab=/home/datameer/datameer.keytab 

Test

• Execute test job
• Gather and review logs, yarn-site.xml, Tomcat/catalina.log, full job trace
• Check that everything is working properly