Workbook fails after Upgrade from 6.3.3 to 6.4.4
I have a workbook which is automatically starte every hour. After upgrading Datameer to version 6.4.4 it fails:
ERROR [2018-02-07 14:02:10.711] [JobExecutionPlanRunner] (ClusterSession.java:252) - Failed to run cluster job 'Workbook job (3720106): tweet_analysis#evaluated_tweets(Disconnected record stream), TweetExportNew(' [23 sec] datameer.com.google.common.base.VerifyException: Finished DAG 'Workbook job (3720106): tweet_analysis#evaluated_tweets(Disconnected record stream), TweetExportNew( (2d554b4c-1ac2-4fbe-b69d-109010cd3858)' (application_1517996577677_0640) with state FAILED and diagnostics: [Vertex failed, vertexName=Map for sheets:[Map side join processor, evaluated_tweets, TweetExportNew, Group by step 1, PartitioningRecordFunction{}] (4ce86072-9690-4c19-84cc-91f9ee7d2c79), vertexId=vertex_1517996577677_0640_9_03, diagnostics=[Task failed, taskId=task_1517996577677_0640_9_03_000001, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3 at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132) at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131) at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163) at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65) at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27) at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025) at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3 at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132) at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131) at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163) at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65) at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27) at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025) at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3 at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132) at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131) at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163) at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65) at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27) at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025) at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.ArrayIndexOutOfBoundsException: 3 at datameer.dap.sdk.common.Record.getFieldContent(Record.java:132) at datameer.dap.common.graphv2.RecordDistributor$SelectedColumnsDistributor.distribute(RecordDistributor.java:131) at datameer.plugin.tez.processing.TezPartitioner.getPartition(TezPartitioner.java:26) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:284) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163) at datameer.plugin.tez.processing.VertexOutput$Partitioned.write(VertexOutput.java:65) at datameer.plugin.tez.processing.CombinedVertexOutput.write(CombinedVertexOutput.java:22) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:38) at datameer.plugin.tez.processing.VertexOutput$VertexOutputConsumer.accept(VertexOutput.java:27) at datameer.dap.sdk.sequence.Sequence.forEach(Sequence.java:1025) at datameer.plugin.tez.processing.SimpleVertexProcessor.run(SimpleVertexProcessor.java:121) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:2, Vertex vertex_1517996577677_0640_9_03 [Map for sheets:[Map side join processor, evaluated_tweets, TweetExportNew, Group by step 1, PartitioningRecordFunction{}] (4ce86072-9690-4c19-84cc-91f9ee7d2c79)] killed/failed due to:OWN_TASK_FAILURE], Vertex killed, vertexName=Reduce for sheets:[Group by operation, CompactTweetExport] (ca31e534-516c-430e-bd2f-ef8a16ba0795), vertexId=vertex_1517996577677_0640_9_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:400, Vertex vertex_1517996577677_0640_9_04 [Reduce for sheets:[Group by operation, CompactTweetExport] (ca31e534-516c-430e-bd2f-ef8a16ba0795)] killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1] at datameer.com.google.common.base.Verify.verify(Verify.java:125) at datameer.plugin.tez.TezJob.runTezDag(TezJob.java:178) at datameer.plugin.tez.TezJob.runImpl(TezJob.java:152) at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:115) at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:229) at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:343) at datameer.dap.common.graphv2.JobExecutionPlanRunner.run(JobExecutionPlanRunner.java:122) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:120) at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:197) at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34) at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at datameer.dap.common.impersonation.ClusterAwareUgiImpersonator.doAs(ClusterAwareUgiImpersonator.java:43) at datameer.dap.common.impersonation.ConfigurableImpersonator.doAs(ConfigurableImpersonator.java:34) at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30) INFO [2018-02-07 14:02:10.711] [JobExecutionPlanRunner] (ClusterSession.java:255) - -------------------------------------------
-
Hi Mark, you can find the job traces here: #14127
-
For future readers, the resolution for this issue is summarized in a KB article: https://datameer.zendesk.com/hc/en-us/articles/360001912632-Post-Upgrade-6-4-3-Workbook-Fails-ArrayIndexOutOfBoundsException-
Please sign in to leave a comment.
Comments
3 comments