How to Change the Stack Size for a Tez Task

Goal

It might happen that a Datameer execution fails with java.lang.StackOverflowError exception.

Error: Encountered an Error while executing task: attempt_1526467016480_0456_1_02_000000_0:java.lang.StackOverflowError
at org.apache.hadoop.io.WritableComparator.readLong(WritableComparator.java:226)
at org.apache.hadoop.io.WritableComparator.readDouble(WritableComparator.java:232)
at org.apache.hadoop.io.DoubleWritable$Comparator.compare(DoubleWritable.java:95)
at datameer.plugin.tez.processing.AggregationVertexRecordProcessor$3.readNextKeyValue(AggregationVertexRecordProcessor.java:140)
at datameer.dap.common.graphv2.hadoop.KeyValueSequenceAdapter.moveCursorForward(KeyValueSequenceAdapter.java:39)
at datameer.dap.common.graphv2.hadoop.KeyValueSequenceAdapter.access$000(KeyValueSequenceAdapter.java:18)
at datameer.dap.common.graphv2.hadoop.KeyValueSequenceAdapter$1.computeNext(KeyValueSequenceAdapter.java:68)
at datameer.dap.common.graphv2.hadoop.KeyValueSequenceAdapter$1.computeNext(KeyValueSequenceAdapter.java:61)
at datameer.dap.sdk.sequence.Sequence$Simple.moveToNext(Sequence.java:158)
at datameer.dap.common.graphv2.ValueStream.moveToNext(ValueStream.java:31)
at datameer.dap.sdk.sequence.Sequence$9.moveToNext(Sequence.java:479)
at datameer.dap.common.graphv2.ValueStream.moveToNext(ValueStream.java:31)

A likely reason for this failure is that workbook's logic requires an optimization. A temporary workaround is that you can increase the stack size for Tez tasks and try to rerun the artefact.

Learn

To change stack size for a Tez task, introduce the custom property tez.task.launch.cmd-opts with appropriate value for -Xss parameter.

You could start with 2MB or 4MB as a stack size is usually relatively small.

tez.task.launch.cmd-opts=-Xss2m
tez.task.launch.cmd-opts=-Xss4m
tez.task.launch.cmd-opts=-Xss8m