Recommendations under Container is Running Beyond Physical Memory Limits were followed and increasing Mapper and Reducer memory partitions to 16G did not improve the issue. The following error message is received:
task:org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException: Record too large for in-memory buffer. Exceeded buffer overflow limit, bufferOverflowRecursion=2, bufferList.size=1, blockSize=104857600
1. First add the following to the
Custom Properties of the job:
2. If the job still fails, remove the above property and add the following:
3. If execution also fails, attempt the job with both properties added.
Regarding the property
tez.runtime.io.sort.mb, you have a look into Hortonworks documentation about Configuring Tez and Demystify Apache Tez Memory Tuning - Step by Step.
Please sign in to leave a comment.