MapBufferTooSmallException: Record too large for in-memory buffer

Problem

Recommendations under Container is Running Beyond Physical Memory Limits were followed and increasing Mapper and Reducer memory partitions to 16G did not improve the issue. The following error message is received:

task:org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTooSmallException: Record too large for in-memory buffer. Exceeded buffer overflow limit, bufferOverflowRecursion=2, bufferList.size=1, blockSize=104857600

Solution

1. First add the following to the Custom Properties of the job:

• tez.runtime.io.sort.mb=1024

2. If the job still fails, remove the above property and add the following:

• tez.runtime.sorter.class=LEGACY

3. If execution also fails, attempt the job with both properties added.

Background Information

Regarding the property tez.runtime.io.sort.mb, you have a look into Hortonworks documentation about Configuring Tez and Demystify Apache Tez Memory Tuning - Step by Step.