Goal
Go beyond Finding the optimal split size / split count and learn how to configure the number of mapper and reducers.
Learn
The property das.splitting.max-split-count applies to mappers only.
As Datameer Custom Properties outlines, it is possible to configure the number of reducers. If you want to enforce a specific reduce count, set the property das.job.reduce.tasks to greater then zero (0).
If the reducer count should be adapted based on the split-count (map-task count), set the property
das.job.reduce.tasks.correct=true
Be aware that setting either property can influence the performance impact on job execution.
Comments
0 comments
Please sign in to leave a comment.