How to monitor Datameer core directories for certain thresholds?
Instead of using CLI commands
hadoop fs -du -s -h /user/datameer hadoop fs -du -h /user/datameer
and review the output, there should be more advanced solutions available. E.g. one could setup cron jobs to generate daily log files
hadoop fs -du -h /user/datameer > /logs/hdfsusage.log.`date +"%Y-%m-%d"`
You are able to leverage quotas for specific directories, as per documentation HDFS Quotas Guide. This should assist with the needs, i.e. for monitoring space thresholds in the Datameer core directory.
You may also have a look into distribution specific documentation.
Please sign in to leave a comment.