Reason behind long running datalink creation jobs
What could be the reason behind the long run of data link import jobs? Please provide a deep knowledge regarding this.
-
Hi Mohit,
It sounds like you are referring to the execution time of an import job or a workbook and not the time to create a link, correct? Here is an article that outlines techniques to optimize. Let us know if that answers your question.
https://datameer.zendesk.com/hc/en-us/articles/205084424-How-to-Optimize-a-Workbook
-
Hello Mohit,
The answer will also depend on the involved infrastructure, components (CPU, RAM, disk, network, file format, data structure) and the configuration of them.
Lets assume your workbook has to read data from disk on a remote system, transfer the data over the network and write results to the local disk. Depending on the disk subsystems and IOps, the network performance, the amount of data, the overall load on your systems, the runtime may be affected.
The same would be true for other resources like CPU and RAM. A good point to start with this complex theme might be general documenation and Hadoop: The Definitive Guide, Fourth Edition by Tom White.
You can gain insight into a Datameer job run by accessing the particular log files. More information about our logs can be found in Log Basics, Job Trace and Debug Logging.
Please sign in to leave a comment.
Comments
2 comments