Reason behind long running datalink creation jobs

Comments

2 comments

  • Venu Reddy

    Hi Mohit,

    It sounds like you are referring to the execution time of an import job or a workbook and not the time to create a link, correct? Here is an article that outlines techniques to optimize. Let us know if that answers your question.

    https://datameer.zendesk.com/hc/en-us/articles/205084424-How-to-Optimize-a-Workbook 

     

    0
    Comment actions Permalink
  • Gido

    Hello Mohit,

    The answer will also depend on the involved infrastructure, components (CPU, RAM, disk, network, file format, data structure) and the configuration of them. 

    Lets assume your workbook has to read data from disk on a remote system, transfer the data over the network and write results to the local disk. Depending on the disk subsystems and IOps, the network performance, the amount of data, the overall load on your systems, the runtime may be affected. 

    The same would be true for other resources like CPU and RAM. A good point to start with this complex theme might be general documenation and Hadoop: The Definitive Guide, Fourth Edition by Tom White.

    You can gain insight into a Datameer job run by accessing the particular log files. More information about our logs can be found in Log Basics, Job Trace and Debug Logging

    0
    Comment actions Permalink

Please sign in to leave a comment.