Cannot create a data link

Follow

Michael Ahn

• October 13, 2016 12:59

When I try to create a datalink with Datameer 6.1.5 I get an exception when I press the final Save button (after entering the file name):

[anonymous] ERROR [2016-10-13 14:51:18.932] [qtp563628874-34] (DasExceptionFilter.java:43) - Unhandled exception handling /import-job/create/save
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -7
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:979)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at datameer.dap.conductor.webapp.filter.MinifyJsCssFilter.doFilter(MinifyJsCssFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.HeaderInjectFilter.doFilter(HeaderInjectFilter.java:24)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.CsrfFilter.doHttpFilter(CsrfFilter.java:87)
at datameer.dap.conductor.webapp.filter.CsrfFilter.doFilter(CsrfFilter.java:66)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.SessionFilter.doFilter(SessionFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RestAuthenticationOriginFilter.doFilter(RestAuthenticationOriginFilter.java:52)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.OptimisticLockRetryFilter.doFilter(OptimisticLockRetryFilter.java:32)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PermissionDeniedExceptionFilter.doFilter(PermissionDeniedExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PersistenceFilter.doFilter(Unknown Source)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:157)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter$InternalFilterChain.doFilter(CompositeAuthenticationFilter.java:64)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter$InternalFilterChain.doFilter(CompositeAuthenticationFilter.java:68)
at datameer.dap.conductor.authentication.CompositeAuthenticationFilter.doFilter(CompositeAuthenticationFilter.java:49)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.webapp.controller.fte.AccessTokenBasedAuthenticationProcessingFilter.doFilter(AccessTokenBasedAuthenticationProcessingFilter.java:94)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at datameer.dap.conductor.webapp.filter.AutoAdminAuthenticationFilter.doFilter(AutoAdminAuthenticationFilter.java:65)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.ClickJackingPreventionFilter.doFilter(ClickJackingPreventionFilter.java:40)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:118)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.OptimisticLockRetryFilter.doFilter(OptimisticLockRetryFilter.java:32)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.PermissionDeniedExceptionFilter.doFilter(PermissionDeniedExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.DeprecatedRestUrlsFilter.doFilter(DeprecatedRestUrlsFilter.java:90)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.BrowserExceptionFilter.doFilter(BrowserExceptionFilter.java:31)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RestExceptionFilter.doFilter(RestExceptionFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.NoResultExceptionFilter.doFilter(NoResultExceptionFilter.java:24)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.DasExceptionFilter.doFilter(DasExceptionFilter.java:29)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.webapp.filter.RequestContextFilter.doFilter(RequestContextFilter.java:43)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at datameer.dap.conductor.authentication.FailedLoginServletFilter.doFilter(FailedLoginServletFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at com.jamonapi.http.JAMonJettyHandlerNew.handle(JAMonJettyHandlerNew.java:36)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -7
at java.lang.String.substring(String.java:1875)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.createPatternPathAppendix(StaticPartitionDefinition.java:61)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.<init>(StaticPartitionDefinition.java:49)
at datameer.dap.common.partition.filerange.StaticPartitionDefinition.forDataSource(StaticPartitionDefinition.java:201)
at datameer.dap.common.partition.filerange.FileRangePartitionProvider.<init>(FileRangePartitionProvider.java:36)
at datameer.dap.common.partition.filerange.FileRangePartitioning.createPartitionProvider(FileRangePartitioning.java:23)
at datameer.dap.common.partition.filerange.FileRangePartitioning.createPartitionProvider(FileRangePartitioning.java:8)
at datameer.dap.common.partition.extension.DatameerPartitioningExtension.partitionProvider(DatameerPartitioningExtension.java:31)
at datameer.dap.conductor.webapp.commandObject.DataSourceCommand.createPartitionDefinition(DataSourceCommand.java:420)
at datameer.dap.conductor.webapp.commandObject.DataSourceCommand.fillDataSource(DataSourceCommand.java:412)
at datameer.dap.conductor.webapp.controller.data.SaveController.save(SaveController.java:80)
at datameer.dap.conductor.webapp.controller.data.importjob.DataSourceW7SaveController.saveDatasource(DataSourceW7SaveController.java:79)
at datameer.dap.conductor.webapp.controller.data.importjob.DataSourceW7SaveController.submitSave(DataSourceW7SaveController.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:177)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
... 125 more

0

• 

  Konsta Danyliuk

  • October 13, 2016 13:16
  • Edited

  Hello Michael.

  Mentioned error might be caused by some mismatches in format of dataset you are truing to load.
  
  

  • Could you please clarify what exactly data are you trying to ingest into Datameer by this DataLink?
    
    
  • Are you able to get correct Data Preview while setting up the DataLink?
    
    
  • Please try to check Drop record strategy under How to handle invalid data section and check if this helps the job to run without the error.
    
    

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 13, 2016 13:24

  I try to link to AVRO data I exported from a Workbook through a HDFS connection. I see the correct data preview in the data link wizard. Changing invalid data handling makes no difference.

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 13, 2016 13:31
  • Edited

  Michael, thank you for additional information.

  What if you will create a base line test for this activity, to narrow down the root cause:

  • create a very simple dataset (lets say 5 or 10 records).
  • export it into HDFS in avro format.
  • ingest it back via the DataLink.
    
    

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 13, 2016 13:40

  I tried with a very simple table:

  

  Almost the same error:

  [anonymous] ERROR [2016-10-13 15:37:55.508] [qtp563628874-15226] (DasExceptionFilter.java:43) - Unhandled exception handling /import-job/create/save
  org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -2

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 13, 2016 13:45
  • Edited

  Thank you, Michael.

  • Do you face the same issues with any other files export from HDFS?
  • Could you save this tiny avro file at local machine and try to upload it directly?

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 13, 2016 13:49

  Doing an import instead of DataLink works with the HDFS connection. But I need the DataLink.

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 13, 2016 14:01
  • Edited

  Just to confirm if I properly understand - you are able to ingest mentioned avgo file from HDFS over an ImportJob, but DataLink gives you the error, right?

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 13, 2016 14:03

  Yes

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 13, 2016 14:50

  Michael.

  Can you try the following, please

  • Create new folder in HDFS
  • Create new HDFS connection and point it into this folder
  • Copy mentioned avro file (or just the test one) into this folder
  • Create new Data Link and give it another try to ingest the file.
    
    

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 14, 2016 06:15

  I tried it with a new HDFS connection, did not work. I tried the same file with an ssh connection, did not work. I tried other file format (Excel), did not work. I tried another user (admin), did not work. I just cannot create any datalink, I always get that exception on save.

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 14, 2016 09:16

  Any news about this? Looks like a bug to me.

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 14, 2016 09:18

  Hello Michael.

  Could you please let me know what Datameer version you are using and what is your Hadoop distribution?

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 14, 2016 09:21

  Datameer 6.1.5 with CDH 5.6.1

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 14, 2016 09:32

  Michael,

  Let me arrange few tests at my environment.

  btw, have you tried to restart Datameer?

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 14, 2016 09:42

  Of course. I also tried with a fresh installed Datameer and failed as well.

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 14, 2016 10:12

  Just made few tests at fresh installed 6.1.5 (but for another Hadoop distribution HDP 2.4) - everything works fine, and I was able to create and use DataLinks for HDFS and Hive.

  To get some more inputs, could you please try to switch Datameer to embedded cluster mode, setup connection to Datameer server file system and try to create a DataLink.  

   

   

  0

  Comment actions Permalink
• 

  Michael Ahn

  • October 14, 2016 10:54

  I found the reason for the exception: The file paths.

  I used "/data" as the path in the connection and "/" as path in the datalink. It does also not work the other way round. But now I'm using "/data" in the connection and "/mydir" in the datalink and it works. So I have a workaround, but you should be able to reproduce the bug and open a ticket.

  0

  Comment actions Permalink
• 

  Konsta Danyliuk

  • October 14, 2016 11:18

  Hello Michael.

  Good to hear that you were able to find workaround. Thank you for pointing us to this issue, I'll check the problem with engineers.

   

   

  0

  Comment actions Permalink

Please sign in to leave a comment.