SFTP Export Job fails with NullPointerException

Problem

Datameer SFTP ExportJob fails with the following NullPointerException during finalization of export process.

INFO [2018-03-14 05:18:57.990] [JobExecutionPlanRunner] (CsvOutputAdapter.java:261) - moving export result from 'sftp://john.smith@example.com@ec2-54-216-187-5.eu-west-1.compute.amazonaws.com:22/datadir/.export-test_export__adapter_/test_export_00000' to 'sftp://john.smith@example.com@ec2-54-216-187-5.eu-west-1.compute.amazonaws.com:22/datadir/test_export_00000'
DEBUG [2018-03-14 05:18:58.058] [JobExecutionPlanRunner] (SftpFileSystem.java:56) - Directory listing entry found: -rw-r--r--    1 514      513         79405 Mar 14 09:18 test_export_00000
ERROR [2018-03-14 05:18:58.063] [JobExecutionPlanRunner] (ClusterSession.java:252) - Failed to run cluster job 'Export job (15): ExportJobTest1#export(Export record processor)' [29 sec]
java.lang.NullPointerException
at datameer.dap.hadoop.filesystem.SftpFileSystem.rename(SftpFileSystem.java:113)
at datameer.dap.common.csv.CsvOutputAdapter.finalizeExport(CsvOutputAdapter.java:262)
at datameer.dap.common.graphv2.hadoop.ServerSideContext.finalizeExport(ServerSideContext.java:65)

Cause

The problem might be caused by the username that contains '@' character. In this case URI will be sftp://user@name@host:port/datadir/ and can't be properly handled by the underlying code, which leads to the NullPointerException. SFTP Import however might work under this account.

According to the standard RFC3986:

• 3.2.1. User Information

The userinfo subcomponent may consist of a user name and, optionally, scheme-specific information about how to gain authorization to access the resource. The user information, if present, is followed by a commercial at-sign ("@") that delimits it from the host.

• Secure File Transfer Protocol (SFTP) URI

sftpURI       =  "sftp:" hier-part
   hier-part     =  "//" authority path [";" s-param *("," s-param)]
   path          =  path-abempty
   path-abempty  =  <as specified in [RFC3986]>
   authority     =  [ ssh-info "@" ] host [ ":" port ]
   host          =  <as specified in [RFC3986]>
   port          =  <as specified in [RFC3986]>
   ssh-info      =  [ userinfo ] [";" c-param *("," c-param)]
   userinfo      =  <as specified in [RFC3986]>
   c-param       =  paramname "=" paramvalue
   paramname     =  *( ALPHA / DIGIT / "-" )
   paramvalue    =  *( ALPHA / DIGIT / "-" )
   s-param       =  paramname "=" paramvalue

Solution

Use another account that complies the standards to connect to a SFTP server as a workaround.

Underlying code will be reviewed for future improvements.