7

I have a large file (200 MB upwards). I need to transfer it via PHP cron job. Using Phpseclib gives the following error:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 4133 bytes) in /app/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php

Is there a way I can do this with PHP cron job?

The code is simple one line where $localFile is an already existing CSV file 

$sftp->put('/Import/coupons/coupons_import_test.csv', $localFile, NET_SFTP_LOCAL_FILE);

PS. This needs to be done after PHP generates that file in the /tmp folder so timing of the transfer script has to come into play.

[Edit] I do not intend on increasing the memory limit as the files later could be of higher sizes. A solution where I can transfer the file in parts (append mode) or use some shell script with PHP cron could be worthwhile

The file size on remote server is 111.4 MB while the actual file is much larger on local.

[Edit after the fix] The issue disappeared after changing to version 2.0.2 from version 1.0 I had to modify the code for put 

$sftp->put('/Import/coupons/coupons_import.csv', $localFile, $sftp::SOURCE_LOCAL_FILE);
Abhishek
  • 1,440
  • 3
  • 18
  • 34
  • 1
    Possible duplicate of [Allowed memory size of 33554432 bytes exhausted (tried to allocate 43148176 bytes) in php](http://stackoverflow.com/questions/415801/allowed-memory-size-of-33554432-bytes-exhausted-tried-to-allocate-43148176-byte) – Jamie Bicknell Jul 05 '16 at 13:51
  • What does your php.ini have as **memory_limit**? Usually something like 64M or 128M, you need to increase this. – ggdx Jul 05 '16 at 14:00
  • Is there a way to do this without increasing the limit? – Abhishek Jul 05 '16 at 14:09
  • What is your version of `Phpseclib`? – revo Jul 14 '16 at 09:21
  • you can do it with a shell script in more simple way, you dont need php and it's limitations. if you consider making sh i can help with a proper answer – Santa's helper Jul 14 '16 at 12:43
  • I asked this before and didn't get an answer so I'll ask it again (since you just posted the bounty): can you post some code? Also, how much of the file is uploaded by the time you get the memory error? Also, Pawel Dubiel's idea is a good one too so try that and then, if not, maybe answer my question. – neubert Jul 14 '16 at 17:00
  • Version of Phpseclib is 1.0. I have edited question for code ( single line of code). – Abhishek Jul 15 '16 at 20:42
  • Maybe try open an issue here https://github.com/phpseclib/phpseclib and post logs over there. – Pawel Dubiel Jul 16 '16 at 03:04
  • Can you also try to transfer this file to different server to see if it's the same problem? – Pawel Dubiel Jul 16 '16 at 13:58
  • I think you should see this. Hope this helps http://unix.stackexchange.com/questions/190537/transfering-large-8-gb-files-over-ssh – Lokesh Pandey Jul 18 '16 at 12:08

3 Answers3

2

Phpseclib should do just fine transferring big files with no need to increase available memory.

I think you probably hit the old bug "SSH2: don't count data length against window size". You most likely may be using older version of the Phpseclib ( The older faulty version is bundled even with relatively new software like for example Magento 1.9.* )

Check your version if is not the latest redownload it from https://github.com/phpseclib/phpseclib

Pawel Dubiel
  • 14,123
  • 3
  • 37
  • 54
  • I am using phpseclib 1.0 by composer require on a heroku set up. – Abhishek Jul 15 '16 at 20:43
  • I don't know if it was that bug in version 1.0 which you pointed out which was causing the issue, but when I changed to version 2.0.2 the issue disappeared. I though have to modify the code a bit. – Abhishek Jul 18 '16 at 14:26
  • @aeonsleo - What version of 1.0 were you using? Seems like `composer update` might have also fixed the issue if 2.0.2 did.. – neubert Jul 18 '16 at 16:06
1

In lieu of seeing the code that you're using my guess is that you're trying to load the 200mb file as a string and you're trying to upload it as a string. eg. $sftp->put('filename.remote', file_get_contents('filename.local'));. If so then try this instead:

$sftp->put('filename.remote', 'filename.local', NET_SFTP_LOCAL_FILE);
neubert
  • 14,208
  • 21
  • 90
  • 172
  • I am doing exactly the way you have mentioned. – Abhishek Jul 06 '16 at 20:59
  • 2
    To comment further I'd need to see some code. You can edit it into your orig post. Also, how much of the file is uploaded by the time you get the memory error? – neubert Jul 06 '16 at 21:20
0

This actually changed a bit with phpseclib version >= 2.0.5. You should include the namespace use phpseclib\Net\SFTP; , then do something like this:

$sftp->put('filename.remote', 'filename.local', SFTP::SOURCE_LOCAL_FILE);
Bitclaw
  • 775
  • 8
  • 8