Can't Run Binary From Within Python Aws Lambda Function

November 30, 2022 Post a Comment

I am trying to run this tool within a lambda function: https://github.com/nicolas-f/7DTD-leaflet The tool depends on Pillow which depends on imaging libraries not available in the

Solution 1:

You may have been misled into what the issue actually is.

I don't think that the first Popen ran successfully. I think that it just dumped a message in standard error and you're not seeing it. It's probably saying that

chmod: map_reader: No such file or directory

I suggest you can try either of these 2:

Extract the map_reader from the package into /tmp. Then reference it with /tmp/map_reader.
Do it as recommended by Tim Wagner, General Manager of AWS Lambda who said the following in the article Running Arbitrary Executables in AWS Lambda:

Including your own executables is easy; just package them in the ZIP file you upload, and then reference them (including the relative path within the ZIP file you created) when you call them from Node.js or from other processes that you’ve previously started. Ensure that you include the following at the start of your function code:

process.env[‘PATH’] = process.env[‘PATH’] + ‘:’ + process.env[‘LAMBDA_TASK_ROOT’]

The above code is for Node JS but for Python, it's like the following

import os os.environ['PATH']

That should make the command command = './map_reader <arguments> work.

If they still don't work, you may also consider running chmod 755 map_reader before creating the package and uploading it (as suggested in this other question).

Solution 2:

I know I'm a bit late for this but if you want a more generic way of doing this (for instance if you have a lot of binaries and might not use them all), this how I do it, provided you put all your binaries in a bin folder next to your py file, and all the libraries in a lib folder :

import shutil
import time
import os
import subprocess

LAMBDA_TASK_ROOT = os.environ.get('LAMBDA_TASK_ROOT', os.path.dirname(os.path.abspath(__file__)))
CURR_BIN_DIR = os.path.join(LAMBDA_TASK_ROOT, 'bin')
LIB_DIR = os.path.join(LAMBDA_TASK_ROOT, 'lib')
### In order to get permissions right, we have to copy them to /tmp
BIN_DIR = '/tmp/bin'

# This is necessary as we don't have permissions in /var/tasks/bin where the lambda function is running
def _init_bin(executable_name):
    start = time.clock()
    if not os.path.exists(BIN_DIR):
        print("Creating bin folder")
        os.makedirs(BIN_DIR)
    print("Copying binaries for "+executable_name+" in /tmp/bin")
    currfile = os.path.join(CURR_BIN_DIR, executable_name)
    newfile  = os.path.join(BIN_DIR, executable_name)
    shutil.copy2(currfile, newfile)
    print("Giving new binaries permissions for lambda")
    os.chmod(newfile, 0775)
    elapsed = (time.clock() - start)
    print(executable_name+" ready in "+str(elapsed)+'s.')

# then if you're going to call a binary in a cmd, for instance pdftotext :

_init_bin('pdftotext')
cmdline = [os.path.join(BIN_DIR, 'pdftotext'), '-nopgbrk', '/tmp/test.pdf']
subprocess.check_call(cmdline, shell=False, stderr=subprocess.STDOUT)

Solution 3:

There were two issues here. First, as per Jeshan's answer, I had to move the binary to /tmp before I could properly access it.

The other issue was that I'd ran pyinstaller on ubuntu, creating a single file. I saw elsewhere some comments about being sure to compile on the same architecture as the lambda container runs. Therefore I ran pyinstaller on ec2 based on the Amazon Linux AMI. The output was multiple .os files, which when moved to tmp, worked as expected.

Solution 4:

copyfile('/var/task/yourbinary', '/tmp/yourbinary')
os.chmod('/tmp/yourbinary', 0555)

Moving the binary to /tmp and making it executable worked for me

Solution 5:

There is no need to copy the files the /tmp. You can just use ld-linux to execute any file including those not marked executable.

So, for running a non-executable on AWS Lambda, you use the following command:

/lib64/ld-linux-x86-64.so.2 /opt/map_reader

P.S. It would make more sense to add the map_reader binary or any other static files in a Lambda Layer, thus the /opt folder.

Python Tutorial for Beginners