During development of an AWS Lambda function utilizing the recently released AWS Cost Explorer API, the latest version of boto3 and botocore was discovered to be unavailable in the Lambda execution environment. The Lambda execution environment supported version of an AWS SDK can lag behind the latest release found on GitHub, which can cause supportability issues when writing Lambda code attempting to use the most recently added features of AWS SDKs.

This blog post will explore using boto3 1.4.8 and botocore 1.8.0 despite (at the time of this writing) the Lambda execution environment defaulting to boto3 1.4.7 and botocore 1.7.37. This will enable boto’s Cost Explorer API functionality without waiting for Amazon to upgrade the default boto versions.

Initial Problem

This problem is shown by running the below lambda function.

 import botocore
 import boto3
 client = boto3.client('ce')
 def lambda_handler(event,context):
     response = client.get_cost_and_usage(
           TimePeriod={
                'Start': '2017-11-01',
                'End': '2017-11-07'
           },
           Metrics=['BlendedCost'],
           Granularity='DAILY',
           GroupBy=[
           {'Type': 'DIMENSION','Key': 'LINKED_ACCOUNT'},
           {'Type': 'DIMENSION','Key': 'USAGE_TYPE'}
           ]
      )
      print(response)

While running this locally on our Cloud9 instance or remotely after deploying to Lambda, we receive the below error.

> module initialization error: Unknown service: ‘ce’.

This error demonstrates that the boto code the Lambda function does not have CostExplorer API functionality.
To confirm what version of the boto3 and botocore modules the Lambda function is using, the below line is inserted into our function following the import statements:

print("boto3 version:"+boto3.__version__)
print("botocore version:"+botocore.__version__)

This yields the output:

> boto3 version:1.4.7
> botocore version:1.7.2
> module initialization error: Unknown service: ‘ce’.

These versions are in line with the versions Amazon has indicated are running on Lambda as of the time of this post.

The Solution

To solve this problem, there are two steps: download the newer versions of boto3 and botocore into the lambda function’s directory, and modify our function to utilize these newer versions.

Placing Newer Versions of boto3 and botocore

Open a shell prompt into the directory containing the lambda handler, in this case, the lambda handler is within the file “lambda_function.py” within the innermost directory “NewBotoVersion” (the name of the function for this example).

Once in this directory create a requirements.txt file requiring the latest version of the boto3 and botocore modules, or add the same to your existing requirements.txt:

boto3==1.4.8
botocore==1.8.0

Once this change is made, issue the command:

> pip install -r requirements.txt -t .

This command installs the boto3 and botocore correct versions and their required dependencies in the directory the command is executed within. The result is a directory such as:

As can be seen, there are folders containing the desired versions of the boto3 and botocore modules.

Utilizing Newer Versions of boto3 and botocore

Although the modules are present in the same folder, the function must be modified to reference these instead of the default modules, which are the older version.

 import os
 import os.path
 import sys
 
 envLambdaTaskRoot = os.environ["LAMBDA_TASK_ROOT"]
 print("LAMBDA_TASK_ROOT env var:"+os.environ["LAMBDA_TASK_ROOT"])
 print("sys.path:"+str(sys.path))
 
 sys.path.insert(0,envLambdaTaskRoot+"/NewBotoVersion")
 print("sys.path:"+str(sys.path))
 import botocore
 import boto3
 
 print("boto3 version:"+boto3.__version__)
 print("botocore version:"+botocore.__version__)
 
 print(“available services”+str(boto3.session.Session().get_available_services()))
 client = boto3.client('ce')
 
 def lambda_handler(event,context):
     response = client.get_cost_and_usage(
         #the start is inclusive, the end is exclusive
         TimePeriod={
             'Start': '2017-11-01',
             'End': '2017-11-07'
         },
         Metrics=[
             'BlendedCost'
         ],
         Granularity='DAILY',
         GroupBy=[
             {
                 'Type': 'DIMENSION',
                 'Key': 'LINKED_ACCOUNT'
             },
             {
                 'Type': 'DIMENSION',
                 'Key': 'USAGE_TYPE'
             }
         ]
     )
     print(response)

This yields the following output:

> LAMBDA_TASK_ROOT env var:/var/task
> sys.path:['/var/task', '/var/runtime/awslambda', '/var/runtime', '/var/lang/lib/python36.zip', '/var/lang/lib/python3.6', '/var/lang/lib/python3.6/lib-dynload', '/var/lang/lib/python3.6/site-packages']
> sys.path:['/var/task/NewBotoVersion', '/var/task', '/var/runtime/awslambda', '/var/runtime', '/var/lang/lib/python36.zip', '/var/lang/lib/python3.6', '/var/lang/lib/python3.6/lib-dynload', '/var/lang/lib/python3.6/site-packages']
> boto3 version:1.4.8
> botocore version:1.8.0
> available services:['acm', 'apigateway', [...] 'budgets', 'ce' ...
> {'GroupDefinitions': [{'Type': 'DIMENSION', 'Key': 'LINKED_ACCOUNT'}, {'Type': 'DIMENSION', 'Key': 'USAGE_TYPE'}], 'ResultsByTime': ...

Here not only are the desired module versions loaded, but the list of available services (abridged in this post for formatting) includes ‘ce’ and the CostExplorer API call produced a successful response.

Why the Solution Works

In the above output, two printings of sys.path are listed, the first being sys.path as it is by default in the Lambda execution environment (notice the first item in the list is ‘/var/task’, the same value as the LAMBDA_TASK_ROOT environment variable). The second printing is the same variable after we add the location of the current version boto3 and botocore modules. This results in the current versions of these modules being used as dependencies instead of the default module versions.