AWS AppSync Pipeline Resolver Templates, Secrets Manager, and External HTTP APIs

UPDATE:  As of July 6, 2021, it seems that the AWS Amplify team has updated the tool to support secrets.  The official announcement is here: AWS Amplify CLI adds support for storing environment variables and secrets accessed by AWS Lambda functions (amazon.com).  The Amplify documentation is here: Functions – Access secret values – Amplify Docs.  Therefore, I recommend using this and save yourself the extra work of using Secrets Manager.

I’ve been working with AWS Amplify and AppSync over the last few weeks and it’s truly been a love-hate relationship with both of these.

As an aside, having worked with both application development on AWS and Azure now, my opinion is that Azure is generally more productive.  I don’t know if the Azure documentation is better or not because I’ve encountered my fair share of gaps in the Azure documentation (one of the reasons I wrote this post on Functions with JWT authentication), but I know that the AWS documentation — at least as it pertains to Amplify and AppSync — is severely lacking in examples.  I think the root cause of the issue is that Azure serverless (Functions) kind of “just works”; the connectivity within the ecosystem is dead simple with function bindings built in and really straight forward inter-service authentication; you just don’t need to muck around as much.  On the other hand, the combination of Amplify, AppSync, and CloudFormation is insanely powerful, but also kind of inaccessible given the tremendous breadth required to use the tools.  If I were to put it another way, because of the breadth of knowledge required to make use of Amplify and AppSync with CloudFormation, it demands far superior documentation.

The Scenario

Recently, I’ve been trying to figure out how to implement pipeline resovlers in AppSync (as part of an Amplify project).

Consider the following scenario:

1. I want to call an external REST API using an AppSync HTTP resolver which allows me to call the REST endpoint without writing a Lambda
2. The REST API requires an API key which I want to keep in AWS Secrets Manager
3. Therefore, I need to retrieve the API key from Secrets Manager and pass that into my HTTP resolver

This is an optimal solution because it means the secret is not exposed to the client side and I don’t need to write a Lambda to handle what is otherwise just an HTTP POST.

To build this solution, I need to provision the following artifacts in AWS via CloudFormation:

1. An AppSync HTTP data source for Secrets Manager (assuming Secrets Manager is already provisioned)
2. An AppSync HTTP data source for the REST API
3. An AppSync resolver function for getting the secret API key
4. An AppSync resolver function for making the REST API call
5. An AWS IAM role for the AppSync HTTP data source for Secrets Manager to authenticate the function call from my function to Secrets Manager
6. An AppSync pipeline resolver for chaining the call to (3) and (4)

In this particular case, I’m making a REST API call to an endpoint in Azure that requires an API key that I’m keeping in Secrets Manager, but this could be any REST endpoint.

Of course, you could write a Lambda that just encapsulates this series of actions (if you aren’t a masochist like me), but then you wouldn’t really be leveraging all of the power of AppSync to entirely bypass Lambdas.

OK, Where Do We Start?

One would assume that this is a really straight forward use case given that this should be a common pattern in AppSync.  Well, you’d be wrong.  The best resources I found for this are a series of articles from Josh Kahn:

1. Invoking even more AWS services directly from AWS AppSync
2. Securely Storing API Secrets for AWS AppSync HTTP Data Sources
3. Invoke AWS services directly from AWS AppSync

Josh’s articles were a good foundation, but I struggled to get this scenario working due to the gaps in each of his articles.  In particular, the biggest gap is that none of the articles demonstrated how to assemble item 2 and item 5 on that list.  So here, I present you the CloudFormation JSON that will provision items 1, 2, and 5:

In this example, the role AppSyncServiceRoleTest is actually the secret sauce.  It’s critical because it’s not possible to configure this in the AWS console UI in any way whatsoever (I really hate this); you must define it as part of a CloudFormation template or update it via the AWS CLI.

You’ll notice that the template for the Secrets Manager HTTP data source is slightly more complex.  This is necessary to allow the AuthorizationConfig on the SecretsManagerDataSourceTest resource to work by authorizing the HTTP resolver to call into Secrets Manager; without it, pushing this via Amplify will fail with the following error:

“The validated string is empty” (Service: AWSAppSync; Status Code: 400…)”

I only found this by reducing by iteratively reducing my template down (until I got it to deploy) and then iteratively building it back up.

My opinion is that this really should be more straight forward; the integration between Secrets Manager and AppSync should be transparent as this will promote best practices (storing secrets away from code).  Requiring these extra steps and artifacts to do something as simple as accessing a secret from an AppSync resolver function is just asking for less initiated developers to fall into bad habits.  Indeed, there is a Github issue on this very topic.

I learned a number of other lessons along the way.

Lesson 1: Use CloudFormation Directly

If you’ve worked with Amplify, you know how mind-numbing it is to perform a push and wait for the results.  To be fair, it’s doing a good bit of magic to pull all of the AWS resources together on your behalf, but half my day was spent waiting for it to deploy and dump out some obscure CloudFormation error (see above).  To that end, one lesson I learned through this process is that it’s way easier to test by directly uploading your stack definition into CloudFormation instead of using Amplify to push during testing:

This approach is MUCH faster for testing CloudFormation steps rather than using Amplify push.

If you’re testing something complex or exotic, then create a simplified standalone copy outside of your Amplify project and just test the deployment of the CloudFormation template by itself.

This allows you to bypass the rest of the CloudFormation artifacts generated by Amplify and significantly cut down the time it takes to test tricky CloudFormation scenarios.

Lesson 2: VTL Templates Are Tricky; Test Edits in the Console

When working with VTL templates for the functions, I found that the fastest way to do so was to edit the functions directly in the AWS console since — once again — this avoided a time consuming push operation.

Listing of AppSync functions (resolvers); click to edit the VTL templates inline

Then you can edit the VTL templates directly:

Edit the templates directly to save time instead of trying to using push.

(Don’t forget to copy your changes back to your codebase!)

To test this, go to the AppSync Queries and test your API call:

Testing the AppSync API call without writing the front-end and wasting time with build/deploy.

Of note is that there is a delay from when you save a change to a function VTL and when this gets picked up by the API call.  This can be anywhere from 30-60 seconds in my experience and this caused me a good number of wasted cycles as I kept scratching my head wondering why my templates were not outputting correctly before I realized there was a delay.

What’s perhaps even more confounding is that pipeline resolvers are not visible anywhere in the console so you really want to make sure each of your individual pipelined functions are executing correctly to avoid having to push to test the full chain over and over.

Lesson 3: Be Wary of VTL Examples

The reason why you need to test is that you can’t actually trust any VTL example you find online.  I have yet to find one that Just Works; every single time it seems like the example is off in some nuanced way.  In this case, Josh’s examples don’t factor that the result from GetSecretValue is actually a JSON object and not a direct string value.  So the VTL template of the REST API call should do something like this:

See the first line?  We take the result from the call to Secrets Manager and convert it into a JS object and then we can access the key-value pairs as $secrets.qnaApiKey below.  Here’s what it looks like in Secrets Manager:

Note that the secrets are stored as JSON key-value pairs under the secret name.

Putting It All Together

Here’s how my Amplify project is organized:

Note the structure of the files in VS Code

Here is the full CustomResources.json CloudFormation template including the resolver functions and pipeline resolver:

It looks intimidating, but it’s just the 6 components I outlined at the outset pulled together in one file.

Here are the relevant portions of the GQL schema:

This is relevant as MutationResult is the target return structure from the AzureQnaQuery.res.vtl (I’m going to replace that with something less generic, I think).

And here are all of the VTL templates:

And there you have it.  Hopefully this fills in the gaps left by the few samples online of how to put together an AppSync pipeline resolver with Secrets Manager.