Problem:
We have a container pod running with a IAM role that has OpenSearch Serverless read/write data access on both collection items & indexes. Java application running within the container tries to access OpenSearch Serverless using standard opensearch-java & AWS Java sdks.
But problem is that the Java client call to OpenSearch Serverless is failing with security exception as below:
Caused by: org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] authentication/authorization failure
There are no other details & apparently all the access policies are configured properly. So you are not sure why the 403 error is thrown by the OpenSearch API.
Solution:
One potential reason might be that you have not included AWS STS library in your dependencies. AWS STS library is required to create short term crdential from the IAM role. The problem is that the exception above doesn’t describe the reason. if you debug & print the AWS credentials generated from DefaultAWSCredentialsProviderChain, you will see one AwsSessionCredentials object which will have AWS secrect id, key & session token. But the thing is that without AWS STS JAVA sdk, session credentials won’t work.
If you are using Gradle, you can add the dependency as below in build.gradle file:
implementation 'software.amazon.awssdk:sts:2.20.31'