We were trying to connect to ES through debezium ElasticsearchSinkConnector but getting 401 Unauthorized We were able to connect to ES instance with IAM role with no master cred and authentication. But not able to connect when IAM role is set Using IAM Authentication with Amazon Elasticsearch Service. Have followed steps mentioned in https://medium.com/dana-engineering/streaming-data-changes-in-mysql-into-elasticsearch-using-debezium-kafka-and-confluent-jdbc-sink-8890ad221ccf and working fine for ES in local with no AWS iam-authentication. All the configuration and access is also provided but not able to create access_key,secret_key and session_token mentioned in the below link https://ruan.dev/blog/2018/08/20/using-iam-authentication-with-amazon-elasticsearch-service Curl which I used to create ElasticsearchSinkConnector is
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
"name": "elastic-sink-connect-v1",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "social.facebook_insight",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"connection.url": "https://search-replaceme.eu-west-1.es.amazonaws.com",
"type.name": "_doc",
"behavior.on.null.values": "delete",
"elastic.security.protocol": "PLAINTEXT",
"elastic.http.auth.type": "IAM",
"elastic.region": "us-west-1",
"schema.ignore": "true",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "false",
"transforms.unwrap.delete.handling.mode": "drop"
}
}'
We also tried with username and password but not able to authenticate.
