1,151
questions
1
vote
0
answers
23
views
Kafka Debezium Oracle Source Connector Intermittent Lag & Data Delay
I am working with Kafka, and we are using Debezium to capture CDC (Change Data Capture) data from an Oracle source database. Below are my Debezium configuration parameters:
{
"name": &...
- 125
0
votes
1
answer
22
views
Debezium KafkaConnect, kafka_connect_run.sh No such file or directory
I am getting the error when attempting to setup Debezium in kubernetes (minikube for now) using Strimzi. What is wrong with my manifest? I may be confusing how to use the debezium connect image. I am ...
- 113
0
votes
1
answer
24
views
Error creating broker listeners from PLAINTEXT
I'm new to debezium and having trouble getting it setup in minikube (kubernetes later). I followed the tutorial to get it running with docker compose, but get errors in minikube. I am missing ...
- 113
0
votes
0
answers
35
views
How to implicitly cast ISO-8601 Avro column as Timestamp column in Delta Live Table pipeline
this one has stumped me for a few days... I have a Postgres server with a table to which a deployed Debezium instance is connected, recording CDC. Debezium serialises using Avro and Confluent Schema ...
- 1,915
0
votes
0
answers
16
views
Enhance Debezium message sent to Apache Kafka with current logged in user identifier
I want to implement Debezium into my application to capture all the changes in selected tables in database (MSSQL). I do not have userId column in any table. Is it possible to intercept the message ...
- 121
1
vote
1
answer
34
views
Debezium MySQL Connector - Incorrect Base64 Encoded Value for Decimal Column
I'm facing an issue with the Debezium MySQL connector where I'm getting incorrect Base64 encoded values for a DECIMAL(10,2) column named Total.
Problem:
In the MySQL database, the Total column has a ...
0
votes
0
answers
21
views
do not commit offset when cdc event not processed correctly - debezium embedded
I have debezium embedded in spring boot app (kafka-less solution). I connect to mysql db and I store offset in the file.
.with("offset.storage","org.apache.kafka.connect.storage....
- 1,025
0
votes
0
answers
47
views
Payload compression at Debezium Server. "Debezium Server - Pub/Sub Pipeline"
We have a setup where we are using Debezium to read the logs from MYSQL, sending them across to Pub/Sub, and then transferring them to BigQuery via BQ Subscriptions. This pipeline is working fine but ...
- 166
1
vote
0
answers
31
views
kafka connector debezium stuck at snapshot of large data
I setup elasticsearch, kibana, mongodb, and kafka on the same linux server for development purposes. The server has 30GB Memory and enough disk space. I'm using a debezium connector and I'm trying ...
- 476
1
vote
0
answers
27
views
Data filtering in Debezium source connector (PostgreSQL)
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "<host>",
"database.port": "<port&...
0
votes
0
answers
42
views
Databases bidirectional synchronization
I would like to synchronize two databases bidirectionally, i.e. from old database to new and from new to old. I can use Debezium to achieve this goal. But what are my options to avoid infinite loops, ...
0
votes
1
answer
41
views
How to receive tombstone messages in debezium connector?
Im using Kafka and Debezium Postgres Connector.
I wanted to extract only record value without other details, so I am using the "transforms.unwrap.type": "io.debezium.transforms....
- 303
0
votes
0
answers
29
views
Debezium connect with Azure Mysql Flexible Server getting Exception Caused by:Certificates do not conform to algorithm constraints
I am trying to register a debezium connector with Debezium connect in which i am trying to connect with Azure Managed Mysql flexible server.
But i am getting below exception:
The last packet sent ...
1
vote
0
answers
45
views
How to create debezium ElasticsearchSinkConnector connecting to AWS ES intance with IAM role
We were trying to connect to ES through debezium ElasticsearchSinkConnector but getting 401 Unauthorized
We were able to connect to ES instance with IAM role with no master cred and authentication. ...
- 135
1
vote
1
answer
31
views
How Kafka Connect Cluster is operated in production (number, distribution, etc.)
Hi I have 4 DBs and I need to connect all of them with Debezium Connector. I'm going to float them on AWS EKS and I'm also going to have S3 Sink Connectors, which I expect to be more than 10, divided ...
- 13