Debezium

When a MySQL Schema Change Breaks a Kafka Connector: Following the Error Across System Boundaries

Nana Zedginidze

— 5 min read
Share

A routine MySQL schema change was followed by an unexpected failure in a Kafka Connect pipeline.

The application had been stopped for the maintenance window. A new column was added to a primary table and its archive table, and the triggers responsible for maintaining the archive were recreated to include it. The column operations were executed with ALGORITHM=INSTANT.

The database change completed successfully. MySQL reported no DDL errors, replication remained healthy, and the expected binary logging configuration was still in place.

Downstream processing, however, remained pending. The Kafka connector reported an error similar to this:

Caused by: org.apache.kafka.connect.errors.DataException:
notification_settings_v2 is not a valid field name

at org.apache.kafka.connect.data.Struct.lookupField
at org.apache.kafka.connect.data.Struct.put
at io.debezium.transforms.ExtractNewRecordState.addFields
at io.debezium.transforms.ExtractNewRecordState.apply
at org.apache.kafka.connect.runtime.TransformationChain.apply

Because the failure appeared immediately after the deployment, the schema change-and particularly ALGORITHM=INSTANT-became the first suspect.

But temporal correlation does not identify the failing component. I needed to follow the event from MySQL DDL execution, through the binary log and Debezium, to the Kafka Connect transformation layer.

The database change

The simplified version of the change looked like this:

ALTER TABLE app.customer
    ADD COLUMN nsettings_v2 VARBINARY(255) NULL,
    ALGORITHM=INSTANT;

ALTER TABLE app.customer_archive
    ADD COLUMN settings_v2 VARBINARY(255) NULL,
    ALGORITHM=INSTANT;

After both columns existed, the archive triggers were recreated so that inserts and relevant updates would include the new field.

The order was intentional:

  1. Stop application writes.
  2. Add the column to the primary table.
  3. Add the column to the archive table.
  4. Recreate the triggers.
  5. Validate the objects before restoring traffic.

This ensured that no trigger referenced a column that did not yet exist and that no application DML could occur during the gaps between the DDL statements.

Was ALGORITHM=INSTANT the problem?

For supported operations, ALGORITHM=INSTANT changes only table metadata. It avoids rebuilding the table and does not rewrite all existing rows. That makes it especially useful for large InnoDB tables where a rebuild would create significantly more I/O, consume additional space, and extend the maintenance window.

INSTANT does not make the DDL invisible to the binary log, nor does it combine related DDL statements into a single transaction. It changes how InnoDB applies the individual ALTER TABLE operation.

If Debezium had failed to understand the DDL itself, I would expect the connector logs to point toward DDL parsing or schema-history processing. That was not where this stack trace failed.

The call path showed that Kafka Connect was already processing a source record and failed while applying ExtractNewRecordState, a Debezium single message transformation (SMT). The specific failure occurred when the transformation attempted to put a field into a Kafka Connect Struct whose schema did not recognize that field.

This does not prove the exact root cause, but it moves the likely failure boundary away from InnoDB’s DDL algorithm and toward Kafka Connect’s transformation schema or configuration.

Atomic DDL is not a transactional deployment

One suggestion during the incident was that the table alterations and trigger changes should have been executed simultaneously.

This is not possible as one user-controlled transaction in MySQL.

MySQL 8 supports atomic DDL, which means an individual supported DDL statement either completes fully or is rolled back by the server if it fails. It does not mean that several DDL statements can be grouped into one transaction and rolled back together.

For example, this does not create one atomic deployment:

START TRANSACTION;

ALTER TABLE app.customer ...;
ALTER TABLE app.customer_archive ...;
DROP TRIGGER app.after_update_customer;
CREATE TRIGGER app.after_update_customer ...;

COMMIT;

Statements such as ALTER TABLE, CREATE TRIGGER, and DROP TRIGGER cause implicit commits. Each statement is therefore a separate operation, even when surrounded by START TRANSACTION and COMMIT.

The practical control is not a transaction wrapper. It is a carefully ordered maintenance procedure, with writes paused when consistency across the intermediate states matters.

What I verified on MySQL

I first checked whether the binary logging settings required by the CDC pipeline were still correct:

SELECT
    @@GLOBAL.log_bin AS log_bin,
    @@GLOBAL.binlog_format AS binlog_format,
    @@GLOBAL.binlog_row_image AS binlog_row_image,
    @@GLOBAL.gtid_mode AS gtid_mode,
    @@GLOBAL.enforce_gtid_consistency AS enforce_gtid_consistency;

The result showed:

log_bin:                  ON
binlog_format:            ROW
binlog_row_image:         FULL
gtid_mode:                ON
enforce_gtid_consistency: ON

I also verified the configuration source through performance_schema.variables_info. The relevant variables were loaded from the MySQL configuration file, and there was no evidence of a runtime or persisted change introduced by the deployment.

Finally, I inspected active binlog clients. The MySQL replica were connected and waiting for new events, which is the normal state when they are caught up. Nothing indicated that the schema deployment had disabled binary logging or broken MySQL replication.

Reading the stack trace by layer

The most useful part of the investigation was not the final error message alone, but where it appeared in the execution path:

AbstractWorkerSourceTask.sendRecords
  -> TransformationChain.apply
     -> ExtractNewRecordState.apply
        -> ExtractNewRecordState.addFields
           -> Struct.put

Debezium’s ExtractNewRecordState SMT extracts the after portion of a change event and produces a flattened Kafka record. It can also add configured metadata fields to that output.

In this case, the transformation tried to add settings_v2, but the output Struct schema did not contain a valid field with that name.

This strongly suggests a schema mismatch within the transformation stage. A stale or cached transformation schema is one possible explanation. An incompatible transforms.*.add.fields configuration, field renaming, or another schema-handling issue could produce a similar symptom. The trace alone is not sufficient to choose between them.

The next checks therefore belonged on the connector side:

  • the complete transforms.* configuration;
  • the configured ExtractNewRecordState options;
  • connector and Debezium versions;
  • task offsets and the event being retried;
  • schema conversion and Schema Registry compatibility settings;
  • connector behavior after a controlled task restart.

Restarting a connector may rebuild in-memory transformation state, but it should not be treated as proof of root cause. If the task replays the same incompatible event from its existing offset, the failure can return.

What the incident demonstrated

The MySQL change and the Kafka failure were related in time, and the new field clearly participated in the error. However, the available evidence did not show ALGORITHM=INSTANT corrupting data, disabling binary logging, or failing to emit a schema change.

The evidence showed something narrower and more actionable: a record reached Kafka Connect, and the connector failed while transforming it because the field and the transformation output schema did not agree.

That distinction matters. Without it, a team can spend time rolling back a healthy database change while leaving the actual transformation problem unresolved.

Lessons learned

  1. Follow the full stack trace. The component named in the last line is not always the component that introduced the condition, but the call path identifies where processing actually stopped.
  2. Separate atomic DDL from transactional deployment. MySQL can make one DDL statement atomic, but it cannot commit multiple table and trigger changes as one user transaction.
  3. Treat CDC as part of schema-change planning. Testing the database objects is not enough; the connector, transformations, schema compatibility, and replay behavior also need validation.
  4. Do not confuse correlation with root cause. A failure immediately after a database deployment justifies investigating the deployment first, not concluding that its DDL algorithm caused the failure.
  5. Document ownership across system boundaries. A production runbook should define who validates MySQL, Debezium, Kafka Connect, transformations, offsets, and downstream consumers before and after a schema change.

Read more