Practical database engineering, performance, and scalable system design

When MySQL LOAD DATA Loads 0 Rows: Check Your Line Endings

While working on a data migration, I ran into a simple but easy-to-miss issue with MySQL LOAD DATA FROM S3. The import statement looked correct, but the result was unexpected: Query OK, 0 rows affected Records: 0 The script contained: LINES TERMINATED BY '\r\n' The

When a MySQL Schema Change Breaks a Kafka Connector: Following the Error Across System Boundaries

A routine MySQL schema change was followed by an unexpected failure in a Kafka Connect pipeline. The application had been stopped for the maintenance window. A new column was added to a primary table and its archive table, and the triggers responsible for maintaining the archive were recreated to include

MySQL Replication: "Can't connect to MySQL server" (110) - Troubleshooting Case

A production replica suddenly stopped receiving updates. SHOW REPLICA STATUS showed: Replica_IO_Running: No Replica_SQL_Running: Yes Last_IO_Error: Can't connect to MySQL server ... (110) The first step was verifying the source. I checked that binary logging was enabled, GTID mode was active, and the

Monitoring Failed Amazon RDS Snapshots: An Unexpected AWS Limitation

One of the recent tasks I worked on sounded quite simple at first.The goal was to send a Slack notification whenever an Amazon RDS or Aurora snapshot creation failed. Since AWS RDS already publishes many events through EventBridge, I assumed there would be a dedicated event for snapshot failures

Troubleshooting Replication Lag on a Percona XtraDB Cluster (PXC) Async Replica

Replication lag is one of the most common problems MySQL DBAs encounter.The replica was almost nine hours behind the source. Both replication threads were running, there were no replication errors, and from a quick glance everything appeared healthy. Yet the replica was barely making any progress. Environment The affected

Why SST Completed But MySQL Still Failed to Start

A few days ago, I came across an interesting recovery scenario while working with a Percona XtraDB Cluster. At first, it looked like a straightforward SST recovery. One node had fallen out of the cluster, SST started automatically, the data transfer completed successfully... and yet MySQL still refused to start.

Aurora Mystery: "Found a row not matching the given partition set"

A few weeks ago we started seeing a strange error in one of our Aurora MySQL testing environments: Found a row not matching the given partition set At first glance it looked like a classic partitioning problem. Maybe a partition was missing. Maybe partition maintenance failed. The interesting part was