MDEV-14992 BACKUP SERVER by dr-m · Pull Request #4817 · MariaDB/server

dr-m · 2026-03-17T14:24:11Z

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory';
BACKUP SERVER TO '/path/to/directory' 1 CONCURRENT;
BACKUP SERVER WITH 'command';
BACKUP SERVER WITH 1 CONCURRENT 'command';

In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to.

For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file.

The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used.

Note: The parameter innodb_log_recovery_start in backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database!

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-39092 Less blocking backup of ENGINE=Aria

The implementation introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_file(): A partial or sparse file-copying service for all systems.

backup_stream_append(): Equivalent to copy_file(), but appending to a stream. On Linux, this uses sendfile(2), which assumes that the source data will not be changed before the data has been consumed from the pipe.

backup_stream_append_async(): A variant of backup_stream_append() where the source file region is guaranteed to be immutable after the call returns. We must not use Linux sendfile(2) for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe.

InnoDB_backup::context: Backup context, attached to backup_sink so that context can continue to exist between the time a BACKUP SERVER releases all locks and another BACKUP SERVER starts executing, with innodb_backup pointing to the new backup, while the old backup is still being finished.

fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup.

fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup.

buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread().

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files.

CLAassistant · 2026-03-17T14:24:21Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

dr-m · 2026-06-26T13:05:54Z

I plan to rebase this once #5070 has been merged up to the 13.0 branch. @grooverdan pushed a merge to 10.11 today, and I pushed to 11.4 and 11.8. I hope that the conflicts for 12.3 and potentially 13.0 will have be resolved by Monday.

The ultimate merge target is main. For testing, it is better to be based on the oldest maintained branch that includes #4405, which forms the fundament for this, innodb_log_archive=ON.

While rebasing, I will write a description based on the commit message of 4769a43, but mentioning actual MDEVs for the outstanding work. Soon after the rebase, we can include #5140 so that this can be tested more conveniently.

dr-m · 2026-06-29T14:29:53Z

+        const uint32_t end{start + fil_space_t::BACKUP_BATCH_SIZE};
+        backup_batch_start(node->space, end);
+        /* TODO: avoid copying freed page ranges */
+        err= copy_file(node->handle, f, start * uint64_t{page_size},
+                       std::min(end, file_size) * uint64_t{page_size});
+        backup_batch_stop(node->space);


If this is a ROW_FORMAT=COMPRESSED table, then the file may be 1024, 2048, or 3172 bytes shorter than calculated, and the copying could fail. This API as well as the one in stream() must be refactored so that we will know how much was actually copied. The reason for this short file is that fil_space_extend_must_retry() will only extend files to integer multiples of 4096 bytes.

In stream() we must pad with field_ref_zero so that the file size will match what was written to the header. The last page will be recovered from the redo log.

Note: We don’t currently keep track of the file size or the allocated file size as of the checkpoint when the backup started. If we did that, we could copy even less. That could be an even more elegant fix of this. I think we would create sparse files that match the current file size.

Thirunarayanan

BACKUP SERVER WITH ''` runs an arbitrary shell command as the mysqld OS user, gated only by global RELOAD + SELECT_ACL. Is a dedicated privilege planned? Should there be a server switch to disable the WITH variant, and is the command captured by the audit log?

Thirunarayanan · 2026-07-27T04:29:07Z

            goto no_space;
          }
+
+          backup_page_end= space->writing_start()


what if we cached backup_page_end before backup server set backup bit? We cache it only once. This
could lead to torn page read?

Thirunarayanan · 2026-07-27T04:29:46Z

          auto p= buf_flush_space(space_id);
          space= p.first;
          last_space_id= space_id;
+          backup_page_end= space && space->writing_start()


same issue here as well

Thirunarayanan · 2026-07-27T05:06:32Z


 	dict_sys.lock(SRW_LOCK_CALL);

+	if (fil_system.have_all_spaces) {


fil_system.have_all_spaces set to true when buf_load() is being called only on partial tablespace

Thirunarayanan · 2026-07-27T11:48:27Z

+  thd->mdl_context.release_lock(mdl_request.ticket);
+  if (!fail)
+  {
+    target_phase->phase= BACKUP_PHASE_FINISH;


It does run lock-free iiuc, what happens if we have second backup server is starting while the first is still finishing?

The logically completed BACKUP SERVER statement will clean up things concurrently with the subsequently starting BACKUP SERVER. This is why the storage engines may save their execution context in backup_sink.

Thirunarayanan · 2026-07-27T11:49:13Z

+
+  if (command)
+  {
+    char cmd[1024];


1024 lines is sufficient for shell script commands?

I think yes. The script should likely be in $PATH anyway. On Linux, extfs and its successors such as ext4 would limit the length of a path component (a file name) to 255 bytes.

Thirunarayanan · 2026-07-27T13:09:01Z

+      logs.clear();
+    else
+    {
+      delete_logs();


we're deleting the logs before backup archiving stops.

We also had executed the following:

this->ctx= nullptr; /* fini() will delete the object */

Therefore, nothing will be recorded in logs on checkpoint completion:

/** Complete the first checkpoint in a new archive log file. */ void checkpoint_complete() noexcept { ut_ad(log_sys.latch_have_wr()); if (ctx) logs.emplace_back(log_sys.get_first_lsn() - log_sys.capacity()); }

However, new log files would have been created. It seems that we should swap some statements to ensure that no garbage log files will be left behind:

log_sys.latch.wr_unlock(); fail= log_sys.backup_stop_archiving(thd); log_sys.latch.wr_lock(); delete_logs(); logs.clear();

Thirunarayanan · 2026-07-27T13:11:59Z

+    const int src{open(path, O_RDONLY)};
+    if (src < 0)
+      goto fail;
+    if (move && unlink(path))


should the unlink() deferred till copy() succeeds?

The log file that we are duplicating here is one in the BACKUP SERVER target directory. If the duplication failed, we did not remove the hard link to the active (last) log of the server, and the user started to use the output directory despite the error, they could end up corrupting the log of the server that is being backed up.

Thirunarayanan · 2026-07-27T13:55:02Z

+        log_sys.archived_checkpoint;
+      start_end= log_sys.archived_lsn;
+#endif
+      ctx= new context{


Do we need to handle bad_alloc() here?

Thirunarayanan · 2026-07-27T15:02:00Z

+  {
+    uint64_t id_limit{0};
+    lsn_t lsn{0};
+    log_sys.latch.wr_lock();


Hotter latch acquired for each archive log and datafile. Can we tweak it? Need to check whether it impact the server performance

We could use a separate latch for protecting queue. However, we must prioritise the copying of the log files that have been collected in logs, because otherwise we could run out of storage space while copying a huge server.

Originally I had a separate mutex in the InnoDB_backup singleton, but in the end, most things ended up being protected by log_sys.latch, most notably the InnoDB_backup::checkpoint_complete().

I checked the size of the critical section between log_sys.latch.wr_lock() and log_sys.latch.wr_unlock() in an optimized build where this code was inlined in innodb_backup_step():

0x00000000010bb199 <+153>: mov 0x153b238(%rip),%rcx # 0x25f63d8 <_ZN12_GLOBAL__N_1L13innodb_backupE+24> 0x00000000010bb1a0 <+160>: mov 0x153b249(%rip),%rax # 0x25f63f0 <_ZN12_GLOBAL__N_1L13innodb_backupE+48> 0x00000000010bb1a7 <+167>: mov 0x153b23a(%rip),%rdx # 0x25f63e8 <_ZN12_GLOBAL__N_1L13innodb_backupE+40> 0x00000000010bb1ae <+174>: mov %rcx,%r10 0x00000000010bb1b1 <+177>: sub 0x153b218(%rip),%r10 # 0x25f63d0 <_ZN12_GLOBAL__N_1L13innodb_backupE+16> 0x00000000010bb1b8 <+184>: sar $0x3,%r10 0x00000000010bb1bc <+188>: cmp %rdx,%rax 0x00000000010bb1bf <+191>: je 0x10bb290 <_Z18innodb_backup_stepP3THDPK13backup_target12backup_phasePK11backup_sink+400> 0x00000000010bb1c5 <+197>: mov %r12,-0x20(%rbp) 0x00000000010bb1c9 <+201>: sub $0x8,%rax 0x00000000010bb1cd <+205>: mov (%rax),%r12 0x00000000010bb1d0 <+208>: mov 0xb8(%rbx),%rcx 0x00000000010bb1d7 <+215>: mov %rax,0x153b212(%rip) # 0x25f63f0 <_ZN12_GLOBAL__N_1L13innodb_backupE+48> 0x00000000010bb1de <+222>: sub %rdx,%rax 0x00000000010bb1e1 <+225>: sar $0x3,%rax 0x00000000010bb1e5 <+229>: test %r10,%r10 0x00000000010bb1e8 <+232>: movl $0x0,0x2c(%rbx) 0x00000000010bb1ef <+239>: cmove %rax,%r10

The log_sys.wr_unlock() is also duplicated below the following (else if (size)):

0x00000000010bb290 <+400>: test %r10,%r10 0x00000000010bb293 <+403>: je 0x10bb390 <_Z18innodb_backup_stepP3THDPK13backup_target12backup_phasePK11backup_sink+656> 0x00000000010bb299 <+409>: mov %r12,-0x20(%rbp) 0x00000000010bb29d <+413>: sub $0x8,%rcx 0x00000000010bb2a1 <+417>: mov (%rcx),%r12 0x00000000010bb2a4 <+420>: mov %r14,-0x10(%rbp) 0x00000000010bb2a8 <+424>: mov %r15,-0x8(%rbp) 0x00000000010bb2ac <+428>: mov %rcx,0x153b125(%rip) # 0x25f63d8 <_ZN12_GLOBAL__N_1L13innodb_backupE+24> 0x00000000010bb2b3 <+435>: dec %r10 0x00000000010bb2b6 <+438>: movl $0x0,0x2c(%rbx)

The "nothing to do" branch at +656 is duplicating the assignment at +438. All in all, we have 0 or 1 taken conditional branches in the likely code path (there is some work to do). We could avoid that if we used a common data structure for logs and queue.

I think that this should only matter when there are huge amounts of tiny files being backed up.

dr-m · 2026-08-03T12:46:51Z

+    if (!ret)
+      return -1;


@mleich1 filed an rr replay trace of a strange error here, while copying the file t2.ibd:

ssh pluto rr replay /data/results/1784559906/NMB-2484/1/rr/latest-trace

The immediate cause of the error is the following:

#0 0x00006527e28644f3 in send_step (offset=0x7726ccdfea48, count=71680, out_fd=120, in_fd=32) at /data/Server/MDEV-39061/sql/sql_backup.cc:856 #1 copy<send_step, true> (in_fd=in_fd@entry=32, out_fd=out_fd@entry=120, offset=<optimized out>, offset@entry=0, end=end@entry=71680) at /data/Server/MDEV-39061/sql/sql_backup.cc:35 #2 0x00006527e2864b92 in backup_stream_append_async (src=src@entry=32, stream=stream@entry=120, start=start@entry=0, end=end@entry=71680) at /data/Server/MDEV-39061/sql/sql_backup.cc:876 #3 0x00006527e2864c00 in copy_file (src=32, dst=120, start=0, end=71680) at /data/Server/MDEV-39061/sql/sql_backup.cc:207 #4 0x00006527e29e239e in (anonymous namespace)::InnoDB_backup::backup (target=<optimized out>, node=0x7726dc130b90, start=64, limit=<optimized out>) at /data/Server/MDEV-39061/storage/innobase/handler/backup_innodb.cc:658 #5 0x00006527e29e190c in (anonymous namespace)::InnoDB_backup::step (this=this@entry=0x6527e43a1620 <(anonymous namespace)::innodb_backup>, target=..., phase=<optimized out>, sink=...) at /data/Server/MDEV-39061/storage/innobase/handler/backup_innodb.cc:403 #6 0x00006527e29e198d in innodb_backup_step (target=<optimized out>, phase=<optimized out>, sink=<optimized out>) at /data/Server/MDEV-39061/storage/innobase/handler/backup_innodb.cc:1170 #7 0x00006527e2865732 in backup_step (thd=0x0, plugin=<optimized out>, arg=0x7726f94576e0) at /data/Server/MDEV-39061/sql/sql_backup.cc:370

We correctly enter the function with *offset=0 as expected. We copy 69632 or 0x11000 bytes, which corresponds to the innodb_page_size=4k and about half the tablespace (35=0x23 pages).

A subsequent call enters with *offset=69632 and sendfile(2) returns 0, and the *offset remains unchanged.

We abort the copying because if (!ret) would hold here, even though the file had not been copied entirely. The target is a mounted file system, not a stream.

The problem seems to be that the size of the file really is 69632 (0x11000) bytes and not 0x23000 as expected. The fil_node_t::size had been repeatedly updated by fil_space_extend_must_retry() as part of an ALTER TABLE t2 ADD COLUMN operation.

This is also the root cause of another failure; the source file name and the is likewise cool_down/t2.ibd at innodb_page_size=4k but the sizes are 59 and 29 pages, respectively, in:

ssh pluto rr replay /data/results/1784559906/NMB-2485/1/rr/latest-trace

I don’t see any obvious race condition here. Actually, as far as I can tell, the file is being extended by fallocate() just fine. Could this be a bug in rr? Is this error reproducible without rr?

For the traces in /data/results/1784559906/NMB-2485/1/rr I can’t find any copy of the server data directory, so I can’t cross-check the file size. In /data/results/1784559906/NMB-2484/data we have cool_down/t2.ibd whose size is 0x11000 (69632) bytes, matching the rr replay trace.

I was suspecting a bug in the rr emulation of sendfile or fallocate. I checked that on the pluto system we are running rr 5.9.0, which is the latest release, from February 2025. There have been 180 changes since then, nothing related to sendfile or fallocate system calls. Support for sendfile was introduced in 1.2.0, and fallocate should be supported since the beginning. Therefore, a bug in rr seems very unlikely.

I checked the file system:

od -Ax -t x1 /data/results/1784559906/NMB-2484/1/data/cool_down/t2.ibd 000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 011000

There is also a test/t2.ibd that had been extended to 2 more innodb_page_size=4k pages:

od -Ax -t x1 /data/results/1784559906/NMB-2484/1/data/test/t2.ibd 000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 013000

In the rr replay trace, by setting some breakpoints and logging, I can see both files being extended to the observed size:

break os_file_set_size condition 1 $_regex(name,".*/t2.ibd") command 1 when continue end break *0x7f27114661d0 condition 2 $_regex(old,".*/t2.ibd")||$_regex(new,".*/t2.ibd") command 2 when continue end display/i $pc continue

The relevant part of the output seems to be the following:

Thread 3 hit Breakpoint 1, os_file_set_size (name=0x7726dd1c9230 "./test/t2.ibd", file=32, size=69632, is_sparse=false) at /data/Server/MDEV-39061/storage/innobase/os/os0file.cc:1550 1550 { 1: x/i $pc => 0x6527e2a522a1 <_Z16os_file_set_sizePKcimb>: endbr64 Completed event: 62006 Thread 3 hit Breakpoint 3, rename (old=0x7726dd1c9230 "./test/t2.ibd", new=0x7726dd1fdf60 "./cool_down/t2.ibd") at ../sysdeps/unix/sysv/linux/rename.c:27 warning: 27 ../sysdeps/unix/sysv/linux/rename.c: No such file or directory 1: x/i $pc => 0x7f27114661d0 <rename>: endbr64 Completed event: 63457 … Thread 3 hit Breakpoint 1, os_file_set_size (name=0x7726dc1418d0 "./test/t2.ibd", file=87, size=57344, is_sparse=false) at /data/Server/MDEV-39061/storage/innobase/os/os0file.cc:1550 1550 { 1: x/i $pc => 0x6527e2a522a1 <_Z16os_file_set_sizePKcimb>: endbr64 Completed event: 94690 2026-07-20 15:35:19 18 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2026-07-20 15:35:19 18 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files 2026-07-20 15:35:21 24 [Note] InnoDB: setting innodb_log_archive=1 at innodb_log_recovery_start=60024 2026-07-20 15:35:21 0 [ERROR] mariadbd: Can't create file './cool_down/t2.ibd' (errno: 2 "No such file or directory") … Thread 3 hit Breakpoint 1, os_file_set_size (name=0x7726dc081430 "./test/t2.ibd", file=57, size=77824, is_sparse=false) at /data/Server/MDEV-39061/storage/innobase/os/os0file.cc:1550 1550 { 1: x/i $pc => 0x6527e2a522a1 <_Z16os_file_set_sizePKcimb>: endbr64 Completed event: 103074 2026-07-20 15:35:30 17 [Warning] Could not read packet: fd: 55 state: 1 read_length: 4 errno: 11 vio_errno: 1158 length: 0 2026-07-20 15:35:30 17 [Warning] Aborted connection 17 to db: 'mysql' user: 'root' host: 'localhost' (Got an error reading communication packets) [Thread 1999175.1999175 exited] … Thread 4 received signal SIGKILL, Killed. [Switching to Thread 1999175.1999372] 0x0000000070000002 in syscall_traced () 1: x/i $pc => 0x70000002: ret

We had indeed extended the file only to 69632 (0x11000) bytes before it had been renamed to cool_down/t2.ibd. A lot later we expect to copy more data from that file. I will dig deeper.

At the time of the rename (old=0x7726dd1c9230 "./test/t2.ibd", new=0x7726dd1fdf60 "./cool_down/t2.ibd"), the tablespace size was 35 (0x23) pages. But, the page size is not innodb_page_size; it actually is 2048 bytes, because this is a ROW_FORMAT=COMPRESSED tablespace. Files are only being extended to multiples of 4096 bytes, because they might be opened in O_DIRECT mode, and the underlying physical sector size might be up to 4096 bytes.

This explains the size difference. Backup is using the correct physical page size of 2048 bytes, but our attempt to copy the 0x11800 bytes fails at 0x11000 because the last page had not yet been written to the file.

The backup started at LSN=60024. The tablespace had been created much later:

Thread 27 hit Hardware watchpoint 5: -location $node.handle Old value = {m_file = -1} New value = {m_file = 32} fil_ibd_create (space_id=103, name=..., path=path@entry=0x7726dc122370 "./test/#sql-alter-1e8147-12-4d.ibd", flags=229, size=size@entry=4, mode=mode@entry=FIL_ENCRYPTION_DEFAULT, key_id=1, err=0x7f270807090c) at /data/Server/MDEV-39061/storage/innobase/fil/fil0fil.cc:2140 2140 node->find_metadata(IF_WIN(,true)); 1: x/i $pc => 0x6527e2bfa407 <_Z14fil_ibd_createj12table_name_tPKcjj16fil_encryption_tjP7dberr_t+2068>: mov $0x1,%esi 3: log_sys.base_lsn = std::atomic<unsigned long> = { 1498624 } Completed event: 60934

The tested revision was 45b94b3, which is ahead of the current 5235e7a by 1 change (#5140).

We should have avoided copying this file in the first place. For some reason, the fil_space_t::create_lsn remains at 0 here, which is causing innodb_backup.init(thd) to enqueue the tablespace ID 103 as well as its allocated size of 35 pages, even though the file had been created after the backup start checkpoint LSN 60024, at LSN 60934.

We fail to assign fil_space_t::create_lsn in fil_ibd_create(). I started writing a fix for this, based on modifying mtr_t::commit_file(). In its current form it is breaking the test innodb.alter_crash as well as making --suite=backup fail due to missing files. I hope to fix this tomorrow.

dr-m · 2026-08-03T13:09:57Z

+    int copy_file(const backup_target &target, const backup_sink &sink,
+                  const char *path, bool is_log) const noexcept
+    {
+#ifndef _WIN32
+      int ret_val{0};
+      int src_fd{openat(is_log ? logdir_fd : datadir_fd, path, O_RDONLY)};
+      if (src_fd < 0)
+      {
+        my_error(ER_CANT_OPEN_FILE, MYF(0), path, errno);
+        return 1;
+      }


@mleich1 filed an rr replay trace:

ssh pluto rr replay /data/results/1784559906/NMB-2482/1/rr/latest-trace

This may fail due to a race condition with a DDL operation. @mariadb-andrzejjarzabek mentioned a race condition in this area. Could it be that we should filter out all #sql* files here?

#0 (anonymous namespace)::Aria_backup::copy_file (this=this@entry=0x71700c03a520, target=..., sink=..., path=0x71700c0b61c0 "test/#sql-alter-5f2e4-25-2.frm", is_log=is_log@entry=false) at /data/Server/MDEV-39061/storage/maria/ma_backup_server.cc:245 #1 0x000062e2587d63b5 in (anonymous namespace)::Aria_backup::copy_database (this=this@entry=0x71700c03a520, target=..., sink=..., dir={...}) at /usr/include/c++/13/bits/basic_string.h:222 #2 0x000062e2587d6464 in (anonymous namespace)::Aria_backup::copy_databases (this=this@entry=0x71700c03a520, target=..., sink=...) at /data/Server/MDEV-39061/storage/maria/ma_backup_server.cc:179 #3 0x000062e2587d689f in (anonymous namespace)::Aria_backup::perform_backup (this=this@entry=0x71700c03a520, target=..., sink=...) at /data/Server/MDEV-39061/storage/maria/ma_backup_server.cc:106 #4 0x000062e2587d6902 in (anonymous namespace)::Aria_backup::end (this=this@entry=0x71700c03a520, target=..., sink=...) at /data/Server/MDEV-39061/storage/maria/ma_backup_server.cc:81 #5 0x000062e2587d6979 in aria_backup_end (thd=<optimized out>, target=0x7170036e86b0, phase=<optimized out>, sink=0x7170036e86b8) at /data/Server/MDEV-39061/storage/maria/ma_backup_server.cc:446 #6 0x000062e2587895ae in backup_end (thd=0x71700c00bab0, plugin=<optimized out>, arg=0x7170036e86b0) at /data/Server/MDEV-39061/sql/sql_backup.cc:346 #7 0x000062e2583101c1 in plugin_foreach_with_mask (thd=thd@entry=0x71700c00bab0, func=func@entry=0x62e258789557 <backup_end(THD*, plugin_ref, void*)>, type=type@entry=1, state_mask=state_mask@entry=10, arg=arg@entry=0x7170036e86b0) at /data/Server/MDEV-39061/sql/sql_plugin.cc:2570 #8 0x000062e2587893af in Sql_cmd_backup::execute (this=0x71700c020e40, thd=0x71700c00bab0) at /data/Server/MDEV-39061/sql/sql_backup.cc:568 #9 0x000062e2582f5edf in mysql_execute_command (thd=thd@entry=0x71700c00bab0, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/Server/MDEV-39061/sql/sql_parse.cc:5907 #10 0x000062e2582f6ac4 in mysql_parse (thd=thd@entry=0x71700c00bab0, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7170036e93c0) at /data/Server/MDEV-39061/sql/sql_parse.cc:7947 #11 0x000062e2582f8118 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x71700c00bab0, packet=packet@entry=0x71700c0164e1 "BACKUP SERVER TO '/data/results/1784559906/26/1_clone/data' 2 CONCURRENT /* E_R Mariabackup3 QNO 2 CON_ID 57 */ ", packet_length=packet_length@entry=112, blocking=blocking@entry=true) at /data/Server/MDEV-39061/sql/sql_parse.cc:1904

The following SQL statements will be introduced: BACKUP SERVER TO '/path/to/directory' [ 1 CONCURRENT ]; BACKUP SERVER WITH [ 1 CONCURRENT ] 'command'; In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to. For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file. The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application. To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used. Note: The parameter innodb_log_recovery_start in backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database! The following will be implemented separately: MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER MDEV-40163 Partial backup and restore MDEV-39091 Back up ENGINE=RocksDB MDEV-39092 Less blocking backup of ENGINE=Aria The implementation introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV. backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles. backup_sink: Wraps a per-thread output stream as well as storage engine specific context. handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement. handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase. copy_entire_file(): A file copying service for POSIX systems. copy_file(): A partial or sparse file-copying service for all systems. backup_stream_append(): Equivalent to copy_file(), but appending to a stream. On Linux, this uses sendfile(2), which assumes that the source data will not be changed before the data has been consumed from the pipe. backup_stream_append_async(): A variant of backup_stream_append() where the source file region is guaranteed to be immutable after the call returns. We must not use Linux sendfile(2) for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe. InnoDB_backup::context: Backup context, attached to backup_sink so that context can continue to exist between the time a BACKUP SERVER releases all locks and another BACKUP SERVER starts executing, with innodb_backup pointing to the new backup, while the old backup is still being finished. InnoDB_backup::queue: Collection of tablespace IDs and payload sizes at the start of the backup. If any file is created or extended while the backup is executing, we must have the corresponding write-ahead-log entries that we are copying since the latest checkpoint that was completed when the backup started. If any tablespaces are deleted during the backup, we may or may not copy them, and the application of a FILE_DELETE record will remove them. Similarly, FILE_RENAME or FILE_CREATE records will take care of renaming or creating files during recovery (applying the backed-up log). fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup. fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress). fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup. log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF. log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup. buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread(). fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files.

innodb_backup_batch_wait(): Ensure that no conflicting write-fixed pages exist

dr-m self-assigned this Mar 17, 2026

dr-m added the MariaDB Corporation label Mar 17, 2026

dr-m force-pushed the MDEV-14992 branch 2 times, most recently from 2723322 to 1703796 Compare March 18, 2026 11:01

vuvova reviewed Mar 18, 2026

View reviewed changes

Comment thread sql/sql_backup.cc

dr-m force-pushed the MDEV-14992 branch 2 times, most recently from 9a529de to 857edeb Compare March 23, 2026 08:28

dr-m changed the base branch from 11.4 to 12.3 March 24, 2026 11:51

dr-m force-pushed the MDEV-14992 branch 3 times, most recently from 8149b3d to c08d121 Compare March 27, 2026 09:48

dr-m force-pushed the MDEV-14992 branch from fcf4ee1 to b182d72 Compare April 8, 2026 08:56

dr-m mentioned this pull request Apr 8, 2026

MDEV-39101 Make BACKUP SERVER mutually exclusive with itself and BACKUP STAGE #4892

Open

dr-m commented Apr 15, 2026

View reviewed changes

Comment thread storage/innobase/handler/backup_innodb.cc Outdated

Comment thread mysql-test/suite/backup/backup_innodb.test

dr-m changed the base branch from 12.3 to main May 5, 2026 10:49

dr-m force-pushed the MDEV-14992 branch from bcbda03 to e0d850e Compare May 5, 2026 12:06

dr-m force-pushed the MDEV-14992 branch from e0d850e to 0c52540 Compare May 18, 2026 09:40

dr-m commented May 19, 2026

View reviewed changes

Comment thread sql/sql_backup.cc Outdated

dr-m force-pushed the MDEV-14992 branch from bdec600 to 45e7902 Compare May 20, 2026 06:52

dr-m mentioned this pull request May 21, 2026

MDEV-39092 Copy Aria data and logs as part of backup #4971

Open

dr-m commented May 27, 2026

View reviewed changes

Comment thread storage/innobase/handler/backup_innodb.cc Outdated

dr-m force-pushed the MDEV-14992 branch from d595ce0 to c0e48fc Compare May 29, 2026 14:17

dr-m mentioned this pull request Jun 5, 2026

MDEV-39861 innodb_log_recovery_target wrongly opens log in read-write mode #5185

Merged

dr-m force-pushed the MDEV-14992 branch from b98be03 to 10539aa Compare June 8, 2026 09:35

dr-m mentioned this pull request Jun 11, 2026

MDEV-39061 mariadb-backup compatible wrappers for BACKUP SERVER #5140

Open

dr-m commented Jun 15, 2026

View reviewed changes

Comment thread storage/innobase/buf/buf0flu.cc

dr-m mentioned this pull request Jun 18, 2026

MDEV-40063 Corruption due to race in SET GLOBAL innodb_log_archive=ON #5253

Merged

dr-m force-pushed the MDEV-14992 branch from e81af7f to 6c8a37f Compare June 18, 2026 09:59

dr-m force-pushed the MDEV-14992 branch from 53cf648 to 87036f8 Compare June 25, 2026 13:10

dr-m force-pushed the MDEV-14992 branch from 87036f8 to 4769a43 Compare June 25, 2026 13:18

dr-m commented Jun 26, 2026

View reviewed changes

Comment thread storage/maria/ma_backup.cc Outdated

dr-m commented Jun 26, 2026

View reviewed changes

Comment thread storage/maria/ma_backup_server.cc Outdated

dr-m commented Jun 26, 2026

View reviewed changes

Comment thread storage/innobase/handler/backup_innodb.cc

dr-m changed the base branch from main to 13.0 June 26, 2026 12:54

dr-m force-pushed the MDEV-14992 branch from a7570c4 to 81b3ae7 Compare June 29, 2026 08:23

dr-m requested a review from Thirunarayanan June 29, 2026 08:34

dr-m changed the title ~~MDEV-14992 BACKUP SERVER to mounted file system~~ MDEV-14992 BACKUP SERVER Jun 29, 2026

dr-m marked this pull request as ready for review June 29, 2026 08:35

dr-m commented Jun 29, 2026

View reviewed changes

dr-m force-pushed the MDEV-14992 branch 2 times, most recently from 9e27d73 to 5235e7a Compare June 30, 2026 12:43

Thirunarayanan reviewed Jul 27, 2026

View reviewed changes

dr-m commented Aug 3, 2026

View reviewed changes

dr-m added 2 commits August 4, 2026 16:05

squash! e16b1b4

bfed32b

innodb_backup_batch_wait(): Ensure that no conflicting write-fixed pages exist

dr-m force-pushed the MDEV-14992 branch from 5235e7a to bfed32b Compare August 4, 2026 13:26


		dict_sys.lock(SRW_LOCK_CALL);

		if (fil_system.have_all_spaces) {

Uh oh!

Uh oh!

Conversation

dr-m commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dr-m commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thirunarayanan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

dr-m commented Mar 17, 2026 •

edited

Loading

CLAassistant commented Mar 17, 2026 •

edited

Loading