Designing a WAL in C: Append-Only, fsync, and Crash Consistency

You can ship fast storage without gambling on power-loss corner cases. The trick is the same one databases have used for decades: a Write-Ahead Log (WAL). We’ll build a compact WAL in C that appends records, survives crashes, and recovers deterministically—while keeping performance honest with group commit and careful fsync semantics.

Our goals:

Durability you can explain: an acknowledged record must survive power loss.
Crash-only design: on startup, scan, accept a torn tail, and continue.
Sequential I/O: one append-only write path with batched fsync.
Portable C: no undefined behavior, robust I/O loops, checksums to detect damage.
Simplicity first: segments on disk, records inside, recovery via scan + redo.

The write path in one picture

At commit time we append an entry to the WAL, flush it durably, then acknowledge the caller. Data files are updated later (checkpointing), not on the critical path.

sequenceDiagram participant App as Application participant WAL as WAL Append API participant OS as Kernel/Page Cache participant Disk as Storage App->>WAL: wal_append(record) WAL->>OS: write(record bytes) OS-->>WAL: bytes accepted (buffered) Note over WAL: Group multiple appends WAL->>OS: fsync/fdatasync OS->>Disk: flush dirty pages + metadata barriers Disk-->>OS: durable OS-->>WAL: fsync ok WAL-->>App: ACK (commit) Note over WAL,Disk: Data files updated later by checkpoint

What makes a durable record?

A record is considered durable when:

Its bytes are fully written into the log file (no partial/truncated header/body), and
A successful fsync/fdatasync on the file descriptor occurs after those bytes were written, and
On segment rotation, the parent directory entry is also durable (directory fsync) before the process advertises the new segment name.

Crash tolerance rule: It’s acceptable to leave a partially written last record in the file. On recovery, we scan until a record fails validation (length bounds and checksum), then truncate the file at the last valid offset and continue.

Record layout (self-describing and scan-friendly)

We want a header that lets us skip forward, a checksum that detects tears or bit-rot, and a type field for future extensions. We’ll keep fields fixed-width and encode lengths in little-endian for portability.

#include <stdint.h>
 
// On-disk record header; all fields little-endian when serialized.
// Keep the header compact and aligned to ease parsing.
struct WalRecordHeader {
  uint32_t total_len;   // bytes: header + payload + trailer (>= sizeof(header)+sizeof(trailer))
  uint16_t type;        // user-defined (e.g., 1=PUT, 2=DEL, ...)
  uint16_t flags;       // future use (compression, etc.)
  uint64_t txn_id;      // optional correlation; 0 if unused
};
 
// Trailer placed immediately after payload
struct WalRecordTrailer {
  uint32_t crc32;       // checksum over header (with total_len=0) + payload
};

Guidelines:

total_len allows scanners to jump to the next record fast.
Checksums are computed over a stable image. Zeroing total_len in the header during CRC avoids self-referential changes across encodings.
Keep payload opaque to the WAL. The WAL is a byte transport with integrity, not a schema.

Segment layout and lifecycle

The log is split into segment files to cap growth and simplify retention/archival:

File name: wal-<epoch>-<seq>.log (monotonic). The name itself provides ordering.
Header (optional): magic, version, creation timestamp. Not required for simple scanners; we’ll include to catch misroutes.
Body: concatenated records. No index is needed for forward scans.
Trailer: none. Crash-only tail is detected via record validation.

Rotation:

When current segment exceeds a size budget (e.g., 256 MiB) or time budget, close current fd.
fsync(old_fd) then fsync(dir_fd) to persist the closed file and its directory entry.
Create the new segment with O_CREAT|O_EXCL|O_CLOEXEC, write a small header, and start appending.

fsync discipline and group commit

fsync is expensive; paying it per append kills throughput. The classic answer is group commit: batch many appends under one fsync and still provide a clear durability boundary.

Patterns:

Commit-every-N-ms: queue appends and flush on a timer (e.g., every 2–10 ms) or when bytes exceed a threshold.
Commit-on-demand: flush immediately when a caller requests durable semantics (e.g., WAL_SYNC) while others ride along if they’re pending.
fdatasync vs fsync: prefer fdatasync for data-only durability; if your filesystem requires metadata durability (e.g., new file or size extension semantics), use fsync or a file-system-specific dance.

Correctness footnotes:

After a successful fsync, a subsequent crash must not lose acknowledged records. If your platform/filesystem has caveats (e.g., write barriers, battery-backed cache), document them.
When you create/rename segment files, fsync the directory to publish the name durably.

Minimal C API shape

We’ll separate the append path (hot), flushing policy, and recovery scan. This keeps the hot path small, lockable, and testable.

#include <stddef.h>
#include <stdint.h>
#include <stdbool.h>
 
typedef struct Wal Wal;
 
struct WalOptions {
  const char *dir_path;       // directory for segments
  size_t     segment_bytes;   // roll at ~this size
  bool       preallocate;     // optional fallocate-like behavior
};
 
// Open (recover if needed) and return a WAL handle positioned at the end.
Wal *wal_open(const struct WalOptions *opt);
 
// Append a payload as a single WAL record. If sync==true, ensure durability
// before returning. Returns an application-level LSN (monotonic).
uint64_t wal_append(Wal *w, uint16_t type, uint16_t flags,
                    uint64_t txn_id, const void *payload, size_t len,
                    bool sync);
 
// Opportunistically flush pending appends (group commit). Returns 0 on success.
int wal_flush(Wal *w);
 
// Close the current segment (flush+fsync) and rotate.
int wal_rotate(Wal *w);
 
// Crash recovery: scan segments, validate records, and invoke a callback
// for each valid record in order. Truncate torn tails and resume.
typedef int (*wal_replay_cb)(uint16_t type, uint16_t flags, uint64_t txn_id,
                             const void *payload, size_t len, void *arg);
int wal_replay(const char *dir_path, wal_replay_cb cb, void *arg);
 
void wal_close(Wal *w);

Robust writes (boring on purpose)

The core of append is a small, reliable write loop that tolerates short writes and signals. We’ll build on it later with buffered headers and checksum calculation.

#include <errno.h>
#include <stdint.h>
#include <stdbool.h>
#include <unistd.h>
 
static bool write_all(int fd, const void *buf, size_t len) {
  const unsigned char *p = (const unsigned char *)buf;
  size_t left = len;
  while (left > 0) {
    ssize_t w = write(fd, p, left);
    if (w > 0) { p += (size_t)w; left -= (size_t)w; continue; }
    if (w == -1 && errno == EINTR) continue;
    return false; // error
  }
  return true;
}

Append logic (sketch):

Serialize header with total_len and zeroed total_len copy used for CRC.
Compute CRC over [header_with_total_len_zero | payload].
Write header → payload → trailer in order using write_all.
Optionally set a “needs_sync” flag and let a flusher thread call fdatasync per policy; if caller requested sync, perform fdatasync immediately.

Recovery logic (sketch):

For each segment in name order, open and scan from offset 0.
At each offset, try to read a full header; validate total_len bounds.
Read payload+trailer, recompute CRC; if mismatch, truncate file at current offset and stop scanning this segment.
Invoke the application callback with the payload.

These two paths—append and scan—are the only ones you need to prove correct. Everything else (rotation, preallocation, group commit) is plumbing around them.

Where we are and what’s next

We now have the core durability story and file formats: append-only records inside rolling segments, a checksum-backed scan, and fsync boundaries that give precise guarantees. Next up, we’ll turn the sketches into concrete append/replay implementations, cover directory fsync on rotations, and wire in a small group-commit loop you can benchmark.

Portable serialization helpers (little-endian)

We’ll serialize integers in little-endian regardless of host. Helpers keep the hot path clean and avoid aliasing pitfalls.

#include <stdint.h>
 
static inline uint16_t le16(uint16_t v) {
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
  return v;
#else
  return (uint16_t)((v<<8) | (v>>8));
#endif
}
 
static inline uint32_t le32(uint32_t v) {
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
  return v;
#else
  return ((v & 0x000000FFu) << 24) | ((v & 0x0000FF00u) << 8)
       | ((v & 0x00FF0000u) >> 8)  | ((v & 0xFF000000u) >> 24);
#endif
}
 
static inline uint64_t le64(uint64_t v) {
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
  return v;
#else
  return ((uint64_t)le32((uint32_t)v) << 32) | (uint64_t)le32((uint32_t)(v >> 32));
#endif
}

When computing the checksum, build a temporary header image with total_len zeroed and pass it along with the payload to the CRC routine to avoid self-influence.

Checksums without heroics

Any fast 32-bit checksum works (CRC32C, CRC32-IEEE, xxHash32). For reproducibility across platforms, prefer a stable implementation rather than vendor intrinsics. Below we assume a function:

// Compute CRC32 over one or two buffers; seed=0 for new streams.
uint32_t crc32_update(uint32_t seed, const void *data, size_t len);

You can bring this from zlib (crc32), a small CRC32C implementation, or xxhash (xxhash32) depending on licensing and performance needs.

Append implementation with writev

Fewer syscalls means less overhead. We’ll format a header, payload, and trailer, then send them via writev.

#include <sys/uio.h>
#include <string.h>
 
struct WalIo {
  int fd;              // current segment fd
  uint64_t lsn_next;   // next LSN to assign
  size_t seg_bytes;    // bytes written to current segment
  size_t seg_limit;    // rotation threshold
  int dirfd;           // open(dir_path, O_RDONLY|O_DIRECTORY)
  int needs_sync;      // 0/1: pending fsync
};
 
static bool wal_write_record(struct WalIo *io,
                             uint16_t type, uint16_t flags,
                             uint64_t txn_id,
                             const void *payload, size_t len,
                             uint64_t *out_lsn) {
  struct WalRecordHeader h = {0};
  struct WalRecordTrailer t = {0};
 
  // Serialize header fields (little-endian)
  uint32_t total = (uint32_t)(sizeof h + len + sizeof t);
  h.total_len = le32(total);
  h.type      = le16(type);
  h.flags     = le16(flags);
  h.txn_id    = le64(txn_id);
 
  // Build checksum over [header with total_len=0] + payload
  struct WalRecordHeader h_crc = h; h_crc.total_len = 0;
  uint32_t crc = 0;
  crc = crc32_update(crc, &h_crc, sizeof h_crc);
  crc = crc32_update(crc, payload, len);
  t.crc32 = le32(crc);
 
  struct iovec iov[3] = {
    { .iov_base = &h,      .iov_len = sizeof h },
    { .iov_base = (void*)payload, .iov_len = len },
    { .iov_base = &t,      .iov_len = sizeof t },
  };
 
  // Single writev for locality; fall back to write_all() if desired
  ssize_t w = writev(io->fd, iov, 3);
  if (w != (ssize_t)(sizeof h + len + sizeof t)) return false;
 
  io->seg_bytes += (size_t)w;
  io->needs_sync = 1;
  *out_lsn = io->lsn_next++;
  return true;
}

Synchronous durability is just a fdatasync after the writev:

#include <unistd.h>
 
static int wal_sync(struct WalIo *io) {
  if (!io->needs_sync) return 0;
  if (fdatasync(io->fd) != 0) return -1;
  io->needs_sync = 0;
  return 0;
}

Recovery scanner with tail truncation

On startup, iterate segments in lexicographic order, scanning each from offset 0. Stop at first invalid record; truncate there; replay all valid ones via callback.

#include <fcntl.h>
#include <sys/stat.h>
 
static int scan_segment(const char *path, wal_replay_cb cb, void *arg) {
  int fd = open(path, O_RDONLY | O_CLOEXEC);
  if (fd < 0) return -1;
 
  off_t off = 0;
  for (;;) {
    struct WalRecordHeader h; ssize_t r = pread(fd, &h, sizeof h, off);
    if (r == 0) break;                 // EOF (clean end)
    if (r != (ssize_t)sizeof h) {      // torn header
      // Truncate tail and stop
      int wfd = open(path, O_WRONLY | O_CLOEXEC);
      if (wfd >= 0) { (void)ftruncate(wfd, off); close(wfd); }
      break;
    }
    uint32_t total = le32(h.total_len);
    if (total < (uint32_t)(sizeof h + sizeof(struct WalRecordTrailer))) {
      int wfd = open(path, O_WRONLY | O_CLOEXEC);
      if (wfd >= 0) { (void)ftruncate(wfd, off); close(wfd); }
      break;
    }
 
    size_t payload_len = (size_t)total - sizeof h - sizeof(struct WalRecordTrailer);
    // Read payload + trailer
    size_t rest = (size_t)total - sizeof h;
    unsigned char *buf = (unsigned char *)malloc(rest);
    if (!buf) { close(fd); return -1; }
    r = pread(fd, buf, rest, off + (off_t)sizeof h);
    if (r != (ssize_t)rest) {
      free(buf);
      int wfd = open(path, O_WRONLY | O_CLOEXEC);
      if (wfd >= 0) { (void)ftruncate(wfd, off); close(wfd); }
      break;
    }
 
    // Validate checksum
    struct WalRecordTrailer t;
    memcpy(&t, buf + payload_len, sizeof t);
    uint32_t got = le32(t.crc32);
    struct WalRecordHeader h_crc = h; h_crc.total_len = 0;
    uint32_t crc = 0;
    crc = crc32_update(crc, &h_crc, sizeof h_crc);
    crc = crc32_update(crc, buf, payload_len);
    if (crc != got) {
      free(buf);
      int wfd = open(path, O_WRONLY | O_CLOEXEC);
      if (wfd >= 0) { (void)ftruncate(wfd, off); close(wfd); }
      break;
    }
 
    // REDO callback
    uint16_t type  = le16(h.type);
    uint16_t flags = le16(h.flags);
    uint64_t txn   = le64(h.txn_id);
    (void)cb(type, flags, txn, buf, payload_len, arg);
 
    free(buf);
    off += (off_t)total;
  }
 
  close(fd);
  return 0;
}

Enumerating segments is platform-specific; simplest is to opendir() the directory, collect file names matching wal-*.log, sort them, then pass to scan_segment.

Segment rotation with directory fsync

Proper publication of new segments requires syncing both the file and the directory entry. A minimal rotation sequence:

#include <stdio.h>
 
static int rotate_segment(struct WalIo *io, const char *dir, const char *next_name) {
  // Close old fd after a final sync
  if (wal_sync(io) != 0) return -1;
  if (fsync(io->fd) != 0) return -1;
  close(io->fd);
 
  // fsync directory to persist the closed file’s metadata
  if (fsync(io->dirfd) != 0) return -1;
 
  // Create new segment
  char path[1024];
  snprintf(path, sizeof path, "%s/%s", dir, next_name);
  int fd = open(path, O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0644);
  if (fd < 0) return -1;
 
  // Optionally write a file header (magic/version) and sync it
  // ... write header bytes ...
  if (fdatasync(fd) != 0) { close(fd); return -1; }
 
  // Publish by syncing the directory
  if (fsync(io->dirfd) != 0) { close(fd); return -1; }
 
  io->fd = fd;
  io->seg_bytes = 0;
  return 0;
}

Preallocation (posix_fallocate) helps avoid late ENOSPC and fragmentation on some filesystems; it’s optional and workload-dependent.

Group commit: a tiny flusher

Two API points suffice: appenders set needs_sync=1; a background flusher calls fdatasync on a cadence or a byte threshold. Threads coordinate with a mutex/condvar.

#include <pthread.h>
 
struct Wal {
  struct WalIo io;
  pthread_mutex_t mu;
  pthread_cond_t  cv;
  int stop;
};
 
static void *flusher_thread(void *arg) {
  struct Wal *w = (struct Wal *)arg;
  struct timespec ts;
  while (1) {
    clock_gettime(CLOCK_REALTIME, &ts);
    ts.tv_nsec += 5 * 1000 * 1000; // ~5ms cadence
    if (ts.tv_nsec >= 1000000000L) { ts.tv_sec++; ts.tv_nsec -= 1000000000L; }
 
    pthread_mutex_lock(&w->mu);
    int rc = 0;
    while (!w->stop && !w->io.needs_sync) {
      rc = pthread_cond_timedwait(&w->cv, &w->mu, &ts);
      if (rc == ETIMEDOUT) break;
    }
    int need = w->io.needs_sync;
    pthread_mutex_unlock(&w->mu);
 
    if (w->stop) break;
    if (need) (void)wal_sync(&w->io);
  }
  return NULL;
}

On each append, signal the flusher if you want low latency; on explicit wal_flush, perform a synchronous fdatasync to establish a boundary.

Concurrency, locking, and LSNs

Guard file descriptor writes and seg_bytes with a mutex to serialize append and rotation. The writev hot path should hold the mutex briefly.
Assign LSNs from an atomic counter or under the same mutex; persist only in memory (they can be reconstructed by replay order).
For multi-segment recovery, the total order is lexicographic by segment name and then by file offset.

A robust writev loop (advance iovecs on partials)

While a single writev often completes, it may legally return a short count. A tiny helper keeps the append path correct under all conditions.

#include <sys/uio.h>
 
static void advance_iovecs(struct iovec **piov, int *piovcnt, size_t bytes) {
  struct iovec *iov = *piov; int n = *piovcnt; size_t left = bytes;
  while (n > 0 && left > 0) {
    if (left >= iov->iov_len) { left -= iov->iov_len; ++iov; --n; }
    else { iov->iov_base = (char*)iov->iov_base + left; iov->iov_len -= left; left = 0; }
  }
  *piov = iov; *piovcnt = n;
}
 
static bool writev_all(int fd, struct iovec *iov, int iovcnt) {
  while (iovcnt > 0) {
    ssize_t w = writev(fd, iov, iovcnt);
    if (w > 0) { advance_iovecs(&iov, &iovcnt, (size_t)w); continue; }
    if (w == -1 && errno == EINTR) continue;
    return false;
  }
  return true;
}

To use in the append path, replace the single writev with writev_all(io->fd, iov, 3).

Putting it together: wal_append

Below is a minimal wrapper that handles rotation, append, and optional synchronous durability. Locking and error handling are explicit.

#include <pthread.h>
 
struct Wal {
  struct WalIo io;
  char dir[512];
  pthread_mutex_t mu;
  pthread_cond_t  cv;
  int stop;
};
 
uint64_t wal_append(Wal *w, uint16_t type, uint16_t flags,
                    uint64_t txn_id, const void *payload, size_t len,
                    bool sync) {
  pthread_mutex_lock(&w->mu);
 
  // Rotate if needed (size-based)
  size_t need = sizeof(struct WalRecordHeader) + len + sizeof(struct WalRecordTrailer);
  if (w->io.seg_bytes + need > w->io.seg_limit) {
    // Compute next segment name (not shown), then rotate
    if (rotate_segment(&w->io, w->dir, /*next_name*/"wal-next.log") != 0) {
      pthread_mutex_unlock(&w->mu); return 0;
    }
  }
 
  uint64_t lsn = 0;
  bool ok = wal_write_record(&w->io, type, flags, txn_id, payload, len, &lsn);
  if (!ok) { pthread_mutex_unlock(&w->mu); return 0; }
 
  if (sync) {
    if (wal_sync(&w->io) != 0) { pthread_mutex_unlock(&w->mu); return 0; }
  } else {
    pthread_cond_signal(&w->cv); // let flusher piggyback
  }
 
  pthread_mutex_unlock(&w->mu);
  return lsn;
}

In production, compute the next segment name based on a monotonic tuple (epoch, sequence) and store it in Wal state; avoid hardcoding.

Opening and closing the WAL cleanly

Initialization establishes directory handles, replays existing segments, and prepares the active segment for appends.

#include <dirent.h>
 
struct seg { unsigned long long a, b; char name[256]; };
 
static int cmp_seg(const void *pa, const void *pb) {
  const struct seg *x = (const struct seg*)pa, *y = (const struct seg*)pb;
  if (x->a != y->a) return (x->a < y->a) ? -1 : 1;
  if (x->b != y->b) return (x->b < y->b) ? -1 : 1;
  return 0;
}
 
static int list_segments(const char *dir, struct seg **out, int *out_n) {
  DIR *d = opendir(dir); if (!d) return -1;
  struct dirent *de; int cap=16, n=0; struct seg *v = (struct seg*)malloc(cap*sizeof *v);
  while ((de = readdir(d))) {
    unsigned long long a=0,b=0;
    if (sscanf(de->d_name, "wal-%llu-%llu.log", &a, &b) == 2) {
      if (n==cap){ cap*=2; v=(struct seg*)realloc(v,cap*sizeof *v);} v[n].a=a; v[n].b=b;
      snprintf(v[n].name, sizeof v[n].name, "%s", de->d_name);
      n++;
    }
  }
  closedir(d);
  qsort(v, n, sizeof *v, cmp_seg);
  *out = v; *out_n = n; return 0;
}
 
Wal *wal_open(const struct WalOptions *opt) {
  Wal *w = (Wal*)calloc(1, sizeof *w);
  snprintf(w->dir, sizeof w->dir, "%s", opt->dir_path);
  w->io.seg_limit = opt->segment_bytes ? opt->segment_bytes : (256u<<20);
  w->io.dirfd = open(opt->dir_path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
  if (w->io.dirfd < 0) { free(w); return NULL; }
 
  pthread_mutex_init(&w->mu, NULL); pthread_cond_init(&w->cv, NULL);
 
  // Enumerate and replay
  struct seg *segs = NULL; int n = 0;
  if (list_segments(opt->dir_path, &segs, &n) != 0) { wal_close(w); return NULL; }
  for (int i = 0; i < n; ++i) {
    char p[1024]; snprintf(p, sizeof p, "%s/%s", opt->dir_path, segs[i].name);
    (void)scan_segment(p, /*cb*/NULL, NULL); // or pass your REDO callback
  }
 
  // Open last segment or create a new one
  char tail[1024];
  if (n > 0) {
    snprintf(tail, sizeof tail, "%s/%s", opt->dir_path, segs[n-1].name);
    w->io.fd = open(tail, O_RDWR | O_APPEND | O_CLOEXEC);
  } else {
    // Create first segment, e.g., wal-<epoch>-00000001.log
    snprintf(tail, sizeof tail, "%s/%s", opt->dir_path, "wal-0-00000001.log");
    w->io.fd = open(tail, O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0644);
    if (w->io.fd >= 0) (void)fdatasync(w->io.fd), fsync(w->io.dirfd);
  }
  free(segs);
  if (w->io.fd < 0) { wal_close(w); return NULL; }
 
  // Determine current size for seg_bytes
  struct stat st; if (fstat(w->io.fd, &st) == 0) w->io.seg_bytes = (size_t)st.st_size;
 
  // Start flusher thread (optional)
  pthread_t th; (void)pthread_create(&th, NULL, flusher_thread, w); (void)pthread_detach(th);
 
  return w;
}
 
void wal_close(Wal *w) {
  if (!w) return;
  pthread_mutex_lock(&w->mu); w->stop = 1; pthread_cond_broadcast(&w->cv); pthread_mutex_unlock(&w->mu);
  (void)wal_sync(&w->io);
  if (w->io.fd >= 0) { (void)fsync(w->io.fd); close(w->io.fd); }
  if (w->io.dirfd >= 0) { (void)fsync(w->io.dirfd); close(w->io.dirfd); }
  pthread_mutex_destroy(&w->mu); pthread_cond_destroy(&w->cv);
  free(w);
}

This skeleton prioritizes safety over brevity: enumerate, replay, open tail, establish sizes, and arrange background flushing.

Checkpointing and retention policy (integration sketch)

The WAL carries durability; checkpoints make startup faster and bound log size. A simple, safe flow:

Apply WAL to data files in memory (or a staging area).
Write a checkpoint file (snapshot) to a temporary name.
fdatasync the checkpoint file, then atomically rename() over the previous checkpoint.
fsync the directory to publish the rename.
Compute a retention watermark (LSN or segment tuple) and delete segments strictly older than the watermark; fsync the directory after unlinks.

Avoid deleting the active segment; only prune fully superseded ones. If multiple processes may read the WAL (e.g., replication), coordinate watermarks.

Filesystem and platform caveats you should measure

fdatasync vs fsync: some filesystems treat them equivalently; others require fsync for extending file length or for metadata like new inodes. Measure on your target.
Journaling modes (e.g., ext4 data=ordered vs writeback) impact when data becomes durable relative to metadata. Your sequence (file fsync, then directory fsync) remains correct across modes.
NVMe write caches and RAID controllers may reorder writes unless flush/FUA is honored. fsync should issue the right barriers; ensure the stack is not lying (test with power-cut simulations if possible).
macOS APFS and BSDs have solid fsync/fcntl(F_FULLFSYNC) semantics; prefer fsync and directory fsync after rename/create.

Testing strategy that builds confidence

Kill -9 testing: run a writer that appends continuously; at random intervals, SIGKILL it, restart, and verify replay idempotency and no corruption.
Torn-write injection: randomly truncate the active segment or flip a byte in the last record while the process is down; expect truncation at last valid offset.
Latency/throughput: vary group-commit cadence; record ops/sec vs p50/p99 append latency.
Disk pressure: fill the filesystem to near capacity and ensure errors surface predictably; optional posix_fallocate to avoid surprises.

Durability modes and API semantics

You can expose durability as a caller choice without complicating the core:

Async append: queue bytes, return immediately; background flusher will establish a real boundary on cadence. Best throughput, bounded latency.
Synchronous append: call wal_append(..., /*sync=*/true) to fdatasync before returning. Higher latency, clear boundary per append.
Manual boundary: callers perform many async appends and then call wal_flush() to force a boundary.

Implementing wal_flush and a safe wal_rotate wrapper:

int wal_flush(Wal *w) {
  pthread_mutex_lock(&w->mu);
  int rc = wal_sync(&w->io);
  pthread_mutex_unlock(&w->mu);
  return rc;
}
 
static void next_segment_name(Wal *w, char *out, size_t cap) {
  // Example: maintain epoch/seq in memory; persist via filename
  static unsigned long long epoch = 0; // bump on start or policy change
  static unsigned long long seq = 1;
  ++seq;
  snprintf(out, cap, "wal-%llu-%08llu.log", epoch, seq);
}
 
int wal_rotate(Wal *w) {
  pthread_mutex_lock(&w->mu);
  char name[64]; next_segment_name(w, name, sizeof name);
  int rc = rotate_segment(&w->io, w->dir, name);
  pthread_mutex_unlock(&w->mu);
  return rc;
}

Preallocation, alignment, and write size

Preallocate with posix_fallocate(fd, 0, segment_bytes) when opening/rotating a segment to reduce ENOSPC surprises and fragmentation.
Favor larger writes: grouping header+payload+trailer into one writev already helps. If you buffer multiple small payloads into a temporary slab and emit a single writev of many records, you can amortize syscall cost further (just keep failure semantics the same).
Padding is optional: the WAL does not require 4K alignment. If you choose to pad for device-friendly sizes, use a distinct record type (e.g., TYPE_PAD) with a CRC so scanning remains uniform.

static int wal_preallocate(int fd, size_t bytes) {
#if defined(_POSIX_C_SOURCE) && _POSIX_C_SOURCE >= 200112L
  return posix_fallocate(fd, 0, (off_t)bytes);
#else
  (void)fd; (void)bytes; return 0; // best-effort noop
#endif
}

Call wal_preallocate after creating a new segment if the option is enabled.

Observability: measure what matters

Track basic counters and latencies. Keep it simple and lock-free where possible.

struct WalMetrics {
  unsigned long long appends;
  unsigned long long bytes;
  unsigned long long fsyncs;
  unsigned long long flush_batches;
  unsigned long long crc_failures;
  unsigned long long tail_truncations;
  unsigned long long append_ns_total;
  unsigned long long fsync_ns_total;
};
 
static inline unsigned long long now_ns(void) {
  struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
  return (unsigned long long)ts.tv_sec*1000000000ull + (unsigned long long)ts.tv_nsec;
}

Wrap the hot paths to accumulate time spent in append and fsync. Export metrics periodically for dashboards.

End-to-end usage example

Open a WAL, append some records, close; then replay on startup.

static int redo_print(uint16_t type, uint16_t flags, uint64_t txn,
                      const void *payload, size_t len, void *arg) {
  (void)flags; (void)arg;
  printf("REDO type=%u txn=%llu len=%zu\n", (unsigned)type, (unsigned long long)txn, len);
  return 0;
}
 
int main(void) {
  struct WalOptions opt = { .dir_path = "./wal", .segment_bytes = 256u<<20, .preallocate = false };
  Wal *w = wal_open(&opt);
  if (!w) { perror("wal_open"); return 1; }
 
  const char msg[] = "hello wal";
  uint64_t lsn = wal_append(w, /*type=*/1, /*flags=*/0, /*txn_id=*/42, msg, sizeof msg, /*sync=*/false);
  if (!lsn) { fprintf(stderr, "append failed\n"); }
  (void)wal_flush(w); // establish durability boundary for all pending
  wal_close(w);
 
  // Recovery path
  (void)wal_replay("./wal", redo_print, NULL);
  return 0;
}

Safety notes for production

Validate all lengths before allocation; clamp to sane maxima.
Treat any short pread/pwrite after partial progress as a tail tear; truncate there.
Always fsync the directory when creating/renaming/removing files that affect recovery.
Keep your REDO idempotent and bounded-time. It must not depend on ambient state outside the committed payload.
Test under kill -9, aggressive truncation, small segment_bytes, and tiny group-commit cadences to surface edge cases.

Tuning cheatsheet

Segment size: 128–1024 MiB is a good range. Larger segments = fewer files, longer recovery; smaller = faster trims.
Group-commit cadence: 2–10 ms often balances p99 latency and throughput.
Checksum: CRC32C is a fast, widely supported choice. Consider xxHash32 for even faster integrity checks (trade auditability).
Preallocation: enable on spinning disks and busy filesystems; optional on NVMe.

Wrap-up

A WAL is a small, sharp tool: append records sequentially, checksum them, draw a durability line with fsync, and accept that the tail might tear. With disciplined rotation, directory fsync, and a conservative recovery scan, you get predictable durability and fast restarts. Add group commit for throughput, keep REDO idempotent, and measure. The result is a compact C module that protects your data without slowing you down.