vfs

package
v0.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 2, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package vfs provides filesystem abstractions including fault injection for testing.

FaultInjectionFS wraps a real filesystem and allows injecting errors and simulating crashes for testing recovery code.

Reference: RocksDB v10.7.5

  • utilities/fault_injection_fs.h
  • utilities/fault_injection_fs.cc

Package vfs provides filesystem abstractions including fault injection for testing.

This file provides goroutine-local fault injection capabilities, allowing different goroutines to have different error injection settings. This is useful for testing concurrent code with targeted fault injection.

Reference: RocksDB v10.7.5

  • utilities/fault_injection_fs.h (thread-local error injection)
  • utilities/fault_injection_fs.cc

lock.go implements file locking on Unix systems.

Reference: RocksDB v10.7.5

  • env/env_posix.cc (PosixEnv::LockFile)
  • env/io_posix.cc

Package vfs provides a virtual filesystem abstraction layer.

This allows RockyardKV to: - Use the real OS filesystem in production - Use a memory filesystem for testing - Use a fault-injection filesystem for crash testing

Reference: RocksDB v10.7.5 include/rocksdb/file_system.h

Index

Constants

View Source
const DefaultBlockSize = 4096

DefaultBlockSize is the default logical block size for Direct I/O alignment. Most modern filesystems use 4KB blocks.

Variables

View Source
var (
	// ErrInjectedReadError is returned when a read error is injected.
	ErrInjectedReadError = errors.New("vfs: injected read error")

	// ErrInjectedWriteError is returned when a write error is injected.
	ErrInjectedWriteError = errors.New("vfs: injected write error")

	// ErrInjectedSyncError is returned when a sync error is injected.
	ErrInjectedSyncError = errors.New("vfs: injected sync error")
)
View Source
var ErrDirectIONotSupported = errors.New("direct I/O not supported")

ErrDirectIONotSupported is returned when Direct I/O is not supported on the current platform or filesystem.

View Source
var ErrNotAligned = errors.New("buffer or offset not aligned for direct I/O")

ErrNotAligned is returned when a buffer or offset is not properly aligned for Direct I/O operations.

Functions

func AlignDown

func AlignDown(value, alignment int) int

AlignDown rounds down the given value to the previous multiple of alignment.

func AlignUp

func AlignUp(value, alignment int) int

AlignUp rounds up the given value to the next multiple of alignment.

func IsAligned

func IsAligned(value, alignment int) bool

IsAligned checks if the given value is aligned to the given alignment.

Types

type AlignedBuffer

type AlignedBuffer struct {
	// contains filtered or unexported fields
}

AlignedBuffer is a buffer that is properly aligned for Direct I/O.

func NewAlignedBuffer

func NewAlignedBuffer(capacity, alignment int) *AlignedBuffer

NewAlignedBuffer creates a new aligned buffer with the given capacity and alignment. The actual capacity may be larger than requested to accommodate alignment.

Note: In Go, we cannot guarantee memory alignment without using unsafe. This implementation provides a buffer of the requested size and relies on the caller to use aligned offsets when performing Direct I/O operations. For true memory-aligned buffers, consider using C allocation via cgo, but this project is pure Go and doesn't use cgo.

func (*AlignedBuffer) Alignment

func (b *AlignedBuffer) Alignment() int

Alignment returns the alignment of the buffer.

func (*AlignedBuffer) Bytes

func (b *AlignedBuffer) Bytes() []byte

Bytes returns the underlying byte slice.

func (*AlignedBuffer) Cap

func (b *AlignedBuffer) Cap() int

Cap returns the capacity of the buffer.

func (*AlignedBuffer) Clear

func (b *AlignedBuffer) Clear()

Clear resets the buffer to zero length.

func (*AlignedBuffer) Len

func (b *AlignedBuffer) Len() int

Len returns the length of the buffer.

func (*AlignedBuffer) Resize

func (b *AlignedBuffer) Resize(size int) error

Resize resizes the buffer to the given size. The size must not exceed the capacity.

type DirectIOFS

type DirectIOFS interface {
	FS

	// CreateWithOptions creates a new writable file with the given options.
	CreateWithOptions(name string, opts FileOptions) (DirectWritableFile, error)

	// OpenWithOptions opens an existing file for reading with the given options.
	OpenWithOptions(name string, opts FileOptions) (DirectSequentialFile, error)

	// OpenRandomAccessWithOptions opens an existing file for random access with options.
	OpenRandomAccessWithOptions(name string, opts FileOptions) (DirectRandomAccessFile, error)

	// IsDirectIOSupported returns true if Direct I/O is supported on this filesystem.
	IsDirectIOSupported() bool

	// GetBlockSize returns the filesystem block size for the given path.
	GetBlockSize(path string) int
}

DirectIOFS extends FS with Direct I/O support.

func NewDirectIOFS

func NewDirectIOFS() DirectIOFS

NewDirectIOFS creates a new filesystem with Direct I/O support.

func WrapWithDirectIO

func WrapWithDirectIO(fs FS) DirectIOFS

WrapWithDirectIO wraps an existing FS with Direct I/O support. If the underlying FS is already a DirectIOFS, it is returned unchanged. Otherwise, a fallback wrapper is created.

type DirectIOFile

type DirectIOFile interface {
	// UseDirectIO returns true if Direct I/O is enabled for this file.
	UseDirectIO() bool

	// GetRequiredBufferAlignment returns the required alignment for buffers
	// used with this file. Typically 4KB for Direct I/O.
	GetRequiredBufferAlignment() int
}

DirectIOFile is a file that supports Direct I/O operations.

type DirectIOHelper

type DirectIOHelper struct {
	// contains filtered or unexported fields
}

DirectIOHelper helps manage Direct I/O operations with alignment requirements. Mirrors RocksDB's DirectIOHelper struct in env/io_posix.h.

func NewDirectIOHelper

func NewDirectIOHelper(alignment int) *DirectIOHelper

NewDirectIOHelper creates a new DirectIOHelper with the given alignment.

func (*DirectIOHelper) AddFinished

func (h *DirectIOHelper) AddFinished(n int64)

AddFinished adds to the finished count.

func (*DirectIOHelper) AlignLength

func (h *DirectIOHelper) AlignLength(length int64) int64

AlignLength aligns a length up to the alignment boundary.

func (*DirectIOHelper) AlignOffset

func (h *DirectIOHelper) AlignOffset(offset int64) int64

AlignOffset aligns an offset down to the alignment boundary.

func (*DirectIOHelper) Alignment

func (h *DirectIOHelper) Alignment() int

Alignment returns the alignment requirement.

func (*DirectIOHelper) Finished

func (h *DirectIOHelper) Finished() int64

Finished returns the total bytes finished.

func (*DirectIOHelper) IsOffsetAndLengthAligned

func (h *DirectIOHelper) IsOffsetAndLengthAligned(offset, length int64) bool

IsOffsetAndLengthAligned checks if an offset and length are properly aligned.

type DirectRandomAccessFile

type DirectRandomAccessFile interface {
	RandomAccessFile
	DirectIOFile
}

DirectRandomAccessFile extends RandomAccessFile with Direct I/O information.

type DirectSequentialFile

type DirectSequentialFile interface {
	SequentialFile
	DirectIOFile
}

DirectSequentialFile extends SequentialFile with Direct I/O information.

type DirectWritableFile

type DirectWritableFile interface {
	WritableFile
	DirectIOFile
}

DirectWritableFile extends WritableFile with Direct I/O information.

type ErrorType

type ErrorType int

ErrorType represents the type of error to inject.

const (
	// ErrorTypeStatus returns an error status
	ErrorTypeStatus ErrorType = iota
	// ErrorTypeCorruption corrupts data (for reads)
	ErrorTypeCorruption
	// ErrorTypeTruncated returns truncated data (for reads)
	ErrorTypeTruncated
)

type FS

type FS interface {
	// Create creates a new writable file.
	// If the file already exists, it is truncated.
	Create(name string) (WritableFile, error)

	// Open opens an existing file for reading.
	Open(name string) (SequentialFile, error)

	// OpenRandomAccess opens an existing file for random access reading.
	OpenRandomAccess(name string) (RandomAccessFile, error)

	// Rename atomically renames a file.
	Rename(oldname, newname string) error

	// Remove deletes a file.
	Remove(name string) error

	// RemoveAll removes a directory and all its contents.
	RemoveAll(path string) error

	// MkdirAll creates a directory and all parent directories.
	MkdirAll(path string, perm os.FileMode) error

	// Stat returns file info.
	Stat(name string) (os.FileInfo, error)

	// Exists returns true if the file exists.
	Exists(name string) bool

	// ListDir lists files in a directory.
	ListDir(path string) ([]string, error)

	// Lock acquires an exclusive lock on a file.
	// Returns a Locker that must be closed to release the lock.
	Lock(name string) (io.Closer, error)

	// SyncDir syncs a directory to ensure metadata changes are durable.
	// This is required after file rename to ensure the rename is durable.
	// Reference: RocksDB file/filename.cc SetCurrentFile calls FsyncWithDirOptions.
	SyncDir(path string) error
}

FS is the main filesystem interface.

func Default

func Default() FS

Default returns the default OS filesystem.

type FaultInjectionFS

type FaultInjectionFS struct {
	// contains filtered or unexported fields
}

FaultInjectionFS wraps an FS and allows injecting errors. It tracks unsynced data per file to simulate data loss on crash.

Directory entry durability: Entries created by Rename are not durable until SyncDir is called on the parent directory. On simulated crash, pending renames (without dir sync) are reverted.

func NewFaultInjectionFS

func NewFaultInjectionFS(base FS) *FaultInjectionFS

NewFaultInjectionFS creates a new fault-injecting filesystem wrapper.

func (*FaultInjectionFS) ClearErrors

func (fs *FaultInjectionFS) ClearErrors()

ClearErrors clears all error injection.

func (*FaultInjectionFS) Create

func (fs *FaultInjectionFS) Create(name string) (WritableFile, error)

Create creates a new writable file with fault injection.

func (*FaultInjectionFS) DeleteUnsyncedFiles

func (fs *FaultInjectionFS) DeleteUnsyncedFiles() error

DeleteUnsyncedFiles removes files that were created but never synced.

func (*FaultInjectionFS) DropUnsyncedData

func (fs *FaultInjectionFS) DropUnsyncedData() error

DropUnsyncedData simulates a crash by dropping all unsynced data. This truncates all files to their last synced position.

func (*FaultInjectionFS) Exists

func (fs *FaultInjectionFS) Exists(name string) bool

Exists returns true if the file exists.

func (*FaultInjectionFS) GetFileState

func (fs *FaultInjectionFS) GetFileState(path string) (syncedPos, currentPos int64, ok bool)

GetFileState returns the tracked state for a file.

func (*FaultInjectionFS) GetFileSyncLiePattern

func (fs *FaultInjectionFS) GetFileSyncLiePattern() string

GetFileSyncLiePattern returns the current file sync lie pattern.

func (*FaultInjectionFS) HasPendingRenames

func (fs *FaultInjectionFS) HasPendingRenames() bool

HasPendingRenames returns true if there are renames waiting for SyncDir.

func (*FaultInjectionFS) InjectReadError

func (fs *FaultInjectionFS) InjectReadError(path string)

InjectReadError sets up read error injection for the given path.

func (*FaultInjectionFS) InjectSyncError

func (fs *FaultInjectionFS) InjectSyncError()

InjectSyncError sets up sync error injection.

func (*FaultInjectionFS) InjectWriteError

func (fs *FaultInjectionFS) InjectWriteError(path string)

InjectWriteError sets up write error injection for the given path.

func (*FaultInjectionFS) IsFileSyncLieModeEnabled

func (fs *FaultInjectionFS) IsFileSyncLieModeEnabled() bool

IsFileSyncLieModeEnabled returns true if file Sync lie mode is active.

func (*FaultInjectionFS) IsRenameDoubleNameModeEnabled

func (fs *FaultInjectionFS) IsRenameDoubleNameModeEnabled() bool

IsRenameDoubleNameModeEnabled returns true if "both names exist" mode is active.

func (*FaultInjectionFS) IsRenameNeitherNameModeEnabled

func (fs *FaultInjectionFS) IsRenameNeitherNameModeEnabled() bool

IsRenameNeitherNameModeEnabled returns true if "neither name exists" mode is active.

func (*FaultInjectionFS) IsSyncDirLieModeEnabled

func (fs *FaultInjectionFS) IsSyncDirLieModeEnabled() bool

IsSyncDirLieModeEnabled returns true if SyncDir lie mode is active.

func (*FaultInjectionFS) ListDir

func (fs *FaultInjectionFS) ListDir(path string) ([]string, error)

ListDir lists files in a directory.

func (*FaultInjectionFS) Lock

func (fs *FaultInjectionFS) Lock(name string) (io.Closer, error)

Lock acquires an exclusive lock on a file.

func (*FaultInjectionFS) MkdirAll

func (fs *FaultInjectionFS) MkdirAll(path string, perm os.FileMode) error

MkdirAll creates a directory and all parent directories.

func (*FaultInjectionFS) Open

func (fs *FaultInjectionFS) Open(name string) (SequentialFile, error)

Open opens an existing file for sequential reading.

func (*FaultInjectionFS) OpenRandomAccess

func (fs *FaultInjectionFS) OpenRandomAccess(name string) (RandomAccessFile, error)

OpenRandomAccess opens an existing file for random access reading.

func (*FaultInjectionFS) PendingRenameCount

func (fs *FaultInjectionFS) PendingRenameCount() int

PendingRenameCount returns the number of pending (unsynced) renames.

func (*FaultInjectionFS) Remove

func (fs *FaultInjectionFS) Remove(name string) error

Remove deletes a file.

func (*FaultInjectionFS) RemoveAll

func (fs *FaultInjectionFS) RemoveAll(path string) error

RemoveAll removes a directory and all its contents.

func (*FaultInjectionFS) Rename

func (fs *FaultInjectionFS) Rename(oldname, newname string) error

Rename atomically renames a file. The new directory entry is NOT durable until SyncDir is called on the parent directory. If a crash occurs before SyncDir, the renamed file may disappear or revert to the old name.

func (*FaultInjectionFS) RevertUnsyncedRenames

func (fs *FaultInjectionFS) RevertUnsyncedRenames() error

RevertUnsyncedRenames simulates crash behavior for directory entry durability. Renames that were not followed by SyncDir are reverted: - If the rename had an original path, the file is renamed back - If the rename was from a new file (no original), the file is deleted

func (*FaultInjectionFS) SetFileSyncLieMode

func (fs *FaultInjectionFS) SetFileSyncLieMode(enabled bool, pattern string)

SetFileSyncLieMode enables or disables file Sync lie mode. When enabled, Sync() returns success but does NOT mark data as synced. On simulated crash (DropUnsyncedData), unsynced data is lost.

Use pattern to target specific file types:

  • "" (empty): lie for ALL files
  • ".log": lie for WAL files only
  • "MANIFEST": lie for MANIFEST files only
  • ".sst": lie for SST files only

func (*FaultInjectionFS) SetFilesystemActive

func (fs *FaultInjectionFS) SetFilesystemActive(active bool)

SetFilesystemActive enables or disables the filesystem. When disabled, all writes fail. Used to simulate crash.

func (*FaultInjectionFS) SetRenameDoubleNameMode

func (fs *FaultInjectionFS) SetRenameDoubleNameMode(enabled bool, pattern string)

SetRenameDoubleNameMode enables/disables "both names exist" rename anomaly mode. When enabled, Rename() records both old and new paths in pendingRenames. After SimulateCrash(), both paths will exist. Use pattern to target specific files.

func (*FaultInjectionFS) SetRenameNeitherNameMode

func (fs *FaultInjectionFS) SetRenameNeitherNameMode(enabled bool, pattern string)

SetRenameNeitherNameMode enables/disables "neither name exists" rename anomaly mode. When enabled, after SimulateCrash(), neither old nor new path exists. Use pattern to target specific files.

func (*FaultInjectionFS) SetSyncDirLieMode

func (fs *FaultInjectionFS) SetSyncDirLieMode(enabled bool)

SetSyncDirLieMode enables or disables SyncDir lie mode. When enabled, SyncDir() returns success but does NOT make renames durable. This simulates filesystems that lie about directory fsync completion. Use for testing recovery behavior under N05 fault models where the OS reports success but data is still lost on crash.

func (*FaultInjectionFS) SimulateCrashWithRenameAnomalies

func (fs *FaultInjectionFS) SimulateCrashWithRenameAnomalies() error

SimulateCrashWithRenameAnomalies applies rename anomaly modes to the filesystem. Call this before DropUnsyncedData to set up the rename anomaly state. For double-name mode: copies new file content to old path. For neither-name mode: deletes both paths.

func (*FaultInjectionFS) Stat

func (fs *FaultInjectionFS) Stat(name string) (os.FileInfo, error)

Stat returns file info.

func (*FaultInjectionFS) SyncDir

func (fs *FaultInjectionFS) SyncDir(path string) error

SyncDir marks the directory as synced. This is important for durability of file creation and rename. After SyncDir, pending renames in this directory become durable.

In "lie mode" (SetSyncDirLieMode(true)), this method returns success but does NOT make renames durable. On simulated crash, RevertUnsyncedRenames() will still revert those renames. This models filesystems that lie about directory fsync completion.

type FileOptions

type FileOptions struct {
	// UseDirectReads enables O_DIRECT for reads.
	// This bypasses the OS page cache for read operations.
	UseDirectReads bool

	// UseDirectWrites enables O_DIRECT for writes.
	// This bypasses the OS page cache for write operations.
	UseDirectWrites bool

	// UseMmapReads enables memory-mapped reads.
	// Cannot be used with UseDirectReads.
	UseMmapReads bool

	// UseMmapWrites enables memory-mapped writes.
	// Cannot be used with UseDirectWrites.
	UseMmapWrites bool

	// BlockSize is the alignment requirement for Direct I/O.
	// If 0, DefaultBlockSize is used.
	BlockSize int
}

FileOptions configures how files are opened. Mirrors RocksDB's EnvOptions.

func (FileOptions) GetBlockSize

func (o FileOptions) GetBlockSize() int

GetBlockSize returns the block size to use, defaulting to DefaultBlockSize.

func (FileOptions) Validate

func (o FileOptions) Validate() error

Validate checks that the options are valid.

type GoroutineFaultContext

type GoroutineFaultContext struct {
	// Error injection settings
	ReadErrorOneIn     int // Inject read error 1 in N times (0 = disabled)
	WriteErrorOneIn    int // Inject write error 1 in N times (0 = disabled)
	MetadataErrorOneIn int // Inject metadata error 1 in N times (0 = disabled)
	SyncErrorOneIn     int // Inject sync error 1 in N times (0 = disabled)

	// Error characteristics
	ErrorType ErrorType // Type of error to inject
	Retryable bool      // Whether injected errors are retryable

	// Statistics
	ReadErrorsInjected  atomic.Uint64
	WriteErrorsInjected atomic.Uint64
	SyncErrorsInjected  atomic.Uint64
	// contains filtered or unexported fields
}

GoroutineFaultContext holds fault injection settings for a goroutine.

func NewGoroutineFaultContext

func NewGoroutineFaultContext(seed int64) *GoroutineFaultContext

NewGoroutineFaultContext creates a new fault context with the given seed.

func (*GoroutineFaultContext) ShouldInjectReadError

func (ctx *GoroutineFaultContext) ShouldInjectReadError() bool

ShouldInjectReadError returns true if a read error should be injected.

func (*GoroutineFaultContext) ShouldInjectSyncError

func (ctx *GoroutineFaultContext) ShouldInjectSyncError() bool

ShouldInjectSyncError returns true if a sync error should be injected.

func (*GoroutineFaultContext) ShouldInjectWriteError

func (ctx *GoroutineFaultContext) ShouldInjectWriteError() bool

ShouldInjectWriteError returns true if a write error should be injected.

type GoroutineFaultManager

type GoroutineFaultManager struct {
	// contains filtered or unexported fields
}

GoroutineFaultManager manages per-goroutine fault injection contexts.

func NewGoroutineFaultManager

func NewGoroutineFaultManager() *GoroutineFaultManager

NewGoroutineFaultManager creates a new goroutine fault manager.

func (*GoroutineFaultManager) ClearContext

func (m *GoroutineFaultManager) ClearContext()

ClearContext clears the fault context for the current goroutine. Stats from the context are accumulated before clearing.

func (*GoroutineFaultManager) DisableGlobal

func (m *GoroutineFaultManager) DisableGlobal()

DisableGlobal disables global error injection.

func (*GoroutineFaultManager) GetContext

GetContext gets the fault context for the current goroutine.

func (*GoroutineFaultManager) SetContext

func (m *GoroutineFaultManager) SetContext(ctx *GoroutineFaultContext)

SetContext sets the fault context for the current goroutine.

func (*GoroutineFaultManager) SetGlobalReadErrorRate

func (m *GoroutineFaultManager) SetGlobalReadErrorRate(oneIn int)

SetGlobalReadErrorRate sets the global read error injection rate.

func (*GoroutineFaultManager) SetGlobalSyncErrorRate

func (m *GoroutineFaultManager) SetGlobalSyncErrorRate(oneIn int)

SetGlobalSyncErrorRate sets the global sync error injection rate.

func (*GoroutineFaultManager) SetGlobalWriteErrorRate

func (m *GoroutineFaultManager) SetGlobalWriteErrorRate(oneIn int)

SetGlobalWriteErrorRate sets the global write error injection rate.

func (*GoroutineFaultManager) ShouldInjectReadError

func (m *GoroutineFaultManager) ShouldInjectReadError() bool

ShouldInjectReadError checks if a read error should be injected for the current goroutine.

func (*GoroutineFaultManager) ShouldInjectSyncError

func (m *GoroutineFaultManager) ShouldInjectSyncError() bool

ShouldInjectSyncError checks if a sync error should be injected for the current goroutine.

func (*GoroutineFaultManager) ShouldInjectWriteError

func (m *GoroutineFaultManager) ShouldInjectWriteError() bool

ShouldInjectWriteError checks if a write error should be injected for the current goroutine.

func (*GoroutineFaultManager) Stats

func (m *GoroutineFaultManager) Stats() (reads, writes, syncs uint64)

Stats returns aggregate statistics across all contexts.

type GoroutineLocalFaultInjectionFS

type GoroutineLocalFaultInjectionFS struct {
	*FaultInjectionFS
	// contains filtered or unexported fields
}

GoroutineLocalFaultInjectionFS extends FaultInjectionFS with goroutine-local capabilities.

func NewGoroutineLocalFaultInjectionFS

func NewGoroutineLocalFaultInjectionFS(base FS) *GoroutineLocalFaultInjectionFS

NewGoroutineLocalFaultInjectionFS creates a new goroutine-local fault injection FS.

func (*GoroutineLocalFaultInjectionFS) Create

Create creates a new file with goroutine-local fault injection.

func (*GoroutineLocalFaultInjectionFS) FaultManager

FaultManager returns the goroutine fault manager.

func (*GoroutineLocalFaultInjectionFS) Open

Open opens a file with goroutine-local fault injection.

func (*GoroutineLocalFaultInjectionFS) OpenRandomAccess

func (fs *GoroutineLocalFaultInjectionFS) OpenRandomAccess(name string) (RandomAccessFile, error)

OpenRandomAccess opens a file with goroutine-local fault injection.

type RandomAccessFile

type RandomAccessFile interface {
	io.ReaderAt
	io.Closer

	// Size returns the file size.
	Size() int64
}

RandomAccessFile is a file that can be read at any offset.

type SequentialFile

type SequentialFile interface {
	io.Reader
	io.Closer

	// Skip skips n bytes.
	Skip(n int64) error
}

SequentialFile is a file that can be read sequentially.

type WritableFile

type WritableFile interface {
	io.Writer
	io.Closer

	// Sync flushes the file contents to stable storage.
	Sync() error

	// Append appends data to the file.
	// For most implementations, this is the same as Write.
	Append(data []byte) error

	// Truncate changes the size of the file.
	Truncate(size int64) error

	// Size returns the current file size.
	Size() (int64, error)
}

WritableFile is a file that can be written to.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL