logs

package
v3.6.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 11, 2025 License: AGPL-3.0 Imports: 33 Imported by: 0

Documentation

Overview

Package logs defines types used for the data object logs section. The logs section holds a list of log records across multiple streams.

Index

Constants

View Source 
const (
	AppendUnordered = iota
	AppendOrdered
)

Variables

This section is empty.

Functions

func CheckSection 

func CheckSection(section *dataobj.Section) bool

CheckSection returns true if section is a logs section.

func CompareForSortOrder 

func CompareForSortOrder(sort SortOrder) func(result.Result[dataset.Row], result.Result[dataset.Row]) bool

CompareForSortOrder returns a comparison function for result rows for the given sort order.

func DecodeRow 

func DecodeRow(columns []*Column, row dataset.Row, record *Record, sym *symbolizer.Symbolizer) error

DecodeRow decodes a record from a dataset.Row, using the provided columns to determine the column type. The list of columns must match the columns used to create the row.

The sym argument is used for reusing metadata strings between calls to DecodeRow. If sym is nil, metadata strings are always allocated.

func Iter 

func Iter(ctx context.Context, obj *dataobj.Object) result.Seq[Record]

Iter iterates over records in the provided decoder. All logs sections are iterated over in order. Results objects returned to yield may be reused and must be copied for further use via DeepCopy().

func IterSection 

func IterSection(ctx context.Context, section *Section) result.Seq[Record]

Types

type AndPredicate 

type AndPredicate struct{ Left, Right Predicate }

An AndPredicate is a Predicate which asserts that a row may only be included if both the Left and Right Predicate are true.

type AndRowPredicate 

type AndRowPredicate struct{ Left, Right RowPredicate }

An AndRowPredicate is a RowPredicate which requires both its Left and Right predicate to be true.

type AppendStrategy 

type AppendStrategy int

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder accumulate a set of [Record]s within a data object.

func NewBuilder 

func NewBuilder(metrics *Metrics, opts BuilderOptions) *Builder

Nwe creates a new logs section. The pageSize argument specifies how large pages should be.

func (*Builder) Append 

func (b *Builder) Append(entry Record)

Append adds a new entry to b.

func (*Builder) EstimatedSize 

func (b *Builder) EstimatedSize() int

EstimatedSize returns the estimated size of the Logs section in bytes.

func (*Builder) Flush 

func (b *Builder) Flush(w dataobj.SectionWriter) (n int64, err error)

Flush flushes b to the provided writer.

After successful encoding, the b is reset and can be reused.

func (*Builder) Reset 

func (b *Builder) Reset()

Reset resets all state, allowing b to be reused.

func (*Builder) SetTenant 

func (b *Builder) SetTenant(tenant string)

SetTenant sets the tenant that owns the builder. A builder can be made multi-tenant by passing an empty string.

func (*Builder) Tenant 

func (b *Builder) Tenant() string

Tenant returns the optional tenant that owns the builder.

func (*Builder) Type 

func (b *Builder) Type() dataobj.SectionType

Type returns the dataobj.SectionType of the logs builder.

func (*Builder) UncompressedSize 

func (b *Builder) UncompressedSize() int

UncompressedSize returns the current uncompressed size of the logs section in bytes.

type BuilderOptions 

type BuilderOptions struct {
	// PageSizeHint is the size of pages to use when encoding the logs section.
	PageSizeHint int

	// PageMaxRowCount is the maximum amount of rows of pages to use when encoding the logs section.
	PageMaxRowCount int

	// BufferSize is the size of the buffer to use when accumulating log records.
	BufferSize int

	// StripeMergeLimit is the maximum number of stripes to merge at once when
	// flushing stripes into a section. StripeMergeLimit must be larger than 1.
	//
	// Lower values of StripeMergeLimit reduce the memory overhead of merging but
	// increase time spent merging. Higher values of StripeMergeLimit increase
	// memory overhead but reduce time spent merging.
	StripeMergeLimit int

	// AppendStrategy is allowed to control how the builder creates the section.
	// When appending logs to the section in strict sort order, the [AppendOrdered] can be used to avoid
	// creating and sorting of stripes.
	AppendStrategy AppendStrategy

	// SortOrder defines the order in which the rows of the logs sections are sorted.
	// They can either be sorted by [streamID ASC, timestamp DESC] ([SortStreamASC]) or [timestamp DESC, streamID ASC] ([SortTimestampDESC]).
	SortOrder SortOrder
}

BuilderOptions configures the behavior of the logs section.

type Column 

type Column struct {
	Section *Section   // Section that contains the column.
	Name    string     // Optional name of the column.
	Type    ColumnType // Type of data in the column.
	// contains filtered or unexported fields
}

A Column represents one of the columns in the logs section. Valid columns can only be retrieved by calling Section.Columns.

Data in columns can be read by using a Reader.

type ColumnStats 

type ColumnStats struct {
	Name             string
	Type             string
	ValueType        string
	RowsCount        uint64
	Compression      string
	UncompressedSize uint64
	CompressedSize   uint64
	MetadataOffset   uint64
	MetadataSize     uint64
	ValuesCount      uint64
	Cardinality      uint64
	ColumnIndex      int64

	Pages []PageStats
}

ColumnStats provides statistics about a column in a section.

type ColumnType 

type ColumnType int

ColumnType represents the kind of information stored in a Column. 

const (
	ColumnTypeInvalid   ColumnType = iota // ColumnTypeInvalid is an invalid column.
	ColumnTypeStreamID                    // ColumnTypeStreamID is a column that contains stream IDs.
	ColumnTypeTimestamp                   // ColumnTypeTimestamp is a column that contains timestamps per log record.

	// ColumnTypeMetadata is a column containing a sequence of structured
	// metadata values per log record. There will be one ColumnTypeMetadata per
	// structured metadata key; the name of the structured metadata key is stored
	// as the column name.
	ColumnTypeMetadata

	ColumnTypeMessage // ColumnTypeMessage is a column that contains log messages.
)

func ParseColumnType 

func ParseColumnType(text string) (ColumnType, error)

ParseColumnType parses a ColumnType from a string. The expected string format is the same as the return value of ColumnType.String.

func (ColumnType) String 

func (ct ColumnType) String() string

String returns the human-readable name of ct.

type ColumnarDataset 

type ColumnarDataset = columnar.Dataset

ColumnarDataset is the exported type alias of the internal columnar.Dataset.

func MakeColumnarDataset 

func MakeColumnarDataset(section *Section) (*ColumnarDataset, error)

MakeColumnarDataset returns the dataset from a section and a set of columns. It returns an error if not all columns are from the provided section.

type DatasetSequence 

type DatasetSequence struct {
	// contains filtered or unexported fields
}

func NewDatasetSequence 

func NewDatasetSequence(r *dataset.Reader, bufferSize int) DatasetSequence

func (*DatasetSequence) At 

func (seq *DatasetSequence) At() result.Result[dataset.Row]

func (*DatasetSequence) Close 

func (seq *DatasetSequence) Close()

func (*DatasetSequence) Next 

func (seq *DatasetSequence) Next() bool

type EqualPredicate 

type EqualPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value to check equality for.
}

An EqualPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is equal to the Value.

type FalsePredicate 

type FalsePredicate struct{}

FalsePredicate is a Predicate which always returns false.

type FuncPredicate 

type FuncPredicate struct {
	Column *Column // Column to check.

	// Keep is invoked with the column and value pair to check. Keep is given
	// the Column instance to allow for reusing the same function across
	// multiple columns, if necessary.
	//
	// If Keep returns true, the row is kept.
	Keep func(column *Column, value scalar.Scalar) bool
}

FuncPredicate is a Predicate which asserts that a row may only be included if the Value of the Column passes the Keep function.

Instances of FuncPredicate are ineligible for page filtering and should only be used when there isn't a more explicit Predicate implementation.

type GreaterThanPredicate 

type GreaterThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be greater than.
}

A GreaterThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is greater than the provided Value.

type InPredicate 

type InPredicate struct {
	Column *Column         // Column to check.
	Values []scalar.Scalar // Values to check for inclusion.
}

An InPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is present in the provided Values.

type LessThanPredicate 

type LessThanPredicate struct {
	Column *Column       // Column to check.
	Value  scalar.Scalar // Value for which rows in Column must be less than.
}

A LessThanPredicate is a Predicate which asserts that a row may only be included if the Value of the Column is less than the provided Value.

type LogMessageFilterRowPredicate 

type LogMessageFilterRowPredicate struct {
	Keep func(line []byte) bool
}

A LogMessageFilterRowPredicate is a RowPredicate that requires the log message of the entry to pass a Keep function.

type MetadataFilterRowPredicate 

type MetadataFilterRowPredicate struct {
	Key  string
	Keep func(key, value string) bool
}

A MetadataFilterRowPredicate is a RowPredicate that requires that metadata with the provided key pass a Keep function.

The key is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of MetadataFilterRowPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type MetadataMatcherRowPredicate 

type MetadataMatcherRowPredicate struct{ Key, Value string }

A MetadataMatcherRowPredicate is a RowPredicate that requires a metadata key named Key to exist with a value of Value.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics instruments the logs section.

func NewMetrics 

func NewMetrics() *Metrics

NewMetrics creates a new set of metrics for the logs section.

func (*Metrics) Observe 

func (m *Metrics) Observe(ctx context.Context, section *Section) error

Observe observes section statistics for a given section.

func (*Metrics) Register 

func (m *Metrics) Register(reg prometheus.Registerer) error

Register registers metrics to report to reg.

func (*Metrics) Unregister 

func (m *Metrics) Unregister(reg prometheus.Registerer)

Unregister unregisters metrics from the provided Registerer.

type NotPredicate 

type NotPredicate struct{ Inner Predicate }

A NotePredicate is a Predicate which asserts that a row may only be included if the inner Predicate is false.

type NotRowPredicate 

type NotRowPredicate struct{ Inner RowPredicate }

A NotRowPredicate is a RowPredicate which requires its Inner predicate to be false.

type OrPredicate 

type OrPredicate struct{ Left, Right Predicate }

An OrPredicate is a Predicate which asserts that a row may only be included if either the Left or Right Predicate are true.

type OrRowPredicate 

type OrRowPredicate struct{ Left, Right RowPredicate }

An OrRowPredicate is a RowPredicate which requires either its Left or Right predicate to be true.

type PageStats 

type PageStats struct {
	UncompressedSize uint64
	CompressedSize   uint64
	CRC32            uint32
	RowsCount        uint64
	Encoding         string
	DataOffset       uint64
	DataSize         uint64
	ValuesCount      uint64
}

PageStats provides statistics about a page in a column.

type Predicate 

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is an expression used to filter column values in a Reader.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

A Reader reads batches of rows from a Section.

func NewReader 

func NewReader(opts ReaderOptions) *Reader

NewReader creates a new Reader from the provided options. Options are not validated until the first call to Reader.Read.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the Reader and releases any resources it holds. Closed Readers can be reused by calling Reader.Reset.

func (*Reader) Read 

func (r *Reader) Read(ctx context.Context, batchSize int) (arrow.Record, error)

Read reads the batch of rows from the section, returning them as an Arrow record.

If ReaderOptions has predicates, only rows that match the predicates are returned. If none of the next batchSize rows matched the predicate, Read returns a nil record with a nil error.

Read will return an error if the next batch of rows could not be read due to invalid options or I/O errors. At the end of the section, Read returns nil, io.EOF.

Read may return a non-nil record with a non-nil error, including if the error is io.EOF. Callers should always process the record before processing the error value.

When a record is returned, it will match the schema specified by Reader.Schema. These records must always be released after use.

func (*Reader) Reset 

func (r *Reader) Reset(opts ReaderOptions)

Reset discards any state and resets r with a new set of optiosn. This permits reusing a Reader rather than allocating a new one.

func (*Reader) Schema 

func (r *Reader) Schema() *arrow.Schema

Schema returns the arrow.Schema used by the Reader. Fields in the schema match the order of columns listed in ReaderOptions.

Names of fields in the schema are guaranteed to be unique per column but are not guaranteed to be stable.

The returned Schema must not be modified.

func (*Reader) Stats 

func (r *Reader) Stats() *dataset.ReaderStats

type ReaderOptions 

type ReaderOptions struct {
	// Columns to read. Each column must belong to the same [Section].
	Columns []*Column

	// Predicates holds a set of predicates to apply when reading the section.
	// Columns referenced in Predicates must be in the set of Columns.
	Predicates []Predicate

	// Allocator to use for allocating Arrow records. If nil,
	// [memory.DefaultAllocator] is used.
	Allocator memory.Allocator
}

ReaderOptions customizes the behavior of a Reader.

func (*ReaderOptions) Validate 

func (opts *ReaderOptions) Validate() error

Validate returns an error if the opts is not valid. ReaderOptions are only valid when:

  • Each Column in Columns and Predicates belongs to the same Section.
  • Scalar values used in predicates are of a supported type: an int64, uint64, timestamp, or a byte array.

type Record 

type Record struct {
	StreamID  int64
	Timestamp time.Time
	Metadata  labels.Labels
	Line      []byte
}

A Record is an individual log record within the logs section.

type RowPredicate 

type RowPredicate interface {
	// contains filtered or unexported methods
}

RowPredicate is an expression used to filter rows in a data object.

type RowReader 

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads the set of logs from an [Object].

func NewRowReader 

func NewRowReader(sec *Section) *RowReader

NewRowReader creates a new WowReader that reads from the provided Section.

func (*RowReader) Close 

func (r *RowReader) Close() error

Close closes the RowReader and releases any resources it holds. Closed RowReaders can be reused by calling RowReader.Reset.

func (*RowReader) MatchStreams 

func (r *RowReader) MatchStreams(ids iter.Seq[int64]) error

MatchStreams provides a sequence of stream IDs for the logs reader to match. RowReader.Read will only return logs for the provided stream IDs.

MatchStreams may be called multiple times to match multiple sets of streams.

MatchStreams may only be called before reading begins or after a call to RowReader.Reset.

func (*RowReader) Read 

func (r *RowReader) Read(ctx context.Context, s []Record) (int, error)

Read reads up to the next len(s) records from the reader and stores them into s. It returns the number of records read and any error encountered. At the end of the logs section, Read returns 0, io.EOF.

func (*RowReader) Reset 

func (r *RowReader) Reset(sec *Section)

Reset resets the RowReader with a new Section to read from. Reset allows reusing a RowReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the RowReader without needing a new object.

func (*RowReader) SetPredicates 

func (r *RowReader) SetPredicates(p []RowPredicate) error

SetPredicate sets the predicates to use for filtering logs. RowReader.Read will only return logs for which the predicate passes.

Predicates may only be set before reading begins or after a call to RowReader.Reset.

type Section 

type Section struct {
	// contains filtered or unexported fields
}

Section represents an opened logs section.

func Open 

func Open(ctx context.Context, section *dataobj.Section) (*Section, error)

Open opens a Section from an underlying dataobj.Section. Open returns an error if the section metadata could not be read or if the provided ctx is canceled.

func (*Section) Columns 

func (s *Section) Columns() []*Column

Columns returns the set of Columns in the section. The slice of returned sections must not be mutated.

Unrecognized columns (e.g., when running older code against newer sterams sections) are skipped.

func (*Section) PrimarySortOrder 

func (s *Section) PrimarySortOrder() (ColumnType, SortDirection, error)

PrimarySortOrder returns the primary sort order information of the section as a tuple of ColumnType and SortDirection.

type SortDirection 

type SortDirection int

SortDirection represents sort direction of a column. 

const (
	SortDirectionUnspecified SortDirection = 0 // Sort direction is unspecified.
	SortDirectionAscending   SortDirection = 1 // SortDirectionAscending represents ascending sort order (smallest values first).
	SortDirectionDescending  SortDirection = 2 // SortDirectionDescending represents descending sort order (largest values first).
)

type SortOrder 

type SortOrder int
const (
	SortStreamASC SortOrder = iota
	SortTimestampDESC
)

type Stats 

type Stats struct {
	UncompressedSize uint64
	CompressedSize   uint64

	Columns []ColumnStats
}

Stats provides statistics about a streams section.

func ReadStats 

func ReadStats(ctx context.Context, section *Section) (Stats, error)

ReadStats returns statistics about the logs section. ReadStats returns an error if the streams section couldn't be inspected or if the provided ctx is canceled.

type TimeRangeRowPredicate 

type TimeRangeRowPredicate struct {
	StartTime, EndTime time.Time
	IncludeStart       bool // Whether StartTime is inclusive.
	IncludeEnd         bool // Whether EndTime is inclusive.
}

A TimeRangeRowPredicate is a RowPredicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

type TruePredicate 

type TruePredicate struct{}

TruePredicate is a Predicate which always returns true.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.