Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ChatGPT Instructions
# Agent Instructions

This file contains instructions and guidelines for ChatGPT when working with this project.
This file contains instructions and guidelines for AI coding agents when working with this project. It mirrors [CLAUDE.md](./CLAUDE.md); keep the two files in sync.

## Project Overview
GoBatch is a Go library for batch data processing. It provides infrastructure for processing batches of data with configurable sources, processors, and pipelines. The library allows for flexible batch processing with configurable timing and item count parameters.
Expand All @@ -22,7 +22,7 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo
- Document interfaces thoroughly with usage examples
- For methods, include descriptive comments that explain parameters, return values, and behavior
- Write tests for new functionality
- Maintain backward compatibility where possible
- This is a version 0 library; breaking changes are acceptable on master. When you change the public API, update the docs in the same change: README.md, CHANGELOG.md, the package docs (doc.go in the root, batch, processor, and source packages), and the affected example tests

## Documentation Style
- Use godoc style comments for all exported types, functions, methods, and constants
Expand All @@ -40,10 +40,10 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo
- `/example_test.go`: Top-level usage examples

## Key Concepts
- **Batch**: Main type that orchestrates the batch processing pipeline
- **Source**: Interface for data providers that read from various origins
- **Processor**: Interface for components that process batches of items
- **Item**: Represents a single data item flowing through the pipeline with unique ID and optional error
- **Batch[T]**: Main type that orchestrates the batch processing pipeline
- **Source[T]**: Interface for data providers that read from various origins
- **Processor[T]**: Interface for components that process batches of items
- **Item[T]**: Represents a single data item flowing through the pipeline with unique ID and optional error
- **Config**: Interface for controlling batch timing and item count parameters
- **ConstantConfig**: Static, unchanging configuration
- **DynamicConfig**: Runtime-adjustable configuration that can be updated while processing
Expand Down
14 changes: 7 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Claude Instructions

This file contains instructions and guidelines for Claude when working with this project.
This file contains instructions and guidelines for Claude when working with this project. [AGENTS.md](./AGENTS.md) mirrors these instructions for all other AI coding agents; keep the two files in sync.

## Project Overview
GoBatch is a Go library for batch data processing. It provides infrastructure for processing batches of data with configurable sources, processors, and pipelines. The library allows for flexible batch processing with configurable timing and item count parameters.
Expand All @@ -22,7 +22,7 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo
- Document interfaces thoroughly with usage examples
- For methods, include descriptive comments that explain parameters, return values, and behavior
- Write tests for new functionality
- Maintain backward compatibility where possible
- This is a version 0 library; breaking changes are acceptable on master. When you change the public API, update the docs in the same change: README.md, CHANGELOG.md, the package docs (doc.go in the root, batch, processor, and source packages), and the affected example tests

## Documentation Style
- Use godoc style comments for all exported types, functions, methods, and constants
Expand All @@ -40,10 +40,10 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo
- `/example_test.go`: Top-level usage examples

## Key Concepts
- **Batch**: Main type that orchestrates the batch processing pipeline
- **Source**: Interface for data providers that read from various origins
- **Processor**: Interface for components that process batches of items
- **Item**: Represents a single data item flowing through the pipeline with unique ID and optional error
- **Batch[T]**: Main type that orchestrates the batch processing pipeline
- **Source[T]**: Interface for data providers that read from various origins
- **Processor[T]**: Interface for components that process batches of items
- **Item[T]**: Represents a single data item flowing through the pipeline with unique ID and optional error
- **Config**: Interface for controlling batch timing and item count parameters
- **ConstantConfig**: Static, unchanging configuration
- **DynamicConfig**: Runtime-adjustable configuration that can be updated while processing
Expand Down Expand Up @@ -74,4 +74,4 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo
- Processors should respect context cancellation
- Items with errors are tracked individually through the Error field
- Batch processing continues despite individual item errors
- Use `errors.As` to check error types (SourceError, ProcessorError)
- Use `errors.As` to check error types (SourceError, ProcessorError)
16 changes: 8 additions & 8 deletions batch/config.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
// Package batch provides a flexible batch processing pipeline for handling data.
package batch

import (
Expand All @@ -14,10 +13,11 @@ import (
// which can be adjusted during runtime. This is useful for tuning the system
// under different load scenarios or adapting to changing performance requirements.
type Config interface {
// Get returns the values for configuration.
//
// If MinItems > MaxItems or MinTime > MaxTime, the min value will be
// set to the maximum value.
// Get returns the values for configuration. Implementations return the
// values as provided; Batch normalizes them before collecting each batch:
// MinItems defaults to 1 if zero, and when a non-zero MaxItems or MaxTime
// is smaller than its corresponding min value, the min is lowered to the
// max. A zero MaxItems or MaxTime means no maximum applies.
//
// If the config values may be modified during batch processing, Get
// must properly handle concurrency issues.
Expand Down Expand Up @@ -105,9 +105,9 @@ func (b *ConstantConfig) Get() ConfigValues {
// If values is nil, the default values are used as described in Batch.
//
// This is useful for:
// - Systems that need to adapt to changing workloads
// - Services that implement backpressure mechanisms
// - Applications that tune batch parameters based on performance metrics
// - Systems that need to adapt to changing workloads
// - Services that implement backpressure mechanisms
// - Applications that tune batch parameters based on performance metrics
func NewDynamicConfig(values *ConfigValues) *DynamicConfig {
if values == nil {
return &DynamicConfig{}
Expand Down
12 changes: 7 additions & 5 deletions batch/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,20 @@
//
// A few examples:
//
// - MinTime = 2s. After 1s the input channel is closed. The items are processed right away.
// - MinItems = 10, MinTime = 2s. After 1s, 10 items have been read. They are not processed until 2s has passed.
// - MaxItems = 10, MinTime = 2s. After 1s, 10 items have been read. They are processed right away.
// - MinTime = 2s. After 1s the input channel is closed. The items are processed right away.
// - MinItems = 10, MinTime = 2s. After 1s, 10 items have been read. They are not processed until 2s has passed.
// - MaxItems = 10, MinTime = 2s. After 1s, 10 items have been read. They are processed right away.
//
// Timers and counters are relative to when the previous batch finished processing.
// Timers and counters are relative to when the previous batch was dispatched for
// processing. Batches are processed concurrently: collection of the next batch
// begins immediately, without waiting for the previous batch to finish processing.
// Each batch starts a new MinTime/MaxTime window and counts new items from zero.
//
// Processors can be chained together. Each processor receives the output items
// from the previous processor:
//
// b.Go(ctx, source, processor1, processor2, processor3)

//
// Basic usage:
//
// cfg := NewConstantConfig(&ConfigValues{MinItems: 1})
Expand Down
8 changes: 8 additions & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Tolerate small run-to-run coverage noise: the suite has scheduler-dependent
# paths, so total coverage varies slightly between identical runs.
coverage:
status:
project:
default:
target: auto
threshold: 1%
14 changes: 7 additions & 7 deletions processor/doc.go
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
// Package processor contains several implementations of the batch.Processor
// interface for common processing scenarios, including:
//
// - Error: For simulating errors with configurable failure rates
// - Filter: For filtering items based on custom predicates
// - Nil: For testing timing behavior without modifying items
// - Transform: For transforming item data values
// - Channel: For writing item data to an output channel
// - Error: For simulating errors with configurable failure rates
// - Filter: For filtering items based on custom predicates
// - Nil: For testing timing behavior without modifying items
// - Transform: For transforming item data values
// - Channel: For writing item data to an output channel
//
// Each processor implementation follows a consistent error handling pattern and
// respects context cancellation.
// See each processor's documentation for its error handling and context
// cancellation behavior.
//
// Basic usage of the Transform processor:
//
Expand Down
6 changes: 3 additions & 3 deletions source/doc.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
// Package source contains several implementations of the batch.Source
// interface for common data source scenarios, including:
//
// - Channel: For using existing channels as batch sources
// - Error: For simulating error-only sources without data
// - Nil: For testing timing behavior without emitting data
// - Channel: For using existing channels as batch sources
// - Error: For simulating error-only sources without data
// - Nil: For testing timing behavior without emitting data
//
// Each source implementation handles context cancellation properly and
// ensures channels are closed appropriately.
Expand Down
Loading