Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,6 @@ coverage.txt
coverage.out

# Gogland project files
.idea
.idea
# Planning/codebase-mapping docs (local only)
.planning/
29 changes: 23 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ package main
import (
"context"
"fmt"
"log"
"time"

"github.com/MasterOfBinary/gobatch/batch"
Expand Down Expand Up @@ -164,7 +165,10 @@ func main() {
ctx := context.Background()

// Start batch processing with processors chained
errs := b.Go(ctx, src, doubleProc, printProc)
errs, err := b.Go(ctx, src, doubleProc, printProc)
if err != nil {
log.Fatal(err)
}

// Ignore errors for this simple example
batch.IgnoreErrors(errs)
Expand Down Expand Up @@ -200,6 +204,8 @@ You can choose between:
- **`ConstantConfig`** for static, unchanging settings.
- **`DynamicConfig`** for runtime-adjustable settings that can be updated while processing.

Passing a `nil` `Config` to `New` (or using the zero-value `&Batch[T]{}`) uses a default configuration, where items are processed immediately as they are read.

Configuration options include:

- `MinItems`: Minimum number of items to process in a batch.
Expand Down Expand Up @@ -255,7 +261,6 @@ You can fine-tune the performance by customizing the internal channel buffer siz
// Configure custom buffer sizes
batchProcessor := batch.New[int](config).WithBufferConfig(batch.BufferConfig{
ItemBufferSize: 1000, // Buffer for incoming items
IDBufferSize: 1000, // Buffer for ID generation
ErrorBufferSize: 500, // Buffer for error reporting
})
```
Expand Down Expand Up @@ -297,18 +302,30 @@ go func() {
Or using helper functions:

```go
// Collect all errors
errs := batch.CollectErrors(batchProcessor.Go(ctx, source, processor))
<-batchProcessor.Done()
// Collect all errors (blocks until processing completes)
pipeErrs, err := batchProcessor.Go(ctx, source, processor)
if err != nil {
// Handle start error (e.g. batch.ErrNilSource, batch.ErrBatchUsed)
log.Fatal(err)
}
errs := batch.CollectErrors(pipeErrs)

// Or use the RunBatchAndWait helper
// Or use the RunBatchAndWait helper, which folds a start error into the slice
errs := batch.RunBatchAndWait(ctx, batchProcessor, source, processor)

for _, err := range errs {
// Handle error
}
```

### Batch lifecycle

A `Batch` is **single-use**: call `Go` exactly once per `Batch`. Calling `Go`
again returns `batch.ErrBatchUsed` (along with a closed, drainable error channel)
instead of starting a second run — create a fresh `Batch` with `New` for each
run. `Go` also returns `batch.ErrNilSource` when the source is nil. Both errors
are checkable with `errors.Is`.

## Documentation

See the [pkg.go.dev docs](https://pkg.go.dev/github.com/MasterOfBinary/gobatch) for documentation
Expand Down
Loading
Loading