Report processing vs receive vs add record time in CDC pull

[ilidemi]

Currently it requires a trace capture + LLM crunching to distinguish a network issue from a CPU issue. Report them as accessible metrics instead.

Only PG and MySQL as #4722 is redoing Mongo processing, and semantics of pipelined/multithreaded processing time would potentially be different

Contributes to DBI-1075

[dtunikov]

hm, not sure I'm following this part
afaiu we want to deduct the time we spent in req.RecordStream.AddRecord from the processTime
isn't it smth like that? i just don't understand why we set processTime back to now() on every addRecord call 🤔

Suggested change

	processTime.Add(int64(addStart.Sub(processStart)))
	addStart := time.Now()
	if err := req.RecordStream.AddRecord(ctx, record); err != nil {
	return err
	}
	addTook := int64(time.Since(addStart))
	addRecordTime.Add(addTook)
	processTime.Add(-addTook);

[ilidemi]

These are kind of two ways of achieving the same thing, just the PR one is without subtractions. If that's less clear, can do -addRecord as well. processTime is set back to now() here so that the next processTime.Add wouldn't count addRecord time

[ilidemi]

Looked a bit further, and alternative approaches either have gaps or introduce long-distance math between variables:

• In the suggested version, processTime is undercounted until processTime.Add(int64(time.Since(processStart))) a few lines below and can get reported on timer
• Alternative is to save the duration of last addRecords and subtract it at processTime.Add time but that's 300 lines of scrolling to match up variable names

In this case it's all contained to the transition points and just stops the old counter and starts the next one

[dtunikov]

lgtm
just this part is a bit unclear

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[itsbilal]

LGTM

Once you merge this, I can go ahead and make this addition to the mongodb connector in #4722. We will need to pass through a processStart with each event and then have the very last processing step do the processTime.Add(int64(time.Since(processStart)) for each batch there, but otherwise this model of 3 counters makes sense to me.