XF 2.0 how to tell job rebuilder to be less optimistic with batch size

[Marcus]

I start a job rebuilder with a batch size of 1, he rebuilds one item fetching data with guzzle and thinks "that was so quick, I scale linear and will fetch 200 more items the next batch". Unfortunately that's when I get the error message like below. Currently I use sleep(5) at the end of the rebuild job function and this works nice, but now that job takes the time of a full lunch break.

GuzzleHttp\Exception\RequestException: No response was received for a request with no body. This could mean that you are saturating your network. in src/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php at line 51

[*]GuzzleHttp\Exception\RequestException::wrapException() insrc/vendor/guzzlehttp/guzzle/src/RequestFsm.php at line 103
[*]GuzzleHttp\RequestFsm->__invoke() insrc/vendor/guzzlehttp/guzzle/src/RequestFsm.php at line 132
[*]GuzzleHttp\RequestFsm->GuzzleHttp\{closure}() insrc/vendor/react/promise/src/FulfilledPromise.php at line 25
[*]React\Promise\FulfilledPromise->then() insrc/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php at line 55
[*]GuzzleHttp\Ring\Future\CompletedFutureValue->then() insrc/vendor/guzzlehttp/guzzle/src/Message/FutureResponse.php at line43
[*]GuzzleHttp\Message\FutureResponse::proxy() insrc/vendor/guzzlehttp/guzzle/src/RequestFsm.php at line 134
[*]GuzzleHttp\RequestFsm->__invoke() insrc/vendor/guzzlehttp/guzzle/src/Client.php at line 165
[*]GuzzleHttp\Client->send() insrc/vendor/guzzlehttp/guzzle/src/Client.php at line 125

 

[Jake B.]

You could do something like:

$this->data['batch'] = min($this->data['batch'], 20);

to set a limit (change 20 to whatever the maximum should be)

 

[Marcus]

it seems guzzle can only do one request per page, this is the main issue

 

[Mike]

We have places that would do multiple requests (like sitemaps or page title conversions), so I doubt it's that.

We'd likely need to see the code in question.