Performance issue

[rlakis]

I have compiled the old php extension for PHP-8 and did a small test; the result was that the old extension is faster than the new one by about 100%

the PHP test code

namespace Aerospike;
$socket = "/tmp/asld_grpc.sock";
$client = Client::connect($socket);

$startTime = microtime(true); // true returns a float
$rp = new ReadPolicy();
$key = new Key("user", "profiles", "facebook-id");
$key1 = new Key("user", "devices", "device-is");

$brp = new BatchReadPolicy();
$br = new BatchRead($brp, $key, []);
$br1 = new BatchRead($brp, $key1, []);

$bp = new BatchPolicy();
for ($i=0;$i<10000;$i++) {
$recs = $client->batch($bp, [$br, $br1]);
}

$endTime = microtime(true);
$elapsedTime = ($endTime - $startTime)*1000.0;
echo "Elapsed time: " . $elapsedTime . " ms\n\n";

$hosts = ["hosts" => [["addr"=>"h1", "port"=>3000], ["addr"=>"h2", "port"=>3000], ["addr"=>"h3", "port"=>3000]]];
$options = [\Aerospike::OPT_POLICY_KEY => \Aerospike::POLICY_KEY_SEND,
\Aerospike::OPT_POLICY_EXISTS => \Aerospike::POLICY_EXISTS_IGNORE,
\Aerospike::OPT_MAX_RETRIES => 3
];

$cluster = new \Aerospike($hosts, true, $options);
$startTime = microtime(true); // true returns a float

$pk=$cluster->initKey("user", "profiles", "facebook-id");
$pk1=$cluster->initKey("user", "devices", "device-id");
for ($i=0;$i<10000;$i++) {
$cluster->getMany([$pk, $pk1], $recs);
}

$endTime = microtime(true);
$elapsedTime = ($endTime - $startTime)*1000.0;
echo "Elapsed time: " . $elapsedTime . " ms\n\n";

Rust Elapsed time: 18366.338968277 ms
Old Elapsed time: 10984.447956085 ms

[hmfaysal]

The performance is abysmal compared to Aerospike Client for PHP 7. 1630.924ms for 1000 reads of the same key for this extension, while the older achieves 631.8309ms for the same test. Same network conditions, same machine.

The older client was 2.58x faster than this one. This needs to be carefully looked into

[justinlee-aerospike]

Thanks @hmfaysal and @rlakis for the feedback. Just to clarify, we are aware of the difference in latency performance between the legacy PHP client (C-based client) and the new PHP8 client.

Our original design for the PHP8 client was a Rust-based client with a very similar architecture to the previous C-based client. Like the original client, it used an FFI between our PHP interface and our Rust client and sent the requests directly to the server. It had similar performance, with both clients able to do base operations in the latency range of 200 to 300 ms per operation.

However, we received feedback from multiple alpha customer users that this architecture was insufficient at large scales. This is due to the fact that PHP cannot support multithreading, so scaling is achieved by forking processes. However, this causes an excess use of connection sockets, which causes server resource issues at high scales. This type of issue is not seen on any of our other clients, all of which support multithreading.

So in response, we created a new version of the PHP8 client that uses a proxy architecture. Instead of directly sending the request through the PHP/Rust lib, we send an IPC request to a a local service that then submits the request to the database. To keep the architecture relatively clean, we chose to use gRPC as an IPC mechanism, with the idea of potentially easily expanding the architecture to work with the service running remotely at a future date if we choose. This architecture allows us to use whatever client we like to create and manage outstanding connections (in this case, the Golang client) which will then allow us to scale at a rate comparable to what you would get if you used one of the other native clients.

With the above mentioned new architecture, we incur an additional overhead, which in our measurements amounted to an increase of approximately 200 to 250 ms, bringing our total latency to 400 to 550 ms. Our performance goal was to hold the average latency well under 1 ms; we achieved that, while also enabling our customers to be able to scale at a level comparable with that of our other clients.

We will not be maintaining to legacy C-based PHP client going forward; we felt that the FFI was too cumbersome to use. We may consider trying to optimize the current PHP8 client further in the future. We do recommend that you look carefully at your production use case and see if the totality of your latency and scalability requirements are being met by the current PHP8 client. If not, please let us know and we can have our support team help you tune and also consider implementing some optimizations if applicable.

[piyush-bansal1989]

Hi @justinlee-aerospike

I had a related query regarding the Aerospike client upgrade.

We’ve noticed that the PHP 7 Aerospike client had significantly lower memory/CPU usage compared to the PHP 8+ client, which requires a separate Aerospike service to run.
Given this, do we need to scale up our server configuration for applications using the PHP 8+ client?

Also, in case the application server starts before the Aerospike server, is there a built-in reconnect or retry mechanism in the PHP 8+ client to handle this scenario gracefully?

Would appreciate your input on how we can best tune the Aerospike client and server configuration to ensure optimal performance.

Thanks!

[piyush-bansal1989]

Hi @justinlee-aerospike

I had a related query regarding the Aerospike client upgrade.

We’ve noticed that the PHP 7 Aerospike client had significantly lower memory/CPU usage compared to the PHP 8+ client, which requires a separate Aerospike service to run. Given this, do we need to scale up our server configuration for applications using the PHP 8+ client?

Also, in case the application server starts before the Aerospike server, is there a built-in reconnect or retry mechanism in the PHP 8+ client to handle this scenario gracefully?

Would appreciate your input on how we can best tune the Aerospike client and server configuration to ensure optimal performance.

Thanks!

Hi @justinlee-aerospike / @paul-brackin :
Kindly help on this query.

[khaf]

@piyush-bansal1989 Sorry it seems you didn't get any feedback on your query. The new client of course uses a secondary service and that makes it more memory hungry. Scans and queries also keep track of their own state to be able to resume, and that also uses more memory. All in all though, the whole setup should be much more stable and reliable.

The connection manager can take care of pooling and retrying connections. The PHP side simply just connects to that service per request. I expect things to just work out, but if you see any issue or have any questions, don't hesitate to ask or just file new tickets.

[kamazee]

We are migrating from PHP7, and also see similar performance penalty.

Was the connection count any different from PHP7 version? I mean, is the issue on the initial version of the PHP8 client more severe than it was on PHP7? If not, can you make the proxy optional for those who's ready to trade connection count for performance and retain in-process pool of persistent connections as it was in PHP7 client?