Skip to content

feat: export via COPY TO STDOUT instead of temporary view (SUPPORT-17025) #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
terezaskalova wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

feat: export via COPY TO STDOUT instead of temporary view (SUPPORT-17025) #171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 10 additions & 39 deletions src/Extractor/CopyAdapter.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,22 +98,17 @@ public function doExport(
$sql = '\encoding UTF8' . PHP_EOL .
implode(PHP_EOL, $this->databaseConfig->getInitQueries()) . PHP_EOL;

if ($this->canUserCreateView() && !$this->isTransactionReadOnly()) {
$viewName = uniqid();
$sql .= 'CREATE TEMP VIEW "' . $viewName . '" AS ' . $trimmedQuery . ';' . PHP_EOL .
sprintf(
"\COPY (%s) TO '%s' WITH CSV DELIMITER ',' FORCE QUOTE *;",
'SELECT * FROM "' . $viewName . '"',
$csvPath,
) . PHP_EOL .
'DROP VIEW "' . $viewName . '";';
} else {
$sql .= sprintf(
"\COPY (%s) TO '%s' WITH CSV DELIMITER ',' FORCE QUOTE *;",
// Export data using a server-side "COPY ... TO STDOUT" statement redirected to the output
// file with the psql "\o" meta-command. Unlike the client-side "\copy" meta-command,
// "COPY ... TO STDOUT" is a regular SQL statement, so it supports multi-line queries without
// wrapping them in a temporary view. This avoids any DDL (CREATE/DROP TEMP VIEW) on the
// source database during export.
$sql .= sprintf("\o '%s'", $csvPath) . PHP_EOL .
sprintf(
"COPY (%s) TO STDOUT WITH CSV DELIMITER ',' FORCE QUOTE *;",
$trimmedQuery,
$csvPath,
);
}
) . PHP_EOL .
'\o';

try {
$this->runPsqlProcess($sql, null);
Expand Down Expand Up @@ -203,28 +198,4 @@ private function getLastFetchedValue(

return $lastExportedRow[$columnIndex];
}

protected function canUserCreateView(): bool
{
try {
return $this->runPsqlProcess(
'SELECT has_schema_privilege(current_schema(), \'CREATE\');',
null,
) === 't';
} catch (CopyAdapterException $e) {
throw new CopyAdapterQueryException($e->getMessage(), 0, $e);
}
}

protected function isTransactionReadOnly(): bool
{
try {
return $this->runPsqlProcess(
'SHOW transaction_read_only;',
null,
) === 'on';
} catch (CopyAdapterException $e) {
throw new CopyAdapterQueryException($e->getMessage(), 0, $e);
}
}
}
2 changes: 1 addition & 1 deletion tests/functional/error-invalid-query/expected-stderr
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Export by "Copy" adapter failed: ERROR: column "something" does not exist LINE 1: CREATE TEMP VIEW "%x" AS SELECT something, fake;%A
Export by "Copy" adapter failed: ERROR: column "something" does not exist LINE 1: COPY (SELECT something, fake) TO STDOUT WITH CSV DELIMITER '...%A
Export by "PDO" adapter failed: [in.c-main.escaping]: DB query failed: SQLSTATE[42703]: Undefined column: 7 ERROR: column "something" does not exist%A Tried 5 times.
[in.c-main.escaping]: DB query failed: SQLSTATE[42703]: Undefined column: 7 ERROR: column "something" does not exist %A Tried 5 times.
4 changes: 1 addition & 3 deletions tests/functional/run-action-row-debug-logging/expected-stdout
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,5 @@
Creating PDO connection to "pgsql:host=%s;port=5432;dbname=postgres;".
Exporting "sales" to "in.c-main.escaping".
Exporting by "Copy" adapter.
[%s] DEBUG: Running query: "SELECT has_schema_privilege(current_schema(), 'CREATE');". [] []
[%s] DEBUG: Running query: "SHOW transaction_read_only;". [] []
[%s] DEBUG: Running query: "\encoding UTF8 CREATE TEMP VIEW "%s" AS SELECT * FROM escaping; \COPY (SELECT * FROM "%s") TO '/tmp/run-%s.%s/out/tables/in.c-main.escaping.csv' WITH CSV DELIMITER ',' FORCE QUOTE *; DROP VIEW "%s";". [] []
[%s] DEBUG: Running query: "\encoding UTF8 \o '/tmp/run-%s.%s/out/tables/in.c-main.escaping.csv' COPY (SELECT * FROM escaping) TO STDOUT WITH CSV DELIMITER ',' FORCE QUOTE *; \o". [] []
Exported "7" rows to "in.c-main.escaping".
Loading