CopywriteR cannot calculate loesses, regardless of bin size

[exeter-matthew-wakeling]

Hi. I'm trying to run CopywriteR with some BAM files from targeted sequencing. We target 75 genes, get around 3-6M reads, and have around 40-70% off-target. When I try to run the software, I get the following message:

"The GC-content and mappability normalization did not work due to a failure to calculate loesses. This can generally be solved by using larger bin sizes. Stopping execution of the remaining part of the script."

I have attached some of the relevant output. I would greatly appreciate any insight you have as to why it isn't working.

CopywriteR.log

read_counts.txt

[PeeperLab2]

Dear Matthew, Thank you for using CopywriteR. We see this error generally when there are not enough sequence reads to generate a DNA copy number profile. Generally, I would suggest to increase the bin size but your clearly already did this and it did not help. I myself have successfully tested CopywriteR on targeted seq data with as low as ~360 genes, never lower. So I am afraid your data does not have enough off-target sequence reads to generate a proper DNA copy number profile. I am sorry we cannot help you any further. Best Oscar On 28 May 2019, at 20:29, exeter-matthew-wakeling <notifications@github.com<mailto:notifications@github.com>> wrote: Hi. I'm trying to run CopywriteR with some BAM files from targeted sequencing. We target 75 genes, get around 3-6M reads, and have around 40-70% off-target. When I try to run the software, I get the following message: "The GC-content and mappability normalization did not work due to a failure to calculate loesses. This can generally be solved by using larger bin sizes. Stopping execution of the remaining part of the script." I have attached some of the relevant output. I would greatly appreciate any insight you have as to why it isn't working. CopywriteR.log<https://github.com/PeeperLab/CopywriteR/files/3229147/CopywriteR.log> read_counts.txt<https://github.com/PeeperLab/CopywriteR/files/3229148/read_counts.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AB7X5RFV3XGBLIWF4GCADOLPXV2YVA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GWJQYKQ>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7X5RH455CRBLR3AAFRW5DPXV2YVANCNFSM4HQF7DTQ>.

[exeter-matthew-wakeling]

Thanks for the quick reply. For reference, the samples I tried were sequenced with 3-6M reads, with 40-70% of the reads being off-target. Out of curiosity, I also ran CopywriteR on a whole exome sequencing sample (Agilent whole exome v5), which has 64M reads, and a bin size of 1000kb, with the same result. Surely this can't have too few off-target reads? Matthew From: PeeperLab [mailto:notifications@github.com] Sent: 29 May 2019 12:04 To: PeeperLab/CopywriteR <CopywriteR@noreply.github.com> Cc: Wakeling, Matthew <M.Wakeling@exeter.ac.uk>; Author <author@noreply.github.com> Subject: Re: [PeeperLab/CopywriteR] CopywriteR cannot calculate loesses, regardless of bin size (#28) Dear Matthew, Thank you for using CopywriteR. We see this error generally when there are not enough sequence reads to generate a DNA copy number profile. Generally, I would suggest to increase the bin size but your clearly already did this and it did not help. I myself have successfully tested CopywriteR on targeted seq data with as low as ~360 genes, never lower. So I am afraid your data does not have enough off-target sequence reads to generate a proper DNA copy number profile. I am sorry we cannot help you any further. Best Oscar On 28 May 2019, at 20:29, exeter-matthew-wakeling <notifications@github.com<mailto:notifications@github.com>> wrote: Hi. I'm trying to run CopywriteR with some BAM files from targeted sequencing. We target 75 genes, get around 3-6M reads, and have around 40-70% off-target. When I try to run the software, I get the following message: "The GC-content and mappability normalization did not work due to a failure to calculate loesses. This can generally be solved by using larger bin sizes. Stopping execution of the remaining part of the script." I have attached some of the relevant output. I would greatly appreciate any insight you have as to why it isn't working. CopywriteR.log<https://github.com/PeeperLab/CopywriteR/files/3229147/CopywriteR.log> read_counts.txt<https://github.com/PeeperLab/CopywriteR/files/3229148/read_counts.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AB7X5RFV3XGBLIWF4GCADOLPXV2YVA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GWJQYKQ>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7X5RH455CRBLR3AAFRW5DPXV2YVANCNFSM4HQF7DTQ>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AHYRD5VBVHTFRIHGYZNHT6DPXZPIZA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWO7CIA#issuecomment-496890144>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHYRD5VOAKUGOQ4B7SLWE4TPXZPIZANCNFSM4HQF7DTQ>.

[PeeperLab2]

Dear Matthew, Exome seq with 64M reads should work without problems. We ran many DNA copy number profiles on Exome with the Agilent V5 kit. I generally run 20kb bins on these samples with good results. These samples should have sufficient off-target reads. Some pipelines for Exome data remove the sequence reads that do not overlap with any exons. Do you have any such filters in your pipeline? If so, please skip this step. This might explain your problems with the Exome seq. data. Let me know if you keep running into errors. Best Oscar On 29 May 2019, at 13:13, exeter-matthew-wakeling <notifications@github.com<mailto:notifications@github.com>> wrote: Thanks for the quick reply. For reference, the samples I tried were sequenced with 3-6M reads, with 40-70% of the reads being off-target. Out of curiosity, I also ran CopywriteR on a whole exome sequencing sample (Agilent whole exome v5), which has 64M reads, and a bin size of 1000kb, with the same result. Surely this can't have too few off-target reads? Matthew From: PeeperLab [mailto:notifications@github.com] Sent: 29 May 2019 12:04 To: PeeperLab/CopywriteR <CopywriteR@noreply.github.com<mailto:CopywriteR@noreply.github.com>> Cc: Wakeling, Matthew <M.Wakeling@exeter.ac.uk<mailto:M.Wakeling@exeter.ac.uk>>; Author <author@noreply.github.com<mailto:author@noreply.github.com>> Subject: Re: [PeeperLab/CopywriteR] CopywriteR cannot calculate loesses, regardless of bin size (#28) Dear Matthew, Thank you for using CopywriteR. We see this error generally when there are not enough sequence reads to generate a DNA copy number profile. Generally, I would suggest to increase the bin size but your clearly already did this and it did not help. I myself have successfully tested CopywriteR on targeted seq data with as low as ~360 genes, never lower. So I am afraid your data does not have enough off-target sequence reads to generate a proper DNA copy number profile. I am sorry we cannot help you any further. Best Oscar On 28 May 2019, at 20:29, exeter-matthew-wakeling <notifications@github.com<mailto:notifications@github.com><mailto:notifications@github.com>> wrote: Hi. I'm trying to run CopywriteR with some BAM files from targeted sequencing. We target 75 genes, get around 3-6M reads, and have around 40-70% off-target. When I try to run the software, I get the following message: "The GC-content and mappability normalization did not work due to a failure to calculate loesses. This can generally be solved by using larger bin sizes. Stopping execution of the remaining part of the script." I have attached some of the relevant output. I would greatly appreciate any insight you have as to why it isn't working. CopywriteR.log<https://github.com/PeeperLab/CopywriteR/files/3229147/CopywriteR.log> read_counts.txt<https://github.com/PeeperLab/CopywriteR/files/3229148/read_counts.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AB7X5RFV3XGBLIWF4GCADOLPXV2YVA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GWJQYKQ>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7X5RH455CRBLR3AAFRW5DPXV2YVANCNFSM4HQF7DTQ>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AHYRD5VBVHTFRIHGYZNHT6DPXZPIZA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWO7CIA#issuecomment-496890144>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHYRD5VOAKUGOQ4B7SLWE4TPXZPIZANCNFSM4HQF7DTQ>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AB7X5RFD2BEXYJAP476BWX3PXZQMVA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWO7XPQ#issuecomment-496892862>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7X5RBFXUAIH2X42Q6IWTTPXZQMVANCNFSM4HQF7DTQ>.

[exeter-matthew-wakeling]

We don't have any filters that remove off-target reads in our pipeline. The files were processed as follows: Agilent V5, sequenced on Hiseq2500 -> fastq bwa mem Picard FixMateInformation VALIDATION_STRINGENCY=SILENT COMPRESSION_LEVEL=0 CREATE_INDEX=true SORT_ORDER=coordinate Picard MarkDuplicates REMOVE_DUPLICATES=true VALIDATION_STRINGENCY=SILENT COMPRESSION_LEVEL=0 CREATE_INDEX=true GATK RealignerTargetCreator GATK IndelRealigner GATK BaseRecalibrator This generates the BAM files that were given to CopywriteR. I can verify that off-target reads are present. Is there any way to find out the exact error that occurs to cause the error message to be emitted? I notice that in CopywriteR.R the exception that is caught is thrown away. This may be hiding the true reason for the failure. Matthew From: PeeperLab [mailto:notifications@github.com] Sent: 29 May 2019 13:23 To: PeeperLab/CopywriteR <CopywriteR@noreply.github.com> Cc: Wakeling, Matthew <M.Wakeling@exeter.ac.uk>; Author <author@noreply.github.com> Subject: Re: [PeeperLab/CopywriteR] CopywriteR cannot calculate loesses, regardless of bin size (#28) Dear Matthew, Exome seq with 64M reads should work without problems. We ran many DNA copy number profiles on Exome with the Agilent V5 kit. I generally run 20kb bins on these samples with good results. These samples should have sufficient off-target reads. Some pipelines for Exome data remove the sequence reads that do not overlap with any exons. Do you have any such filters in your pipeline? If so, please skip this step. This might explain your problems with the Exome seq. data. Let me know if you keep running into errors. Best Oscar

[PeeperLab2]

Hi, This is basically what we run for Exome seq data and should be fine. Would it be possible to share one of your bam files? I would like to run it myself and if the same error occurs run the CopywriteR code step by step. Best Oscar On 29 May 2019, at 15:06, exeter-matthew-wakeling <notifications@github.com<mailto:notifications@github.com>> wrote: We don't have any filters that remove off-target reads in our pipeline. The files were processed as follows: Agilent V5, sequenced on Hiseq2500 -> fastq bwa mem Picard FixMateInformation VALIDATION_STRINGENCY=SILENT COMPRESSION_LEVEL=0 CREATE_INDEX=true SORT_ORDER=coordinate Picard MarkDuplicates REMOVE_DUPLICATES=true VALIDATION_STRINGENCY=SILENT COMPRESSION_LEVEL=0 CREATE_INDEX=true GATK RealignerTargetCreator GATK IndelRealigner GATK BaseRecalibrator This generates the BAM files that were given to CopywriteR. I can verify that off-target reads are present. Is there any way to find out the exact error that occurs to cause the error message to be emitted? I notice that in CopywriteR.R the exception that is caught is thrown away. This may be hiding the true reason for the failure. Matthew From: PeeperLab [mailto:notifications@github.com] Sent: 29 May 2019 13:23 To: PeeperLab/CopywriteR <CopywriteR@noreply.github.com<mailto:CopywriteR@noreply.github.com>> Cc: Wakeling, Matthew <M.Wakeling@exeter.ac.uk<mailto:M.Wakeling@exeter.ac.uk>>; Author <author@noreply.github.com<mailto:author@noreply.github.com>> Subject: Re: [PeeperLab/CopywriteR] CopywriteR cannot calculate loesses, regardless of bin size (#28) Dear Matthew, Exome seq with 64M reads should work without problems. We ran many DNA copy number profiles on Exome with the Agilent V5 kit. I generally run 20kb bins on these samples with good results. These samples should have sufficient off-target reads. Some pipelines for Exome data remove the sequence reads that do not overlap with any exons. Do you have any such filters in your pipeline? If so, please skip this step. This might explain your problems with the Exome seq. data. Let me know if you keep running into errors. Best Oscar — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#28?email_source=notifications&email_token=AB7X5RDC7IEHKQ4KRGI5UTDPXZ5WBA5CNFSM4HQF7DT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPIMXI#issuecomment-496928349>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7X5RBIJJAYS63DPM7XHTTPXZ5WBANCNFSM4HQF7DTQ>.

[exeter-matthew-wakeling]

I'll bring this up at our lab meeting tomorrow morning. I need to get approval, since our samples are personal data. I will let you know the answer as soon as I know. Matthew From: PeeperLab [mailto:notifications@github.com] Sent: 29 May 2019 14:15 To: PeeperLab/CopywriteR <CopywriteR@noreply.github.com> Cc: Wakeling, Matthew <M.Wakeling@exeter.ac.uk>; Author <author@noreply.github.com> Subject: Re: [PeeperLab/CopywriteR] CopywriteR cannot calculate loesses, regardless of bin size (#28) Hi, This is basically what we run for Exome seq data and should be fine. Would it be possible to share one of your bam files? I would like to run it myself and if the same error occurs run the CopywriteR code step by step.

[exeter-matthew-wakeling]

On a hunch, I tried running CopywriteR on one of the targeted samples, with a bin size of 1000kb, but with an X screen available. It worked. So, something inside the tryCatch() is trying to use X, and is failing if X is not available. My guess is that it is whatever is generating the read.counts.compensated.*.png figure, as this file has zero size when it fails.

Please could you double-check - if you ssh onto a server without using a "-X" or "-Y" option, and try running CopywriteR, does it fail for you?

[dtemko]

Thank you exeter-matthew-wakeling, this was incredibly helpful