bug in get.otureps

mothur bugs
1

Hello,

I’ve been using mothur in command-prompt mode via .pbs script for processing on the WestGrid computer cluster (bugaboo cluster).

Submitting the following command and getting the output as below:

mothur > get.oturep(column=stability.trim.contigs.good.unique.good.filter.unique.precluster.pick.pick.dist, fasta=stability.trim.contigs.good.unique.good.filter.unique.precluster.pick.pick.fasta, list=stability.trim.contigs.good.unique.good.filter.unique.precluster.pick.pick.opti_mcc.0.03.pick.list, count=stability.trim.contigs.good.unique.good.filter.unique.precluster.count_table)
********************#****#****#****#****#****#****#****#****#****#****#
Reading matrix:     |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||[ERROR]: std::bad_allocRAM used: 13.7773Gigabytes . Total Ram: 15.5756Gigabytes.

 has occurred in the SparseDistanceMatrix class function addCell. This error indicates your computer is running out of memory.  This is most commonly caused by trying to process a dataset too large, using multiple processors, or a file format issue. If you are running our 32bit version, your memory usage is limited to 4G.  If you have more than 4G of RAM and are running a 64bit OS, using our 64bit version may resolve your issue.  If you are using multiple processors, try running the command with processors=1, the more processors you use the more memory is required. Also, you may be able to reduce the size of your dataset by using the commands outlined in the Schloss SOP, http://www.mothur.org/wiki/Schloss_SOP. If you are unable to resolve the issue, please contact Pat Schloss at mothur.bugs@gmail.com, and be sure to include the mothur.logFile with your inquiry.program finished with exit code 1 at: Thu Jul 20 17:07:20 PDT 2017

Submitting the job as:

qsub -l procs=6,pmem=4000mb,walltime=12:00:00 mothurTop30.pbs

I’m using the top 30 OTU’s from a sub-set of my samples, and needing the fasta files from this step for further processing of my data, but I keep hitting this same error.

Would this error not happen if I used an accnos file with fewer than 30 OTU’s? I can’t request more processors with this command, and can’t get any more memory out of the processors. Requesting more processors wouldn’t make a difference since it’s a single-processor limited command, right?

Very appreciative of the help so far, and hoping to resolve this issue!

2

Just an update from what I’ve tried and what the problem might be (after moving to a more powerful computer cluster (Cedar on Westgrid) but I’m still running into the memory limit error described above.

The issue is likely due to the size of my dist file (stability.trim.contigs.good.unique.good.filter.unique.precluster.pick.pick.dist), which is ~33 gb, and requests 15.6 gb of RAM to read the dist file when when the error message hits. Since this is a single processor command, is there a way to “shrink” or split up the .dist file so that it doesn’t require so much RAM to read the dist file all at once? I’ve requested as much as 96 gb of ram on two nodes, with 12 cores per node, but the memory seems to max out in around 2 hours each time, no matter how much memory or how many extra cores I request. I’m trying a request for 16 gb of RAM from a single node on the server, but that will likely take an unreasonable amount of time.

Any advice is appreciated!

3

You want to omit the distance matrix and use method=abundance. https://mothur.org/wiki/Get.oturep#method