Preface: I am trying to replicate results from a publication that used mothur v.1.37.4. However, I ran into a bug(?) that resulted in variable results when using the command “make.contigs()” with different numbers of processors.
Overview: After running the command “make.contigs(ffastq=/home//Downloads/testFlax_local/oldMoth/fastq/ERR2179953_1.fastq, rfastq=/home//Downloads/testFlax_local/oldMoth/fastq/ERR2179953_2.fastq, oligos=/home//Downloads/testFlax_local/oldMoth/wd/primers.oligo, pdiffs=0, insert=30, processors=128)” the output indicates 150769 counts from 170196 sequences. Running the same command, but with 16 processors, outputs 150806 counts from the same number of sequences. Neither of these commands resulted in the expected number of counts for this group (as suggested by the publication). However, when running the command with the processors set to 1 I get 154399 counts which is expected. The log is provided below:
Linux version
Running 64Bit Version
mothur v.1.37.4
Last updated: 5/11/2016
by
Patrick D. Schloss
Department of Microbiology & Immunology
University of Michigan
pschloss@umich.edu
http://www.mothur.org
When using, please cite:
Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41.
Distributed under the GNU General Public License
Type 'help()' for information on the commands that are available
Type 'quit()' to exit program
Interactive Mode
mothur >
mothur > set.dir(output=/home/<usr>/Downloads/testFlax_local/oldMoth/output)
Mothur's directories:
outputDir=/home/<usr>/Downloads/testFlax_local/oldMoth/output/
mothur >
mothur > make.contigs(ffastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_1.fastq, rfastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_2.fastq, oligos=/home/<usr>/Downloads/testFlax_local/oldMoth/wd/primers.oligo, pdiffs=0, insert=30, processors=128)
Using 128 processors.
Making contigs...
Done.
It took 65 secs to process 170196 sequences.
Group count:
16S 150769
Total of all groups is 150769
Output File Names:
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.report
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.groups
[WARNING]: your sequence names contained ':'. I changed them to '_' to avoid problems in your downstream analysis.
mothur >
mothur > make.contigs(ffastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_1.fastq, rfastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_2.fastq, oligos=/home/<usr>/Downloads/testFlax_local/oldMoth/wd/primers.oligo, pdiffs=0, insert=30, processors=1)
Using 1 processors.
Making contigs...
Done.
It took 76 secs to process 170196 sequences.
Group count:
16S 154399
Total of all groups is 154399
Output File Names:
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.report
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.groups
mothur >
mothur > make.contigs(ffastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_1.fastq, rfastq=/home/<usr>/Downloads/testFlax_local/oldMoth/fastq/ERR2179953_2.fastq, oligos=/home/<usr>/Downloads/testFlax_local/oldMoth/wd/primers.oligo, pdiffs=0, insert=30, processors=16)
Using 16 processors.
Making contigs...
Done.
It took 58 secs to process 170196 sequences.
Group count:
16S 150806
Total of all groups is 150806
Output File Names:
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.fasta
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.trim.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.scrap.contigs.qual
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.report
/home/<usr>/Downloads/testFlax_local/oldMoth/output/ERR2179953_1.contigs.groups
I also ran similar commands with v.1.48.3 comparing processors=1 and processors=128. I got the same results where the command with processors=1 returned the expected counts per group (according to the publication) and the command with processors=128 returned supposedly deflated counts. Log file provided below:
Linux version
Using Boost,HDF5,GSL
mothur v.1.48.3
Last updated: 5/21/25
by
Patrick D. Schloss
Department of Microbiology & Immunology
University of Michigan
http://www.mothur.org
When using, please cite:
Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41.
Distributed under the GNU General Public License
Type 'help()' for information on the commands that are available
For questions and analysis support, please visit our forum at https://forum.mothur.org
Type 'quit()' to exit program
[NOTE]: Setting random seed to 19760620.
Interactive Mode
mothur >
mothur > set.dir(input=/home/<usr>/testFlax/data/fastqFiles/early;/home/<usr>/testFlax/data/reference;/home/<usr>/testFlax/data/reference/trainset19_072023.pds;/home/<usr>/testFlax/data/testBatch)
Mothur's directories:
inputdir=
/home/<usr>/testFlax/data/fastqFiles/early/
/home/<usr>/testFlax/data/reference/
/home/<usr>/testFlax/data/reference/trainset19_072023.pds/
/home/<usr>/testFlax/data/testBatch/
mothur >
mothur > set.dir(output=/home/<usr>/testFlax/data/mothurDir)
Mothur's directories:
outputDir=/home/<usr>/testFlax/data/mothurDir/
mothur >
mothur > make.file(inputdir=/home/<usr>/testFlax/data/fastqFiles/early, type=fastq, prefix=stability)
Setting input directories to:
/home/<usr>/testFlax/data/fastqFiles/early/
Output File Names:
/home/<usr>/testFlax/data/mothurDir/stability.files
mothur >
mothur > make.contigs(file=/home/<usr>/testFlax/data/oldMothPipe/fileList.single.file, oligos=/home/<usr>/testFlax/data/mothur_old/primers.oligo, pdiffs=0, insert=30, processors=1)
Using 1 processors.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_2.fastq (files 1 of 7) <<<<<
Making contigs...
Done.
It took 135 secs to assemble 192608 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_2.fastq (files 2 of 7) <<<<<
Making contigs...
Done.
It took 117 secs to assemble 168659 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_2.fastq (files 3 of 7) <<<<<
Making contigs...
Done.
It took 118 secs to assemble 170196 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_2.fastq (files 4 of 7) <<<<<
Making contigs...
Done.
It took 90 secs to assemble 129539 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_2.fastq (files 5 of 7) <<<<<
Making contigs...
Done.
It took 102 secs to assemble 148287 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_2.fastq (files 6 of 7) <<<<<
Making contigs...
Done.
It took 71 secs to assemble 103859 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_2.fastq (files 7 of 7) <<<<<
Making contigs...
Done.
It took 194 secs to assemble 279553 reads.
Group count:
ERR2179951 172692
ERR2179952 153136
ERR2179953 154399
ERR2179954 117611
ERR2179955 133806
ERR2179956 94244
ERR2179957 255084
Total of all groups is 1080972
It took 831 secs to process 1192701 sequences.
Output File Names:
/home/<usr>/testFlax/data/mothurDir/fileList.single.trim.contigs.fasta
/home/<usr>/testFlax/data/mothurDir/fileList.single.scrap.contigs.fasta
/home/<usr>/testFlax/data/mothurDir/fileList.single.contigs_report
/home/<usr>/testFlax/data/mothurDir/fileList.single.contigs.count_table
mothur >
mothur > make.contigs(file=current, maxambig=0, maxlength=500, maxhomop=8, oligos=primers.oligo, pdiffs=0, insert=30, processors=128)
Using /home/<usr>/testFlax/data/mothurDir/stability.files as input file for the file parameter.
Using 128 processors.
Unable to open ERR2179951_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_1.fastq.
Unable to open ERR2179951_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_2.fastq.
Unable to open ERR2179952_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_1.fastq.
Unable to open ERR2179952_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_2.fastq.
Unable to open ERR2179953_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_1.fastq.
Unable to open ERR2179953_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_2.fastq.
Unable to open ERR2179954_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_1.fastq.
Unable to open ERR2179954_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_2.fastq.
Unable to open ERR2179955_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_1.fastq.
Unable to open ERR2179955_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_2.fastq.
Unable to open ERR2179956_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_1.fastq.
Unable to open ERR2179956_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_2.fastq.
Unable to open ERR2179957_1.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_1.fastq.
Unable to open ERR2179957_2.fastq. Trying input directory /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_2.fastq.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179951_2.fastq (files 1 of 7) <<<<<
Making contigs...
Done.
It took 9 secs to assemble 192608 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179952_2.fastq (files 2 of 7) <<<<<
Making contigs...
Done.
It took 9 secs to assemble 168659 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179953_2.fastq (files 3 of 7) <<<<<
Making contigs...
Done.
It took 9 secs to assemble 170196 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179954_2.fastq (files 4 of 7) <<<<<
Making contigs...
Done.
It took 6 secs to assemble 129539 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179955_2.fastq (files 5 of 7) <<<<<
Making contigs...
Done.
It took 8 secs to assemble 148287 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179956_2.fastq (files 6 of 7) <<<<<
Making contigs...
Done.
It took 5 secs to assemble 103859 reads.
>>>>> Processing file pair /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_1.fastq - /home/<usr>/testFlax/data/fastqFiles/early/ERR2179957_2.fastq (files 7 of 7) <<<<<
Making contigs...
Done.
It took 14 secs to assemble 279553 reads.
Group count:
ERR2179951 134480
ERR2179952 121728
ERR2179953 121072
ERR2179954 92847
ERR2179955 105730
ERR2179956 75469
ERR2179957 204189
Total of all groups is 855515
It took 64 secs to process 1192701 sequences.
Output File Names:
/home/<usr>/testFlax/data/mothurDir/stability.trim.contigs.fasta
/home/<usr>/testFlax/data/mothurDir/stability.scrap.contigs.fasta
/home/<usr>/testFlax/data/mothurDir/stability.contigs_report
/home/<usr>/testFlax/data/mothurDir/stability.contigs.count_table
mothur >
mothur > quit()
Logfile : mothur.1750845829.logfile
Question: What causes this behavior? I’m confused because I would assume the number of processors is unrelated to the computation underlying the formation of contigs and just specifies the computational power that can be allocated to it so these differing results make me wonder. To note, the systems I am using are capable of using these numbers of processors. I apologize if this is a duplicate post, if so please refer me to the OP.