cluster.split error

mothur bugs
1

is this a RAM issue? I have 24gb ram and am testing using a 240gb ssd as swap. I usually get to clustering before it crashes though

mothur > cluster.split(column=all_bac.unique.good.filter.unique.precluster.pick.dist, name=bac_final.names, taxonomy=bac_final.0.03cons.taxonomy, splitmethod=classify, processors=4, taxlevel=3, method=average)

Using 4 processors.
Using splitmethod classify.
Splitting the file...
It took 3867 seconds to split the distance file.
[ERROR]: unable to spawn the necessary processes.
2

It is most likely a RAM issue. When you run the cluster.split command with multiple processors, each processor clusters a separate file from the split. The clustering is very memory intensive. Have you tried running it with processors=1?

3

I reran it with one processor and get the same error. I was watching RAM usage and it never got about 5gb (physical ram 24GB) before crashing with the same error

4

It shouldn’t be possible to get “unable to spawn the necessary processes” for processors=1. Are you getting a different error?

5

as always looking for new and exciting ways to break programs :slight_smile:

mothur > cluster.split(column=all_bac.unique.good.filter.unique.precluster.pick.dist, name=bac_final.names, taxonomy=bac_final.0.03cons.taxonomy, splitmethod=classify, taxlevel=3, method=average)

Using 1 processors.
Using splitmethod classify.
Splitting the file...
It took 3872 seconds to split the distance file.
[ERROR]: unable to spawn the necessary processes.
6

:slight_smile: , so I found a way you could hit that. Mothur has a check, if your number of split files is smaller than the number of processors we reduce the number of processors. If your split had an error, it could return 0 split files therefore reducing your processors to 0. I noticed you used bac_final.0.03cons.taxonomy as your taxonomy file. The constaxonomy file looks like:

Otu001 744 Bacteria;“Bacteroidetes”;“Bacteroidia”;“Bacteroidales”;“Porphyromonadaceae”;unclassified;

the taxonomy file:

GQY1XT001C296C Bacteria(100);“Bacteroidetes”(100);“Bacteroidia”(99);“Bacteroidales”(99);“Porphyromonadaceae”(89);unclassified;

If your file does have the format of a cons.taxonomy file the reading / splitting will fail.

7

oh, not sure why I was using the cons taxonomy. trying it again