# Problems when using cluster.split on huge .dist file **URL:** \ **Category:** Commands in mothur\ **Created:** [August 19, 2019, 2:10pm UTC](https://forum.mothur.org/t/problems-when-using-cluster-split-on-huge-dist-file/20073 "2019-08-19T14:10:54Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![mafernandez](https://avatars.discourse-cdn.com/v4/letter/m/45deac/32.png) [@mafernandez](https://forum.mothur.org/u/mafernandez)\ **Post date:** [August 19, 2019, 2:10pm UTC](https://forum.mothur.org/t/problems-when-using-cluster-split-on-huge-dist-file/20073/1 "2019-08-19T14:10:54Z")
Hello there. This is a continuation of the topic posted at: > [@HUGE dist file when running Eukarya analysis](https://forum.mothur.org/t/huge-dist-file-when-running-eukarya-analysis/20028): > > Hello there. I am running a MiSeq analysis on 11 samples of 18S rDNA (Eukarya) following MiSeqSOP (Mothur v.1.42) on a 16 processors server (64GB RAM). As it has happened to me before, I know that with my pair of primers we amplify lots of diversity (from algae to insects), so the dist. file should always be very big, even when running it with cutoff=0.03. However, in this experiment it is extremely big for such a small number of samples. It took 61 hours to complete the command (dist.seqs(fas… As it has been closed due to inactivity. I have a problem when trying to cluster eukaryotic sequences using either cluster or cluster.split. After several tries where the system collapsed I ran dist.seqs with the column output option and then tried to run cluster.split like this: ``` cluster.split(column=Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist, count=Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.denovo.vsearch.pick.count_table, cutoff=0.03, large=T) ``` It is, with all the requierments to avoid the use of too much memory. However, I got something like this: ``` Using 16 processors. Splitting the file... It took 529530 seconds to split the distance file. Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.33.temp Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.209.temp Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.115.temp Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.451.temp Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.39.temp (...) Clustering Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.807.temp Clustering Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.658.temp Clustering Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.724.temp Clustering Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.1118.temp Clustering Eukarya_rocas.trim.contigs.good.unique.good.filter.unique.precluster.pick.dist.1293.temp (...) tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score 0 1428050402 0 1 0 1428050402 0 1 0 1428050402 0 1 0 1428050402 0 1 0 1428050402 0 1 tp tn fp fn sensitivity specificity ppv npv fdr accuracy mcc f1score (...) ``` And then, the process stops. Any idea of what is happening? Could it be a problem of disk space? I am running mothur on a windows server with 16 processors and 64gb of RAM Thanks ---
**Author:** ![westcott](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.mothur.org/westcott/32/18_2.png) [@westcott](https://forum.mothur.org/u/westcott)\ **Post date:** [August 20, 2019, 6:46pm UTC](https://forum.mothur.org/t/problems-when-using-cluster-split-on-huge-dist-file/20073/2 "2019-08-20T18:46:17Z")
Here are a few suggestions: 1. Run cluster.split with fasta, count, taxonomy files and set cluster=f. [https://mothur.org/wiki/Cluster.split#file](https://mothur.org/wiki/Cluster.split#file) mothur \> cluster.split(fasta=final.fasta, name=final.names, taxonomy=final.taxonomy, taxlevel=4, cluster=f, cutoff=0.03) - splits files and creates distance matrices. By running the splitting separately from the clustering you can increase the processors used in the splitting and reduce the processors used in the clustering which is much more memory intensive. You also eliminated the need to re-split in the clustering fails which can be a major time saver. mothur \> cluster.split(file=final.file, processors=4) - you may need to run with with processors=1 so only one distance matrix is loaded into memory at a time. 1. The command could be failing when it tries to find the final sensspec results. Try running with runsensspec=f. This will also stop mothur from creating the complete distance matrix which can use a lot of hard disk space. 2. Use phylotype clustering. This may be required if your dataset is too large to clustering by distance. 3. Try running mothur using Amazon, [https://mothur.org/wiki/Mothur\_AMI](https://mothur.org/wiki/Mothur_AMI) ---
**Author:** ![system](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.mothur.org/system/32/2_2.png) [@system](https://forum.mothur.org/u/system)\ **Post date:** [August 30, 2019, 6:55pm UTC](https://forum.mothur.org/t/problems-when-using-cluster-split-on-huge-dist-file/20073/3 "2019-08-30T18:55:41Z")
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.