Negative mcc in the *opti_mcc.sensspec output of cluster.split

1

I’m running a 16S rRNA amplicon analysis using the MiSeq SOP (V4 region) with mothur v.1.42.3. Everything looks good until the cluster.split command. Here, I split a pretty big distance matrix and get a weird result (*opti_mcc.sensspec output): a negative Matthews correlation coefficient (-0.26).

My job ran with 8 processors (the same thing happened with more processors), I copy the logfile below. Any help is more than welcomed.

Thanks!

--
Linux version

Using ReadLine,Boost,HDF5
mothur v.1.42.3
Last updated: 08/08/2019
by
Patrick D. Schloss

Department of Microbiology & Immunology

University of Michigan
http://www.mothur.org

When using, please cite:
Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41.

Distributed under the GNU General Public License

Type 'help()' for information on the commands that are available

For questions and analysis support, please visit our forum at https://forum.mothur.org

Type 'quit()' to exit program

[NOTE]: Setting random seed to 19760620.

Script Mode


mothur > set.dir(input=/home/apolo\-jsescobarr/uchuva/uchuva_only)
Mothur's directories:
inputDir=/home/apolo\-jsescobarr/uchuva/uchuva_only/

mothur > cluster.split(column=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist, count=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.denovo.uchime.pick.pick.count_table, taxonomy=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.gg.wang.pick.taxonomy, splitmethod=classify, taxlevel=4, large=T, cutoff=0.03, processors=8)

Using 8 processors.
Splitting the file...
It took 2611 seconds to split the distance file.
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.1.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.13.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.18.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.5.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.12.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.64.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.51.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.47.temp
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.65.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.49.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.61.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.65.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.71.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.36.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.58.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.53.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	14	0	1	0	1	0	0.933333	1	0.933333	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
2	19	0	0	1	1	1	1	1	1	1	1	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1	4	0	1	0.5	1	1	0.8	1	0.833333	0.632456	0.666667	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
2	19	0	0	1	1	1	1	1	1	
Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.59.temp
1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.50.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.47.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1	1	0	1	0.5	1	1	0.5	1	0.666667	0.5	0.666667	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.34.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	60	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.38.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	52	0	3	0	1	0	0.945455	1	0.945455	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.66.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.40.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
3	5	0	2	0.6	1	1	0.714286	1	0.8	0.654654	0.75	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
5	113	1	1	0.833333	0.991228	0.833333	0.991228	0.833333	0.983333	0.824561	0.833333	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	6	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.42.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	4	0	6	0	1	0	0.4	1	0.4	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.57.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	6	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.51.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
7	57	0	2	0.777778	1	1	0.966102	1	0.969697	0.86684	0.875	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.67.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.44.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.43.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	8	0	1	0.857143	1	1	0.888889	1	0.933333	0.872872	0.923077	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	2	0	2	0.75	1	1	0.5	1	0.8	0.612372	0.857143	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.28.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
9	4	1	1	0.9	0.8	0.9	0.8	0.9	0.866667	0.7	0.9	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.69.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	5	0	10	0	1	0	0.333333	1	0.333333	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
11	1263	1	0	1	0.999209	0.916667	1	0.916667	0.999216	0.957048	0.956522	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.29.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	15	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.64.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
39	663	1	0	1	0.998494	0.975	1	0.975	0.998578	0.986677	0.987342	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.62.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
17	38	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.24.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.56.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
31	73	0	1	0.96875	1	1	0.986486	1	0.990476	0.977578	0.984127	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.21.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.45.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
34	1442	0	9	0.790698	1	1	0.993797	1	0.993939	0.88645	0.883117	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
54	79	2	1	0.981818	0.975309	0.964286	0.9875	0.964286	0.977941	0.954453	0.972973	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
61	792	5	3	0.953125	0.993726	0.924242	0.996226	0.924242	0.990708	0.933567	0.938462	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.46.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
64	663	6	8	0.888889	0.991031	0.914286	0.988077	0.914286	0.981107	0.891071	0.901408	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.12.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.54.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
98	336	27	4	0.960784	0.92562	0.784	0.988235	0.784	0.933333	0.827353	0.863436	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
60	656	8	17	0.779221	0.987952	0.882353	0.97474	0.882353	0.966262	0.810887	0.827586	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.35.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
81	548	1	0	1	0.998179	0.987805	1	0.987805	0.998413	0.992978	0.993865	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.23.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.26.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
97	2211	2	36	0.729323	0.999096	0.979798	0.983979	0.979798	0.983802	0.837875	0.836207	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.14.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
138	447	7	38	0.784091	0.984581	0.951724	0.921649	0.951724	0.928571	0.819352	0.859813	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
121	4004	19	42	0.742331	0.995277	0.864286	0.989619	0.864286	0.985428	0.793629	0.79868	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.5.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
166	734	22	24	0.873684	0.970899	0.882979	0.968338	0.882979	0.951374	0.847943	0.878307	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.30.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.32.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
186	4808	11	45	0.805195	0.997717	0.944162	0.990727	0.944162	0.988911	0.866392	0.869159	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
415	1132	67	39	0.914097	0.94412	0.860996	0.966695	0.860996	0.935874	0.842816	0.886752	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.20.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.22.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.39.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
183	588	19	71	0.720472	0.968699	0.905941	0.892261	0.905941	0.89547	0.741685	0.802632	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.31.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
574	866	55	158	0.784153	0.940282	0.91256	0.845703	0.91256	0.871143	0.741156	0.843497	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
615	15476	64	135	0.82	0.995882	0.905744	0.991352	0.905744	0.987784	0.855526	0.860742	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
434	73402	39	45	0.906054	0.999469	0.917548	0.999387	0.917548	0.998864	0.911211	0.911765	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.18.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
707	15871	108	150	0.824971	0.993241	0.867485	0.990637	0.867485	0.984676	0.837929	0.845694	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
951	4087	21	194	0.830568	0.994888	0.978395	0.954683	0.978395	0.959071	0.877619	0.898441	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.37.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1204	6158	102	537	0.691557	0.983706	0.921899	0.919791	0.921899	0.920135	0.753898	0.790286	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.15.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.16.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
4122	75308	553	2232	0.648725	0.99271	0.881711	0.971215	0.881711	0.966125	0.73966	0.747484	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.19.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6242	376043	1678	1918	0.764951	0.995558	0.788131	0.994925	0.788131	0.990681	0.7717	0.776368	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
15077	104985	2080	5118	0.746571	0.980573	0.878767	0.953516	0.878767	0.943439	0.777939	0.807293	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.7.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.10.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
24765	486905	2648	10482	0.702613	0.994591	0.903403	0.978926	0.903403	0.974981	0.784324	0.790456	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
23080	709302	4109	9540	0.707541	0.99424	0.848873	0.986729	0.848873	0.981705	0.765774	0.77179	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.13.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.6.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
48952	1602299	11355	23760	0.673231	0.992963	0.811713	0.985388	0.811713	0.979177	0.728714	0.736015	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.9.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
100864	418660	16420	21096	0.827025	0.96226	0.859998	0.952028	0.859998	0.932651	0.800575	0.843189	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.8.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
130547	1058643	10002	32473	0.800804	0.99064	0.928836	0.970239	0.928836	0.965514	0.843545	0.860081	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.25.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1512	332486	285	688	0.687273	0.999144	0.841402	0.997935	0.841402	0.997095	0.759035	0.756567	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.17.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
282	10311	51	87	0.764228	0.995078	0.846847	0.991633	0.846847	0.98714	0.797911	0.803419	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.33.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
57	901	7	25	0.695122	0.992291	0.890625	0.973002	0.890625	0.967677	0.770499	0.780822	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.41.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
37	314	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.55.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
8	19	1	0	1	0.95	0.888889	1	0.888889	0.964286	0.918937	0.941176	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.70.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	4	0	6	0	1	0	0.4	1	0.4	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.48.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.4.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
108423	72899124	25692	56856	0.656	0.999648	0.808433	0.999221	0.808433	0.998871	0.727692	0.724283	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.2.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
265663	20862410	71104	98424	0.729669	0.996603	0.788863	0.995304	0.788863	0.99204	0.754665	0.758112	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
601063	654051	51691	37175	0.941754	0.926757	0.920811	0.946219	0.920811	0.933878	0.86777	0.931165	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.11.temp

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.3.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1425288	17531093	103012	219552	0.866521	0.994158	0.932597	0.987631	0.932597	0.983269	0.889956	0.898345	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1174170	5968595	139168	454278	0.721036	0.977215	0.894035	0.929272	0.894035	0.92329	0.758205	0.79827	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.1.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
33399680	2548492530	7740631	8526814	0.796625	0.996972	0.811848	0.996665	0.811848	0.993739	0.80102	0.804164	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist.0.temp

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
86350507	61835371972	22151524	66747858	0.56402	0.999642	0.795842	0.998922	0.795842	0.998566	0.669312	0.660171	

It took 3126 seconds to cluster
Merging the clustered files...
It took 5 seconds to merge.
/******************************************/
Running command: sens.spec(cutoff=0.03, list=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.list, column=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist, count=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.denovo.uchime.pick.pick.count_table)

NOTE: sens.spec assumes that only unique sequences were used to generate the distance matrix.

label	cutoff	numotus	tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0.03	0.03	70839	123686728	108456868966	30332600	76882436	0.616679	0.99972	0.80306	0.999292	0.80306	0.999014	-0.263971	0.697635

It took 1573 to run sens.spec.

Output File Names: 
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.sensspec

/******************************************/
Done.


label	cutoff	numotus	tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0.03	0.03	70839	123686728	108456868966	30332600	76882436	0.6167	0.9997	0.8031	0.9993	0.8031	0.999	-0.264	0.6976

Output File Names: 
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.list
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.sensspec


mothur > quit()
2

Hmmm, that is weird. I wonder if the distance file is corrupted somehow. Could you try running the following instead? It should be much faster and use less RAM…

cluster.split(fasta=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta, count=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.denovo.uchime.pick.pick.count_table, taxonomy=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.gg.wang.pick.taxonomy, splitmethod=classify, taxlevel=4, cutoff=0.03, processors=8)
3

Dear Pat, thank you for your answer. I tried the command you suggested, but I got the same result (negative MCC). Below is the full logfile of that run.

Looking at the logfile, I found something weird in it: throughout the parallelization, MCC values look reasonably high, it’s only at the end of the process (sens.spec command) that I got the unexpected output (MCC = -0.264382). I thought it could be a parallelization problem and ran my original cluster.split command with 1 processor. Disappointingly enough, I got the same result.

FYI, all my upstream controls look pretty well. I indeed inspected the *pick.pick.fasta alignment with ARB and looks quite good. Any idea of what can be happening?

Linux version

Using ReadLine,Boost,HDF5
mothur v.1.42.3
Last updated: 08/15/2019
[NOTE]: Setting random seed to 19760620.

Script Mode

mothur > cluster.split(fasta=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta, count=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.denovo.uchime.pick.pick.count_table, taxonomy=uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.gg.wang.pick.taxonomy, splitmethod=classify, taxlevel=4, cutoff=0.03)

Using 32 processors.
Splitting the file...
/******************************************/
Running command: dist.seqs(fasta=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.0.temp, processors=32, cutoff=0.03)

*** etc. (for reasons of space I cut this chunk of the logfile)

It took 6530 seconds to split the distance file.
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.1.dist
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.12.dist
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.65.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.21.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.65.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.71.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.56.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.24.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.29.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.41.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.64.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.62.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.69.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.28.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.43.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.57.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.44.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.55.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.67.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.51.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.42.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.40.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.66.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.38.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.50.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.70.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.34.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.47.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.59.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.49.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.61.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.36.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.58.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.53.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.33.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	4	0	6	0	1	0	0.4	1	0.4	0	0	

0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	5	0	10	0	1	0	0.333333	1	0.333333	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
9	4	1	1	0.9	0.8	0.9	0.8	0.9	0.866667	0.7	0.9	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	6	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
2	19	0	0	1	1	1	1	1	1	1	1	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	14	0	1	0	1	0	0.933333	1	0.933333	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	8	0	1	0.857143	1	1	0.888889	1	0.933333	0.872872	0.923077	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.45.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1	1	0	1	0.5	1	1	0.5	1	0.666667	0.5	0.666667	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.12.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.22.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
54	79	2	1	0.981818	0.975309	0.964286	0.9875	0.964286	0.977941	0.954453	0.972973	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
17	38	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.31.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
8	19	1	0	1	0.95	0.888889	1	0.888889	0.964286	0.918937	0.941176	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.26.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
3	5	0	2	0.6	1	1	0.714286	1	0.8	0.654654	0.75	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.54.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	6	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.18.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1	4	0	1	0.5	1	1	0.8	1	0.833333	0.632456	0.666667	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.14.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	52	0	3	0	1	0	0.945455	1	0.945455	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
31	73	0	1	0.96875	1	1	0.986486	1	0.990476	0.977578	0.984127	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	2	0	2	0.75	1	1	0.5	1	0.8	0.612372	0.857143	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.32.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
2	19	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.39.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.16.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	4	0	6	0	1	0	0.4	1	0.4	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.23.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.25.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	15	0	0	0	0	1	0	0	0	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.5.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
7	57	0	2	0.777778	1	1	0.966102	1	0.969697	0.86684	0.875	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.46.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
37	314	0	0	1	1	1	1	1	1	1	1	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
5	113	1	1	0.833333	0.991228	0.833333	0.991228	0.833333	0.983333	0.824561	0.833333	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6	60	0	0	1	1	1	1	1	1	1	1	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.35.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
61	792	5	3	0.953125	0.993726	0.924242	0.996226	0.924242	0.990708	0.933567	0.938462	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.17.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
57	901	7	25	0.695122	0.992291	0.890625	0.973002	0.890625	0.967677	0.770499	0.780822	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
39	663	1	0	1	0.998494	0.975	1	0.975	0.998578	0.986677	0.987342	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.37.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
34	1442	0	9	0.790698	1	1	0.993797	1	0.993939	0.88645	0.883117	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
11	1263	1	0	1	0.999209	0.916667	1	0.916667	0.999216	0.957048	0.956522	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
64	663	6	8	0.888889	0.991031	0.914286	0.988077	0.914286	0.981107	0.891071	0.901408	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.20.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
60	656	8	17	0.779221	0.987952	0.882353	0.97474	0.882353	0.966262	0.810887	0.827586	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.30.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
615	15476	64	135	0.82	0.995882	0.905744	0.991352	0.905744	0.987784	0.855526	0.860742	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.48.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
138	447	7	38	0.784091	0.984581	0.951724	0.921649	0.951724	0.928571	0.819352	0.859813	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
707	15871	108	150	0.824971	0.993241	0.867485	0.990637	0.867485	0.984676	0.837929	0.845694	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
81	548	1	0	1	0.998179	0.987805	1	0.987805	0.998413	0.992978	0.993865	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
166	734	22	24	0.873684	0.970899	0.882979	0.968338	0.882979	0.951374	0.847943	0.878307	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
951	4087	21	194	0.830568	0.994888	0.978395	0.954683	0.978395	0.959071	0.877619	0.898441	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
183	588	19	71	0.720472	0.968699	0.905941	0.892261	0.905941	0.89547	0.741685	0.802632	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.19.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
574	866	55	158	0.784153	0.940282	0.91256	0.845703	0.91256	0.871143	0.741156	0.843497	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
121	4004	19	42	0.742331	0.995277	0.864286	0.989619	0.864286	0.985428	0.793629	0.79868	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
186	4808	11	45	0.805195	0.997717	0.944162	0.990727	0.944162	0.988911	0.866392	0.869159	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
98	336	27	4	0.960784	0.92562	0.784	0.988235	0.784	0.933333	0.827353	0.863436	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
97	2211	2	36	0.729323	0.999096	0.979798	0.983979	0.979798	0.983802	0.837875	0.836207	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.15.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1512	332486	285	688	0.687273	0.999144	0.841402	0.997935	0.841402	0.997095	0.759035	0.756567	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
282	10311	51	87	0.764228	0.995078	0.846847	0.991633	0.846847	0.98714	0.797911	0.803419	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1204	6158	102	537	0.691557	0.983706	0.921899	0.919791	0.921899	0.920135	0.753898	0.790286	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
415	1132	67	39	0.914097	0.94412	0.860996	0.966695	0.860996	0.935874	0.842816	0.886752	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0	0	0	1	0	0	0	0	1	0	0	0	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
434	73402	39	45	0.906054	0.999469	0.917548	0.999387	0.917548	0.998864	0.911211	0.911765	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
6242	376043	1678	1918	0.764951	0.995558	0.788131	0.994925	0.788131	0.990681	0.7717	0.776368	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
4122	75308	553	2232	0.648725	0.99271	0.881711	0.971215	0.881711	0.966125	0.73966	0.747484	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
15077	104985	2080	5118	0.746571	0.980573	0.878767	0.953516	0.878767	0.943439	0.777939	0.807293	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.10.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.7.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
24765	486905	2648	10482	0.702613	0.994591	0.903403	0.978926	0.903403	0.974981	0.784324	0.790456	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.13.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
23080	709302	4109	9540	0.707541	0.99424	0.848873	0.986729	0.848873	0.981705	0.765774	0.77179	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
48952	1602299	11355	23760	0.673231	0.992963	0.811713	0.985388	0.811713	0.979177	0.728714	0.736015	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.6.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.9.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
100864	418660	16420	21096	0.827025	0.96226	0.859998	0.952028	0.859998	0.932651	0.800575	0.843189	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.8.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
130547	1058643	10002	32473	0.800804	0.99064	0.928836	0.970239	0.928836	0.965514	0.843545	0.860081	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.4.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
108423	72899124	25692	56856	0.656	0.999648	0.808433	0.999221	0.808433	0.998871	0.727692	0.724283	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.2.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
265663	20862410	71104	98424	0.729669	0.996603	0.788863	0.995304	0.788863	0.99204	0.754665	0.758112	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
601063	654051	51691	37175	0.941754	0.926757	0.920811	0.946219	0.920811	0.933878	0.86777	0.931165	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.3.dist

Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.11.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1425288	17531093	103012	219552	0.866521	0.994158	0.932597	0.987631	0.932597	0.983269	0.889956	0.898345	


tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
1174170	5968595	139168	454278	0.721036	0.977215	0.894035	0.929272	0.894035	0.92329	0.758205	0.79827	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.1.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
33399680	2548492530	7740631	8526814	0.796625	0.996972	0.811848	0.996665	0.811848	0.993739	0.80102	0.804164	


Clustering /home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.fasta.0.dist

tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
86350507	61835371972	22151524	66747858	0.56402	0.999642	0.795842	0.998922	0.795842	0.998566	0.669312	0.660171	

It took 3033 seconds to cluster
Merging the clustered files...
It took 5 seconds to merge.
/******************************************/
Running command: sens.spec(cutoff=0.03, list=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.list, column=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist, count=/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.denovo.uchime.pick.pick.count_table)

NOTE: sens.spec assumes that only unique sequences were used to generate the distance matrix.

label	cutoff	numotus	tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0.03    0.03    70869   123686728       108457501401    30332600        76250001        0.618629        0.99972 0.80306 0.999297        0.80306 0.999019        -0.264382       0.698882

It took 1486 to run sens.spec.

Output File Names: 
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.sensspec

/******************************************/
Done.


label	cutoff	numotus	tp	tn	fp	fn	sensitivity	specificity	ppv	npv	fdr	accuracy	mcc	f1score
0.03	0.03	70869	123686728	108457501401	30332600	76250001	0.6186	0.9997	0.8031	0.9993	0.8031	0.999	-0.2644	0.6989

Output File Names: 
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.dist
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.list
/home/apolo\-jsescobarr/uchuva/uchuva_only/uchuva_only.trim.contigs.trim.trim.good.unique.good.filter.pick.unique.precluster.pick.pick.opti_mcc.sensspec


mothur > quit()
4

Thanks for reporting this issue. The mcc value is negative due to a datatype overflow in mothur. The large values stored in tp, tn, fp, fn result in values too large for a long long type, and need to be stored in a double. I will fix this in mothur for our next release. Your clustering and mcc values look good, matthewsCorrCoef = ((123686728 * 108457501401) - (30332600 * 76250001)) / sqrt((123686728 + 76250001) * (30332600 + 108457501401) * (123686728 + 30332600) * (108457501401 + 76250001)) = 0.704370.

The equation for mcc is:

p = tp + fn;
n = fp + tn;
pPrime = tp + fp;
nPrime = tn + fn;
matthewsCorrCoef = ((tp * tn) - (fp * fn)) / sqrt(p * n * pPrime * nPrime);

5

Thanks Sarah for your answer. That’s great news. And happy to contribute with improvements in Mothur.