Note

Go to the end to download the full example code.

Tutorial part 3: statistical analysis of group-level RSA results#

In this tutorial, we’ll dive into cluster-based permutation testing for group-level statistical analysis.

The statistical test is described in Maris & Oostenveld, 2007. This functionality is not part of the MNE-RSA package, but rather MNE-Python, but it is a large part of a proper RSA analysis so I figures any decent RSA tutorial should cover this.

Cluster-based permutation testing is becoming the standard “go-to” method for testing for significant differences between experimental conditions in MEG and EEG studies. However, applying the technique correctly is not always straightforward and there are a lot of caveats to beware of. On top of this, the API of MNE-Python regarding these tests could use a bit more work. (If you are inspired by this tutorial, could you lend a hand with this? ❤)

The purpose of cluster-based permutation testing#

When exploring differences between conditions, a good first step is to perform separate t-tests for each channel/sourcepoint/timepoint and visualize them. While this gives you a good feel of where and when significant differences may occur, one cannot always make solid statistical inferences based on these values. Cluster-based permutation testing is designed to solve two problems that arise when using mass t-tests:

The multiple-comparisons problem: solved by using a cluster-based statistic
The non-Gaussian data problem: solved by using random permutations

The solutions to these problems come at a hefty price though: the test can only infer the data is different between conditions. The test can not be used to infer when and where there is a significant difference. Researchers get this wrong a lot. The Fieldtrip manual even has a dedicated page about this which I encourage you to read.

Now that I’ve scared you enough, let’s do this!

In the cell below, update the data_path variable to point to where you have extracted the `rsa-data.zip <wmvanvliet/neuroscience_tutorials>`__ file to.

# ruff: noqa: E402
# sphinx_gallery_thumbnail_number=2

import mne

mne.set_log_level("warning")  # Make the output less verbose

# Set this to where you've extracted `rsa-data.zip` to
data_path = "data"

Cluster permutation testing on a single channel#

In the classic EEG event-related potential (ERP) literature, statistical testing between two conditions was done by first identifying one or more sensors and stretches of time that seem likely to hold a significant difference, then averaging the signal across those sensors and time region, followed by a single t-test or ANOVA test. Time would tell whether the chosen channels and time region would replicate across studies.

Let’s do this for our sensor-level RSA result that we obtained in Tutorial part 1: RSA on sensor-level MEG data. I’ve ran the same analysis across multiple subjects and placed it in the group_analysis subfolder of the data package you’ve downloaded. Executing the call below will load them.

import mne

subjects = [2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19]

# RSA against the RDM based on the pixels in each image
rsa_pixels = [
    mne.read_evokeds(
        f"{data_path}/group_analysis/sub-{subject:02d}-rsa-ave.fif", condition="pixels"
    )
    for subject in subjects
]

# RSA against the RDM based on the FaceNet embeddings of each image
rsa_facenet = [
    mne.read_evokeds(
        f"{data_path}/group_analysis/sub-{subject:02d}-rsa-ave.fif", condition="facenet"
    )
    for subject in subjects
]

rsa_pixels and rsa_facenet are now lists of mne.Evoked objects. Here are the first 5 of them:

rsa_pixels[:5]

[<Evoked | 'pixels' (average, N=450), 0 – 0.9 s, baseline off, 1 ch, ~8 KiB>, <Evoked | 'pixels' (average, N=450), 0 – 0.9 s, baseline off, 1 ch, ~8 KiB>, <Evoked | 'pixels' (average, N=450), 0 – 0.9 s, baseline off, 1 ch, ~8 KiB>, <Evoked | 'pixels' (average, N=448), 0 – 0.9 s, baseline off, 1 ch, ~8 KiB>, <Evoked | 'pixels' (average, N=450), 0 – 0.9 s, baseline off, 1 ch, ~8 KiB>]

Every mne.Evoked` object has data defined for a single channel, for the time region 0 – 0.9 seconds. The cell below compares the grand averages for pixels versus FaceNet and gives the 95% confidence intervals for the means:

mne.viz.plot_compare_evokeds(
    dict(pixels=rsa_pixels, facenet=rsa_facenet),
    picks=0,
    show_sensors=0,
    ylim=dict(misc=[-0.02, 0.15]),
)

[<Figure size 800x600 with 1 Axes>]

Based on the above plot, there might be a significant difference very early on: 0 – 0.16 seconds. Let’s run a t-test in this time-window.

import numpy as np
from scipy.stats import ttest_rel

# Construct arrays with the data of all participants combined
data_pixels = np.array(
    [ev.data[0] for ev in rsa_pixels]
)  # shape: 16 x 199 (participants x timepoints)
data_facenet = np.array(
    [ev.data[0] for ev in rsa_facenet]
)  # shape: 16 x 199 (participants x timepoints)

# Time region we are interested in
tmin, tmax = 0, 0.16  # seconds

# Convert tmin/tmax to sample indices
tmin, tmax = np.searchsorted(rsa_pixels[0].times, [tmin, tmax])

# Average the data across the time window
data_pixels_avg = data_pixels[:, tmin:tmax].mean(axis=1)
data_facenet_avg = data_facenet[:, tmin:tmax].mean(axis=1)

# Perform the t-test
tval, pval = ttest_rel(data_pixels_avg, data_facenet_avg)

assert np.isscalar(tval), "We want a single t-value"
assert np.isscalar(pval), "We want a single p-value"
assert pval < 0.05, f"p-value is not significant {pval=}"

print(f"{tval=}, {pval=}   Looking great! 😊")

tval=np.float64(2.9768295878272895), pval=np.float64(0.009406024901266791)   Looking great! 😊

The “cluster” part of cluster permutation tests#

In essence, the permutation cluster test automates what we’ve just done above, but with a twist that we will get to later.

First, it must determine “clusters”, i.e. regions of interest where there might be a significant difference between the conditions. We eyeballed this based on the confidence interval of the mean, but the algorithm must use some form of automated function. We can specify the function that the algorithm should use, but it comes with a useful default: a t-test: mne.stats.ttest_1samp_no_p()

Let’s do this step manually first, so you get an understanding what is going on: - First, we do the t-test for every timepoint - This is a t-test against 0. To perform a paired t-test, we test data_pixels - data_facenet - Then, we set a threshold - Everything above the threshold becomes a cluster - We compute the sum of all the t-values inside the cluster

import matplotlib.pyplot as plt
from mne.stats import ttest_1samp_no_p

tvals = ttest_1samp_no_p(data_pixels - data_facenet)


# Plot the t-values and form a cluster
def plot_cluster(tvals, threshold, data_cond1, data_cond2):
    """Plot a cluster."""
    times = rsa_pixels[0].times

    # Make the cluster
    cluster = np.where(tvals > threshold)[0]

    # This is an important statistec: the sum of all t-values in the cluster
    cluster_tvals_sum = tvals[cluster].sum()

    # Show the t-values and how the cluster is determined
    plt.figure(figsize=(8, 4))
    plt.plot(times, tvals, label="t-values")
    plt.axhline(0, color="black", linewidth=1)
    plt.axhline(2, color="black", linestyle="--", linewidth=1, label="threshold")
    plt.fill_between(
        times[cluster],
        tvals[cluster],
        threshold * np.ones(len(cluster)),
        color="C1",
        alpha=0.5,
        label="cluster",
    )
    plt.legend()
    plt.xlabel("Time samples")
    plt.ylabel("t-value")
    return cluster_tvals_sum


cluster_tvals_sum = plot_cluster(
    tvals, threshold=2, data_cond1=data_pixels, data_cond2=data_facenet
)
print(f"Sum of t-values in the cluster: {cluster_tvals_sum:.3f}")

Sum of t-values in the cluster: 105.259

Based on the figure above, a threshold value of 2 seems to obtain a reasonable cluster. We could have chosen a different threshold, and there is no reason to choose a threshold that corresponds to some p-value, other than perhaps providing a sensible default value. The statistical function and the threshold used to form clusters are completely up to the researcher. The idea here is to use your domain knowledge to increase statistical power: when clusters “make sense” regarding to what we know about the experimental setup and how MEG data behaves, we are likely to obtain better p-values in the end.

The “permutation” part of the cluster permutation test#

Now that we have our cluster, we can “permute” it to find what random clusters looks like. In order to generate random clusters, we mix up the class labels. Read the code in the cell below carefully to find out how this is done:

# Throw the data onto one big pile
big_pile = np.vstack((data_pixels, data_facenet))

# Shuffle the rows. This is what a "random permutation" means.
np.random.shuffle(big_pile)

# Divide again into the "pixels" and "facenet" conditions
permuted_pixels, permuted_facenet = big_pile[: len(subjects)], big_pile[len(subjects) :]

# Compute t-values, which should be nice and low
permuted_tvals = ttest_1samp_no_p(permuted_pixels - permuted_facenet)

# See what the clusters look like now
permuted_cluster_tvals_sum = plot_cluster(
    permuted_tvals, threshold=2, data_cond1=permuted_pixels, data_cond2=permuted_facenet
)
print("Sum of t-values in cluster:", permuted_cluster_tvals_sum.round(3))

Sum of t-values in cluster: 0.0

You can run the cell above multiple times to see what different permutations look like. If it is very rare that we get a cluster with larger max t-value than the one we found in the original data, we can say that there is a significant difference in the original data. Finally, the percentage of times you find a larger max t-value in the randomly permuted data is your p-value.

And that is the cluster permutation test!

So, with all of this in mind, we can now use the MNE-Python mne.stats.permutation_cluster_1samp_test() to perform everything we have done above in a single line of code. When studying the line of code, keep the following in mind:

Read the documentation of mne.stats.permutation_cluster_1samp_test() first!
The first parameter, X is your data matrix (n_participants x n_times). Since we are using paired t-tests to define clusters, you need to set this to data_pixels - data_facenet like we did before
The threshold to form clusters is 2 like we used before - We set tail=1 to only look for positive t-values to mimic what we did before. You can also experiment with -1 and 0 to look for clusters with negative t-values or both positive and negative.

from mne.stats import permutation_cluster_1samp_test

t_obs, clusters, pvals, H0 = permutation_cluster_1samp_test(
    X=data_pixels - data_facenet, threshold=2, tail=1
)

Executing the cell below will explain what all the return values mean:

times = rsa_pixels[0].times
print(
    f"t_obs (an array of length {len(t_obs)}) contains the initial t-values that are "
    "used to create the clusters."
)
print(
    f"clusters (a list of length {len(clusters)}) contains for each cluster, the "
    "indices in the data array that are part of the cluster."
)
print(
    f"pvals (an array of length {len(pvals)}) contains for each cluster a p-value, "
    "more about this later."
)
print(
    f"H0 (an array of lenth {len(H0)}) contains the largest "
    "`permuted_cluster_tvals_sum` found in each random permutation."
)
print()
print(f"The cluster permutation test found {len(clusters)} clusters.")
for i, cluster in enumerate(clusters):
    print(
        f"Cluster {i}: {times[cluster[0].min()]:.3f}-{times[cluster[0].max()]:.3f} "
        "seconds."
    )

t_obs (an array of length 199) contains the initial t-values that are used to create the clusters.
clusters (a list of length 1) contains for each cluster, the indices in the data array that are part of the cluster.
pvals (an array of length 1) contains for each cluster a p-value, more about this later.
H0 (an array of lenth 1024) contains the largest `permuted_cluster_tvals_sum` found in each random permutation.

The cluster permutation test found 1 clusters.
Cluster 0: 0.000-0.164 seconds.

Computing the p-value#

Depending on the value of tail you used in your call to mne.stats.permutation_cluster_1samp_test() you have either 1, 2, or 3 clusters. The clusters that your test found each have an associated p-value.

If the sum-of-t-values for the cluster was positive, then the p-value is the fraction of random permutations that produced a cluster which a sum-of-t-values that was equal or greater
If the sum-of-t-values for the cluster was negative, then the p-value is the fraction of random permutations that produced a cluster which a sum-of-t-values that was equal or smaller

To understand this better, lets use the t_obs, clusters and H0 return values that you produced above to compute this p-value ourselves to test our knowledge of what these return values mean. We will:

Loop over all the clusters you found
Determine the sum_of_t_values inside the cluster
Compute the fraction of H0 whose absolute value is equal or larger than the absolute sum_of_t_values

print(f"The cluster p-values should be:\t{pvals}")
our_pvals = [(H0 >= t_obs[cluster].sum()).mean() for cluster in clusters]
print(f"Our manually computed cluster p-values are:\t{our_pvals}")

The cluster p-values should be: [0.03613281]
Our manually computed cluster p-values are:     [np.float64(0.0361328125)]

Interpreting the p-value#

Now comes the twist that I mentioned in the beginning. The final part of the cluster permutation test is that you use the smallest cluster p-value as final outcome of the test. You might be tempted to judge the significance of each cluster separately based on its associated p-value, but that would be statistically incorrect.

So what can you do? Sassenhagen & Draschkow 2019 offer some good advice. Personally, I would go for something like:

The cluster-based permutation test revealed a significant difference between the conditions (p=SMALLEST_P_VALUE). This result was mostly driven by clusters at X (CLUSTER_PVAL) and Y (CLUSTER_PVAL), which correspond roughly to DESCRIPTION_OF_ROIS”.

Clusters across space and time#

The formation of clusters becomes a bit more tricky in the case of mne.SourceEstimate objects, where data is defines across vertices in space as well as samples in time.

In this case, the first order of business is to morph everything to a template brain, so that the vertices are aligned. This is common practice in MEG group analysis, so I’m not going to cover that process in this tutorial.

Instead, I have run the spatio-temporal RSA analysis that we did Tutorial part 2: RSA on sourse-level MEG data across multiple subjects and placed the results in the group_analysis folder of your data package. Let’s load them and see if we can run a spatio-temporal cluster-based permutation test (quite a mouthful).

from mne import read_source_estimate

stc_pixels = [
    read_source_estimate(
        f"{data_path}/group_analysis/sub-{subject:02d}-rsa-pixels-morphed"
    )
    for subject in subjects
]
stc_facenet = [
    read_source_estimate(
        f"{data_path}/group_analysis/sub-{subject:02d}-rsa-facenet-morphed"
    )
    for subject in subjects
]

Now, stc_pixels and stc_facenet are two lists containing the mne.SourceEstimate’s for all subjects. Here are 5 of them:

stc_pixels[:5]

[<SourceEstimate | 20484 vertices, tmin : 4.545454502105713 (ms), tmax : 495.45454072952265 (ms), tstep : 4.545454502105713 (ms), data shape : (20484, 109), ~8.7 MiB>, <SourceEstimate | 20484 vertices, tmin : 4.545454502105713 (ms), tmax : 495.45454072952265 (ms), tstep : 4.545454502105713 (ms), data shape : (20484, 109), ~8.7 MiB>, <SourceEstimate | 20484 vertices, tmin : 4.545454502105713 (ms), tmax : 495.45454072952265 (ms), tstep : 4.545454502105713 (ms), data shape : (20484, 109), ~8.7 MiB>, <SourceEstimate | 20484 vertices, tmin : 4.545454502105713 (ms), tmax : 495.45454072952265 (ms), tstep : 4.545454502105713 (ms), data shape : (20484, 109), ~8.7 MiB>, <SourceEstimate | 20484 vertices, tmin : 4.545454502105713 (ms), tmax : 495.45454072952265 (ms), tstep : 4.545454502105713 (ms), data shape : (20484, 109), ~8.7 MiB>]

Let’s proceed as before by first looking at the difference between the means of the two conditions. A neat feature of the mne.SourceEstimate class is that it supports common math operations such as +, -, * and /. For example, computing the difference between the pixels and facenet RSA results of the first subject can simply be computed as stc_diff = stc_pixels[0] - stc_facenet[0]. In the cell below, we compute the difference between the two conditions in a pairwise manner for each subject, then compute the mean across the pairwise differences. If you want a nice elegant way to write this, remember that the `zip() <https://docs.python.org/3/library/functions.html#zip>`__ function exists.

stc_pairwise_diff = np.mean(
    [stc1 - stc2 for stc1, stc2 in zip(stc_pixels, stc_facenet)]
)

# Visualize the result.
stc_pairwise_diff.plot(
    "fsaverage",
    subjects_dir=f"{data_path}/freesurfer",
    hemi="both",
    initial_time=0.121,
    views="caudal",
)

False

<mne.viz._brain._brain.Brain object at 0x7bbcf9edd1c0>

The figure shows us clear differences between the pixel versus FaceNet RSA scores across the visual cortex.

Let’s explore this difference with a t-value map. In order to compute t-values, we need to pull the data out of the mne.SourceEstimate objects. As before, since we have a within-subject contrast between the two conditions, we want to perform pairwise tests. So, as a first step, create a data array X of shape (n_participants x n_vertices x n_times) where each slice i along the first dimension is the difference between stc_pixels[i].data - stc_facenet[i].data. The next step is running X through the mne.stats.ttest_1samp_no_p() function to obtain t-values.

from mne.stats import ttest_1samp_no_p

X = np.array([stc1.data - stc2.data for stc1, stc2 in zip(stc_pixels, stc_facenet)])
tvals = ttest_1samp_no_p(X)

# Some sanity checks for our result
assert X.shape == (16, 20484, 109), "Your `X` array has the wrong shape. 🤔"
assert -0.17 < X.min() < -0.16, "Your `X` array does not have the correct values. 🤔"
assert 0.29 < X.max() < 0.30, "Your `X` array does not have the correct values. 🤔"

assert tvals.shape == (20484, 109), "Your `tvals` array has the wrong shape. 🤔"
assert -6.6 < tvals.min() < -6.5, (
    "Your `tvals` array does not have the correct values. 🤔"
)
assert 7.4 < tvals.max() < 7.5, (
    "Your `tvals` array does not have the correct values. 🤔"
)

print("All good! 😊")

All good! 😊

Now we can pack the tvals into a new mne.SourceEstimate object in order to visualize them:

stc_tvals = mne.SourceEstimate(
    data=tvals,
    vertices=stc_pixels[0].vertices,
    tmin=stc_pixels[0].tmin,
    tstep=stc_pixels[0].tstep,
    subject="fsaverage",
)
stc_tvals.plot(
    "fsaverage",
    subjects_dir=f"{data_path}/freesurfer",
    hemi="both",
    initial_time=0.121,
    views="caudal",
)

False

<mne.viz._brain._brain.Brain object at 0x7bbcf0d48530>

From this figure, it seems that setting a threshold of 4 may result in useful clusters for our cluster-based permutation test. The function to perform the test in both space and time is called mne.stats.spatio_temporal_cluster_1samp_test()

But wait! Before you go and run the test, there’s two things we need to take care of.

The other thing to keep in mind is that in order to form clusters across the cortex, we need to make sure that only neighbouring vertices are part of the same cluster. This is a similar problem as we faced when computing searchlight patches in space during our RSA analysis. The solution is found within the mne.SourceSpaces object that keeps track of the distances between source points. Since all of our data has been morphed to the fsaverage template brain, we need to load the mne.SourceSpaces object for that brain. Then, we can compute the spatial adjacency matrix from the mne.SourceSpaces object through the mne.spatial_src_adjacency() function.

src = mne.read_source_spaces(
    f"{data_path}/freesurfer/fsaverage/bem/fsaverage-ico-5-src.fif"
)
adj = mne.spatial_src_adjacency(src)

assert adj.shape == (20484, 20484), "You've `adj` seems to have the wrong shape. 🤔"
print("All good! 😊")

All good! 😊

Now it’s time to run the big spatio-temporal cluster-based permutation test! Read its documentation first (:func:`mne.stats.spatio_temporal_cluster_1samp_test) and then go for it!

from mne.stats import spatio_temporal_cluster_1samp_test

t_obs, clusters, pvals, H0 = spatio_temporal_cluster_1samp_test(
    X.transpose(0, 2, 1), threshold=4, tail=1, adjacency=adj, n_jobs=-1, verbose=True
)

# What is the smallest cluster p-value that we found? Was it significant? (p < 0.05?)
print("Smallest p-value:", pvals.min())
if pvals.min() < 0.05:
    print("It was significant! 🎉")
else:
    print("It was not significant 🤔")

stat_fun(H1): min=-6.571207046508789 max=7.4570088386535645
Running initial clustering …
Found 99 clusters

  0%|          | Permuting : 0/1023 [00:00<?,       ?it/s]
  0%|          | Permuting : 1/1023 [00:02<40:24,    2.37s/it]
  0%|          | Permuting : 2/1023 [00:02<21:02,    1.24s/it]
  0%|          | Permuting : 3/1023 [00:02<14:19,    1.19it/s]
  0%|          | Permuting : 4/1023 [00:02<10:55,    1.56it/s]
  0%|          | Permuting : 5/1023 [00:02<08:37,    1.97it/s]
  1%|          | Permuting : 6/1023 [00:02<07:14,    2.34it/s]
  1%|          | Permuting : 7/1023 [00:02<06:10,    2.74it/s]
  1%|          | Permuting : 8/1023 [00:02<05:26,    3.10it/s]
  1%|          | Permuting : 9/1023 [00:03<04:47,    3.52it/s]
  1%|          | Permuting : 10/1023 [00:03<04:20,    3.89it/s]
  1%|          | Permuting : 11/1023 [00:03<03:54,    4.32it/s]
  1%|          | Permuting : 12/1023 [00:03<03:32,    4.77it/s]
  1%|▏         | Permuting : 14/1023 [00:03<03:00,    5.60it/s]
  1%|▏         | Permuting : 15/1023 [00:03<02:48,    5.98it/s]
  2%|▏         | Permuting : 16/1023 [00:03<02:37,    6.40it/s]
  2%|▏         | Permuting : 18/1023 [00:03<02:15,    7.40it/s]
  2%|▏         | Permuting : 19/1023 [00:03<02:11,    7.66it/s]
  2%|▏         | Permuting : 20/1023 [00:03<02:03,    8.10it/s]
  2%|▏         | Permuting : 22/1023 [00:03<01:49,    9.15it/s]
  2%|▏         | Permuting : 23/1023 [00:03<01:44,    9.58it/s]
  2%|▏         | Permuting : 25/1023 [00:03<01:35,   10.49it/s]
  3%|▎         | Permuting : 27/1023 [00:03<01:27,   11.38it/s]
  3%|▎         | Permuting : 30/1023 [00:03<01:17,   12.76it/s]
  3%|▎         | Permuting : 31/1023 [00:03<01:15,   13.09it/s]
  3%|▎         | Permuting : 32/1023 [00:03<01:14,   13.38it/s]
  3%|▎         | Permuting : 33/1023 [00:03<01:11,   13.76it/s]
  3%|▎         | Permuting : 35/1023 [00:03<01:06,   14.85it/s]
  4%|▎         | Permuting : 36/1023 [00:03<01:04,   15.22it/s]
  4%|▍         | Permuting : 40/1023 [00:04<00:57,   17.04it/s]
  4%|▍         | Permuting : 42/1023 [00:04<00:54,   17.84it/s]
  4%|▍         | Permuting : 44/1023 [00:04<00:52,   18.81it/s]
  4%|▍         | Permuting : 46/1023 [00:04<00:49,   19.80it/s]
  5%|▍         | Permuting : 47/1023 [00:04<00:48,   19.98it/s]
  5%|▍         | Permuting : 49/1023 [00:04<00:46,   20.98it/s]
  5%|▍         | Permuting : 51/1023 [00:04<00:44,   21.84it/s]
  5%|▌         | Permuting : 52/1023 [00:04<00:44,   22.04it/s]
  5%|▌         | Permuting : 55/1023 [00:04<00:40,   23.93it/s]
  6%|▌         | Permuting : 57/1023 [00:04<00:38,   24.96it/s]
  6%|▌         | Permuting : 58/1023 [00:04<00:38,   25.10it/s]
  6%|▌         | Permuting : 61/1023 [00:04<00:35,   27.03it/s]
  6%|▌         | Permuting : 63/1023 [00:04<00:35,   26.92it/s]
  7%|▋         | Permuting : 68/1023 [00:04<00:31,   29.91it/s]
  7%|▋         | Permuting : 69/1023 [00:04<00:32,   29.76it/s]
  7%|▋         | Permuting : 71/1023 [00:04<00:31,   30.58it/s]
  7%|▋         | Permuting : 74/1023 [00:04<00:29,   32.25it/s]
  7%|▋         | Permuting : 76/1023 [00:04<00:28,   33.00it/s]
  8%|▊         | Permuting : 78/1023 [00:04<00:27,   33.84it/s]
  8%|▊         | Permuting : 81/1023 [00:04<00:26,   35.65it/s]
  8%|▊         | Permuting : 82/1023 [00:04<00:26,   35.45it/s]
  8%|▊         | Permuting : 84/1023 [00:04<00:25,   36.26it/s]
  9%|▉         | Permuting : 90/1023 [00:04<00:24,   38.81it/s]
  9%|▉         | Permuting : 95/1023 [00:05<00:22,   41.32it/s]
  9%|▉         | Permuting : 96/1023 [00:05<00:22,   40.74it/s]
  9%|▉         | Permuting : 97/1023 [00:05<00:23,   40.03it/s]
 10%|▉         | Permuting : 102/1023 [00:05<00:21,   43.44it/s]
 10%|█         | Permuting : 104/1023 [00:05<00:21,   43.76it/s]
 10%|█         | Permuting : 107/1023 [00:05<00:20,   45.14it/s]
 11%|█         | Permuting : 110/1023 [00:05<00:19,   46.41it/s]
 11%|█         | Permuting : 111/1023 [00:05<00:19,   45.62it/s]
 11%|█         | Permuting : 114/1023 [00:05<00:19,   47.15it/s]
 11%|█▏        | Permuting : 116/1023 [00:05<00:19,   47.59it/s]
 12%|█▏        | Permuting : 119/1023 [00:05<00:18,   49.10it/s]
 12%|█▏        | Permuting : 121/1023 [00:05<00:18,   49.48it/s]
 12%|█▏        | Permuting : 125/1023 [00:05<00:17,   50.04it/s]
 13%|█▎        | Permuting : 130/1023 [00:05<00:17,   52.03it/s]
 13%|█▎        | Permuting : 131/1023 [00:05<00:17,   50.94it/s]
 13%|█▎        | Permuting : 135/1023 [00:05<00:16,   53.11it/s]
 13%|█▎        | Permuting : 137/1023 [00:05<00:16,   53.11it/s]
 14%|█▎        | Permuting : 140/1023 [00:05<00:16,   54.13it/s]
 14%|█▍        | Permuting : 144/1023 [00:05<00:15,   56.30it/s]
 14%|█▍        | Permuting : 146/1023 [00:05<00:15,   56.01it/s]
 14%|█▍        | Permuting : 148/1023 [00:05<00:15,   56.14it/s]
 15%|█▍        | Permuting : 151/1023 [00:05<00:15,   57.39it/s]
 15%|█▍        | Permuting : 153/1023 [00:05<00:15,   57.47it/s]
 15%|█▌        | Permuting : 156/1023 [00:05<00:14,   58.70it/s]
 16%|█▌        | Permuting : 161/1023 [00:05<00:14,   59.76it/s]
 16%|█▌        | Permuting : 163/1023 [00:06<00:14,   57.55it/s]
 16%|█▋        | Permuting : 167/1023 [00:06<00:14,   59.50it/s]
 16%|█▋        | Permuting : 168/1023 [00:06<00:14,   58.11it/s]
 17%|█▋        | Permuting : 172/1023 [00:06<00:14,   60.06it/s]
 17%|█▋        | Permuting : 174/1023 [00:06<00:14,   59.90it/s]
 17%|█▋        | Permuting : 177/1023 [00:06<00:13,   60.67it/s]
 17%|█▋        | Permuting : 179/1023 [00:06<00:14,   60.25it/s]
 18%|█▊        | Permuting : 180/1023 [00:06<00:14,   59.06it/s]
 18%|█▊        | Permuting : 184/1023 [00:06<00:13,   61.41it/s]
 18%|█▊        | Permuting : 186/1023 [00:06<00:13,   61.33it/s]
 18%|█▊        | Permuting : 187/1023 [00:06<00:13,   60.05it/s]
 19%|█▉        | Permuting : 193/1023 [00:06<00:13,   60.50it/s]
 19%|█▉        | Permuting : 197/1023 [00:06<00:13,   60.62it/s]
 20%|█▉        | Permuting : 201/1023 [00:06<00:13,   62.46it/s]
 20%|█▉        | Permuting : 203/1023 [00:06<00:13,   62.09it/s]
 20%|██        | Permuting : 205/1023 [00:06<00:13,   61.63it/s]
 20%|██        | Permuting : 207/1023 [00:06<00:13,   61.29it/s]
 21%|██        | Permuting : 210/1023 [00:06<00:13,   62.00it/s]
 21%|██        | Permuting : 211/1023 [00:06<00:13,   60.38it/s]
 21%|██        | Permuting : 214/1023 [00:06<00:13,   60.69it/s]
 21%|██        | Permuting : 217/1023 [00:06<00:13,   61.80it/s]
 21%|██▏       | Permuting : 218/1023 [00:06<00:13,   60.53it/s]
 22%|██▏       | Permuting : 220/1023 [00:06<00:13,   60.48it/s]
 22%|██▏       | Permuting : 222/1023 [00:06<00:13,   60.44it/s]
 22%|██▏       | Permuting : 227/1023 [00:07<00:13,   61.19it/s]
 23%|██▎       | Permuting : 232/1023 [00:07<00:12,   63.34it/s]
 23%|██▎       | Permuting : 233/1023 [00:07<00:12,   61.66it/s]
 23%|██▎       | Permuting : 234/1023 [00:07<00:13,   60.16it/s]
 23%|██▎       | Permuting : 238/1023 [00:07<00:12,   62.11it/s]
 24%|██▎       | Permuting : 242/1023 [00:07<00:12,   63.97it/s]
 24%|██▍       | Permuting : 245/1023 [00:07<00:12,   62.49it/s]
 24%|██▍       | Permuting : 248/1023 [00:07<00:12,   63.52it/s]
 24%|██▍       | Permuting : 249/1023 [00:07<00:12,   62.19it/s]
 25%|██▍       | Permuting : 251/1023 [00:07<00:12,   62.08it/s]
 25%|██▌       | Permuting : 259/1023 [00:07<00:12,   62.11it/s]
 25%|██▌       | Permuting : 260/1023 [00:07<00:12,   60.59it/s]
 26%|██▌       | Permuting : 262/1023 [00:07<00:12,   60.34it/s]
 26%|██▌       | Permuting : 264/1023 [00:07<00:12,   59.97it/s]
 26%|██▌       | Permuting : 266/1023 [00:07<00:12,   59.72it/s]
 26%|██▋       | Permuting : 270/1023 [00:07<00:12,   61.59it/s]
 26%|██▋       | Permuting : 271/1023 [00:07<00:12,   60.37it/s]
 27%|██▋       | Permuting : 272/1023 [00:07<00:12,   59.19it/s]
 27%|██▋       | Permuting : 274/1023 [00:07<00:12,   59.20it/s]
 27%|██▋       | Permuting : 276/1023 [00:07<00:12,   59.20it/s]
 27%|██▋       | Permuting : 281/1023 [00:07<00:12,   58.79it/s]
 28%|██▊       | Permuting : 284/1023 [00:08<00:12,   58.10it/s]
 28%|██▊       | Permuting : 286/1023 [00:08<00:12,   57.82it/s]
 28%|██▊       | Permuting : 288/1023 [00:08<00:12,   57.63it/s]
 29%|██▊       | Permuting : 292/1023 [00:08<00:12,   59.59it/s]
 29%|██▊       | Permuting : 294/1023 [00:08<00:12,   59.25it/s]
 29%|██▉       | Permuting : 295/1023 [00:08<00:12,   58.11it/s]
 29%|██▉       | Permuting : 296/1023 [00:08<00:12,   57.01it/s]
 29%|██▉       | Permuting : 298/1023 [00:08<00:12,   57.09it/s]
 29%|██▉       | Permuting : 301/1023 [00:08<00:12,   58.36it/s]
 30%|██▉       | Permuting : 305/1023 [00:08<00:12,   56.78it/s]
 30%|███       | Permuting : 309/1023 [00:08<00:12,   57.43it/s]
 30%|███       | Permuting : 310/1023 [00:08<00:12,   56.07it/s]
 31%|███       | Permuting : 313/1023 [00:08<00:12,   57.08it/s]
 31%|███       | Permuting : 316/1023 [00:08<00:12,   57.95it/s]
 31%|███       | Permuting : 317/1023 [00:08<00:12,   56.55it/s]
 31%|███▏      | Permuting : 320/1023 [00:08<00:12,   57.46it/s]
 31%|███▏      | Permuting : 322/1023 [00:08<00:12,   57.20it/s]
 32%|███▏      | Permuting : 323/1023 [00:08<00:12,   55.86it/s]
 32%|███▏      | Permuting : 328/1023 [00:08<00:11,   59.47it/s]
 32%|███▏      | Permuting : 329/1023 [00:08<00:11,   58.29it/s]
 32%|███▏      | Permuting : 330/1023 [00:08<00:12,   57.15it/s]
 33%|███▎      | Permuting : 337/1023 [00:08<00:11,   58.46it/s]
 33%|███▎      | Permuting : 342/1023 [00:09<00:11,   60.02it/s]
 34%|███▎      | Permuting : 344/1023 [00:09<00:11,   59.66it/s]
 34%|███▎      | Permuting : 345/1023 [00:09<00:11,   58.30it/s]
 34%|███▍      | Permuting : 349/1023 [00:09<00:11,   60.25it/s]
 34%|███▍      | Permuting : 352/1023 [00:09<00:11,   61.00it/s]
 35%|███▍      | Permuting : 357/1023 [00:09<00:10,   61.29it/s]
 35%|███▌      | Permuting : 361/1023 [00:09<00:10,   63.06it/s]
 35%|███▌      | Permuting : 363/1023 [00:09<00:10,   60.68it/s]
 36%|███▌      | Permuting : 364/1023 [00:09<00:11,   59.55it/s]
 36%|███▌      | Permuting : 367/1023 [00:09<00:10,   60.64it/s]
 36%|███▋      | Permuting : 372/1023 [00:09<00:10,   60.80it/s]
 37%|███▋      | Permuting : 376/1023 [00:09<00:10,   61.00it/s]
 37%|███▋      | Permuting : 379/1023 [00:09<00:10,   61.66it/s]
 37%|███▋      | Permuting : 382/1023 [00:09<00:10,   62.30it/s]
 37%|███▋      | Permuting : 383/1023 [00:09<00:10,   60.87it/s]
 38%|███▊      | Permuting : 388/1023 [00:09<00:09,   63.71it/s]
 38%|███▊      | Permuting : 390/1023 [00:09<00:09,   63.31it/s]
 38%|███▊      | Permuting : 391/1023 [00:09<00:10,   61.72it/s]
 39%|███▊      | Permuting : 395/1023 [00:09<00:09,   63.62it/s]
 39%|███▉      | Permuting : 397/1023 [00:09<00:09,   63.45it/s]
 39%|███▉      | Permuting : 400/1023 [00:09<00:09,   64.43it/s]
 39%|███▉      | Permuting : 402/1023 [00:09<00:09,   64.24it/s]
 40%|███▉      | Permuting : 405/1023 [00:09<00:09,   65.21it/s]
 40%|███▉      | Permuting : 407/1023 [00:10<00:09,   62.79it/s]
 40%|████      | Permuting : 411/1023 [00:10<00:09,   63.07it/s]
 41%|████      | Permuting : 415/1023 [00:10<00:09,   64.82it/s]
 41%|████      | Permuting : 416/1023 [00:10<00:09,   63.22it/s]
 41%|████      | Permuting : 420/1023 [00:10<00:09,   64.98it/s]
 41%|████▏     | Permuting : 422/1023 [00:10<00:09,   64.50it/s]
 41%|████▏     | Permuting : 423/1023 [00:10<00:09,   62.76it/s]
 42%|████▏     | Permuting : 427/1023 [00:10<00:09,   64.67it/s]
 42%|████▏     | Permuting : 430/1023 [00:10<00:09,   65.24it/s]
 42%|████▏     | Permuting : 432/1023 [00:10<00:09,   65.01it/s]
 43%|████▎     | Permuting : 435/1023 [00:10<00:08,   65.97it/s]
 43%|████▎     | Permuting : 438/1023 [00:10<00:09,   64.32it/s]
 43%|████▎     | Permuting : 444/1023 [00:10<00:08,   64.76it/s]
 44%|████▍     | Permuting : 448/1023 [00:10<00:08,   64.61it/s]
 44%|████▍     | Permuting : 451/1023 [00:10<00:08,   65.15it/s]
 44%|████▍     | Permuting : 454/1023 [00:10<00:08,   65.66it/s]
 45%|████▍     | Permuting : 456/1023 [00:10<00:08,   65.06it/s]
 45%|████▍     | Permuting : 460/1023 [00:10<00:08,   66.71it/s]
 45%|████▌     | Permuting : 462/1023 [00:10<00:08,   66.05it/s]
 45%|████▌     | Permuting : 465/1023 [00:10<00:08,   64.21it/s]
 46%|████▌     | Permuting : 468/1023 [00:10<00:08,   65.13it/s]
 46%|████▌     | Permuting : 470/1023 [00:10<00:08,   64.91it/s]
 46%|████▌     | Permuting : 473/1023 [00:11<00:08,   65.83it/s]
 47%|████▋     | Permuting : 476/1023 [00:11<00:08,   63.04it/s]
 47%|████▋     | Permuting : 483/1023 [00:11<00:08,   66.02it/s]
 47%|████▋     | Permuting : 484/1023 [00:11<00:08,   64.35it/s]
 48%|████▊     | Permuting : 489/1023 [00:11<00:07,   67.11it/s]
 48%|████▊     | Permuting : 491/1023 [00:11<00:08,   66.45it/s]
 48%|████▊     | Permuting : 493/1023 [00:11<00:08,   63.21it/s]
 49%|████▊     | Permuting : 498/1023 [00:11<00:07,   66.00it/s]
 49%|████▉     | Permuting : 501/1023 [00:11<00:08,   63.73it/s]
 49%|████▉     | Permuting : 505/1023 [00:11<00:07,   65.66it/s]
 49%|████▉     | Permuting : 506/1023 [00:11<00:08,   62.15it/s]
 50%|█████     | Permuting : 512/1023 [00:11<00:08,   63.54it/s]
 50%|█████     | Permuting : 514/1023 [00:11<00:08,   61.56it/s]
 51%|█████     | Permuting : 518/1023 [00:11<00:07,   63.47it/s]
 51%|█████     | Permuting : 521/1023 [00:11<00:08,   61.69it/s]
 51%|█████▏    | Permuting : 525/1023 [00:11<00:07,   63.28it/s]
 52%|█████▏    | Permuting : 527/1023 [00:11<00:08,   60.75it/s]
 52%|█████▏    | Permuting : 531/1023 [00:11<00:07,   62.35it/s]
 52%|█████▏    | Permuting : 533/1023 [00:12<00:08,   60.24it/s]
 52%|█████▏    | Permuting : 537/1023 [00:12<00:07,   62.18it/s]
 53%|█████▎    | Permuting : 538/1023 [00:12<00:07,   61.08it/s]
 53%|█████▎    | Permuting : 545/1023 [00:12<00:07,   62.70it/s]
 54%|█████▍    | Permuting : 551/1023 [00:12<00:07,   62.27it/s]
 54%|█████▍    | Permuting : 553/1023 [00:12<00:07,   61.96it/s]
 55%|█████▍    | Permuting : 559/1023 [00:12<00:07,   63.06it/s]
 55%|█████▍    | Permuting : 560/1023 [00:12<00:07,   61.78it/s]
 55%|█████▌    | Permuting : 567/1023 [00:12<00:07,   63.55it/s]
 56%|█████▌    | Permuting : 568/1023 [00:12<00:07,   62.50it/s]
 56%|█████▌    | Permuting : 574/1023 [00:12<00:07,   62.25it/s]
 57%|█████▋    | Permuting : 580/1023 [00:12<00:07,   61.48it/s]
 57%|█████▋    | Permuting : 581/1023 [00:12<00:07,   60.29it/s]
 57%|█████▋    | Permuting : 586/1023 [00:12<00:07,   60.66it/s]
 57%|█████▋    | Permuting : 587/1023 [00:12<00:07,   59.57it/s]
 57%|█████▋    | Permuting : 588/1023 [00:12<00:07,   58.43it/s]
 58%|█████▊    | Permuting : 593/1023 [00:12<00:07,   60.96it/s]
 58%|█████▊    | Permuting : 594/1023 [00:13<00:07,   60.00it/s]
 58%|█████▊    | Permuting : 596/1023 [00:13<00:07,   58.18it/s]
 59%|█████▊    | Permuting : 601/1023 [00:13<00:06,   60.96it/s]
 59%|█████▉    | Permuting : 608/1023 [00:13<00:06,   59.80it/s]
 60%|█████▉    | Permuting : 609/1023 [00:13<00:07,   58.65it/s]
 60%|█████▉    | Permuting : 613/1023 [00:13<00:06,   60.22it/s]
 60%|██████    | Permuting : 614/1023 [00:13<00:06,   59.01it/s]
 60%|██████    | Permuting : 615/1023 [00:13<00:07,   57.92it/s]
 61%|██████    | Permuting : 619/1023 [00:13<00:06,   59.55it/s]
 61%|██████    | Permuting : 620/1023 [00:13<00:06,   58.60it/s]
 61%|██████    | Permuting : 622/1023 [00:13<00:07,   56.79it/s]
 61%|██████    | Permuting : 626/1023 [00:13<00:06,   58.77it/s]
 62%|██████▏   | Permuting : 630/1023 [00:13<00:06,   57.39it/s]
 62%|██████▏   | Permuting : 633/1023 [00:13<00:06,   56.86it/s]
 62%|██████▏   | Permuting : 635/1023 [00:13<00:06,   56.67it/s]
 62%|██████▏   | Permuting : 639/1023 [00:13<00:06,   58.35it/s]
 63%|██████▎   | Permuting : 643/1023 [00:13<00:06,   57.87it/s]
 63%|██████▎   | Permuting : 647/1023 [00:13<00:06,   59.58it/s]
 63%|██████▎   | Permuting : 648/1023 [00:13<00:06,   58.34it/s]
 64%|██████▎   | Permuting : 651/1023 [00:14<00:06,   58.62it/s]
 64%|██████▍   | Permuting : 654/1023 [00:14<00:06,   59.61it/s]
 64%|██████▍   | Permuting : 655/1023 [00:14<00:06,   58.62it/s]
 64%|██████▍   | Permuting : 658/1023 [00:14<00:06,   59.64it/s]
 65%|██████▍   | Permuting : 662/1023 [00:14<00:05,   61.32it/s]
 65%|██████▍   | Permuting : 663/1023 [00:14<00:06,   58.33it/s]
 65%|██████▌   | Permuting : 669/1023 [00:14<00:05,   60.56it/s]
 66%|██████▌   | Permuting : 671/1023 [00:14<00:05,   60.20it/s]
 66%|██████▌   | Permuting : 675/1023 [00:14<00:05,   61.87it/s]
 66%|██████▋   | Permuting : 678/1023 [00:14<00:05,   62.58it/s]
 66%|██████▋   | Permuting : 679/1023 [00:14<00:05,   61.11it/s]
 67%|██████▋   | Permuting : 682/1023 [00:14<00:05,   61.76it/s]
 67%|██████▋   | Permuting : 685/1023 [00:14<00:05,   62.47it/s]
 67%|██████▋   | Permuting : 688/1023 [00:14<00:05,   62.63it/s]
 68%|██████▊   | Permuting : 691/1023 [00:14<00:05,   63.57it/s]
 68%|██████▊   | Permuting : 693/1023 [00:14<00:05,   63.42it/s]
 68%|██████▊   | Permuting : 694/1023 [00:14<00:05,   62.17it/s]
 68%|██████▊   | Permuting : 696/1023 [00:14<00:05,   62.07it/s]
 69%|██████▊   | Permuting : 701/1023 [00:14<00:05,   62.61it/s]
 69%|██████▉   | Permuting : 704/1023 [00:14<00:05,   61.35it/s]
 69%|██████▉   | Permuting : 707/1023 [00:14<00:05,   62.01it/s]
 70%|██████▉   | Permuting : 711/1023 [00:14<00:05,   61.26it/s]
 70%|██████▉   | Permuting : 714/1023 [00:15<00:04,   62.00it/s]
 70%|███████   | Permuting : 717/1023 [00:15<00:05,   60.81it/s]
 70%|███████   | Permuting : 719/1023 [00:15<00:05,   60.75it/s]
 70%|███████   | Permuting : 720/1023 [00:15<00:05,   59.64it/s]
 71%|███████   | Permuting : 727/1023 [00:15<00:04,   60.78it/s]
 71%|███████▏  | Permuting : 731/1023 [00:15<00:04,   60.46it/s]
 72%|███████▏  | Permuting : 733/1023 [00:15<00:04,   60.10it/s]
 72%|███████▏  | Permuting : 737/1023 [00:15<00:04,   59.55it/s]
 72%|███████▏  | Permuting : 740/1023 [00:15<00:04,   60.22it/s]
 72%|███████▏  | Permuting : 741/1023 [00:15<00:04,   59.00it/s]
 73%|███████▎  | Permuting : 746/1023 [00:15<00:04,   61.72it/s]
 73%|███████▎  | Permuting : 747/1023 [00:15<00:04,   60.42it/s]
 74%|███████▎  | Permuting : 752/1023 [00:15<00:04,   61.36it/s]
 74%|███████▎  | Permuting : 754/1023 [00:15<00:04,   61.29it/s]
 74%|███████▍  | Permuting : 755/1023 [00:15<00:04,   60.18it/s]
 74%|███████▍  | Permuting : 760/1023 [00:15<00:04,   61.29it/s]
 75%|███████▍  | Permuting : 763/1023 [00:15<00:04,   58.88it/s]
 75%|███████▍  | Permuting : 767/1023 [00:15<00:04,   60.63it/s]
 75%|███████▌  | Permuting : 768/1023 [00:15<00:04,   59.28it/s]
 75%|███████▌  | Permuting : 770/1023 [00:15<00:04,   59.25it/s]
 76%|███████▌  | Permuting : 773/1023 [00:16<00:04,   60.29it/s]
 76%|███████▌  | Permuting : 779/1023 [00:16<00:04,   60.23it/s]
 76%|███████▌  | Permuting : 780/1023 [00:16<00:04,   59.20it/s]
 77%|███████▋  | Permuting : 784/1023 [00:16<00:04,   59.22it/s]
 77%|███████▋  | Permuting : 785/1023 [00:16<00:04,   58.23it/s]
 77%|███████▋  | Permuting : 786/1023 [00:16<00:04,   57.25it/s]
 77%|███████▋  | Permuting : 789/1023 [00:16<00:04,   58.35it/s]
 77%|███████▋  | Permuting : 791/1023 [00:16<00:03,   58.38it/s]
 77%|███████▋  | Permuting : 792/1023 [00:16<00:04,   57.35it/s]
 78%|███████▊  | Permuting : 793/1023 [00:16<00:04,   56.35it/s]
 78%|███████▊  | Permuting : 797/1023 [00:16<00:03,   58.64it/s]
 78%|███████▊  | Permuting : 799/1023 [00:16<00:03,   58.67it/s]
 78%|███████▊  | Permuting : 801/1023 [00:16<00:03,   58.69it/s]
 78%|███████▊  | Permuting : 803/1023 [00:16<00:03,   58.72it/s]
 79%|███████▊  | Permuting : 805/1023 [00:16<00:03,   58.74it/s]
 79%|███████▉  | Permuting : 807/1023 [00:16<00:03,   58.76it/s]
 79%|███████▉  | Permuting : 808/1023 [00:16<00:03,   57.61it/s]
 79%|███████▉  | Permuting : 811/1023 [00:16<00:03,   58.86it/s]
 79%|███████▉  | Permuting : 812/1023 [00:16<00:03,   57.68it/s]
 80%|███████▉  | Permuting : 815/1023 [00:16<00:03,   58.96it/s]
 80%|███████▉  | Permuting : 817/1023 [00:16<00:03,   58.97it/s]
 80%|████████  | Permuting : 819/1023 [00:16<00:03,   58.98it/s]
 80%|████████  | Permuting : 820/1023 [00:16<00:03,   57.75it/s]
 80%|████████  | Permuting : 821/1023 [00:16<00:03,   56.57it/s]
 81%|████████  | Permuting : 825/1023 [00:16<00:03,   59.21it/s]
 81%|████████  | Permuting : 828/1023 [00:16<00:03,   60.48it/s]
 81%|████████  | Permuting : 829/1023 [00:16<00:03,   59.15it/s]
 81%|████████▏ | Permuting : 832/1023 [00:17<00:03,   60.45it/s]
 82%|████████▏ | Permuting : 835/1023 [00:17<00:03,   61.70it/s]
 82%|████████▏ | Permuting : 836/1023 [00:17<00:03,   60.29it/s]
 82%|████████▏ | Permuting : 839/1023 [00:17<00:02,   61.56it/s]
 82%|████████▏ | Permuting : 841/1023 [00:17<00:02,   61.46it/s]
 82%|████████▏ | Permuting : 843/1023 [00:17<00:02,   61.36it/s]
 83%|████████▎ | Permuting : 845/1023 [00:17<00:02,   61.27it/s]
 83%|████████▎ | Permuting : 848/1023 [00:17<00:02,   62.52it/s]
 83%|████████▎ | Permuting : 851/1023 [00:17<00:02,   63.72it/s]
 83%|████████▎ | Permuting : 852/1023 [00:17<00:02,   62.16it/s]
 84%|████████▎ | Permuting : 855/1023 [00:17<00:02,   63.39it/s]
 84%|████████▍ | Permuting : 858/1023 [00:17<00:02,   64.57it/s]
 84%|████████▍ | Permuting : 861/1023 [00:17<00:02,   65.68it/s]
 84%|████████▍ | Permuting : 863/1023 [00:17<00:02,   65.38it/s]
 85%|████████▍ | Permuting : 866/1023 [00:17<00:02,   66.47it/s]
 85%|████████▍ | Permuting : 869/1023 [00:17<00:02,   67.51it/s]
 85%|████████▌ | Permuting : 873/1023 [00:17<00:02,   66.77it/s]
 86%|████████▌ | Permuting : 875/1023 [00:17<00:02,   66.42it/s]
 86%|████████▌ | Permuting : 877/1023 [00:17<00:02,   66.08it/s]
 86%|████████▌ | Permuting : 880/1023 [00:17<00:02,   67.11it/s]
 86%|████████▌ | Permuting : 882/1023 [00:17<00:02,   66.74it/s]
 86%|████████▋ | Permuting : 883/1023 [00:17<00:02,   65.04it/s]
 87%|████████▋ | Permuting : 885/1023 [00:17<00:02,   64.76it/s]
 87%|████████▋ | Permuting : 889/1023 [00:17<00:01,   67.24it/s]
 87%|████████▋ | Permuting : 890/1023 [00:17<00:02,   65.47it/s]
 87%|████████▋ | Permuting : 895/1023 [00:17<00:01,   66.26it/s]
 88%|████████▊ | Permuting : 897/1023 [00:17<00:01,   65.94it/s]
 88%|████████▊ | Permuting : 902/1023 [00:18<00:01,   66.65it/s]
 88%|████████▊ | Permuting : 904/1023 [00:18<00:01,   63.58it/s]
 89%|████████▊ | Permuting : 907/1023 [00:18<00:01,   64.63it/s]
 89%|████████▉ | Permuting : 909/1023 [00:18<00:01,   64.40it/s]
 89%|████████▉ | Permuting : 910/1023 [00:18<00:01,   62.89it/s]
 89%|████████▉ | Permuting : 911/1023 [00:18<00:01,   61.47it/s]
 90%|████████▉ | Permuting : 916/1023 [00:18<00:01,   65.22it/s]
 90%|████████▉ | Permuting : 917/1023 [00:18<00:01,   63.67it/s]
 90%|█████████ | Permuting : 921/1023 [00:18<00:01,   63.32it/s]
 90%|█████████ | Permuting : 923/1023 [00:18<00:01,   63.14it/s]
 91%|█████████ | Permuting : 926/1023 [00:18<00:01,   61.63it/s]
 91%|█████████ | Permuting : 928/1023 [00:18<00:01,   61.53it/s]
 91%|█████████ | Permuting : 929/1023 [00:18<00:01,   60.21it/s]
 91%|█████████ | Permuting : 930/1023 [00:18<00:01,   58.94it/s]
 91%|█████████▏| Permuting : 934/1023 [00:18<00:01,   61.46it/s]
 91%|█████████▏| Permuting : 935/1023 [00:18<00:01,   57.68it/s]
 92%|█████████▏| Permuting : 937/1023 [00:18<00:01,   57.76it/s]
 92%|█████████▏| Permuting : 940/1023 [00:18<00:01,   59.06it/s]
 92%|█████████▏| Permuting : 943/1023 [00:18<00:01,   57.90it/s]
 92%|█████████▏| Permuting : 946/1023 [00:18<00:01,   59.17it/s]
 93%|█████████▎| Permuting : 948/1023 [00:18<00:01,   56.86it/s]
 93%|█████████▎| Permuting : 951/1023 [00:18<00:01,   58.15it/s]
 93%|█████████▎| Permuting : 952/1023 [00:18<00:01,   57.00it/s]
 93%|█████████▎| Permuting : 954/1023 [00:19<00:01,   54.89it/s]
 94%|█████████▎| Permuting : 957/1023 [00:19<00:01,   56.24it/s]
 94%|█████████▎| Permuting : 959/1023 [00:19<00:01,   54.21it/s]
 94%|█████████▍| Permuting : 961/1023 [00:19<00:01,   54.42it/s]
 94%|█████████▍| Permuting : 965/1023 [00:19<00:01,   52.83it/s]
 95%|█████████▍| Permuting : 967/1023 [00:19<00:01,   49.45it/s]
 95%|█████████▍| Permuting : 969/1023 [00:19<00:01,   49.79it/s]
 95%|█████████▌| Permuting : 972/1023 [00:19<00:01,   47.82it/s]
 95%|█████████▌| Permuting : 974/1023 [00:19<00:01,   45.18it/s]
 95%|█████████▌| Permuting : 975/1023 [00:19<00:01,   44.69it/s]
 95%|█████████▌| Permuting : 976/1023 [00:19<00:01,   42.80it/s]
 96%|█████████▌| Permuting : 978/1023 [00:19<00:01,   43.34it/s]
 96%|█████████▌| Permuting : 979/1023 [00:19<00:01,   40.26it/s]
 96%|█████████▌| Permuting : 980/1023 [00:19<00:01,   39.93it/s]
 96%|█████████▌| Permuting : 982/1023 [00:19<00:01,   38.12it/s]
 96%|█████████▌| Permuting : 983/1023 [00:20<00:01,   35.68it/s]
 96%|█████████▌| Permuting : 984/1023 [00:20<00:01,   35.51it/s]
 96%|█████████▋| Permuting : 985/1023 [00:20<00:01,   33.33it/s]
 96%|█████████▋| Permuting : 986/1023 [00:20<00:01,   33.23it/s]
 96%|█████████▋| Permuting : 987/1023 [00:20<00:01,   32.17it/s]
 97%|█████████▋| Permuting : 988/1023 [00:20<00:01,   31.18it/s]
 97%|█████████▋| Permuting : 989/1023 [00:20<00:01,   29.43it/s]
 97%|█████████▋| Permuting : 990/1023 [00:20<00:01,   27.14it/s]
 97%|█████████▋| Permuting : 991/1023 [00:20<00:01,   25.20it/s]
 97%|█████████▋| Permuting : 992/1023 [00:20<00:01,   24.10it/s]
 97%|█████████▋| Permuting : 993/1023 [00:20<00:01,   22.56it/s]
 97%|█████████▋| Permuting : 994/1023 [00:21<00:01,   21.71it/s]
 97%|█████████▋| Permuting : 995/1023 [00:21<00:01,   20.47it/s]
 97%|█████████▋| Permuting : 996/1023 [00:21<00:01,   19.79it/s]
 97%|█████████▋| Permuting : 997/1023 [00:21<00:01,   18.78it/s]
 98%|█████████▊| Permuting : 998/1023 [00:21<00:01,   18.23it/s]
 98%|█████████▊| Permuting : 999/1023 [00:21<00:01,   17.73it/s]
 98%|█████████▊| Permuting : 1000/1023 [00:21<00:01,   16.93it/s]
 98%|█████████▊| Permuting : 1001/1023 [00:21<00:01,   16.52it/s]
 98%|█████████▊| Permuting : 1002/1023 [00:22<00:01,   15.84it/s]
 98%|█████████▊| Permuting : 1003/1023 [00:22<00:01,   15.22it/s]
 98%|█████████▊| Permuting : 1004/1023 [00:22<00:01,   14.66it/s]
 98%|█████████▊| Permuting : 1005/1023 [00:22<00:01,   14.41it/s]
 98%|█████████▊| Permuting : 1006/1023 [00:22<00:01,   13.92it/s]
 98%|█████████▊| Permuting : 1007/1023 [00:22<00:01,   13.48it/s]
 99%|█████████▊| Permuting : 1008/1023 [00:22<00:01,   13.29it/s]
 99%|█████████▊| Permuting : 1009/1023 [00:22<00:01,   13.12it/s]
 99%|█████████▊| Permuting : 1010/1023 [00:22<00:01,   12.74it/s]
 99%|█████████▉| Permuting : 1011/1023 [00:23<00:00,   12.60it/s]
 99%|█████████▉| Permuting : 1012/1023 [00:23<00:00,   12.26it/s]
 99%|█████████▉| Permuting : 1013/1023 [00:23<00:00,   12.14it/s]
 99%|█████████▉| Permuting : 1014/1023 [00:23<00:00,   12.04it/s]
 99%|█████████▉| Permuting : 1015/1023 [00:23<00:00,   11.93it/s]
 99%|█████████▉| Permuting : 1016/1023 [00:23<00:00,   11.64it/s]
 99%|█████████▉| Permuting : 1017/1023 [00:23<00:00,   11.56it/s]
100%|█████████▉| Permuting : 1018/1023 [00:23<00:00,   11.29it/s]
100%|█████████▉| Permuting : 1019/1023 [00:24<00:00,   11.23it/s]
100%|█████████▉| Permuting : 1020/1023 [00:24<00:00,   10.99it/s]
100%|█████████▉| Permuting : 1021/1023 [00:24<00:00,   10.94it/s]
100%|█████████▉| Permuting : 1022/1023 [00:24<00:00,   10.72it/s]
100%|██████████| Permuting : 1023/1023 [00:24<00:00,   10.74it/s]
100%|██████████| Permuting : 1023/1023 [00:24<00:00,   41.82it/s]
Smallest p-value: 0.0009765625
It was significant! 🎉

MNE-Python comes with a function called mne.stats.summarize_clusters_stc() that shows the spatial extent of the clusters:

from mne.stats import summarize_clusters_stc

stc_all_cluster_vis = summarize_clusters_stc(
    (t_obs, clusters, pvals, H0),  # output of spatio_temporal_cluster_1samp_test
    tstep=stc_pixels[0].tstep,  # difference in time between two time samples
    vertices=stc_pixels[0].vertices,  # vertex numbers
    subject="fsaverage",  # FreeSurfer subject whose brain to draw
    p_thresh=0.05,  # Only show clusters with an associated p-value below this threshold
)

# Plot the summary created by summarize_clusters_stc
stc_all_cluster_vis.plot(
    hemi="both",
    views="ventral",
    subjects_dir=f"{data_path}/freesurfer",
    time_label="temporal extent (ms)",
    clim=dict(kind="value", pos_lims=[0, 0.01, 0.11]),
)

False

<mne.viz._brain._brain.Brain object at 0x7bbcf9f07bc0>

In the figure above, the first time point shows all the clusters and the subsequent timepoints show each cluster individually. The colors indicate the extent of the cluster in time (=number of samples).

🎉 CONGRATULATIONS 🎉 You’ve reached the end of this tutorial series and mastered the spatio-temporal cluster-based permutation test!

Total running time of the script: (0 minutes 28.255 seconds)

Gallery generated by Sphinx-Gallery