Using cross-validation when computing RDMs

This example demonstrates how to perform cross-validation when computing dissimilarity matrices (RDMs). When the data has repeated measurements of the same stimulus type, cross-validation can be used to provide much more robust distance estimates between stimulus types. Repeated measurements can for example be actual repetitions of the same stimulus within the same recording, or recordings on multiple volunteers with the same stimuli.

The dataset will be the kiloword dataset 1: approximately 1,000 words were presented to 75 participants in a go/no-go lexical decision task while event-related potentials (ERPs) were recorded.

This dataset as provided does not have repeated measurements of the same stimuli. To illustrate cross-validation, we will treat words with the same number of letters as being repeated measurements of the same stimulus type.

1

Dufau, S., Grainger, J., Midgley, KJ., Holcomb, PJ (2015). A thousand words are worth a picture: Snapshots of printed-word processing in an event-related potential megastudy. Psychological science.

# Import required packages
import mne
import mne_rsa

MNE-Python contains a build-in data loader for the kiloword dataset. We use it here to read it as 960 epochs. Each epoch represents the brain response to a single word, averaged across all the participants. For this example, we speed up the computation, at a cost of temporal precision, by downsampling the data from the original 250 Hz. to 100 Hz.

data_path = mne.datasets.kiloword.data_path(verbose=True)
epochs = mne.read_epochs(data_path / "kword_metadata-epo.fif")
epochs = epochs.resample(100)

Reading C:\Users\wmvan\mne_data\MNE-kiloword-data\kword_metadata-epo.fif ...
Isotrak not found
    Found the data of interest:
        t =    -100.00 ...     920.00 ms
        0 CTF compensation matrices available
Adding metadata with 8 columns
960 matching events found
No baseline correction applied
0 projection items activated

The epochs object contains a .metadata field that contains information about the 960 words that were used in the experiment. Let’s have a look at the metadata for the 10 random words:

epochs.metadata.sample(10)

	WORD	Concreteness	WordFrequency	OrthographicDistance	NumberOfLetters	BigramFrequency	ConsonantVowelProportion	VisualComplexity
278	bran	5.722222	1.662758	1.25	4.0	827.500000	0.750000	66.373901
789	chest	6.000000	2.891537	1.75	5.0	568.000000	0.800000	66.065358
95	snout	5.900000	1.672098	1.70	5.0	202.600000	0.600000	62.797616
750	cult	4.500000	2.209515	1.70	4.0	294.500000	0.750000	53.059937
461	barbecue	5.750000	1.431364	3.35	8.0	521.750000	0.500000	71.607727
809	broil	4.700000	0.301030	1.95	5.0	571.800000	0.600000	55.141335
839	minute	3.950000	3.121231	2.00	6.0	585.000000	0.500000	65.119208
708	pacifist	4.200000	1.653213	3.05	8.0	504.375000	0.625000	59.056773
878	economy	4.300000	3.148911	2.90	7.0	161.285714	0.428571	72.410834
881	brother	4.750000	3.185259	1.90	7.0	924.857143	0.714286	62.340593

The kiloword dataset as provided does not have repeated measurements of the same stimuli. To illustrate cross-validation, we will treat words with the same number of letters as being repeated measurements of the same stimulus type.

To denote which epochs are repetitions of the same stimulus, we create a list y that contains integer labels for each epoch. Repetitions of the same stimulus have the same label in the y list. This scheme is lifted from the machine learning literature (and the Scikit-Learn API). In this example, we use the number of letters in the stimulus words as the labels for the epochs.

y = epochs.metadata.NumberOfLetters.astype(int)

Many high-level functions in the MNE-RSA module can take the y list as a parameter to enable cross-validation. Notably the functions for performing RSA and computing RDMs. In this example, we will restrict the analysis to computing RDMs using a spatio-temporal searchlight on the sensor-level data.

rdms = mne_rsa.rdm_epochs(
    epochs,  # The EEG data
    y=y,  # Set labels to enable cross validation
    n_folds=5,  # Number of folds to use during cross validation
    dist_metric="sqeuclidean",  # Distance metric to compute the RDMs
    spatial_radius=0.45,  # Spatial radius of the searchlight patch in meters.
    temporal_radius=0.05,  # Temporal radius of the searchlight path in seconds.
    tmin=0.15,
    tmax=0.25,
)  # To save time, only analyze this time interval

Plotting the cross-validated RDMs

mne_rsa.plot_rdms_topo(rdms, epochs.info)

Creating spatio-temporal searchlight patches

<Figure size 640x480 with 29 Axes>

For performance reasons, the low-level functions of MNE-RSA do not take a y list for cross-validation. Instead, they require the data to be already split into folds. The create_folds() function can create these folds.

X = epochs.get_data()
y = epochs.metadata.NumberOfLetters.astype(int)
folds = mne_rsa.create_folds(X, y, n_folds=5)

rdm = mne_rsa.compute_rdm_cv(folds, metric="euclidean")
mne_rsa.plot_rdms(rdm)

C:\Users\wmvan\projects\mne-rsa\examples\plot_cross_validation.py:98: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
  X = epochs.get_data()

<Figure size 200x200 with 2 Axes>