_prepare_random_label breaks when assessing spatial indexes

[renanxcortes]

_prepare_random_label seems to be breaking when assessing spatial indexes because here:

segregation/segregation/inference/comparative.py

Line 343 in 3d0ab2f

data_1.columns = ["group", "total", "grouping_variable"]

And here:

segregation/segregation/inference/comparative.py

Line 357 in 3d0ab2f

data_2.columns = ["group", "total", "grouping_variable"]

It is not taking into consideration the geometry column. I'll try to add here later on a minimal reproducible example.

[knaaptime]

ok so it took me awhile to remember why we did it this way, but the short version is: the permutation doesnt work if we ship the geometries along.

Awhile ago i refactored the library to orient toward two kinds of segregation indices: 'spatially-explicit' and 'spatially-implicit'. The former includes space as part of its original formula (i.e. "relative concentration" or "distance decay isolation"). The latter uses the Reardon and Osullivan logic to say that any segregation index is a spatial index if you transform the data properly. The problem is the permutational inference we came up with for the computational social science paper doesn't really work for the 'spatially-explicit' indices

If your two dataframes (df1 and df2) are different places, then the permutations are composed of geometries from both places and the spatial relationships are functionally broken. That is, when you compute some spatial operation (distance, contiguity, centrality, or whatever) on this dataset, each respective place will account for "nearby" observations, but the observations from each place will never interact with each other because they're too far away. One 'synthetic place' is still made up of observations from two distinct spatial systems. In the extreme case imagine I'm comparing the U.S. and Brazil using county-level data. A synthetic realization has observations from both countries; the Brazil observations are so far away from the U.S. observations, they're ultimately irrelevant, so the index is based on an average the the within-country spatial relationships (which themselves have also become sparse enough to be weird)

If df1 and df2 represent the same place at different time periods, then the permutation potentially has two copies of the same neighborhood from different times. Are they neighbors with each other contemporaneously? Are other (spatially proximate) observations neighbors with both future and past copies of the focal neighborhood?

But we also have what we call "spatially-implicit" indices (following the Reardon and Osullivan logic), where you manipulate the data ahead of time (i.e. so it's "spatialized"), then compute a "normal" segregation index like dissimilarity or Information theory, whatever. In this case, the permutations already "have space" (because the input data is put through a kernel function), but we dont need to send the geometry along, and we don't "break" the spatial relationships during the permutation process.

that was the compromise i decided on at the time, and i still think that's probably the right logic. Definitely happy to chat if you disagree or want me to clarify any of this

[knaaptime]

Here's an example with new jersey and west virginia. If we compute something like DxIsolation on this synthetic realization, the distance between the two states is so large that the decay function wipes out the possibility of them influencing each other. And even the within-state distances are large enough (because of the sparsity) to make the index act a little funny

import geosnap as gsp
import pandas as pd
from libpysal.graph import Graph

d = gsp.DataStore()

wv = gsp.io.get_acs(d, state_fips='54', years=2021)
wv['state'] = 'WV'
nj = gsp.io.get_acs(d, state_fips='34', years=2021)
nj['state'] = 'NJ'

df = pd.concat([wv, nj])

samp = df.sample(1500)
samp = samp.to_crs(5070)
samp = samp.reset_index()

samp_graph = Graph.build_distance_band(samp.set_geometry(samp.centroid), threshold=3000,)

ax= samp.explore('state', tiles='cartodb positron')
samp_graph.explore(samp, m=ax, nodes=False)

To avoid this, i still think we dont ship the geometries for spatially-explicit indices. And we should probably go ahead and raise when a user passes a SpatialExplicit class to the inference methods

[renanxcortes]

Hey! Yeah, it makes sense what you're saying, but originally the implementation was a little bit different. After I've read the first reply I took a look closer in the paper of "Interregional Inequality Dynamics in Mexico" (2010) of Sergio J. Rey & Myrna L. Sastré-Gutiérrez to make a double-check of their approach and how it was implemented originally and the topics I'm currently trying to wrap my head aroud are:

• Originally, it seems like Rey & Myrna (2010) developed some kind of "pairwised" random-label approach where value could only be swapped between its corresponding geometry in a harmonized map between two periods of time. (2^n possible iterations) right @sjsrey ?

• In segregation, what was developed was a "generalized" version of this is a way that the user can "random label" any data point between the maps. However (this is important) the segregation measure for each iteration is calculated on its original maps. That is, the polygons are not swap, but only the data points are swapped between the maps. For example, in your previous example, any iteration would be calculated in each state (NJ and WV) in each original map (and therefore, not having the within-state, or within-country, distance problem) but each map can present data from its original state OR data from the other state. At least, this original implementation was the one that we used to compare LA and NY in the Journal of Computation Social Sciences paper.

• The point here that I'm trying to understand better is that this original approach makes sense in light of Rey & Myrna (2010) since the random_label implementation of segregation is not a "pairwise" comparison, but rather monte carlo permutation of all values of both maps and assigning them to both maps (but keeping each spatial context for each).

• I got the difference of spatial explicit and spatial implicit approach of Reardon and Osullivan logic you mention and I think it is right approach, and your implementation is the best approach. However, my head is wrapping in a step even before this.

• Perhaps, a pairwise approach for the random_label (which seems like it is not implemented yet) could allow spatially explicit indexes since the polygons for each map are the same (but for this, the module has to make a check before continuing or raise an error as you mentioned).

[knaaptime]

Perhaps, a pairwise approach for the random_label (which seems like it is not implemented yet) could allow spatially explicit indexes since the polygons for each map are the same (but for this, the module has to make a check before continuing or raise an error as you mentioned).

ok, yeah that would be neat. The idea there would be to stack the two dataframes, then permute the values, then break back apart into their respective areas (with geometries) and proceed. That's straightforward; we should definitely add it

in the aspatial context shuffling values versus observations is irrelevant. But for spatial indices, we should shuffle only the values, leaving both original "locations" intact. The inference is then testing against a random allocation of values conditional on the original spatial supports

that's definitely what we should do

[knaaptime]

so right here, instead of permuting the label (randomizing group membership across the whole row) we should be permuting the data (randomizing the location of each value, leaving spatial relationships intact)

[sjsrey]

Hey! Yeah, it makes sense what you're saying, but originally the implementation was a little bit different. After I've read the first reply I took a look closer in the paper of "Interregional Inequality Dynamics in Mexico" (2010) of Sergio J. Rey & Myrna L. Sastré-Gutiérrez to make a double-check of their approach and how it was implemented originally and the topics I'm currently trying to wrap my head aroud are:

* Originally, it seems like Rey & Myrna (2010) developed some kind of "pairwised" random-label approach where value could only be swapped between its corresponding geometry in a harmonized map between two periods of time. (2^n possible iterations) right [@sjsrey](https://github.com/sjsrey) ?

In the 2010 paper we first calculate a spatial inequality statistic for the states using two different regional schemes.
Then we permute the values, each time recalculating the difference in the same spatial inequality statistic:

The results in Table 5 are based on marginal tests involving each partition scheme alone against its own distribution under the null of spatial randomness. We also explicitly compare the differences between each pair of inequality shares for a given year using the same random permutation strategy. Here we calculate the observed difference in the interregional inequality shares from the two sets of regional definitions, then we repeat that calculation for the random maps. We then compare the observed difference to the expected value of the differences obtained from the random permutations to develop a test of the differences.

So it isn't random labeling there, rather spatial permutations.

[sjsrey]

so right here, instead of permuting the label (randomizing group membership across the whole row) we should be permuting the data (randomizing the location of each value, leaving spatial relationships intact)

I'm trying to understand what the null and alternative hypotheses would be here.

[sjsrey]

Here's an example with new jersey and west virginia. If we compute something like DxIsolation on this synthetic realization, the distance between the two states is so large that the decay function wipes out the possibility of them influencing each other. And even the within-state distances are large enough (because of the sparsity) to make the index act a little funny

import geosnap as gsp
import pandas as pd
from libpysal.graph import Graph

d = gsp.DataStore()

wv = gsp.io.get_acs(d, state_fips='54', years=2021)
wv['state'] = 'WV'
nj = gsp.io.get_acs(d, state_fips='34', years=2021)
nj['state'] = 'NJ'

df = pd.concat([wv, nj])

samp = df.sample(1500)
samp = samp.to_crs(5070)
samp = samp.reset_index()

samp_graph = Graph.build_distance_band(samp.set_geometry(samp.centroid), threshold=3000,)

ax= samp.explore('state', tiles='cartodb positron')
samp_graph.explore(samp, m=ax, nodes=False)

But isn't the inference question here concerned with the difference in the DxIsolation indices between NJ and WV, rather than the index for the union of the two areas?

[knaaptime]

in the aspatial case, shuffling the label and shuffling the values are equivalent. In both cases we're just randomly changing which partition each value belongs to (so the null and alternative are unchanged)

in the spatial case, we can choose to permute the population values rather than the label, leaving the geometries "where they belong". This is still a random labelling exercise, just that when you're assigned a label, it also comes along with a neighboring structure. Since the values of the neighbors are also random, all this does is force the distances/contiguity between the original observations to be enforced. This is similar on concept to shuffling the values in a W/Graph... the links in the graph are unchanged, so the realization is random, conditional on the existing spatial structure.

in the end, i dont think its really all that different. Conceptually its 'what if you emptied the population of every neighborhood in nboth cities, then randomly partitioned the population into new neighborhoods, but allowed each neighborhood to remain where it is located in the city'

so i think the null is still interpreted the same... its still a random reallocation of population, just that the population needs to fit into an existing slot

But isn't the inference question here concerned with the difference in the DxIsolation indices between NJ and WV, rather than the index for the union of the two areas?

yes, so the example above is one synthetic draw. So one observation in the current framework is a difference between two of these draws. The point im trying to make in that figure is the premutations (in existing code) are always made up of observations from both places

[renanxcortes]

To visualize what I was referring imagine these two maps:

Now, for each iteration iterate the values of both maps at once (in a nut shell, put the data points (the pair: total population + minority population) in a urn and pull without replacement these data in a polygon of a map:

Calculate the segregation for each iteration and we were assuming that the null hypothesis was the absence of segregation between the maps.

[knaaptime]

@renanxcortes that image above is what's implemented in #261

the original implementation was shuffling the labels A and B resulting in new maps that are a random assortment of both sets of polygons

so the original implementation was "this neighborhood (including geometry) could belong in either city". The new implementation in #261 is "this population could belong in any neighborhood (where it already exists) among both cities"

[knaaptime]

i think i've confused myself. I think the original implementation did this, as intentioned

import geosnap as gsp
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

d = gsp.DataStore()

wv = gsp.io.get_acs(d, state_fips='54', years=2021)
wv['state'] = 'WV'
nj = gsp.io.get_acs(d, state_fips='34', years=2021)
nj['state'] = 'NJ'

df = pd.concat([wv, nj])

rand_label = df.copy()
rand_label['state'] = np.random.permutation(rand_label.state.values)

rand_label.explore('state')

if we're just shuffling labels, the geoms stay where they should be. So yes, a synthetic realization has geometries from both places, but the places can still influence each other