# Defining your own data objects¶

## Background¶

By default, data objects in glue are instances of the
`Data`

class, and this class assumes that the data are
stored in one or more local n-dimensional Numpy arrays. However, glue now
includes a way of defining a wider variety of data objects, which may rely for
example on large remote datasets, or datasets that are not inherently stored as
regular n-dimensional arrays.

The base class for all datasets is `BaseData`

, which is
intended to represent completely arbitrary datasets. However, glue does not yet
recognize classes that inherit directly from `BaseData`

.
Instead, for now, the base class that can be used to define custom data objects
is `BaseCartesianData`

, which inherits from
`BaseData`

and requires data objects to present an
interface that looks like n-dimensional arrays (although the storage of the data
could still be unstructured). In future, we will also make it possible to
support a more generic interface for data access based on the
`BaseData`

class.

## Main data interface¶

Before we dive in, we recommend that you take a look at the Working with Data objects
tutorial to understand how the default `Data`

objects
work. The most important takeaway from this which is relevant here is that glue
data objects are collections of attributes (*components* in glue-speak) that are
assumed to be aligned (for regular cartesian datasets, this means they are on
the same grid). For example a table consists of a collection of 1-d attributes
that are all the same length. A traditional image consists of a single 2-d
attribute. Attributes/components are identified in data objects by
`ComponentID`

objects (though these objects don’t
contain the actual values – they are just a reference to that attribute).

To define your own data object, you should write a class that inherits from
`BaseCartesianData`

. You will then need to define a few
properties and methods for the data object to be usable in glue. The properties
you need to define are:

`label`

: the name of the dataset, as a string`shape`

: the shape of a single component, given as a tuple`main_components`

: a list of all`ComponentID`

that your data object recognizes, excluding coordinate components (more on this later).

The methods you need to define are:

`get_kind()`

: given a`ComponentID`

, return a string that should be either`'numerical'`

(for e.g. floating-point and integer attributes),`'categorical'`

(for e.g. string attributes), or`'datetime'`

(for attributes that use the`np.datetime64`

type).`get_data()`

: given a`ComponentID`

and optionally a*view*, return a Numpy array. A view can be anything that can be used to slice a Numpy array. For example a single integer (`view=4`

), a tuple of slices (`view=[slice(1, 14, 2), slice(4, 50, 3)]`

), a list of tuples of indices (`view=[(1, 2, 3), (4, 3, 4)]`

), and so on. If a view is specified, only that subset of values should be returned. For example if the data has an overall shape of`(10,)`

and`view=slice(1, 6, 2)`

,`get_data`

should return an array with shape`(3,)`

.`get_mask()`

: given a`SubsetState`

object (described in Subset states) and optionally a`view`

, return a boolean array describing which values are in the specified subset (where True indicates values inside the subset).`compute_statistic()`

: given a statistic name (e.g.`'mean'`

) and a`ComponentID`

, as well as optional keyword arguments (see`compute_statistic()`

), return a statistic for the required component. In particular one of the keyword arguments is`subset_state`

, which can be used to indicate that the statistic should only be computed for a subset of values.`compute_histogram()`

: given a list of`ComponentID`

objects, as well as optional keyword arguments (see`compute_histogram()`

), compute an n-dimensional histogram of the required attributes. At the moment glue only makes use of this for one or two dimensions, though we may also use it for three dimensions in future.`compute_fixed_resolution_buffer()`

: given a list of bounds of the form`(min, max, n)`

along each dimension of the data, return a fixed resolution buffer that goes from the respective`min`

to`max`

in`n`

steps. Bounds can also be single scalars in cases where the fixed resolution buffer is a lower-dimensional slice. This method can optionally take a target dataset in the case where the fixed resolution buffer should be computed in the frame of reference of a different dataset, in which case the bounds should be interpreted in the frame of reference of the target dataset (but this is only needed if data linking is used). See`compute_fixed_resolution_buffer()`

for a full list of arguments.

## Subset states¶

In the above section, we mentioned the concept of a
`SubsetState`

. In glue parlance, a *subset state* is an
abstract representation of a subset in the data – for instance *the subset of
data where a > 3* or *the subset of data inside a polygon with vertices vx and
vy*. Several of the methods in Main data interface can
take subset states, and so in your data object, you should decide how to
most efficiently implement each kind of subset state.

You can find a full list of subset states defined in glue `here`

, and in particular you can look at the documentation of each
one to understand how to access the relevant information. For example, if you
have an `InequalitySubsetState`

, you can access the
relevant information as follows:

```
>>> subset_state
<InequalitySubsetState: (x > 1.2)>
>>> subset_state.left
x
>>> subset_state.right
1.2
>>> subset_state.operator
<built-in function gt>
```

`SubsetState`

objects have a
`to_mask()`

method that can take a data
object and a view and return a mask:

```
>>> subset_state.to_mask(d)
array([False, True, True])
```

In this case, the subset state essentially accesses the data using
`get_data()`

, so this may be very
inefficient for large datasets. Therefore, you may choose to re-interpret the
subset states and compute a mask yourself.

While developing your data class, one way to make sure that glue doesn’t crash if you haven’t yet implemented support for a specific subset state is to interpret any unimplemented subset state as simply indicating an empty subset.

## Using your data object¶

Assuming you have written your own data class, there are several ways that you can start using it in glue:

You can define your own data loader which is a function that takes a filename and should return an instance of a subclass of

`BaseCartesianData`

.You can define your own data importer, which is a function that can do anything you need, for example showing a dialog, and should return a list of instances of

`BaseCartesianData`

. This is more general than a data loader since a data importer doesn’t need to rely on a filename. It might include for example opening a dialog in which you can log in to a remote service and browse available datasets.You can start up glue programmatically, including constructing your data object. This is particularly useful when initially developing your custom data object:

# Construct your data object d = MyCustomData(...) # Create glue application and start dc = DataCollection([d]) ga = GlueApplication(dc) ga.start()

## Example¶

As an example of a minimal custom data class, the following implements a (very uninteresting) dataset that simply generates values randomly in the range [0:1] on-the-fly, and does not take subset states into account. A glue session is then created with one of these data objects:

```
import numpy as np
from glue.core.component_id import ComponentID
from glue.core.data import BaseCartesianData
from glue.utils import view_shape
class RandomData(BaseCartesianData):
def __init__(self):
super(RandomData, self).__init__()
self.data_cid = ComponentID(label='data', parent=self)
@property
def label(self):
return "Random Data"
@property
def shape(self):
return (512, 512, 512)
@property
def main_components(self):
return [self.data_cid]
def get_kind(self, cid):
return 'numerical'
def get_data(self, cid, view=None):
if cid in self.pixel_component_ids:
return super(RandomData, self).get_data(cid, view=view)
else:
return np.random.random(view_shape(self.shape, view))
def get_mask(self, subset_state, view=None):
return subset_state.to_mask(self, view=view)
def compute_statistic(self, statistic, cid,
axis=None, finite=True,
positive=False, subset_state=None,
percentile=None, random_subset=None):
if axis is None:
if statistic == 'minimum':
return 0
elif statistic == 'maximum':
if cid in self.pixel_component_ids:
return self.shape[cid.axis]
else:
return 1
elif statistic == 'mean' or statistic == 'median':
return 0.5
elif statistic == 'percentile':
return percentile / 100
elif statistic == 'sum':
return self.size / 2
else:
final_shape = tuple(self.shape[i] for i in range(self.ndim)
if i not in axis)
return np.random.random(final_shape)
def compute_histogram(self, cid,
range=None, bins=None, log=False,
subset_state=None, subset_group=None):
return np.random.random(bins) * 100
# We now create a data object using the above class,
# and launch a a glue session
from glue.core import DataCollection
from glue.app.qt.application import GlueApplication
d = RandomData()
dc = DataCollection([d])
ga = GlueApplication(dc)
ga.start()
```