If we add the following properties we can make things way more arbitrary:

dim_names: dict that maps numerical dim index to a dimension name, ex: {0: "t", 1: "z"}. Only need to name scrollable dims.

index: dict that maps dim name to current index, ex: {"t": 123, "z": 5}

image_dims: tuple[int, int] | tuple[int, int, int] numerical dim indices that define the image displayed in the ImageGraphic or ImageVolumeGraphic.

Also, for window_funcs, if a window func is provided for only one dim then it should apply the window only along that dim. If window_funcs are defined for multiple dimensions, then apply the window for all provided dims. Can also have a way to define the order, ex. do "t" first and "z" etc.

Histogram calculation can also be done within this widget anytime the window funcs or frame_apply funcs are changed. We can then just allow force-setting the HistogramLUTGraph.

Make a few ready-made subclasses of ImageWidgetArray, for example IWA_Dask, IWA_Lazy etc.

In ImageWidget.__init__(), parse each data array in the list use the appropriate IWA type based on the data array type.

IW also has a new kwarg iwa_types which can explicitly define the IWA type which corresponds to the data array.

Make the window functions more flexible. Allow options kwargs that include the full data array, i.e. IWA.data, and the current index. Useful for things like computing a window function and then using that on the original data or current frame. Example, use a rolling filter on the current window, use this to subtract from the original frame.

new kwarg on ImageWidget, "adaptive_histogram", List[bool] | bool which creates an event handler for current_index so that any time the slider is moved or the index is changed it sets the histogram based on the current frame/volume.

If the data doesn't change when moving through dims, example: if there's a simple 2D image amongst multiple txy videos, then would be good to handle this efficiently so that Texture doesn't isn't uploaded when it doesn't change.

ok so another issue is that you can't use a window function when you have a mix of movies and images (no t dim) 😞 . So this separation also needs to fix this issue.

Also need to make it possible for an ImageWidget subplot to be blank, set that data index as None and just ignore when scrolling.

argument to specify a transpose tuple of size 2 or 3 that is applied on the final image/volume after slicing, window functions, etc. could be useful.

ok I think I got a good way to apply window funcs, clean, understandable.

Thoughts on how sliders should map onto dims:

if we have this array, let's say it's [t, z, m, n]:
[1_000, 30, 100, 100]

then, do we assume the following smaller array is [t, m, n] or [z, m, n]?
[n, 100, 100]

Another way to think about this:

We want to allow arbitrary numbers of dims, so if we have the following array with ndim 6:
[a, b, c, d, m, n]

[m, n] are the image dims (it could also be RGB or volumetric, that stuff is easy to deal with, ignore it for now)

Do we assume a smaller array with ndim 4 is:
[a, b, m, n]

or:

[c, d, m, n]

Sliders will be made for a, b, c, d so this is important to decide.

One way to thing of it is: right-most are image dims, so slider dims accumulate as soon as image dims finish?
This would mean we interpret the shorter array above as [c, d, m, n]

Another way to think of it is, slider dims accumulate from left to right until they reach the image dims.
This would mean we interpret the above array as [a, b, m, n]

EDIT: We have chosen to do [c, d, m, n]

I think accumulating the slider dims from left to right makes sense.

However, depending on the use-case, I would want to be able to choose between [a b m n] and [c d m n]. For microscopy, [t z m n] is the most popular but I've also seen libraries use [z t m n] (e.g. suite3d). These two dim orders behave completely differently and take a lot of fine-tuning in dask/numpy depending on which you use.

What about defaulting to "left to right" mapping with an optional semantic argument, e.g.:

Another important thing: other than slider dims, nothing else has to be the same between iw arrays. For example, we can allow each individual array in the ImageWidget to have their own window function or frame apply function. Not sure what's the best API for this though.

Maybe the constructor can take a window funca arg, which if it's a list it defines the window func per data array. This would of course be symmetric with the properties.

display_dims should be a mutable property. When changed from 2 or 3, delete the existing graphic and replace it.

ok I got the basics down!

I now realize the histogram is not gonna be trivial 😄

We can't sub-sample the original array and then pass it through the window funcs because that won't correspond to windows in the original data.

I think we need to do something like this:

• if a window function is used for a dim, use a small number of indices from this dim depending on the window size, but at least 5 or 10. Apply the window function with these indices.
• if a dim has no window function, use some number of indices from this dim, but can be somewhat generous.
• if using a finalizer_func that's applied before displaying the frame or volume, do not subsample these spatial dims, for example a Gaussian kernel would provide very different results on a full spatial sample vs. a subsample.

I think we can use some of the logic in utils.subsample_array and do something like;

1. Create a new func get_subsample_indices that returns the indices necessary to subsample the array.
2. Determine a number of sample indices, call get() with the indices.
3. Concatenate all these arrays and compute the histogram.

@apasarkar @FlynnOConnell

Thoughts on how sliders should map onto dims:

if we have this array, let's say it's [t, z, m, n]: [1_000, 30, 100, 100]

then, do we assume the following smaller array is [t, m, n] or [z, m, n]? [n, 100, 100]

Another way to think about this:

We want to allow arbitrary numbers of dims, so if we have the following array with ndim 6: [a, b, c, d, m, n]

[m, n] are the image dims (it could also be RGB or volumetric, that stuff is easy to deal with, ignore it for now)

Do we assume a smaller array with ndim 4 is: [a, b, m, n]

or:

[c, d, m, n]

Sliders will be made for a, b, c, d so this is important to decide.

One way to thing of it is: right-most are image dims, so slider dims accumulate as soon as image dims finish? This would mean we interpret the shorter array above as [c, d, m, n]

Another way to think of it is, slider dims accumulate from left to right until they reach the image dims. This would mean we interpret the above array as [a, b, m, n]

EDIT: We have chosen to do [c, d, m, n]

ok so both are actually easy to do and this is something that ImageWidget can handle before calling get() on each of the individual NDImageViews:

right -> left ordering

out:

left -> right ordering

out:

@FlynnOConnell

my all time favorite commit in fastplotlib 😄 : 7770ee0

beauty lol

ok the basics work! 🥳

Needs tests, more manual testing, some cleanup and tweaks and will be ready to go! I anticipate there will be a few growing pains, lots of parsing since this has to deal with a diverse range of complex datasets and compute possibilities so it may take a while to iron out all the bugs.

@apasarkar @FlynnOConnell @clewis7 gonna need a review from all of you :D

Allow the spatial func to be a compute shader or a python function.

@kushalkolar I'm confused at where in the code we expect custom lazy arrays to be called on a set of indices.

This line is causing me problems:

.. where data[:] is causing my arrays getitem to load fully into RAM.

These are 4D TZXY, so when texture = pygfx.Texture(self.value[data_slice], dim=2) is called, my array's 4D structure errors:

In the ImageGraphic.data docstring , it mentions data must implement memoryview(), but the call to memoryview doesn't happen until the Texture() definition here:

The dim=2 here is throwing me off, so we expect this to always be 2D at this point?

EDIT

ValueError: Can't map shape (71, 14, 448, 448) on 2D tex. Maybe also specify size?

This happened because my array.ndim was returning the wrong value.

Everything is working flawlessly so far!

@kushalkolar YESYESYESYES THANK YOU

More to test on Sunday!

Gives TypeError: issubclass() arg 1 must be a class when I pass a list of processors.

Could do:

Works with a single processor:

More ideas:

• two modes for histograms: 1) update on every frame, 2) global histogram that represents full data. A kwargs or something.

right-click histogram to change the colormap?

I just learned python has builtin sigfig formatting! https://docs.cems.umn.edu/intro/Prt_01_Lssn_04_Using_Strings_and_Print%28%29.html#general-format-notation

@kushalkolar some thoughts re: the multi-session use case.

I think I understand what you're suggesting above re: arrays with dynamical shapes. Conceptually this would allow us to define a (num_sessions, <other_dimensions>) array. As you scroll through the "num_sessions" axis, you display data from a different session, and this movie can have a different spatial FOV (and number of frames).

I wonder whether it would be more intuitive to allow the "data" parameter in ImageWidget to be multi-dimensional. For e.g.:

Here, imagewidget would recognize the nested list structure and add an extra scrollable dimension to allow the user to move between groups of arrays.

This probably also relates to the multi-dimensional-everything vis.

Re: the parameters for the constructor, I wonder whether we can get rid of n_display_dims by adding a "volume" boolean. This boolean is by default False (in the same way that rgb is).

By inspecting the values for volume and rgb, the imagewidget constructor can track what the number of display dimensions should be automatically. (2, 3, or 4 in the case where rgb and volume are both true).

This might be easier for the user (since they are specifying parameters directly related to what they expect to see in the data).

ok this is ready to merge into the ndwidget branch, but I will do that tomorrow to make sure it goes well 🥳