NDWidget#971

kushalkolar

Internals

RangeContinuous

Just a dataclass that stores the start, stop, and step of a reference range. A reference range is usually in scientific units, such as seconds/ms for time. Depth is another example of a reference range. A reference range just specifies the min and max of a dimension that will be used as a slider dimension. The imgui sliders use the reference range to determine the min, max, and step size of the UI elements (sliders, step-one button, etc.).

RangeDiscrete is mostly a placeholder for now, not fully implemented yet, waiting for a proper usecase. Genes might be an example? Session index for multi-session data could be another potential usecase.

Users must define a reference range for every dimension they want to use as a slider dimension.

TODO: Would be useful to have an automatic reference range that's generated for a slider dimension when it's not specified. For example, sometimes you just want to dump a bunch of calcium arrays and look at them, and they're already in the same time-space. Could have a simple auto-reference range mode that just sets the min and max for eacher slider dim based on the current arrays in the NDWidget.

ReferenceIndex

This manages the reference index for an NDWidget. It's the only place where indices should be set.

TODO: pushing/popping reference ranges.

NDProcessor

This manages n-dimensional data of any type. Subclasses must implement a get() method which takes a dict mapping dim names -> index for that dim. If specified, window functions are applied on slider dimensions. _get_slider_dims_indexer takes the indices dict and outputs a dict of dim_names -> slice. _apply_window_functions() uses these slices and applies the window functions. The final output after window functions has the same number of dims as the data, but with 1 element in any dimensions that were sliced (i.e. all slider dims, spatial dims remain untouched). In the end, the get() method must return an array that can be directly mapped to graphic data.

Key properties in an NDProcessor:

data - usually an xarray, but subclasses can manage data that is of any type. The NDProcessor NDPP_Pandas for positional data stored in a pandas dataframe is an example of this.

shape, ndim, dims, spatial_dims, slider_dims - self explanatory, dims are always named

window_funcs - window functions, dict {dim_name: (func, window_size)}. A window function must take axis and keep_dims as kwargs.
slider_dim_transforms - dict of functions for each slider dim that maps a reference range value to the local array index value. If a user doesn't provide a transform for a dim (for example, they just want to look at many arrays which are already in the same time-space, such as calcium movies) the it just uses an identify mapping.

NDPositionsProcessor and NDImageProcessor have logic specific to those types of data.

NDPositionsProcessor has an additional display_window properties which acts on the datapoints dimension. The datapoints dim is also both a slider dim and spatial dim, but it is excluded for window_funcs. datapoints_window_func is a separate property on NDPositionsProcessor that applies window funcs for that dimension since it's a bit different from other slider dims.

NDGraphic

A n-dimensional graphical representation. It has an NDProcessor and fpl.Graphic that it manages. ReferenceIndex sets the indices on an NDGraphic, which then call get() on its NDProcessor to retrieve the new graphic data, and then sets it on the graphic. It has property aliases for the various processor properties, so that users can set window funcs, slider dim transforms, etc.

NDPositions and NDImage are two NDGraphic subclasses that have specific code for positional and image specific representations.

NDPositions also manages a linear selector for timeseries representations, and allows swapping between scatter, line stack, line collection, and heatmap representations. It also manages an "auto x range" mode for timeseries representations

NDImage also manages the HistogramLUTTool and swapping between ImageGraphic and ImageVolumeGraphic. It also sets the camera and controller when these are swapped.

NDWSubplot

The main entry point for users to add NDGraphics to a subplot in an NDWidget. It mainly has add_nd<...> methods for users to provide data and kwargs to add NDGraphics.

Need to cleanup this subclass so the args and kwargs for each add method are properly specified and documented.

NDWidget

Just a very simple class that has a fpl.Figure, an instance of ReferenceIndex, and the NDWSubsplots.

Very simplified diagram:

s1, ... sn are slider dims, d1, d2, ... are spatial dims.

User API

The user mainly interacts via the NDW subplots and NDGraphics. For example with images:

data = np.random.rand(1000, 30, 64, 64)

# must define a reference range for each dim
ref = {
    "time": (0, 1000, 1),
    "depth": (0, 30, 1),
}


ndw = fpl.NDWidget(ref_ranges=ref, size=(700, 560))
ndw.show()

ndi = ndw[0, 0].add_nd_image(
    data,
    ("time", "depth", "m", "n"),  # specify all dim names
    ("m", "n"),  # specify spatial dims IN ORDER, rest are auto slider dims
    name="4d-image",
)

# change spatial dims on the fly
# ndi.spatial_dims = ("depth", "m", "n")

Positional data:

# generate some toy timeseries data
n_datapoints = 100_000  # number of datapoints per line
n_freqs = 20  # number of frequencies
n_ampls = 15  # number of amplitudes
n_lines = 8

xs = np.linspace(0, 1000 * np.pi, n_datapoints)

data = np.zeros(shape=(n_freqs, n_ampls, n_lines, n_datapoints, 2), dtype=np.float32)

for freq in range(data.shape[0]):
    for ampl in range(data.shape[1]):
        ys = np.sin(xs * (freq + 1)) * (ampl + 1) + np.random.normal(
            0, 0.1, size=n_datapoints
        )
        line = np.column_stack([xs, ys])
        data[freq, ampl] = np.stack([line] * n_lines)


# must define a reference range, this would often be your time dimension and corresponds to your x-dimension
ref = {
    "freq": (1, n_freqs + 1, 1),
    "ampl": (1, n_ampls + 1, 1),
    "angle": (0, xs[-1], 0.1),
}

ndw = fpl.NDWidget(ref_ranges=ref, size=(700, 560))

nd_lines = ndw[0, 0].add_nd_timeseries(
    data,
    ("freq", "ampl", "n_lines", "angle", "d"),
    ("n_lines", "angle", "d"),
    slider_dim_transforms={
        "angle": xs,
        "ampl": lambda x: int(x + 1),
        "freq": lambda x: int(x + 1),
    },
    x_range_mode="view-range",
    name="nd-sine"
)

nd_lines.graphic.cmap = "tab10"

subplot = ndw.figure[0, 0]
subplot.controller.add_camera(subplot.camera, include_state={"x", "width"})

ndw.show(maintain_aspect=False)

Main things left todo

docs, docstrings
tests, and test edge cases such as when NDProcessor.get() returns empty arrays
lots of examples
user input validation, see what Edo did recently with nemos
generalize reference range to a reference dim class with other property, such as UI slider throttling. Range would be one such property. Units could be another and we could use pint?
auto reference ranges
push/pop reference range on ReferenceIndex
other graphical features such as colors, cmap, sizes, etc. Also maintain things like cmap across lines/scatters when switching graphical representations.
if data is set to None, delete the graphic, indices setter should just ignore. Might need to update ReferenceIndex to also ignore it.
NDSubplot.delete_ndgraphic()
NDSubplot as a subclass of Subplot
vmin, vmax kwargs to add_nd_timeseries, each of which are a collection_size array of float that indicate the min and max of that array to use for auto-line stacking as a substitute for the y-maxes of each line. Since LineStack is initialized with the first display_window portion of the full array, it may not reflect the actual min and max of the full data.
clear install instructions since this will be a new toggle due to xarray dep.
the pandas NDProcessor should just be a default one that's available since pandas is an xarray dependency.
Maybe reorganize subpackage so it's just fastplotlib.ndwidget?
optimize pandas processor get()
property to get the current display range in ref units on NDPositionsGraphic.
nice repr, this seems like a good tutorial https://dev.to/dcodeyt/creating-beautiful-html-tables-with-css-428l
renaming things for consistency, clarity
- ReferenceIndex -> ReferenceIndices, it manages multiple indices
- indices -> ref_indices, keep constructor and properties symmetric like the rest of fpl

implements #951

kushalkolar

Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D

Kooha-2025-12-27-03-50-45.mp4

kushalkolar

Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test.

Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching.

Kooha-2025-12-27-17-47-16.mp4

kushalkolar

So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now).

If we have:

l: number of timeseries
p: number of datapoints in a timeseries

We can have the following shapes:

p: only y-values p, 2: yz vals l, p: l-timeseries with y values l, p 2: l-timeseries with yz values

Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is $d_1, d_2, ... d_n$, then the above becomes:

$$\begin{align*} d_1, ... d_n, p\\\ d_1, ... d_n, p, 2\\\ d_1, ... d_n, l, p\\\ d_1, ... d_n, l, p, 2\\\ \end{align*}$$

I don't think we can auto-detect if l is present or not and the user should specify, something like:

multi_timeseries: bool = True

Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories.

So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x).

kushalkolar

I made a more generalist NDPositions which can map data that is:

[s1, s2, ... sn, l, p, 2 | 3]

where:
s1, s2, .... sn are slider dims
l is number of lines or number of scatters, this dimension is optional and the user must specify whether or not it exists
2 | 3 is the last dim, indicating xy or xyz positions.

It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line).

I think this is a much more elegant way to deal with things, and NDTimeSeries is not necessary. The user can provide a slider mapping (to map from reference units to array index) for the p dimension which is the same as the "x-axis" for time series data!

Example if we have data that is [n_timepoints, 2], and the x-positions here (in the last dim) are in seconds. The NDWidget reference units for the slider can also be in seconds, and we can provide a mapping function that goes from the slider reference units to the n_timepoints index.

I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using [n_timepoints, 2] data.

EDIT: I think that the NDPositions will also work for PolygonGraphic ! Can think about meshes in general later.

kushalkolar

For positions graphics, I should actually do [n_datapoints, n_lines, 2 | 3] so everything before the last 1 or 2 dims is always sliders.

kushalkolar

Some more ideas:

Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray).
API like:

add_nd_<scatter|lines|heatmap>( data=<array>, # array like, defaults to last 2-3 dims are graphical, first (n - 2 | 3) dims are sliders and in order x=<int | str | None> # optionally specify x graphical dim using the integer dim index or dim name y = ... # same for other graphical dims slider_dims=<None | tuple[int | str]> # specify slider dims, None is auto first (n - 2 | 3), or specify a tuple of slider dims as int or str, examples: (0, 1, 2, 3), ("time", "plane") ... # other args )

We interpret the given order of the slider_dims passed as $$s_1, s_2, ... s_n$$, regardless of their order within the actual array. This will make it clear to users which dims they are syncing when they use the sliders.

EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like:

add_nd_<...>( data=<array>, display_dims=<tuple[int | str]>, # specify display dims in [x, y, z] order, OR display dims that collapse to xyz after the finalizer function ... )

An example for images would be collapsing [z, m, n] to display a projection over z as slider dims are moved. But we could also have > 3 dims that are used, and then collapsed to 3 or fewer dims for the "final graphic/display dims".

kushalkolar

We can use LineCollections to display multiple lines, like behavior tracks of keypoints, with shape [n_keypoints, n_timepoints, 2]. This works well with NDPositions and display windows.

I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a ScatterCollection that behaves like a LineCollection so the same array with the same shape can be given, the only difference is the graphical representation would be a scatter instead of lines. For typical behavior keypoint viz, the display_window would usually be just 1, but it can be greater than 1 for any viz that needs to show a window of scatter points.

kushalkolar

ok I think stuff is working

ndpositions-2026-01-29_22.55.54.mp4

kushalkolar

I think I need to make a PolygonCollection too 🤔 . Would be very similar to the ScatterCollection.

kushalkolar

so the basics are all working 🥳

LineStack and heatmap representations are swappable:

nd_positions_swap_linestack_heatmap-2026-01-30_00.19.27.mp4

Scatter and line collection to show behavior trajectories:

nd_positions_behavior-2026-01-30_00.29.15.mp4

kushalkolar

A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:

graphical representation, dropdown menu to choose scatter, line, heatmap
display_window
multi bool

kushalkolar

Stuff I should finish before implementing the orchestrator:

merge auto-replace buffers #974 , so I can implement changing the display_window
implement logic to allow the display_window to centered, on the left or on the right
adding a linear selector, it's "default" positions and its position when the main slider for the n_datapoints dimension changes should correspond whether the display_window is in "center", "start" or "end" mode. Moving the linear selector should probably change the index in all other NDobjects but not its "parent" object. So that the visual representation of the parent object doesn't change, ex: if the linear selector is moved we don't want a linestack to also "move"
spatial function
implement mapping from a slider reference index with units (such as time) to array index.
figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics
auto display_window? Zoom in/out and set graphic data at different subsample levels

Things that make me "uncomfortable" that I need to settle:

dim shapes

Dim shape for nd-positions is [s1, s2, ... sn, n, p, 2 | 3] where:

n: number of lines, scatters, or heatmap rows (optional, can be 1)
p: number of datapoints, often corresponds to a time index.

p would be a slider dim, but there's this n dim that's in between p and all the other slider dims. We could instead use a shape of [s1, ... sn, p, n, 2 | 3] but that feels weird and we'd have to do a transpose to get the array for the graphical representation, i.e. [p, n, 2 | 3] -> [n, p, 2 | 3] is required for the graphical representation.

Do we just document this well, that the n dimension is not a slider dim but p is?

When using nd-positions data in conjunction with nd-image data, we'd have something like this:

# nd-positions [s1, ... sn, n, p, 2 | 3] # nd-image [s1, ... sn, p, r, c, 1 | 3 | 4]

Where r: rows, cols: columns. 1 | 3 | 4 denotes grayscale or RGB(A).

The p dimension in the nd-image array above would correspond to the p in the nd-positions. nd-images don't have any n dimension, it doesn't make sense there.

kushalkolar

Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of slider_dims and n_slider_dims properties. Need to figure out how to properly separate the p n_datapoints dimension from other slider dims, and also apply the window funcs on the p dim. Will do tomorrow.

kushalkolar

Window funcs working for p (n_datapoints) dim and all graphical representations

ndp_windows-2026-02-01_03.39.52.mp4

kushalkolar

ok no longer need xarray! 🥳

kushalkolar

I'm also going to look into making the entire get data -> window funcs -> spatial funcs -> set on graphic async, probably just by threading. So we shouldn't block the main render thread while wait for this full pipeline.

And need to have a way to mark that all new graphic data has been set and we are ready to render the next frame.

Few more things:

Don't convert to a numpy until until the very end, i.e. right before setting graphic.data. This way window_funcs, spatial_func etc. can also operate in torch/cupy/jax/etc. and on the GPU and things are moved off only at the very end and also remains fully async.
NDProcessor.pipeline property that takes tuple(dict[dim, window_size], func) which does a fully customizable pipeline that takes precedence over window_funcs and spatial_func if provided. The pipeline func takes a sliced array with the given windows, the dims, spatial_dims, and must return the final array that maps to the graphic.

kushalkolar

Idea: When the NDProcessor is wrapping async data that uses shared memory we could use that shared memory directly as the graphic buffer. This has some limitations, ex: cannot tile images. Needs some thought to implement properly.

EDIT: see #1042

Add to the docstring whether this works or not

uniform_size -> size_mode

add note here: sharing ref idxs between ndwidgets syncs them

list with kwarg

FlynnOConnell

why is the throttle on RangeContinuous defaulted to 200ms? That makes scrubbing through T noticably worse - I can set the throttle myself, just curious what the benefit is

kushalkolar

why is the throttle on RangeContinuous defaulted to 200ms? That makes scrubbing through T noticably worse - I can set the throttle myself, just curious what the benefit is

These aren't final values, high throttle is necessary for video decoding on cpu. I'll probably set a 100ms throttle when a video is added, maybe 50ms or lower throttle as default. Something to play with.

Also throttling is only for the imgui slider. All other methods of updating the indices are not time-throttled. If you click the "play" button it will actually display every single frame, similarly if you use the linear selector or pan/zoom on a timeseries those updates are not throttled and they are immediate; it relies on the IO being fast and reliable in these cases, else it will block rendering if the IO is slow/inefficient/bad but that's on the IO not on NDWidget.

start ndprocessors

9a01cd5

kushalkolar changed the title NWidget NDWidget Dec 25, 2025

kushalkolar added 2 commits December 27, 2025 03:52

basic timeseries

c46455f

add __init__

d93fa5d

heatmap for timeseries works!

fddefb8

NDPositions, basics work, reorganize, increase default scatter size

d5e4c7d

black

074669b

kushalkolar added 2 commits January 29, 2026 20:27

NDPositions working with multi-dim stack of lines, need to test windo… …

ff5c578

…w funcs

scatter collection

3f412c5

kushalkolar added 2 commits January 30, 2026 00:06

progress, need to change to other branch so committing

dc30151

Merge branch 'main' into ndwidget

2b3c9b6

better

db98bde

kushalkolar added 2 commits January 30, 2026 20:45

interpolation for heatmap

3629f70

better place for check

87ea418

kushalkolar added 2 commits February 1, 2026 03:43

window functions working on n_datapoints dim

e5a8d40

p dim window funcs working for single and multiple dims I think

8d050a7

apasarkar and others added 4 commits April 9, 2026 12:59

async NDProcessor (#1026 ) …

b918626

* async NDProcessor established, NOT TESTED * ASYNC NDPROC IS WORKING :D :D CELEBRATE * comments * type annot * fix * fix * polish async integration, cuda also integrated * no longer using xarray, allow simpler ArrayProtocol * comments * comments * docs

forgot to set back to False

29d4790

remove xarray from pyproject.toml

e8a21a7

throttling

61f39bf

apasarkar and others added 8 commits April 15, 2026 14:33

Adds kwargs so user can customize vector field estimates (#1036 ) …

b7c15de

* Adds kwargs so user can customize vector field estimates * Does the kwarg organization the way ndpositions does * fixes graphic constructor

cleanup

bfff113

compute capabilities (#1040 ) …

427baae

* allow setting texture usage directly * ImageHighlightSelector, VisibilitySelector and ImageVisibilitySelector can handle 'None' as placeholders in the selection * fix example

comments

35de191

Merge branch 'ndwidget' of https://github.com/fastplotlib/fastplotlib … …

f35f185

…into ndwidget

clewis7 reviewed May 6, 2026

View reviewed changes

better axes padding for tick labels, settable axis label (#1049 ) …

71cf24f

* better axes padding for tick labels, settable axis label * cleanup * move axes to tools * update * move stuff around * update docs * update * update

kushalkolar marked this pull request as ready for review May 13, 2026 06:22

kushalkolar marked this pull request as draft May 18, 2026 00:51

kushalkolar added 5 commits May 29, 2026 05:14

remove __array_ufunc__ from required attrs

c3fe667

do not require __array__() to be implemented for ndwidget

9169f43

auto ref ranges

6c9e2b9

fix imgui

a80884a

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

NDWidget#971

NDWidget#971kushalkolar wants to merge 133 commits intomainfastplotlib/fastplotlib:mainfrom ndwidgetfastplotlib/fastplotlib:ndwidgetCopy head branch name to clipboard

Conversation

kushalkolar commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Internals

RangeContinuous

ReferenceIndex

NDProcessor

NDGraphic

NDWSubplot

NDWidget

User API

Main things left todo

Uh oh!

kushalkolar commented Dec 27, 2025

Uh oh!

kushalkolar commented Dec 28, 2025

Uh oh!

kushalkolar commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Dec 29, 2025

Uh oh!

kushalkolar commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stuff I should finish before implementing the orchestrator:

Things that make me "uncomfortable" that I need to settle:

dim shapes

Uh oh!

kushalkolar commented Jan 31, 2026

Uh oh!

kushalkolar commented Feb 1, 2026

Uh oh!

kushalkolar commented Apr 10, 2026

Uh oh!

kushalkolar commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clewis7 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

clewis7 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

clewis7 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

clewis7 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

FlynnOConnell commented May 11, 2026

Uh oh!

kushalkolar commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

NDWidget#971
kushalkolar wants to merge 133 commits into
mainfrom
ndwidget

kushalkolar commented Dec 25, 2025 •
edited
Loading

kushalkolar commented Dec 29, 2025 •
edited
Loading

kushalkolar commented Dec 29, 2025 •
edited
Loading

kushalkolar commented Jan 27, 2026 •
edited
Loading

kushalkolar commented Jan 30, 2026 •
edited
Loading

kushalkolar commented Jan 30, 2026 •
edited
Loading

kushalkolar commented Apr 10, 2026 •
edited
Loading

kushalkolar commented Apr 11, 2026 •
edited
Loading

kushalkolar commented May 12, 2026 •
edited
Loading