SDP Data Access Library

https://readthedocs.org/projects/ska-telescope-sdp-dal-prototype/badge/?version=latestDocumentation Status

Prototyping of the SDP data access library - implementing the memory data models from the SDP architecture.

Installation

You will require at least Python 3.6. Do the following:

$ pip install -r requirements.txt
$ pip install .

Example

The prototype comes with a number of example programs.

Streamer

Simply streaming data in chunks from one process to another. First start an Apache Plasma store:

$ plasma_store -s /tmp/plasma -m 1000000000
/arrow/cpp/src/plasma/store.cc:1242: Allowing the Plasma store to use up to 1GB of memory.
/arrow/cpp/src/plasma/store.cc:1269: Starting object store with directory /dev/shm and huge page support disabled

Note that plasma_store is the binary installed by the pyarrow package. It is generally called plasma-store-server if installed as part of an Apache Arrow distribution.

Leave the store running in the background, start the processor:

$ python scripts/stream_processor.py /tmp/plasma
StreamProcessor waiting for calls at prefix 00000000
0.0 GB/s
0.0 GB/s
0.0 GB/s
...

It should show no traffic at the beginning (0.0 GB/s). This can be fixed by adding a streamer process:

$ python scripts/streamer.py /tmp/plasma
Store using prefix 00000001 for objects
Found processor StreamProcessor at prefix 00000000
Paylod size 80.0 MB

At which point you should be able to see data coming out the other end.

Nifty gridder

A highly accurate gridder implementation using an analytical kernel and 3D (de)gridding (see https://gitlab.mpcdf.mpg.de/ift/nifty_gridder). We utilise it here to check that we can correctly integrate a non-trivial processing component.

First start an Apache Plasma store and the processor as above:

$ plasma_store -s /tmp/plasma -m 1000000000 &
$ python scripts/ng_processor.py /tmp/plasma &
NiftyProcessor waiting for calls at prefix 00000000

Now we can run the demo_wstack.py test from the original repository, using the processor:

$ python scripts/demo_wstack.py /tmp/plasma
Store using prefix 00000001 for objects
Found processor NiftyProcessor at prefix 00000000
[...]
L2 error between explicit transform and gridder: 2.5686169932859304e-13
[...]
Testing adjointness of the gridding/degridding operation
adjointness test: 1.2946522165420342e-15
[...]
Testing adjointness of the gridding/degridding operation
adjointness test: 1.3392328015604763e-13

Demonstrating that all data gets transferred correctly.