Welcome to Biggus’s documentation!¶
Virtual arrays of arbitrary size, with arithmetic and statistical operations, and conversion to NumPy ndarrays.
Virtual arrays can be stacked to increase their dimensionality, or tiled to increase their extent.
Includes support for easily wrapping data sources which produce NumPy ndarray objects via slicing. For example: netcdf4python Variable instances, and NumPy ndarray instances.
All operations are performed in a lazy fashion to avoid overloading system resources. Conversion to a concrete NumPy ndarray requires an explicit method call.
# Wrap two large data sources (e.g. 52000 x 800 x 600). measured = OrthoArrayAdapter(netcdf_var_a) predicted = OrthoArrayAdapter(netcdf_var_b) # No actual calculations are performed here. error = predicted - measured # *Appear* to calculate the mean over the first dimension, and # return a new biggus Array with the correct shape, etc. # NB. No data are read and no calculations are performed. mean_error = biggus.mean(error, axis=0) # *Actually* calculate the mean, and return a NumPy ndarray. # This is when the data are read, subtracted, and the mean derived, # but all in a chunk-by-chunk fashion which avoids using much # memory. mean_error = mean_error.ndarray()