-
Notifications
You must be signed in to change notification settings - Fork 262
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
re: Discussion #2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
- Loading branch information
1 parent
f121e0b
commit 3ffe7be
Showing
96 changed files
with
2,419 additions
and
1,324 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,7 +7,7 @@ | |
| name: Run MSYS2, MinGW64-based Tests | ||
|
|
||
|
|
||
| on: [ pull_request ] | ||
| on: [pull_request] | ||
|
|
||
| jobs: | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Searches for an installation of the bz2 library. On success, it sets the following variables: | ||
| # | ||
| # Bz2_FOUND Set to true to indicate the bz2 library was found | ||
| # Bz2_INCLUDE_DIRS The directory containing the header file bz2/bz2.h | ||
| # Bz2_LIBRARIES The libraries needed to use the bz2 library | ||
| # | ||
| # To specify an additional directory to search, set Bz2_ROOT. | ||
| # | ||
| # Author: Siddhartha Chaudhuri, 2009 | ||
| # | ||
|
|
||
| # Look for the header, first in the user-specified location and then in the system locations | ||
| SET(Bz2_INCLUDE_DOC "The directory containing the header file bz2.h") | ||
| FIND_PATH(Bz2_INCLUDE_DIRS NAMES bz2.h bz2/bz2.h PATHS ${Bz2_ROOT} ${Bz2_ROOT}/include DOC ${Bz2_INCLUDE_DOC} NO_DEFAULT_PATH) | ||
| IF(NOT Bz2_INCLUDE_DIRS) # now look in system locations | ||
| FIND_PATH(Bz2_INCLUDE_DIRS NAMES bzlib.h DOC ${Bz2_INCLUDE_DOC}) | ||
| ENDIF(NOT Bz2_INCLUDE_DIRS) | ||
|
|
||
| SET(Bz2_FOUND FALSE) | ||
|
|
||
| IF(Bz2_INCLUDE_DIRS) | ||
| SET(Bz2_LIBRARY_DIRS ${Bz2_INCLUDE_DIRS}) | ||
|
|
||
| IF("${Bz2_LIBRARY_DIRS}" MATCHES "/include$") | ||
| # Strip off the trailing "/include" in the path. | ||
| GET_FILENAME_COMPONENT(Bz2_LIBRARY_DIRS ${Bz2_LIBRARY_DIRS} PATH) | ||
| ENDIF("${Bz2_LIBRARY_DIRS}" MATCHES "/include$") | ||
|
|
||
| IF(EXISTS "${Bz2_LIBRARY_DIRS}/lib") | ||
| SET(Bz2_LIBRARY_DIRS ${Bz2_LIBRARY_DIRS}/lib) | ||
| ENDIF(EXISTS "${Bz2_LIBRARY_DIRS}/lib") | ||
|
|
||
| # Find Bz2 libraries | ||
| FIND_LIBRARY(Bz2_DEBUG_LIBRARY NAMES bz2d bz2_d libbz2d libbz2_d libbz2 | ||
| PATH_SUFFIXES Debug ${CMAKE_LIBRARY_ARCHITECTURE} ${CMAKE_LIBRARY_ARCHITECTURE}/Debug | ||
| PATHS ${Bz2_LIBRARY_DIRS} NO_DEFAULT_PATH) | ||
| FIND_LIBRARY(Bz2_RELEASE_LIBRARY NAMES bz2 libbz2 | ||
| PATH_SUFFIXES Release ${CMAKE_LIBRARY_ARCHITECTURE} ${CMAKE_LIBRARY_ARCHITECTURE}/Release | ||
| PATHS ${Bz2_LIBRARY_DIRS} NO_DEFAULT_PATH) | ||
|
|
||
| SET(Bz2_LIBRARIES ) | ||
| IF(Bz2_DEBUG_LIBRARY AND Bz2_RELEASE_LIBRARY) | ||
| SET(Bz2_LIBRARIES debug ${Bz2_DEBUG_LIBRARY} optimized ${Bz2_RELEASE_LIBRARY}) | ||
| ELSEIF(Bz2_DEBUG_LIBRARY) | ||
| SET(Bz2_LIBRARIES ${Bz2_DEBUG_LIBRARY}) | ||
| ELSEIF(Bz2_RELEASE_LIBRARY) | ||
| SET(Bz2_LIBRARIES ${Bz2_RELEASE_LIBRARY}) | ||
| ENDIF(Bz2_DEBUG_LIBRARY AND Bz2_RELEASE_LIBRARY) | ||
|
|
||
| IF(Bz2_LIBRARIES) | ||
| SET(Bz2_FOUND TRUE) | ||
| ENDIF(Bz2_LIBRARIES) | ||
| ENDIF(Bz2_INCLUDE_DIRS) | ||
|
|
||
| IF(Bz2_FOUND) | ||
| # IF(NOT Bz2_FIND_QUIETLY) | ||
| MESSAGE(STATUS "Found Bz2: headers at ${Bz2_INCLUDE_DIRS}, libraries at ${Bz2_LIBRARY_DIRS}") | ||
| MESSAGE(STATUS " library is ${Bz2_LIBRARIES}") | ||
| # ENDIF(NOT Bz2_FIND_QUIETLY) | ||
| ELSE(Bz2_FOUND) | ||
| IF(Bz2_FIND_REQUIRED) | ||
| MESSAGE(FATAL_ERROR "Bz2 library not found") | ||
| ENDIF(Bz2_FIND_REQUIRED) | ||
| ENDIF(Bz2_FOUND) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Searches for an installation of the zstd library. On success, it sets the following variables: | ||
| # | ||
| # Zstd_FOUND Set to true to indicate the zstd library was found | ||
| # Zstd_INCLUDE_DIRS The directory containing the header file zstd/zstd.h | ||
| # Zstd_LIBRARIES The libraries needed to use the zstd library | ||
| # | ||
| # To specify an additional directory to search, set Zstd_ROOT. | ||
| # | ||
| # Author: Siddhartha Chaudhuri, 2009 | ||
| # | ||
|
|
||
| # Look for the header, first in the user-specified location and then in the system locations | ||
| SET(Zstd_INCLUDE_DOC "The directory containing the header file zstd.h") | ||
| FIND_PATH(Zstd_INCLUDE_DIRS NAMES zstd.h zstd/zstd.h PATHS ${Zstd_ROOT} ${Zstd_ROOT}/include DOC ${Zstd_INCLUDE_DOC} NO_DEFAULT_PATH) | ||
| IF(NOT Zstd_INCLUDE_DIRS) # now look in system locations | ||
| FIND_PATH(Zstd_INCLUDE_DIRS NAMES zstd.h zstd/zstd.h DOC ${Zstd_INCLUDE_DOC}) | ||
| ENDIF(NOT Zstd_INCLUDE_DIRS) | ||
|
|
||
| SET(Zstd_FOUND FALSE) | ||
|
|
||
| IF(Zstd_INCLUDE_DIRS) | ||
| SET(Zstd_LIBRARY_DIRS ${Zstd_INCLUDE_DIRS}) | ||
|
|
||
| IF("${Zstd_LIBRARY_DIRS}" MATCHES "/include$") | ||
| # Strip off the trailing "/include" in the path. | ||
| GET_FILENAME_COMPONENT(Zstd_LIBRARY_DIRS ${Zstd_LIBRARY_DIRS} PATH) | ||
| ENDIF("${Zstd_LIBRARY_DIRS}" MATCHES "/include$") | ||
|
|
||
| IF(EXISTS "${Zstd_LIBRARY_DIRS}/lib") | ||
| SET(Zstd_LIBRARY_DIRS ${Zstd_LIBRARY_DIRS}/lib) | ||
| ENDIF(EXISTS "${Zstd_LIBRARY_DIRS}/lib") | ||
|
|
||
| # Find Zstd libraries | ||
| FIND_LIBRARY(Zstd_DEBUG_LIBRARY NAMES zstdd zstd_d libzstdd libzstd_d libzstd | ||
| PATH_SUFFIXES Debug ${CMAKE_LIBRARY_ARCHITECTURE} ${CMAKE_LIBRARY_ARCHITECTURE}/Debug | ||
| PATHS ${Zstd_LIBRARY_DIRS} NO_DEFAULT_PATH) | ||
| FIND_LIBRARY(Zstd_RELEASE_LIBRARY NAMES zstd libzstd | ||
| PATH_SUFFIXES Release ${CMAKE_LIBRARY_ARCHITECTURE} ${CMAKE_LIBRARY_ARCHITECTURE}/Release | ||
| PATHS ${Zstd_LIBRARY_DIRS} NO_DEFAULT_PATH) | ||
|
|
||
| SET(Zstd_LIBRARIES ) | ||
| IF(Zstd_DEBUG_LIBRARY AND Zstd_RELEASE_LIBRARY) | ||
| SET(Zstd_LIBRARIES debug ${Zstd_DEBUG_LIBRARY} optimized ${Zstd_RELEASE_LIBRARY}) | ||
| ELSEIF(Zstd_DEBUG_LIBRARY) | ||
| SET(Zstd_LIBRARIES ${Zstd_DEBUG_LIBRARY}) | ||
| ELSEIF(Zstd_RELEASE_LIBRARY) | ||
| SET(Zstd_LIBRARIES ${Zstd_RELEASE_LIBRARY}) | ||
| ENDIF(Zstd_DEBUG_LIBRARY AND Zstd_RELEASE_LIBRARY) | ||
|
|
||
| IF(Zstd_LIBRARIES) | ||
| SET(Zstd_FOUND TRUE) | ||
| ENDIF(Zstd_LIBRARIES) | ||
| ENDIF(Zstd_INCLUDE_DIRS) | ||
|
|
||
| IF(Zstd_FOUND) | ||
| # IF(NOT Zstd_FIND_QUIETLY) | ||
| MESSAGE(STATUS "Found Zstd: headers at ${Zstd_INCLUDE_DIRS}, libraries at ${Zstd_LIBRARY_DIRS}") | ||
| MESSAGE(STATUS " library is ${Zstd_LIBRARIES}") | ||
| # ENDIF(NOT Zstd_FIND_QUIETLY) | ||
| ELSE(Zstd_FOUND) | ||
| IF(Zstd_FIND_REQUIRED) | ||
| MESSAGE(FATAL_ERROR "Zstd library not found") | ||
| ENDIF(Zstd_FIND_REQUIRED) | ||
| ENDIF(Zstd_FOUND) |
Oops, something went wrong.