Installer un Serveur DAP Hyrax pour les données météo

Motivations

Remerciements

Carole Larose, Éric le Jan et Charles-Henri Eyraud pour les relectures conseils et observations. Charles-Henri Eyraud pour la contribution sur l'installation de l'image sur un autre OS que linux.

Sommaire

  1. Environnement et téléchargement des ressources

    a. Particularités de la Raspberry Pi par rapport à un PC portable
    2. Recompilation finale et configuration

a. Recompilation

b. Configuration

3. Mise en production

Téléchargements

  1. Environnement de création Hyrax
  2. Proj-6.0.0
  3. cd gdal/ ./configure --with-proj=/usr/local

git clone https://github.com/OSGeo/GDAL.git

1. Environnement dépendances et téléchargement des ressources

L'installation se fait en suivant les instructions du site OpenDAP dans la version manuelle. Les sources du logiciel se trouvent sur le dépôt github Hyrax. Il existe d'imprtants problèmes de compatibilité entre les dépendances du logiciel car il intéresse une communauté restreinte, exigeante sur les fonctions disponibles, très active et disposant de compétences numériques avancées ce qui se traduit dans les usages par des compilations systématiques. Il est possible qu'il soit nécessaire suivant la configuration de votre serveur de recompiler certaines ou toutes les dépendances. Le point de départ de ce travail est un serveur dedian stretch (stable.

Installation des dépendances

apt-get install gfortran bash-completion

HDF4 et HDF5

curl -o hdf-4.2.14.tar.bz2 https://support.hdfgroup.org/ftp/HDF/releases/HDF4.2.14/src/hdf-4.2.14.tar.bz2 ./configure --enable-production --enable-java --prefix=/usr/local --enable-shared --disable-netcdf --disable-fortran

curl -o hdf5-1.10.5.tar.bz2 https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 2016 tar xvfj hdf-4.2.14.tar.bz2 2017 ls 2018 tar xvfj hdf5-1.10.5.tar.bz2 ./configure --enable-production --enable-java --prefix=/usr/local --enable-shared --disable-fortran --with-szlib 2043 make -j2

proj-6

Cette dépendance est requise pour compiler gdal, Les sources sont téléchargeables sur le site osgeo.org.

wget https://download.osgeo.org/proj/proj-6.0.0.tar.gz

La configuration et la compilation sont automatisées

cd proj-6.0.0
./configure --prefix=/usr/local
make
sudo make install

gdal

Cette dépendance est requise lar le serveur bes d'opendap dans une version suffisamment récente. On télécharge depuis github et on compile ordinairement (ne pas oublier de préciser le lien pour utiliser proj-6

git clone https://github.com/OSGeo/GDAL.git
cd gdal/
./configure --with-proj=/usr/local

git clone https://github.com/Unidata/netcdf-c.git 2002 git clone https://github.com/Unidata/netcdf4-python.git 2003 wget https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/latest_release/HDF-EOS2.20v1.00.tar.Z 2004 wget https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos5/latest_release/HDF-EOS5.1.16.tar.Z

cp hyrax/hyrax-dependencies/downloads/STARE-0.3.0.tar.bz2 . sudo apt-get install libfl-dev 2025 sudo apt-get install libjpeg-dev 2028 sudo apt-get install zlib1g-dev 2029 sudo apt-get install zlib1g-dev 2036 sudo apt-get install libsz2 2038 sudo make install 2040 sudo make install 2042 sudo apt-get install pkg-config 2043 sudo apt-get install sqlite3 2044 sudo apt-get install sqlite3-dev 2048 sudo apt-get install libsqlite3-dev 2050 sudo make install 2069 sudo apt-get install cmake 2070 sudo apt-get install bison byacc 2073 sudo apt-get install --reinstall libjpeg-dev 2075 sudo apt-get install libopenjp2-7-dev 2082 sudo apt-get install libicu-dev icu-devtools 2086 sudo apt-get install libccfits-dev

Téléchargement des sources et des dépendances

git clone https://github.com/opendap/hyrax
source spath.sh /usr/local
./hyrax_clone.sh -v

Compilation des dépendances

La variable jobs permet de paralléliser la compilation en fonction du nombre de coeurs.

cd hyrax-dependencies/
 make --jobs=3

genuca.cpp:398:25: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
         fprintf(stderr, "Error: unable to create %s"INVC_DATA_NAME", error %s\n", outputDir, u_errorName(*status));
                         ^
genuca.cpp:745:25: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
         fprintf(stderr, "Error: unable to create %s"UCA_DATA_NAME", error %s\n", outputDir, u_errorName(*status));
                         ^
genuca.cpp: In function ‘int32_t write_uca_table(const char*, const char*, const char*, UErrorCode*)’:
genuca.cpp:901:5: error: narrowing conversion of ‘4127195136u’ from ‘unsigned int’ to ‘int32_t {aka int}’ inside { } [-Wnarrowing]
     };
     ^
genuca.cpp:901:5: error: narrowing conversion of ‘4160749568u’ from ‘unsigned int’ to ‘int32_t {aka int}’ inside { } [-Wnarrowing]

pour éviter cette erreur passer la variable en uint32 :

struct {
      UChar32 start;
      UChar32 end;
      uint32_t value;
    } ranges[] =
.../...

À l'erreur suivante correspondant à la création d'un répertoire dans /usr/local relancer la compilation depuis root. Puis retourner dans l'espace racine de Hyrax.

Compilation de libdap4

Entrer dans le répertoire libdap4 et lancer la série de commandes suivante :

cd libdap4/
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j2
make check -j2

Compilation de bes

cd bes/
git submodule update --init
autoreconf --force --install --verbose
 ./configure --prefix=$prefix --with-dependencies=$prefix/deps 
 make -j2

rapatrier et compiler HDF5 en dernière version sinon il manque des dépendances. On trouve les sources sur le site HDF5

./configure --prefix=/usr/local --enable-build-mode=production  --enable-optimization=high --with-gnu-ld

2. Recompilation finale et configuration

Recompilation

.shell /hyrax_build.sh -p /usr/local

Configuration

3. Mise en production

1849 ls modules/cmr_module/tests/testsuite.dir 1850 ls modules/cmr_module/tests/testsuite.dir/1 1851 cat modules/cmr_module/tests/testsuite.dir/1/test 1852 sudo make install 1853 make -j12 1854 make check -j12 1855 ls modules/httpd_catalog_module/tests/testsuite.dir 1856 ls modules/httpd_catalog_module/tests/testsuite.dir/02/ 1857 cat modules/httpd_catalog_module/tests/testsuite.dir/02/bes.log 1858 cat modules/httpd_catalog_module/tests/testsuite.dir/02/testsuite.log 1859 sudo make install 1860 make 1861 make check 1862 cat modules/httpd_catalog_module/tests/testsuite.log 1863 more modules/httpd_catalog_module/tests/testsuite.log 1864 cd .. 1865 ls 1866 cd . 1867 cd µ.. 1868 cd .. 1869 ls 1870 cd bes 1871 make -j12 1872 git submodule update --init 1873 autoreconf --force --install --verbose 1874 ./configure --prefix=$prefix --with-dependencies=$prefix/deps 1875 make -j12 1876 make clean 1877 make -j12 1878 cd .. 1879 ls 1880 cd hyrax-dependencies/ 1881 ls 1882 sudo rm -r bes 1883 make -j12 1884 cd bes 1885 make -j12 1886 cd ../bes/ 1887 make -j12 1888 make check -j12 1889 sudo make install 1890 cd .. 1891 besctl start 1892 mkdir /home/vidal/Logiciels/Hyrax/hyrax/build/var/run 1893 ll /home/vidal/Logiciels/Hyrax/hyrax/build/var/run 1894 sudo chown -R vidal.vidal /home/vidal/Logiciels/Hyrax/hyrax/build 1895 besctl start 1896 sudo besctl start 1897 ls 1898 ./hyrax_build.sh 1899 dpkg -l | grep openjdk 1900 ./hyrax_build.sh 1901 which javac 1902 ll /usr/bin/javac 1903 ll /etc/alternatives/javac 1904 ./hyrax_build.sh 1905 ls 1906 ls build/ 1907 echo £Classpath 1908 echo $Classpath 1909 printenv 1910 printenv | grep JAVA 1911 readlink -f /usr/bin/java | sed "s:bin/java::" 1912 export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/ 1913 ./hyrax_build.sh 1914 sudo galternatives 1915 sudo apt-get install galternatives 1916 sudo galternatives 1917 bg 1918 ./hyrax_build.sh 1919 which javac 1920 ll /usr/bin/javac 1921 ll /etc/alternatives/javac 1922 exit 1923 sudo dpkg -l | grep libdap 1924 sudo apt-get purge libdap-dev libdap-doc libdap25 libdapclient6v5 libdapserver7v5 1925 sudo apt-get purge libdap-dev libdap25 libdapclient6v5 libdapserver7v5 1926 sudo apt-get purge libdap-dev libdap25 libdapserver7v5 1927 sudo apt-get purge libdap-dev libdap25 1928 sudo apt-get purge libdap-dev
1929 jed genuca.cpp 1930 ls 1931 ls src/ge 1932 ls scripts/ 1933 ls src/ 1934 jed /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/src/icu-3.6/source/tools/genuca/genuca.cpp 1935 man g++ 1936 jed /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/src/icu-3.6/source/tools/genuca/genuca.cpp 1937 ls src 1938 rgrep -e MFLAGS 1939 rgrep -e MFLAGS= 1940 cd .. 1941 ls 1942 m spath.sh 1943 ls 1944 cd hyrax-dependencies/ 1945 ls 1946 rgrep -e 'MFLAGS=' 1947 jed src/icu-3.6/source/configure 1948 cd src/icu-3.6/source/ 1949 ls 1950 cd tools/ 1951 ls 1952 cd genuca 1953 ls 1954 g++ -D_REENTRANT -I../../common -I../../i18n -I./../toolutil -Wno-narrowing -g -O2 -c -o genuca.o genuca.cpp 1955 apt-cache search readline 1956 dpkg -l | grep libreadline 1957 apt-cache search readline | grep dev 1958 sudo apt-get install libreadline-dev 1959 cd /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/bes/modules/hdf4_handler 1960 rgrep -e HUGE 1961 cd .. 1962 rgrep -e HUGE 1963 cd /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/bes/modules/hdf4_handler 1964 ls 1965 jed HDFEOS2.h 1966 cat freeform_handler/FreeFormCPP.h 1967 cat ../freeform_handler/FreeFormCPP.h 1968 more ../freeform_handler/FreeFormCPP.h 1969 jed HDFEOS2.h 1970 jed HDFEOS2.cc 1971 cd ../../../bes 1972 cd modules/ 1973 ls 1974 rgrep -e HUGE 1975 rgrep -e HUGE | grep HUGE 1976 jed HDFEOS2.cc 1977 cd /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/bes/modules/hdf4_handler 1978 jed HDFEOS2.cc 1979 ls 1980 \rm .o 1981 ls 1982 jed HDFEOS2.cc 1983 cd 1984 cd /home/vidal/Logiciels/Hyrax/hyrax/hyrax-dependencies/bes/modules/hdf4_handler 1985 ls 1986 pwd 1987 cd 1988 cd /home/vidal/Logiciels/Hyrax/hyrax/bes/modules/hdf4_handler 1989 jed HDFEOS2.cc 1990 apt-cache search javax 1991 apt-cache search javax.xml 1992 sudo apt-get install jaxws libjaxws-java 1993 dpkg -l | rep xml 1994 dpkg -l | grep xml 1995 apt-cache search jaxb 1996 dpkg -l | grep jaxb 1997 sudo apt-get install jaxb 1998 pip list 1999 cd Logiciels/ 2000 mkdir NetCDF4 2001 cd NetCDF4 2002 git clone https://github.com/Unidata/netcdf4-python.git 2003 python -V 2004 python3 -V 2005 sudo pip3 install pip --upgrade 2006 sudo pip3 install Cython 2007 dpkg -l | grep hdf5 2008 apt-cache search hdf5 2009 dpkg -l | grep libcurl 2010 apt-cache search hdf4 2011 sudo apt-get install libhdf4-dev 2012 wget https://github.com/Unidata/netcdf-c/releases/tag/v4.7.0 2013 ls 2014 file v4.7.0 2015 rm v4.7.0 2016 wget https://github.com/Unidata/netcdf-c/archive/v4.7.0.tar.gz 2017 ls netcdf4-python/ 2018 ls 2019 history 2020 ls 2021 tar xvfz v4.7.0.tar.gz 2022 ls 2023 ls netcdf4-python/ 2024 ls 2025 cd netcdf-c-4.7.0/ 2026 ls 2027 m README.md 2028 m INSTALL.md 2029 dpkg -l | grep zlib 2030 apt-cache search zlib 2031 apt-cache search zlib | grep zlib 2032 m INSTALL.md 2033 apt-cache search szlib 2034 apt-cache search szip 2035 sudo apt-get install libsz2 2036 sudo apt-get install --reinstall libsz2 2037 mls 2038 ls 2039 cd .. 2040 ls 2041 tar xvfz szip-2.1.tar.gz 2042 tar xvfz hdf5-1.8.14.tar.bz2 2043 tar xvfJ hdf5-1.8.14.tar.bz2 2044 tar xvfj hdf5-1.8.14.tar.bz2 2045 ls 2046 mkdir hdf5-1.8.14 2047 mv include/ lib/ share/ hdf5-1.8.14 2048 ls 2049 cd szip-2.1/ 2050 ls 2051 ./configure 2052 make -j12 2053 ls 2054 m INSTALL 2055 ./configure --prefix=/usr/local 2056 make -j12 2057 make check 2058 sudo make install 2059 ls 2060 cd .. 2061 ls 2062 cd netcdf-c-4.7.0/ 2063 ls 2064 m INSTALL.md 2065 ls 2066 cd .. 2067 ls 2068 cd hdf5-1.8.14/ 2069 ls 2070 ls include/ 2071 ls lib/ 2072 tar tvfj hdf5-1.8.14.tar.bz2 2073 cd .. 2074 ls 2075 tar tvfj hdf5-1.8.14.tar.bz2 2076 ls 2077 cd hdf5-1.8.14/ 2078 ls 2079 cd share/ 2080 ls 2081 m RELEASE.txt 2082 ls 2083 ls cmake/ 2084 ls cmake/hdf5/ 2085 ls 2086 m COPYING 2087 cd .. 2088 ls 2089 ls include/ 2090 ls lib/ 2091 file lib/ 2092 cd .. 2093 ls 2094 rm -r hdf5-1.8.14/ hdf5-1.8.14.tar.bz2 2095 tar tvfj hdf5-1.10.5.tar.bz2 2096 ll 2097 tar tvfj hdf5-1.10.5.tar.bz2 2098 tar xvfj hdf5-1.10.5.tar.bz2 2099 ls 2100 ls hdf5-1.10.5 2101 ls 2102 ll 2103 tar tvfz CMake-hdf5-1.10.5.tar.gz 2104 file CMake-hdf5-1.10.5.tar.gz 2105 gunzip CMake-hdf5-1.10.5.tar.gz 2106 tar tvf CMake-hdf5-1.10.5.tar 2107 tar xvf CMake-hdf5-1.10.5.tar 2108 ls 2109 cd CMake-hdf5-1.10.5/ 2110 ls 2111 m README_HDF5_CMake 2112 ls 2113 m HDF5options.cmake 2114 jed HDF5options.cmake 2115 ls 2116 jed HDF5config.cmake 2117 ./build-unix.sh 2118 ls 2119 ls hdf5-1.10.5/ 2120 m build-unix.sh 2121 ls 2122 m HDF5config.cmake 2123 ls 2124 ls build 2125 m hdf5.log 2126 ls 2127 cd .. 2128 ls 2129 cd hdf5-1.10.5/ 2130 ls 2131 ./configure 2132 ls 2133 m CMakeLists.txt 2134 ls 2135 m configure.ac 2136 rgrep -e C++ 2137 rgrep -e g++ 2138 ./configure 2139 rgrep -e parallel 2140 ls 2141 ls config 2142 cat config/linux-gnu 2143 m README.txt 2144 ls 2145 m configure 2146 ls 2147 m config/linux-gnulibc1 2148 ls 2149 cp config/linux-gnulibc1 . 2150 chmod a+x linux-gnulibc1 2151 ./linux-gnulibc1 2152 rm linux-gnulibc1 2153 m autogen.sh 2154 make -j12 2155 sudo make install 2156 ls 2157 ls hdf5/ 2158 ./configure --help 2159 ./configure --prefix=/usr/local --enable-cxx --enable-hl --enable-production --enable-parallel --with-gnu-ld 2160 ./configure --prefix=/usr/local --enable-cxx --enable-hl --enable-production --with-gnu-ld 2161 ./configure --prefix=/usr/local --enable-cxx --enable-hl --enable-build-mode=production --with-gnu-ld 2162 make -j12 2163 sudo make install 2164 cd .. 2165 ls 2166 cd netcdf-c-4.7.0/ 2167 ls 2168 ./configure --help 2169 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-dynamic-loading --enable-hdf4 --with-gnu-ld 2170 ls apt-cache search hdf4 2171 sudo apt-get install libhdf4-dev 2172 dpkg -L libhdf4-dev 2173 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-dynamic-loading --enable-hdf4 --with-gnu-ld 2174 cd .. 2175 ls 2176 ll 2177 tar xvfj hdf-4.2.14.tar.bz2 2178 cd hdf-4.2.14 2179 ls 2180 ./configure --help 2181 ./configure --prefix=/usr/local --enable-netcdf --enable-production --with-gnu-ld
2182 make -j12 2183 sudo make install 2184 cd .. 2185 ls 2186 cd netcdf-c-4.7.0/ 2187 ls 2188 ./configure --help 2189 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --enable-pnetcdf --with-gnu-ld
2190 cd ../hdf-4.2.14/ 2191 history 2192 make -j12 2193 sudo make install 2194 cd ../netcdf-c-4.7.0/ 2195 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --enable-pnetcdf --with-gnu-ld
2196 dpkg -l | grep HDF4 2197 dpkg -l | grep hdf4 2198 sudo apt-get purge libhdf4-0 2199 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --enable-pnetcdf --with-gnu-ld
2200 cd ../hdf-4.2.14/ 2201 history 2202 ./configure --prefix=/usr/local --disable-netcdf --enable-production --with-gnu-ld
2203 make -j12 2204 sudo make install 2205 cd ../netcdf-c-4.7.0/ 2206 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --enable-pnetcdf --with-gnu-ld
2207 apt-cache search lpnetcdf 2208 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --with-gnu-ld
2209 make -j12 2210 ./configure --help 2211 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --with-gnu-ld --enable-FEATURE=-fPIC 2212 make -j12 2213 ls 2214 m config.f 2215 m config.h 2216 ls 2217 make clean 2218 make -j12 2219 ls 2220 ls libhdf4/ 2221 m libhdf4/Makefile 2222 rgrep -e CPP 2223 ls 2224 ls libhdf4/ 2225 make -j12 2226 ls liblib/ 2227 jed liblib/Makefile 2228 make -j12 2229 make clean 2230 make -j12 2231 ls 2232 find . -name libmfhdf 2233 find . -name libmfhdf 2234 find . -name mfhdf 2235 find . -name mfhdf 2236 find . -name mfhdf. 2237 find . -name mfsd 2238 find . -name dfconv.o 2239 find . -name dfconv 2240 find . -name dfconv 2241 ls /usr/local/lib 2242 ll /usr/local/lib/libdf.a 2243 ll /usr/local/lib/ 2244 cd ../hdf-4.2.14/ 2245 ls 2246 ls mfhdf/ 2247 cd hdf 2248 jed Makefile 2249 cd ../mfhdf/ 2250 ls 2251 jed Makefile 2252 history 2253 ./configure --prefix=/usr/local --disable-netcdf --enable-production --with-gnu-ld
2254 cd .. 2255 ./configure --prefix=/usr/local --disable-netcdf --enable-production --with-gnu-ld
2256 make clean 2257 make -j12 2258 sudo make install 2259 cd ../netcdf-c-4.7.0/ 2260 history 2261 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --enable-pnetcdf --with-gnu-ld
2262 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --with-gnu-ld
2263 make -j12 2264 ll /usr/local/lib/ 2265 make -j12 2266 sudo rm /usr/local/lib/libdf.a /usr/local/lib/libmfhdf.a 2267 cd .. 2268 ls 2269 ls hdf-4.2.14/ 2270 ls hdf 2271 cd hdf-4.2.14/ 2272 ls hdf 2273 ls hdf/src/ 2274 ls hdf/CMakeLists.txt 2275 cat ! 2276 cat hdf/CMakeLists.txt 2277 ls 2278 ./configure --help 2279 ls 2280 grep -e CFLAGS ./ 2281 history 2282 ./configure --prefix=/usr/local --disable-netcdf --enable-production --with-gnu-ld
2283 make clean 2284 jed Makefile 2285 make -j12 2286 jed Makefile 2287 rgrep -e -fomit-frame-pointer 2288 ll 2289 rgrep -e -fomit-frame-pointer 2290 jed config/linux-gnu 2291 ./configure --prefix=/usr/local --disable-netcdf --enable-production --with-gnu-ld
2292 rgrep -e PIC 2293 make -j12 2294 sudo make install 2295 cd .. 2296 ls 2297 cd netcdf-c-4.7.0/ 2298 history 2299 ./configure --prefix=/usr/local --enable-doxygen --enable-dot --enable-hdf4 --with-gnu-ld
2300 make -j12 2301 sudo make install 2302 cd .. 2303 ls 2304 cd netcdf4-python/ 2305 ls 2306 python3 setup.py build 2307 ls 2308 ls build/ 2309 sudo python3 setup.py install 2310 pip list 2311 pip3 list 2312 pip3 uninstall netCDF4 2313 sudo pip3 uninstall netCDF4 2314 sudo pip uninstall netCDF4 2315 sudo python3 setup.py install 2316 pip3 list | grep net 2317 python3 2318 python -V 2319 python3 -V 2320 pip3 list 2321 python3 run_all.py 2322 ls 2323 ls test/ 2324 python3 test/run_all.py 2325 cd test 2326 python3 run_all.py 2327 ip a 2328 sudo python3 run_all.py 2329 python3 run_all.py 2330 python3 2331 sudo apt-get autoremove 2332 df -h 2333 cd ../.. 2334 cd .. 2335 history > listcommands

Webographie

https://docs.opendap.org/index.php/Hyrax_GitHub_Source_Build

Commentaires