norwin
/
ecmwf-dataset-crawl
mirror of https://github.com/52North/ecmwf-dataset-crawl
1
0
Fork 0
Code Issues Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
develop
secure_kibana
skip_unrelated_outlinks
crawler-test-manual
v1.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
ecmwf-dataset-crawl/crawler/README.md

1.0 KiB
Raw Permalink Blame History

Crawler

Based on Apache Storm + Flux. Depends on Running elasticsearch instance and a python 3.6 installation. Can be configured via environment variables. All variables in .env must be set.

To run the topology without a storm executable:

env $(cat .env | xargs) \
  mvn compile exec:java -Dexec.mainClass=org.apache.storm.flux.Flux -Dexec.args="\
  --local --sleep 99999999 --env-filter es-crawler.flux"

To build a bundle that can be run with storm:

mvn package
env $(cat .env | xargs) storm jar target/crawler-alpha.jar org.apache.storm.flux.Flux --local --sleep 99999999 --env-filter ./es-crawler.flux

installation

The crawler makes use of storm's multilang support to use a classifier written in python. The resources for this classifier are located in src/main/resources/resources (due to classpath weirdness in the generated jar) with all dependencies vendored in that directory. To reinstall these sources, run

export target=src/main/resources/resources; pip3 install -r $target/requirements.txt --target $target