norwin
/
ecmwf-dataset-crawl
mirror of https://github.com/52North/ecmwf-dataset-crawl
1
0
Fork 0
Code Issues Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
develop
secure_kibana
skip_unrelated_outlinks
crawler-test-manual
v1.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
ecmwf-dataset-crawl/controller
History
Norwin c0d520ea8a show query score in result list 6 years ago
..
src show query score in result list 6 years ago
.dockerignore controller: add boilerplate (docker, yarn, ...) 7 years ago
.gitignore add Crawl model & convert it to JSON schema for swagger 7 years ago
Dockerfile document api key setup, closes #23 7 years ago
README.md fix docker build of controller 6 years ago
package.json restrict google results to specified country by default 6 years ago
swagger.yaml allow to filter result subset of the original crawl langs 6 years ago
testcrawl.json restrict google results to specified country by default 6 years ago
tsconfig.json fixup! controller: use typescript 7 years ago
tslint.json set up linter 7 years ago
yarn.lock update dependencies 6 years ago

README.md

ecmwf-dataset-crawl controller

Component responsible for management of crawls + communication with frontend

Contains business logic for

  • creating / starting / crawls
  • generating seed URLs for a crawl from keyword list

Provides an HTTP API for crawl management + result retrieval.

howto build

see Dockerfile

howto dev

# configure custom config options via environment variables, see src/config
vi .env

# continuously compile typescript
yarn build:watch

# run with hot reload
yarn start:dev

# lint code
yarn lint

note that for a successful build, the app may not have any type errors, as npm run build:types will fail otherwise!