OpenTripPlanner/test/performance

Speed test

This folder contains configuration and expectations to run the OTP speed test. The test runs automatically after each merged PR and the results are visualised on a Grafana instance.

Each run is listed under the GitHub Actions.

If you need to run the test locally you need to download the fixed datasets (OSM, transit) listed below and build a graph from them first.

After the graph is built, copy it into ./test/performance/${location} and execute the following command to run the speed test (also make sure the tests are compiled with e.g. mvn test-compile):

mvn --projects application exec:java -Dexec.mainClass="org.opentripplanner.transit.speed_test.SpeedTest" -Dexec.classpathScope=test -Dexec.args="--dir=test/performance/${location} -p md -n 4 -i 3 -0"

The results will be displayed in the console.

CI

The test is run after every merge to dev-2.x. Its GitHub Actions workflow is defined in performance-test.yml.

Running the test manually

Sometimes it is desirable to run the test configuration in the CI environment manually, for example, when making changes to the tests. To do this, perform these steps:

1. Create a local branch in git that contains the desired performance test configuration.
2. In performance-test.yml add your test branch name to the configuration and change all those locations that you want to run on the branch to core:

name: Performance test

on:
  push:
    branches:
      - dev-2.x
+     - test-branch-name

jobs:
  perf-test:
    if: github.repository_owner == 'opentripplanner' && !startsWith(github.event.head_commit.message ,'Bump serialization version id for') && !startsWith(github.event.head_commit.message ,'Upgrade debug client to version')
    runs-on: performance-test
    strategy:
      fail-fast: false
      matrix:
        include:
          ...
          - location: helsinki
            iterations: 1
            jfr-delay: "50s"
-           profile: extended
+           profile: core
...

3. Commit the changes to performance-test.yml.
4. Push the changes to a branch in the upstream OpenTripPlanner repository with the same name you added to performance-test.yml.
5. The tests will run after the push.

Note In order to visualise before and after, it is helpful to push a commit that is the same as the current dev-2.x and run the speed test on that.

Instrumentation

Each run on CI is instrumented with Java Flight Recorder. The results are then saved as an artifact and can be downloaded. IntelliJ for example can display a very useful flame graph from those jrf files that shows bottlenecks in the code.

Configure the test

• Pick a valid "testDate" for your data set and set it in the speed-test-config.json.
• Make sure build-config "transitServiceStart" and "transitServiceEnd" include the "testDate".

Data files

All data input files are located at https://otp-performance.leonard.io/data/

Norway

📊 Dashboard

Data used:

• Norwegian NeTEx data
• Norway OSM data

Contact: T.Gran, Entur

build-config

If the link above do not work you should be able to find it on the ENTUR web:

• https://www.entur.org/

Baden-Württemberg, Germany

📊 Dashboard

Data used:

• Tidied GTFS data
• BW OSM data

build-config

Helsinki, Finland

📊 Dashboard

Data used:

• HSL GTFS data
• Helsinki metropolitan area OSM data

build-config

Germany

📊 Dashboard

Data used:

• Tidied GTFS data

build-config

Skånetrafiken

📊 Dashboard

Data used:

• Skånetrafiken NeTEx data
• Sweden OSM data
• Denmark GTFS data
• Denmark OSM data

build-config

Switzerland

📊 Dashboard

Data used:

• Switzerland OSM data
• Switzerland GTFS national data

build-config

Washington State

This data set tests a lot of overlapping flex routes in and around Seattle in Washington State.

📊 Dashboard

build-config