orcaLog in

Fleet

The EC2 instances orca launches for runs: head and worker hosts, their uptime, load and network. The one machine that runs the scheduler, this UI and Postgres is Control.

This scheduler's runner is local: it runs every run on its own machine and launches no instances, so there are none to list and no per-instance cost. The machine is Control; its price is the server's fixed price.

Hosts

HostThe instance: its role (head or worker) and its id.TypeThe EC2 instance type.RunA run: one start of a pipeline over some partitions, with its steps and their attempts. Click for its page.UptimeHow long the instance has been up.CPUCPU used: for a step, its latest attempt's user and system seconds; for a machine, its busiest moment as a share of all cores.10 minThe host's CPU over its last ten minutes.Busiest coreThe busiest single core's use at its highest.MemoryMemory used at its highest, of the machine's total.Net in / out MB/sThe host's network traffic in and out, in MB a second.DiskThe fullest disk the run writes to, at its fullest.StateWhere it stands now.
control—87%91%19%32.17 / 58.1712%control

Range: 24h 7d (UTC, by the hour)

Hosts over time

Hosts that ran during each hour, by role.

1.0011-07 11:00 UTC; heads 0; workers 0; control 111-07 12:00 UTC; heads 0; workers 0; control 111-07 13:00 UTC; heads 0; workers 0; control 111-07 14:00 UTC; heads 0; workers 0; control 111-07 15:00 UTC; heads 0; workers 0; control 111-07 16:00 UTC; heads 0; workers 0; control 111-07 17:00 UTC; heads 0; workers 0; control 111-07 18:00 UTC; heads 0; workers 0; control 111-07 19:00 UTC; heads 0; workers 0; control 111-07 20:00 UTC; heads 0; workers 0; control 111-07 21:00 UTC; heads 0; workers 0; control 111-07 22:00 UTC; heads 0; workers 0; control 111-07 23:00 UTC; heads 0; workers 0; control 111-08 00:00 UTC; heads 0; workers 0; control 111-08 01:00 UTC; heads 0; workers 0; control 111-08 02:00 UTC; heads 0; workers 0; control 111-08 03:00 UTC; heads 0; workers 0; control 111-08 04:00 UTC; heads 0; workers 0; control 111-08 05:00 UTC; heads 0; workers 0; control 111-08 06:00 UTC; heads 0; workers 0; control 111-08 07:00 UTC; heads 0; workers 0; control 111-08 08:00 UTC; heads 0; workers 0; control 111-08 09:00 UTC; heads 0; workers 0; control 111-08 10:00 UTC; heads 0; workers 0; control 111:0017:0023:0005:00

heads   workers   control  

Runs per hour

Runs started and runs that ended each hour; failures (failed runs, and every failed step attempt, even one a retry fixed) are overlaid on the finished bar.

1000011-07 11:00 UTC; started 0; finished 0; failures (runs and attempts) 011-07 12:00 UTC; started 15; finished 15; failures (runs and attempts) 111-07 13:00 UTC; started 27; finished 27; failures (runs and attempts) 111-07 14:00 UTC; started 14; finished 15; failures (runs and attempts) 011-07 15:00 UTC; started 19; finished 19; failures (runs and attempts) 011-07 16:00 UTC; started 18; finished 18; failures (runs and attempts) 011-07 17:00 UTC; started 20; finished 20; failures (runs and attempts) 011-07 18:00 UTC; started 20; finished 20; failures (runs and attempts) 011-07 19:00 UTC; started 20; finished 20; failures (runs and attempts) 011-07 20:00 UTC; started 22; finished 762; failures (runs and attempts) 011-07 21:00 UTC; started 53; finished 52; failures (runs and attempts) 011-07 22:00 UTC; started 42; finished 42; failures (runs and attempts) 011-07 23:00 UTC; started 64; finished 63; failures (runs and attempts) 011-08 00:00 UTC; started 68; finished 69; failures (runs and attempts) 011-08 01:00 UTC; started 62; finished 62; failures (runs and attempts) 011-08 02:00 UTC; started 61; finished 60; failures (runs and attempts) 011-08 03:00 UTC; started 91; finished 92; failures (runs and attempts) 011-08 04:00 UTC; started 252; finished 249; failures (runs and attempts) 1311-08 05:00 UTC; started 134; finished 135; failures (runs and attempts) 011-08 06:00 UTC; started 145; finished 145; failures (runs and attempts) 011-08 07:00 UTC; started 143; finished 143; failures (runs and attempts) 011-08 08:00 UTC; started 138; finished 138; failures (runs and attempts) 011-08 09:00 UTC; started 133; finished 133; failures (runs and attempts) 011-08 10:00 UTC; started 104; finished 104; failures (runs and attempts) 111:0017:0023:0005:00

started   finished   failures (runs and attempts)  

Fleet CPU

The vCPUs of all sampled hosts: busy, and available.

5.0011-07 11:00 UTC; vCPUs busy 0.13; vCPUs available 4.0011-07 12:00 UTC; vCPUs busy 0.81; vCPUs available 4.0011-07 13:00 UTC; vCPUs busy 1.97; vCPUs available 4.0011-07 14:00 UTC; vCPUs busy 1.62; vCPUs available 3.9911-07 15:00 UTC; vCPUs busy 2.21; vCPUs available 4.0011-07 16:00 UTC; vCPUs busy 2.16; vCPUs available 4.0011-07 17:00 UTC; vCPUs busy 2.23; vCPUs available 4.0011-07 18:00 UTC; vCPUs busy 2.22; vCPUs available 4.0011-07 19:00 UTC; vCPUs busy 2.22; vCPUs available 4.0011-07 20:00 UTC; vCPUs busy 2.22; vCPUs available 4.0011-07 21:00 UTC; vCPUs busy 2.33; vCPUs available 4.0011-07 22:00 UTC; vCPUs busy 2.32; vCPUs available 4.0011-07 23:00 UTC; vCPUs busy 3.61; vCPUs available 4.0011-08 00:00 UTC; vCPUs busy 3.63; vCPUs available 4.0011-08 01:00 UTC; vCPUs busy 3.61; vCPUs available 4.0011-08 02:00 UTC; vCPUs busy 3.64; vCPUs available 4.0011-08 03:00 UTC; vCPUs busy 3.52; vCPUs available 3.9911-08 04:00 UTC; vCPUs busy 3.69; vCPUs available 4.0011-08 05:00 UTC; vCPUs busy 3.43; vCPUs available 4.0011-08 06:00 UTC; vCPUs busy 3.54; vCPUs available 4.0011-08 07:00 UTC; vCPUs busy 3.49; vCPUs available 4.0011-08 08:00 UTC; vCPUs busy 3.53; vCPUs available 4.0011-08 09:00 UTC; vCPUs busy 3.58; vCPUs available 4.0011-08 10:00 UTC; vCPUs busy 3.54; vCPUs available 3.9711:0017:0023:0005:00

vCPUs busy   vCPUs available   vCPUs

Fleet network

Throughput summed over all sampled hosts.

200011-07 11:00 UTC; in (MB/s) 1.40; out (MB/s) 2.4911-07 12:00 UTC; in (MB/s) 12.17; out (MB/s) 15.6711-07 13:00 UTC; in (MB/s) 27.54; out (MB/s) 35.6211-07 14:00 UTC; in (MB/s) 20.57; out (MB/s) 27.3411-07 15:00 UTC; in (MB/s) 28.26; out (MB/s) 36.9611-07 16:00 UTC; in (MB/s) 27.63; out (MB/s) 36.4511-07 17:00 UTC; in (MB/s) 28.68; out (MB/s) 37.4811-07 18:00 UTC; in (MB/s) 29.04; out (MB/s) 38.1111-07 19:00 UTC; in (MB/s) 29.53; out (MB/s) 38.8711-07 20:00 UTC; in (MB/s) 29.95; out (MB/s) 39.3611-07 21:00 UTC; in (MB/s) 31.92; out (MB/s) 42.1011-07 22:00 UTC; in (MB/s) 31.17; out (MB/s) 41.5511-07 23:00 UTC; in (MB/s) 50.37; out (MB/s) 66.9211-08 00:00 UTC; in (MB/s) 50.45; out (MB/s) 67.0411-08 01:00 UTC; in (MB/s) 50.64; out (MB/s) 67.4911-08 02:00 UTC; in (MB/s) 52.48; out (MB/s) 70.2811-08 03:00 UTC; in (MB/s) 50.88; out (MB/s) 76.4911-08 04:00 UTC; in (MB/s) 59.30; out (MB/s) 113.2611-08 05:00 UTC; in (MB/s) 50.08; out (MB/s) 94.7211-08 06:00 UTC; in (MB/s) 51.49; out (MB/s) 97.2111-08 07:00 UTC; in (MB/s) 51.39; out (MB/s) 97.1211-08 08:00 UTC; in (MB/s) 53.96; out (MB/s) 101.9711-08 09:00 UTC; in (MB/s) 53.53; out (MB/s) 101.3611-08 10:00 UTC; in (MB/s) 52.49; out (MB/s) 99.1711:0017:0023:0005:00

in (MB/s)   out (MB/s)   MB/s