orcaLog in

Pipelines

gharchive

Progress

Runs at onceRuns of this pipeline going now, out of the most the scheduler starts at once (max_runs); the others wait in the queue. Fill runs come on top, up to fill_max_runs. Docs ↗
2 running of 1, 0 queued
FillWith fills = "all" in orca.toml the scheduler works through the whole range by itself, one partition per fill run, within a daily budget and a window. This is how far it got, how fast, and what it spent today. Docs ↗
1003 / 1407 done (71%), 2 fill runs going, 2 in live runs; 63.9 an hour, ETA 6h 19m; today $0.00, no budget (estimated)
Idle: at fill_max_runs: 2 of 2 going
Next runWhen the schedule fires next, in UTC, and the partition that run fills. Next runs, further down, lists the next five. Docs ↗
2026-11-09T03:00:00Z, fills 2026-11-08
CoverageWhich days or months a run has done, failed or not reached yet, as a calendar on the Operate tab. Docs ↗
The calendar

Config

PartitionsHow the work is split: each run does some partitions (months, days, hours or named keys), from the first one on. Backfills and the coverage calendar count in partitions. Docs ↗
daily from 2023-01-01
ScheduleThe cron line the scheduler fires on, in its time zone, and the partition each fire runs. None means runs start only from a sensor, a backfill or a person. Docs ↗
0 3 * * * UTC, fills the previous partition
Run retriesHow many more times the scheduler starts a run that failed before leaving it failed for a person. A step's own retries, inside a run, come first. Docs ↗
3
PlacementWhere its runs go: control is the scheduler's own host, ec2 an instance launched for the run, worker:<pool> a remote orca worker of that pool, and default the scheduler's runner (ORCA_RUNNER). Docs ↗
default (the scheduler's runner)
BinaryThe program each run starts: the pipeline's Mojo code built into one executable, as orca register recorded it. Docs ↗
/opt/orca/pipelines/gharchive

CodeThe functions this pipeline is made of, at the commit it's registered with. Hover a name to see its source; click it to open the file on its host. Docs ↗

PipelineWhere the pipeline is declared: its Pipeline[...] line. Docs ↗
gharchive
StepsEach step's function, in the order the steps run. The Steps table below has what each reads and writes. Docs ↗
load_day
TasksThe dista tasks its steps fan out to, each a function. Docs ↗
index_hour · hour_task · merge_repo_hour

Code hash 0xe6cdc4870d3f3535A hash of what the binary says about itself: its steps, partitions and schedule. It changes when they do, not on every build. Docs ↗

ResourcesWhat its latest runs used, as the median (p50) and the most of them: what a run of it needs. A run's peak is its attempts' largest, its CPU their sum. Docs ↗

MeasuredHow many of its latest runs these are: the last 20 that recorded what they used.
its last 20 runs
Peak memoryThe most memory its processes held at once: the run's process and what it started (dista's local workers too), sampled every second by the scheduler or worker that started it, and at the end the largest single process. Not measured on EC2 or Kubernetes. Docs ↗
p50 810 MB · max 930 MB
CPU usedThe CPU time its processes used, user and system, and the most cores they kept busy over one sample's second. macOS gives only the time, and only for a process that made one attempt. Docs ↗
p50 194.2 s · max 233.1 s
Peak coresThe most cores a run's processes kept busy over one sample's second: the CPUs to ask for. Not measured on macOS. Docs ↗
p50 2.9 · max 3.2

Steps docs

GraphTable
StepA step of the pipeline: a function that runs once per partition (mapped) or once over all of them (gather).KindHow the step runs: mapped, once per partition, or gather, once over all of them. Hover a kind for more.AfterThe steps this one waits for: it starts once they have succeeded.ReadsThe datasets the step reads.WritesThe datasets the step commits to.
load_day pipelines/gharchive.mojo:135mappedgharchive.events, gharchive.candidates, gharchive.repo_hour, gharchive.releases

Next runs docs

UTCWhen the schedule fires, in UTC.Wall clockThe same moment in the pipeline's time zone.FillsThe partition that run fills.
2026-11-09T03:00:00Z2026-11-09T03:00:002026-11-08
2026-11-10T03:00:00Z2026-11-10T03:00:002026-11-09
2026-11-11T03:00:00Z2026-11-11T03:00:002026-11-10
2026-11-12T03:00:00Z2026-11-12T03:00:002026-11-11
2026-11-13T03:00:00Z2026-11-13T03:00:002026-11-12

Configuration

Registered from /tmp/orca.toml @ ac0c766d3f15 of https://github.com/magmalake/mojo-census, sha256 9ac40fcb5922.

KeyThe setting's name, as orca.toml and orca register's flags call it. Hover the name for what it sets.ValueThe value it is registered with.FromWhere the value came from: the code's default, orca.toml (and which table), a flag of orca register, or the scheduler's environment. A link opens the line that sets it.
placementWhere its runs start: default (the scheduler's runner), control, ec2, or worker:<pool>.defaultcode default the scheduler's runner, local
head_typeThe EC2 instance type of a run's dista head.c7g.largecode default the ec2 runner's
worker_typeThe EC2 instance type of each dista worker.c7g.xlargecode default the ec2 runner's
workersHow many dista workers a run asks for, or min..max to scale between.8code default the pipeline's code the Pipeline[...] line
worker_procsDista worker processes on each worker instance; auto for one per core.1code default
max_runsThe most runs of the pipeline the scheduler starts at once.1orca.toml [defaults]
fillsall: the scheduler fills the whole range by itself. since_registration: from registration on.allorca.toml [pipelines.gharchive]
fill_orderWhich end of the range the fill starts from.oldestorca.toml [pipelines.gharchive]
fill_max_runsFill runs at once, on top of max_runs.2orca.toml [pipelines.gharchive]
fill_budget_per_dayThe most the fill may spend a day, in dollars.nonecode default fill runs on ec2 or kubernetes don't start without one
fill_windowThe hours of the day, UTC, in which fill runs start.0..24code default all day
env.GHARCHIVE_EVENTSA variable set in its runs' environment.offorca.toml [pipelines.gharchive]
env.DISTA_WORKER_SLOTSA variable set in its runs' environment.2orca.toml [pipelines.gharchive]
env.GHARCHIVE_PARALLELA variable set in its runs' environment.2orca.toml [pipelines.gharchive]
env.ORCA_PUBLIC_WAREHOUSEA variable set in its runs' environment.s3://census-mlake-dev/warehouseenvironment ORCA_RUN_ENV