orcaLog in

Pipelines

gharchive_index paused

Progress

Runs at onceRuns of this pipeline going now, out of the most the scheduler starts at once (max_runs); the others wait in the queue. Fill runs come on top, up to fill_max_runs. Docs ↗
0 running of 1, 0 queued
CoverageWhich days or months a run has done, failed or not reached yet, as a calendar on the Operate tab. Docs ↗
The calendar

Config

PartitionsHow the work is split: each run does some partitions (months, days, hours or named keys), from the first one on. Backfills and the coverage calendar count in partitions. Docs ↗
daily from 2023-01-01
ScheduleThe cron line the scheduler fires on, in its time zone, and the partition each fire runs. None means runs start only from a sensor, a backfill or a person. Docs ↗
none
Run retriesHow many more times the scheduler starts a run that failed before leaving it failed for a person. A step's own retries, inside a run, come first. Docs ↗
3
PlacementWhere its runs go: control is the scheduler's own host, ec2 an instance launched for the run, worker:<pool> a remote orca worker of that pool, and default the scheduler's runner (ORCA_RUNNER). Docs ↗
default (the scheduler's runner)
BinaryThe program each run starts: the pipeline's Mojo code built into one executable, as orca register recorded it. Docs ↗
/opt/orca/pipelines/gharchive_index

CodeThe functions this pipeline is made of, at the commit it's registered with. Hover a name to see its source; click it to open the file on its host. Docs ↗

PipelineWhere the pipeline is declared: its Pipeline[...] line. Docs ↗
gharchive_index
StepsEach step's function, in the order the steps run. The Steps table below has what each reads and writes. Docs ↗
index_day
TasksThe dista tasks its steps fan out to, each a function. Docs ↗
index_hour

Code hash 0xdac1ead002f9378cA hash of what the binary says about itself: its steps, partitions and schedule. It changes when they do, not on every build. Docs ↗

ResourcesWhat its latest runs used, as the median (p50) and the most of them: what a run of it needs. A run's peak is its attempts' largest, its CPU their sum. Docs ↗

None of its runs has been measured yet: the scheduler measures the runs it starts here (local and control-placed), a worker the ones it runs; runs on EC2 and Kubernetes aren't.

Steps docs

GraphTable
StepA step of the pipeline: a function that runs once per partition (mapped) or once over all of them (gather).KindHow the step runs: mapped, once per partition, or gather, once over all of them. Hover a kind for more.AfterThe steps this one waits for: it starts once they have succeeded.ReadsThe datasets the step reads.WritesThe datasets the step commits to.
index_day pipelines/gharchive_index.mojo:51mappedgharchive.eventsgharchive.candidates, gharchive.repo_hour, gharchive.releases

Configuration

Registered from /tmp/orca.toml @ ac0c766d3f15 of https://github.com/magmalake/mojo-census, sha256 9ac40fcb5922.

KeyThe setting's name, as orca.toml and orca register's flags call it. Hover the name for what it sets.ValueThe value it is registered with.FromWhere the value came from: the code's default, orca.toml (and which table), a flag of orca register, or the scheduler's environment. A link opens the line that sets it.
placementWhere its runs start: default (the scheduler's runner), control, ec2, or worker:<pool>.defaultcode default the scheduler's runner, local
head_typeThe EC2 instance type of a run's dista head.c7g.largecode default the ec2 runner's
worker_typeThe EC2 instance type of each dista worker.c7g.xlargecode default the ec2 runner's
workersHow many dista workers a run asks for, or min..max to scale between.8code default the pipeline's code the Pipeline[...] line
worker_procsDista worker processes on each worker instance; auto for one per core.1code default
max_runsThe most runs of the pipeline the scheduler starts at once.1orca.toml [defaults]
fillsall: the scheduler fills the whole range by itself. since_registration: from registration on.since_registrationcode default
env.DISTA_WORKER_SLOTSA variable set in its runs' environment.1orca.toml [pipelines.gharchive_index]
env.ORCA_PUBLIC_WAREHOUSEA variable set in its runs' environment.s3://census-mlake-dev/warehouseenvironment ORCA_RUN_ENV