orcaLog in

Pipelines

mojo_repos

Progress

Runs at onceRuns of this pipeline going now, out of the most the scheduler starts at once (max_runs); the others wait in the queue. Fill runs come on top, up to fill_max_runs. Docs ↗
1 running of 1, 0 queued
FollowingA sensor runs the partitions of the tables it reads as they get snapshots. This is how many of those partitions it has done, and the ones it hasn't: going, failed, or not queued yet (a backfill queues them). They are behind on Coverage. Docs ↗
gharchive (gharchive.candidates): 632 of 1004 days done; 372 behind: 1 running or queued and 371 not queued yet
CoverageWhich days or months a run has done, failed or not reached yet, as a calendar on the Operate tab. Docs ↗
The calendar

Config

PartitionsHow the work is split: each run does some partitions (months, days, hours or named keys), from the first one on. Backfills and the coverage calendar count in partitions. Docs ↗
daily from 2023-01-01
ScheduleThe cron line the scheduler fires on, in its time zone, and the partition each fire runs. None means runs start only from a sensor, a backfill or a person. Docs ↗
none
Run retriesHow many more times the scheduler starts a run that failed before leaving it failed for a person. A step's own retries, inside a run, come first. Docs ↗
3
PlacementWhere its runs go: control is the scheduler's own host, ec2 an instance launched for the run, worker:<pool> a remote orca worker of that pool, and default the scheduler's runner (ORCA_RUNNER). Docs ↗
default (the scheduler's runner)
BinaryThe program each run starts: the pipeline's Mojo code built into one executable, as orca register recorded it. Docs ↗
/opt/orca/pipelines/mojo_repos

CodeThe functions this pipeline is made of, at the commit it's registered with. Hover a name to see its source; click it to open the file on its host. Docs ↗

PipelineWhere the pipeline is declared: its Pipeline[...] line. Docs ↗
mojo_repos
StepsEach step's function, in the order the steps run. The Steps table below has what each reads and writes. Docs ↗
find_repos

Code hash 0x627ffec531716e6eA hash of what the binary says about itself: its steps, partitions and schedule. It changes when they do, not on every build. Docs ↗

ResourcesWhat its latest runs used, as the median (p50) and the most of them: what a run of it needs. A run's peak is its attempts' largest, its CPU their sum. Docs ↗

MeasuredHow many of its latest runs these are: the last 20 that recorded what they used.
its last 20 runs
Peak memoryThe most memory its processes held at once: the run's process and what it started (dista's local workers too), sampled every second by the scheduler or worker that started it, and at the end the largest single process. Not measured on EC2 or Kubernetes. Docs ↗
p50 102 MB · max 104 MB
CPU usedThe CPU time its processes used, user and system, and the most cores they kept busy over one sample's second. macOS gives only the time, and only for a process that made one attempt. Docs ↗
p50 15.0 s · max 15.8 s
Peak coresThe most cores a run's processes kept busy over one sample's second: the CPUs to ask for. Not measured on macOS. Docs ↗
p50 0.4 · max 0.5

Steps docs

GraphTable
StepA step of the pipeline: a function that runs once per partition (mapped) or once over all of them (gather).KindHow the step runs: mapped, once per partition, or gather, once over all of them. Hover a kind for more.AfterThe steps this one waits for: it starts once they have succeeded.ReadsThe datasets the step reads.WritesThe datasets the step commits to.
find_repos pipelines/mojo_repos.mojo:191mappedgharchive.candidatesmojo.repos, mojo.checked

Configuration

Registered from /tmp/orca.toml @ ac0c766d3f15 of https://github.com/magmalake/mojo-census, sha256 9ac40fcb5922.

KeyThe setting's name, as orca.toml and orca register's flags call it. Hover the name for what it sets.ValueThe value it is registered with.FromWhere the value came from: the code's default, orca.toml (and which table), a flag of orca register, or the scheduler's environment. A link opens the line that sets it.
placementWhere its runs start: default (the scheduler's runner), control, ec2, or worker:<pool>.defaultcode default the scheduler's runner, local
head_typeThe EC2 instance type of a run's dista head.c7g.largecode default the ec2 runner's
worker_typeThe EC2 instance type of each dista worker.c7g.xlargecode default the ec2 runner's
workersHow many dista workers a run asks for, or min..max to scale between.0code default the pipeline has no tasks
worker_procsDista worker processes on each worker instance; auto for one per core.1code default
max_runsThe most runs of the pipeline the scheduler starts at once.1orca.toml [defaults]
fillsall: the scheduler fills the whole range by itself. since_registration: from registration on.since_registrationcode default
secrets.GITHUB_TOKENA secret put in its runs' environment, by its id in the secret store; its value is never shown.orca/github-tokenorca.toml [pipelines.mojo_repos]
env.ORCA_PUBLIC_WAREHOUSEA variable set in its runs' environment.s3://census-mlake-dev/warehouseenvironment ORCA_RUN_ENV