Docs / The model / ScrapeRun

ScrapeRun.

One run of an extraction template or of an inline scope: pages walked and denied, rows made, bytes, cost and time, with partial results at block boundaries; a whole site's crawl from its sitemap is one run

Figure 1ScrapeRunspecified

ScrapeRun is a work at follows scope. The drawing shows its fields and the objects that they point at.

kind: work
scope: follows
key: the template’s version and the scope’s hash
store: postgres
family: sources

Fields#

field	type	required	note
`template`	`id`	yes
`template_version`	`count`	yes
`scope_hash`	`hash`	yes
`pages_walked`	`count`	yes
`pages_refused`	`count`	yes
`rows_made`	`count`	yes
`bytes`	`count`	yes
`cost`	Cost	yes
`failure`	FailureClass	no
`hops`	HopStamp	yes
`request`	ScrapeRequest	yes	the request the run carries out; the beat runs it from this field alone
`pages_found`	`count`	yes	the pages the sitemaps and the seeds gave the frontier
`gaps`	CoverageGap	yes	one gap per page that could not be read, with its reason, and one for the pages past the budget

Routes that use it#

/v1/act/crawl returns it
/v1/act/set_aside_run returns it