SourceProcedure.
How a source is read, derived from its row and its last reading: how it opens, every table with its count, population and control, the joins, the lineage and the pitfalls
Figure 1SourceProcedurespecified
- kind
- row
- scope
- platform
- key
- the source and the reading
- store
- postgres
- family
- sources
Fields#
| field | type | required | note |
|---|---|---|---|
source | id | yes | |
kind | one of lake | catalogue | folder of files | hypergraph | embedding store | predicate set | derived table | yes | one row for every kind |
path | text | yes | |
hash | hash | yes | the file at the last reading |
generator | text | yes | the program that made it, with its path and commit |
engine | text | yes | |
opens | text | yes | the statement that attaches it read-only |
tables | SourceTable | yes | every table of the artefact with its count, population and control |
joins | SourceJoin | yes | to other sources, by identifier |
pitfalls | [text] | yes | each with the reading that showed it |
read_at | time | yes | |
stale | bool | yes | past the parameter’s age |
version | count | yes | |
state | Readiness | no | ready when the artefact opened and every table counted; not ready with the reason, such as a missing file or a table with no control; absent on a procedure not yet read |
bytes | bytes | no | the size of the file, or of every file of the folder, at the reading; a changed size says the artefact was rebuilt without hashing it again |
modified | time | no | when the file, or the newest file of the folder, was last changed, at the reading; a later time says the artefact was rebuilt without hashing it again |
Routes that use it#
/v1/read/source_procedurereturns it