Usage

Simple specialized pipeline library - probably not useful to many.

Synopsis

❯ putki

 Usage: putki [OPTIONS] COMMAND [ARGS]...

 Pipeline (Finnish: putki) - discovering and executing a specific task description.

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮
│ --version  -V        Display the application version and exit                                   │
│ --help     -h        Show this message and exit.                                                │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────╮
│ structures       Verify the structure definition against the file system.                       │
│ tasks            Verify the structure definition against the file system.                       │
│ version          Display the application version and exit.                                      │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯

Structures

Given a location that does contain structures pointing to structures pointing to structure files with complete existing artifacts:

❯ python -m putki structures -d example/s --component-folder-name c --verbose 2>&1 | cut -c 34-
INFO [PUTKI]: Verifying structures below example/s
INFO [PUTKI]: Structures from structures.yml(root):
INFO [PUTKI]: - x:
INFO [PUTKI]:   + d:
INFO [PUTKI]:     * f:
INFO [PUTKI]:       approvals -> approvals.yml
INFO [PUTKI]:       bind      -> bind.txt
INFO [PUTKI]:       changes   -> changes.yml
INFO [PUTKI]:       meta      -> meta.yml
INFO [PUTKI]:       render    -> True
INFO [PUTKI]:   + r:
INFO [PUTKI]:     * f:
INFO [PUTKI]:       approvals -> approvals.yml
INFO [PUTKI]:       bind      -> bind.txt
INFO [PUTKI]:       changes   -> changes.yml
INFO [PUTKI]:       meta      -> meta.yml
INFO [PUTKI]:       render    -> True
INFO [PUTKI]: - y:
INFO [PUTKI]:   + d:
INFO [PUTKI]:     * f:
INFO [PUTKI]:       approvals -> approvals.yml
INFO [PUTKI]:       bind      -> bind.txt
INFO [PUTKI]:       changes   -> changes.yml
INFO [PUTKI]:       meta      -> meta.yml
INFO [PUTKI]:       render    -> True
INFO [PUTKI]:   + r:
INFO [PUTKI]:     * f:
INFO [PUTKI]:       approvals -> approvals.yml
INFO [PUTKI]:       bind      -> bind.txt
INFO [PUTKI]:       changes   -> changes.yml
INFO [PUTKI]:       meta      -> meta.yml
INFO [PUTKI]:       render    -> True
INFO [PUTKI]: Components (from folders):
INFO [PUTKI]: - x
INFO [PUTKI]: - y
INFO [PUTKI]: Verifying any component structures not declared in root level structures.yml
INFO [PUTKI]: OK - no missing declarations

Structures - Help

(WIP)

❯ putki structures --help

 Usage: putki structures [OPTIONS] [DOC_ROOT_POS]

 Verify the structure definition against the file system.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────╮
│   doc_root_pos      [DOC_ROOT_POS]                                                              │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮
│ --document-root          -d      TEXT  Root of the document tree to visit. Optional (default:   │
│                                        positional tree root value)                              │
│ --structures                     TEXT  structures mapping file (default:                        │
│                                        {DEFAULT_STRUCTURES_NAME})                               │
│                                        [default: structures.yml]                                │
│ --component-folder-name          TEXT  component folder name (default: component)               │
│                                        [default: component]                                     │
│ --verbose                -v            Verbose output (default is False)                        │
│ --strict                 -s            Ouput noisy warnings on console (default is False)       │
│ --help                   -h            Show this message and exit.                              │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯

Tasks

Given a location that does not contain tasks files:

❯ putki tasks -d putki --verbose
2023-10-08T19:27:17.855331+00:00 DEBUG [git.cmd]: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/some/where/putki, universal_newlines=False, shell=None, istream=None)
2023-10-08T19:27:17.865118+00:00 INFO [PUTKI]: Identified tasks default root at /some/where/putki/example/basic/tasks
2023-10-08T19:27:17.865504+00:00 ERROR [PUTKI]: No tasks files found

Given a location that does indeed contain tasks files:

❯ python -m putki tasks -d example/minimal-tree/tasks/wun --verbose 2>&1 | cut -c 34-
DEBUG [git.cmd]: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/some/where/putki, universal_newlines=False, shell=None, istream=None)
INFO [PUTKI]: Identified tasks default root at /some/where/putki/example/basic/tasks
INFO [PUTKI]: Mapped tasks below specified root at example/minimal-tree/tasks/wun
INFO [PUTKI]: The 1 tasks files collected below specified root at example/minimal-tree/tasks/wun are:
INFO [PUTKI]: - example/minimal-tree/tasks/wun
INFO [PUTKI]: Collected the following 2 tasks from 1 tasks files:
INFO [PUTKI]: id: /wun
INFO [PUTKI]: source:
INFO [PUTKI]:   path: /local/path/to/another/root
INFO [PUTKI]: ---
INFO [PUTKI]: branch: another-branch-name
INFO [PUTKI]: discover: true
INFO [PUTKI]: id: /two
INFO [PUTKI]: source:
INFO [PUTKI]:   path: git@example.com:orga/repo
INFO [PUTKI]: target:
INFO [PUTKI]:   globs:
INFO [PUTKI]:   - structure.yml
INFO [PUTKI]:   name: structures.yml
INFO [PUTKI]:   root: local/path/from/repo/root/to/folder/that/should/host/a/liitos/structures/file
INFO [PUTKI]:
INFO [PUTKI]: OK

Tasks - Help

(WIP)

❯ putki tasks --help

 Usage: putki tasks [OPTIONS] [DOC_ROOT_POS]

 Verify the structure definition against the file system.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────╮
│   doc_root_pos      [DOC_ROOT_POS]                                                              │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮
│ --document-root  -d      TEXT  Root of the document tree to visit. Optional (default:           │
│                                positional tree root value)                                      │
│ --structure      -s      TEXT  structure mapping file (default: {DEFAULT_STRUCTURE_NAME})       │
│                                [default: structure.yml]                                         │
│ --target         -t      TEXT  target document key                                              │
│ --facet          -f      TEXT  facet key of target document                                     │
│ --verbose        -v            Verbose output (default is False)                                │
│ --strict         -s            Ouput noisy warnings on console (default is False)               │
│ --help           -h            Show this message and exit.                                      │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯

Version

Asking for the version:

❯ putki version
Pipeline (Finnish: putki) - discovering and executing a specific task description. version 2023.10.8+parent.gd4187e22

Version - Help

For completeness:

❯ putki version --help

 Usage: putki version [OPTIONS]

 Display the application version and exit.

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮
│ --help  -h        Show this message and exit.                                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯

Discovery

Given a git repository URL discover the tasks pointing to other local or remote git repositories, branches, and folder trees.

Convention suggests a top level /tasks folder in the disptach repository to recursively collect the tasks.yml files defining the tasks.

The order of execution is parallel per sub folders of the /tasks folder and lexically sequential per tasks files entries.

The resulting tasks file will maintain the uniqueness of the collected ids by prefixing with a path.

Example: Collecting task with id top in top level tasks file as well as another task with id top from a tasks file at /tasks/some/other/tasks.yml results in the two task ids:

- id: '/tasks/top'
- id: '/tasks/some/other/top'

The tasks.yml files offer the following example structure and shall adhere to the schema at /schema/1/tasks.json:

---
schema: https://git.sr.ht/~sthagen/putki/blob/default/schema/1/tasks.json
tasks:
- id: wun
  source:
    path: "/local/path/to/root"
- id: two
  source:
    path: git@example.com:orga/repo
  branch: another-branch-name
  target:
    root: local/path/from/repo/root/to/folder/that/should/host/a/liitos/structures/file
    name: structures.yml
    globs:
    - structure.yml
  discover: true
- id: three-complicated-kind-of
  source:
    path_elements:
      protocol: https://
      host: example.com
      port: 7999
      user:
      token:
      service_root: "/some/funny/path/"
      address_template: "{{protocol}}{{host}}:{{port}}{{service_root}}project/orga/repos/repo"
  branch: branch-name-too

The default values for the key branch is default. The discover value is assumed to be false per default.

The addressing is declared within a source object to cover two use cases:

1. For simple source addressing needs the path key shall be set to the local path or remote clone URL of a public repository.

2. Alternatively for more complex addressing the object path_elements is required providing the keys protocol, host, port, service_root, user, token, and address_template.

   The former keys are expected to be present in the latter key value (each inserted between the usual pairs of {{ and }}).

As an example for the second use case some on-prem server product ...

https://username@your.needbucket.domain:7999/yourproject/repo.git

... could be represented as:

---
schema: https://git.sr.ht/~sthagen/putki/blob/default/schema/1/tasks.json
tasks:
- id: fourth-complicated-kind-of
  source:
    path_elements:
      protocol: https://
      host: your.needbucket.domain
      port: 7999
      user: username
      token: MAGIC_PLACE
      service_root: "/"
      address_template: "{{protocol}}{{user}}@{{host}}:{{port}}{{service_root}}yourproject/repo.git"
  branch: another-branch-name-too

The putkiconfiguration provides the policy for processing upstream access credentials.

In case the executing process environment does not provide a token in the PUTKI_TOKEN_MAGIC_PLACE(derived from the task token key value by adding the PUTKI_TOKEN_ prefix, the processor may override the given user value with a functional local default user as fallback (using some locally available token or pass phrase).

The target object caters the use case when the targeted locations deviate from the convention to offer a structures.yml file in the top level directory of the folder tree that points to the individual structure.yml files within the tree.

The expectation is that these files provide structural information that the tools navigaattori and liitos understand.