Skip to main content

Creating Use Case Templates

To create your template, you will need to define two files:

  • A YAML file that configures the fields that a user needs to fill when creating a new component that the scaffolder wizard will render (i.e. template.yaml)
  • All the files and folders that the component created by this template will be made of; we will refer to them as _ skeleton files_. Inside the skeleton directory, there must be at least a YAML file containing the definitions of the component that your template will scaffold (i.e. catalog-info.yaml)

Those files together with all files and folders related to a template must be stored in your configured Git provider, the one configured within witboost (e.g. GitLab). This allows witboost to register your template by using the repository URL, pointing to the catalog-info.yaml file.

A common folder structure for templates is as follows:

template.yaml
mkdocs.yml
docs/
├──── index.md
skeleton/
├──────── README.md
├──────── catalog-info.yaml
├──────── mkdocs.yml
├──────── .gitlab-ci.yaml   <--- CI/CD pipeline which will be executed by Use case templates
├──────── docs/
│         └──── index.md
├──────── environments/
│         ├──────────── dev/
│         │             └─── configurations.yaml
│         ├──────────── prod/
│         │             └──── configurations.yaml
│         └──────────── # any other env config goes here
└──────── # any other skeleton files here
tip

All configurations.yaml files found under environments/<env_name> will be part of the full Data Product descriptor and will be visible under the components[i].configuration key. Where i is the corresponding index of the component that you are building. configurations.yaml acts as environment-specific configurations for your components.

Step one. template.yaml Definition​

Templates are entities, and as such, they can be stored inside the Catalog. This is why they must be compliant with a well-defined structure.

A template definition describes both the parameters that are rendered in the frontend part of the scaffolding wizard and the steps that are executed when scaffolding that component.

A template.yaml definition generally contains these sections, as shown in the example below:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
# data about the template, useful for the catalog

spec:
  parameters:
  # defines the form fields for the user to insert the data required by the component

  steps:
  # steps performed by the scaffolder engine to effectively create your component

  output:
  # some extra metadata for the scaffolder actions

Let's breakdown more those sections, leveraging an example template that we created.

The metadata section defines information about the template that is meaningful for the Catalog, for display purposes:

apiVersion: backstage.io/v1beta2 # internal, can be leaved as it is
kind: Template # kind will be always `template` when defining a template.yaml
metadata:
  name: example-template # unique id for this template inside the catalog
  title: Example Template # display name
  description: Template definition of an Example Template
  mesh:
    icon: https://<icon_url>.png # icon appearing inside the catalog when displaying this template
  annotations:
    backstage.io/techdocs-ref: dir:. # this is used by TechDocs to know where to look for docs, can be leaved as that
  tags: # tags meaningful for users, they can be used for filtering templates inside the catalog
    - example

The spec section defines:

  • template owner (spec.owner) and the type of the scaffolded entity (spec.type). The spec.type field of the template is used in the UI to group templates on the "Template" page (templates with the same type field will be grouped). You can put types in camel case and they will be displayed using spaces in the UI groupings.
  • the parameters (spec.parameters) that are presented to the user in a form, detailing the mandatory/optional ones (" required")
spec:
  owner: agilelab
  type: component # or any other available type
  parameters:
    - title: Example Page # title of the scaffolder page
      required:
        - fieldone
        - fieldtwo
      properties:
        fieldone:
          title: Name
          type: string
          description: Name of the example component
        fieldtwo:
          title: Owner
          type: string
          description: Owner of the example component
          ui:field: EntityPicker # a custom picker
          ui:options: # with custom options values
            allowArbitraryValues: false
            allowedKinds:
              - User
        fieldthree:
          title: Description
          type: string
          description: Description of the example component

    - title: Example Page for Location Selection
      required:
        - repoUrl
      properties:
        repoUrl:
          title: Repository Location
          type: string
          ui:field: RepoUrlPicker # another custom picker
          ui:options:
            allowedHosts:
              - gitlab.com

Moreover, spec also defines:

  • Steps performed by the scaffolder
  • Values spec.steps[template].values passed to all files in the skeleton (including the catalog-info.yaml) when a new component is cloned. So that inside the skeleton you can refer to those values with the assigned key ( e.g. parameters.fieldone)
  • spec.steps[template].useCaseTemplateId: It is the unique Id that will be used to register the Use Case Template to the Provisioning Coordinator
  • spec.steps[template].infrastructureTemplateId: It is the unique ID that will be used to register a Tech Adapter to the Provisioning Coordinator.
info

Remember: all URNs are treated as case-insensitive values, except for infrastructureTemplateId. This is deprecated, in the future it will be all case-insensitive.

spec:
  # we are still in the same section as in the above example
  steps:
    - id: template # fetches values from UI
      name: Fetch Skeleton + Template #
      action: fetch:template
      input:
        url: ./skeleton # this is used to locate catalog-info.yaml
        copyWithoutRender: # you can also specify files that will not be affected by variables replacement
          - .gitlab-ci.yml
        values:
          fieldone: '{{ parameters.fieldone }}'
          fieldtwo: '{{ parameters.fieldtwo }}'
          fieldthree: '{{ parameters.fieldthree }}'
          useCaseTemplateId: urn:dmb:utm:aws-cdp-outputport-impala-template:0.0.0 # Specify any well-formatted Id you desire
          infrastructureTemplateId: urn:dmb:itm:aws-cdp-outputport-impala-provisioner:1 # Specify any well-formatted Id you desire
          useCaseTemplateVersion: 0.0.0
    - id: publish # custom action that publishes the example component to GitLab
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: ['gitlab.com']
        description: 'This is {{ parameters.fieldone}}'
        repoUrl: '{{ parameters.repoUrl }}'

    - id: register # register the example component into the witboost catalog
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '{{ steps.publish.output.repoContentsUrl }}'
        catalogInfoPath: '/{{ parameters.rootDirectory }}/catalog-info.yaml'

  output:
    remoteUrl: '{{ steps.publish.output.remoteUrl }}'
    entityRef: '{{ steps.register.output.entityRef }}'
info

If you are not using Gitlab, as in the example, please refer to the Git Providers section below

template.yaml has some requirements to which it must be compliant:

  • Field metadata.title and metadata.description are mandatory fields
  • Field spec.steps.values.useCaseTemplateId: Must match the following URN format: urn:dmb:utm:{name}:{version} (note that is utm)
  • Field spec.steps.values.infrastructureTemplateId must match the URN identifier urn:dmb:itm:{name}:{version}
  • Field spec.steps.values.useCaseTemplateVersion must be not empty and should equal to the {version} in the useCaseTemplateId field.

Moreover:

  • allowed characters in {name} section are alphanumeric ([a-z] and/or [A-Z] and/or [0-9]) and/or dashes (-) and/or underscores (_) only
  • allowed format for {version} section must be a single positive number (e.g. 13) or a dot-separated sequence of three numbers (e.g. 1.11.16)
  • Use case template URN starts with urn:dmb:utm while an Infrastructure template URN starts with urn:dmb:itm
warning

Notice that the registration step will reject your request of creating a new component if such mandatory requirements are not fulfilled.

All input fields are defined in template.yaml will be fetched by the scaffolder and will be available to be referenced from the catalog-info.yaml and all files inside the skeleton folder using the parameters variable ( e.g. parameters.fieldone).

So, now that we have a template.yaml ready, we should now map all user inputs to catalog-info.yaml which will define the structure of your component.

Publish step​

The publishing step is in charge of creating the repository if that does not exist, and publishing all files and contents generated from the scaffolding phase into a destination SCM provider.

When integrating with an SCM provider (like Gitlab), the templates for all the entities handled by witboost must use specific actions for publication and registration. This is because each provider needs different input parameters, and the data used to perform the operations can differ ( e.g. for Bitbucket all the URLs must end with ?at={branchName}).

Gitlab​

An example of valid Gitlab template's actions in the template.yaml file looks like the following:

- id: publish
  name: Publish
  action: witboostMeshComponent:publish:gitlab
  input:
    allowedHosts: ['gitlab.com']
    description: 'This is ${{ parameters.name }}'
    repoUrl: '${{ parameters.repoUrl }}'
    rootDirectory: '${{ parameters.rootDirectory }}'
    dataproduct: '${{ parameters.name }}'

- id: register
  name: Register
  action: catalog:register
  input:
    repoContentsUrl: '${{ steps.publish.output.repoContentsUrl }}'
    catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

The publish action is a custom action that takes care of creating the repository for a data product on the Gitlab target repository. The inputs that must be passed, as shown above, are the target host, the description, the repository URL and root directory (extracted from the RepoUrlPicker), and the Data Product name. Please note that the extract above is valid for a Data Product template, while when used for a Component template the dataproduct line should be changed to refer to the Data Product chosen in the steps above (e.g. with an EntityPicker with name dataproduct):

dataproduct: '${{ parameters.dataproduct }}'

For other components the parentRef line should be used instead of dataproduct. The behaviour will be the same.

The register action simply takes the output of the publish action and registers the published repository as an entity in the database.

Bitbucket Server​

An example of valid on-premise Bitbucket template actions in the template.yaml file looks like the following:

- id: publish
  name: Publish
  action: witboostMeshComponent:publish:bitbucketServer
  input:
    allowedHosts: ['mybitbucket.com']
    description: 'This is ${{ parameters.name }}'
    repoUrl: '${{ parameters.repoUrl }}'
    rootDirectory: '${{ parameters.rootDirectory }}'
    dataproduct: '${{ parameters.name }}'

- id: register
  name: Register
  action: catalog:register
  input:
    catalogInfoUrl: '${{ steps.publish.output.catalogInfoUrl }}'

The publish action is a custom action that takes care of creating the repository for a data product on the on-premise Bitbucket target repository (you can see that the host does not reference the global bitbucket.org cloud). The inputs that must be passed, as shown above, are the target host, the description, the repository URL and root directory ( extracted from the RepoUrlPicker), and the Data Product name. Please note that the extract above is valid for a Data Product template, while when used for a Component template the dataproduct line should be changed to refer to the Data Product chosen in the steps above (e.g. with an EntityPicker with name dataproduct):

dataproduct: '${{ parameters.dataproduct }}'

The register action simply takes the output of the publish action and registers the published repository as an entity in the database. Note that the register action is the Backstage default one, so you can refer to the default documentation to change its behavior. Anyway, in this case, it is mandatory to use the URL generated by the publish step as input; this is because the URL must have the ?at={branchName} suffix, and the publish action already creates the URL that way.

Azure DevOps​

An example of valid Azure DevOps template actions in the template.yaml file looks like the following:

- id: publish
  name: Publish
  action: witboostMeshComponent:publish:azure
  input:
    allowedHosts: ['dev.azure.com']
    description: This is ${{ parameters.name }}
    rootDirectory: ${{ parameters.rootDirectory }}
    repoUrl: ${{ parameters.repoUrl }}
    dataproduct: '${{ parameters.name }}'

- id: register
  name: Register
  action: catalog:register
  input:
    catalogInfoUrl: '${{ steps.publish.output.catalogInfoUrl }}'

The publish action is a custom action that takes care of creating the repository for a data product on Azure target repository. The inputs that must be passed, as shown above, are the target host, the description, the repository URL and root directory ( extracted from the RepoUrlPicker), and the Data Product name. Please note that the extract above is valid for a Data Product template, while when used for a Component template the dataproduct line should be changed to refer to the Data Product chosen in the steps above (e.g. with an EntityPicker with name dataproduct):

- id: publish
  name: Publish
  action: witboostMeshComponent:publish:azure
  input:
    allowedHosts: ['dev.azure.com']
    description: This is ${{ parameters.name }}
    rootDirectory: ${{ parameters.rootDirectory }}
    repoUrl: ${{ parameters.repoUrl }}
    dataproduct: '${{ parameters.dataproduct }}'

The register action simply takes the output of the publish action and registers the published repository as an entity in the database.

Default values​

When creating a new data product or component from a template in Azure, you can specify some default values from the configuration, in order to customize the creation of the corresponding project, if not existent. The values are the following, to be put in the values.yaml:

mesh:
  builder:
    scaffolder:
      azure:
        defaultValues:
          projectDescription: test description
          projectVisibility: 1
          projectTemplateType: basic

where:

  • projectDescription: default description value when a new project is created by the action. If not provided, default is ''.
  • projectVisibility: default visibility value when a new project is created by the action. If not provided, default is Organization, with fallback to Private. Available visibilities are the following:
    • Private = 0 : The project is only visible to users with explicit access
    • Organization = 1 : Enterprise level project visibility
    • Public = 2 : The project is visible to all.
    • SystemPrivate = 3
  • projectTemplateType: default template type value when a new project is created by the action. If not provided, default is Basic. Available template types are:
    • basic : This template is flexible for any process and great for teams getting started with Azure DevOps.
    • agile : This template is flexible and will work great for most teams using Agile planning methods, including those practicing Scrum.
    • cmmi : This template is for more formal projects requiring a framework for process improvement and an auditable record of decisions.
    • scrum : This template is for teams who follow the Scrum framework.

Publishing in multiple repositories​

It is possible to publish to more than one repository starting from one template. In that case, you would probably want to create two separate fetch, publish and register phases. Here we are introducing the input.sourcePath value in the publish phase which is used to read from the folder specified in the input.targetPath of the fetch phase so these two variables must be an exact match. An example of this is:

  steps:
    - id: templateOne
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}'
        values:
          ...
          identifier: '${{ parameters.identifier }}One'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}One'
          ...
    - id: publishOne
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: [ 'gitlab.com' ]
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'

    - id: registerOne
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishOne.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

    - id: templateTwo
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}/two'
        values:
          ...
          identifier: '${{ parameters.identifier }}Two'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}Two'
          ...
    - id: publishTwo
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: [ 'gitlab.com' ]
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}Two'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'
        sourcePath: 'two'
    - id: registerTwo
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishTwo.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

  output:
    links:
      - title: Repository
        url: '${{ steps.publish.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.register.output.entityRef }}'
      - title: Repository
        url: '${{ steps.publishTwo.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.registerTwo.output.entityRef }}'
caution

Be really careful NOT to use special characters(for example -) in the id of the publishing phase!

Step two. catalog-info.yaml Definition​

The catalog-info.yaml file, that must be defined inside the skeleton folder, serves as a collector of all the metadata and the structure of what your template will become after cloning it.

This file contains variables, which are filled by the scaffolding phase (when a template is cloned). During this scaffolding phase, all variables are resolved by filling them with the frontend values taken from the user input (as defined in template.yaml).

You will notice that in our examples metadata.name field is structured as ${{ values.domain + "." + values.identifier + "." + values.version }}. That is because, as said above, it is important for the metadata.name field to be unique.

In the catalog-info.yaml you must add under the spec.mesh.specific field all the fields that your specific technology requires. You can leave them empty (the user will fill them directly in the repository) or read them from the UI (in that case you can use variables as done for the other "common" fields).

For the example above, we could define a catalog-info.yaml like this:

apiVersion: backstage.io/v1alpha1
kind: Component # this is one of the allowed catalog kinds, and should not be changed, since it will affect how entities are handled inside witboost
metadata:
  name: ${{ values.fieldone }}
  description: ${{ values.fieldthree }}

spec:
  # fixed fields
  type: component
  lifecycle: experimental
  owner: ${{ values.fieldtwo }}
  mesh:
    # custom fields
    componentOwner: ${{ values.fieldtwo }}
    useCaseTemplateId: ${{ values.useCaseTemplateId }}
    infrastructureTemplateId: ${{ values.infrastructureTemplateId }}
    version: 0.0.0 # Specify whatever component version you desire

    specific: # all extra fields goes here
tip

When creating a new template, you will not need to start from scratch, but you can just start by cloning a meta-template and editing it. This is the easiest and most recommended way to create a new template. There are different types of templates that you can create, and you can find them in the Template Kinds section of this documentation.

Attention:

  • The metadata.name field supports only [a-z0-9+#] separated by [-], no spaces or other special characters are allowed here
  • tags field is following the same rule

Adding extra fixed-value fields in catalog-info.yaml​

If you feel like you do not have enough fields in the fixed structure to define your component, you can always add extra fields by defining them under spec.mesh.specific, as in this example, where we create a custom field called mycustomfield:

apiVersion: backstage.io/v1alpha1
kind: System
metadata:
# ...
spec:
  # ...
  mesh:
    # ...
    specific:
      mycustomfield: custom

Step three. skeleton folder​

The skeleton folder and its contents will be copied to the repository when the Use Case Template is scaffolded; so this folder should contain all the things needed for that component to be deployed: metadata, code, documentation, etc.

  1. Then you can add your Use Case Template to the Builder, and you can use it to create multiple components for your Data Products.

  2. When you perform a commit operation in the Builder UI we will generate a Data Product Descriptor, and we will call the resulting file descriptor.yaml. Descriptor serves as a combination of all catalog-info.yaml files of the components of the selected Data product. It merges the Data product info and all of its components info into one single file which can be later sent to the Provisioner. The most important conversion that happens in this step is that metadata.name is used as id field inside the descriptor. Other changes that happen in this step are the additions of:

    • dataProductOwnerDisplayName (taken from the Builder organization structure) that serves for display purposes
    • name (taken from the spec.mesh.name field)
    • Everything else that is inside the spec.mesh field of catalog-info.yaml files is copied as is to the descriptor.yaml
tip

Don't know where to start creating your Use Case Template ? Try to duplicate one of the existing templates for the component kind (or Data Product kind) that you want to create. These templates are already pre-filled with all the basic "common" details.

Adding extra editable-value fields​

In case the value of your custom field must be filled by the user from the template wizard, you will need to define a field inside the template.yaml, as shown in this example, where we are creating mycustomfield:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
# ...
spec:
  owner: agilelab
  type: workload
  parameters:
    # ...
    - title: Spark infrastructure details
      required:
        - mycustomfield
      properties:
        # ...
        mycustomfield:
          title: Artifact bucket
          type: string
          description: S3 Bucket name where spark artifacts will be stored
    # ...
  steps:
    - id: template
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        # ...
        values:
          # ...
          mycustomfield: '${{ parameters.mycustomfield }}'
          # ...
    # ...

To reference that field, you must use the values variable from inside the catalog-info.yaml or any other skeleton file. As shown in the example below:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
# ...
spec:
  # ...
  mesh:
    # ...
    tags: [ ${{ values.mycustomfield | dump }} ]
    readsFrom: { % if values.readsFrom | length > 0 % }{ % for i in values.readsFrom % }
                 - ${{ i }}{% endfor %}{% else %}[]{% endif %}
    specific:
      mycustomfield: ${{ values.mycustomfield | dump }}
      # ...
tip

In the example above, notice how you can define complex logic on top of values. To know more about the syntax, please refer to Nunjucks docs

Enabling docs generation in your components​

In the template definition, there are two customizable points that you can leverage to increase the user experience for witboost users:

  1. in the template repository, you can define a mkdocs.yaml file and a docs directory to let users better understand what the template will create once selected for creation.
  • Create a mkdocs.yml file in the root of your repository that will have the following content:
site_name: 'example-docs'
nav:
  - Home: index.md
plugins:
  - techdocs-core
  • Update your template definition by adding the following lines to its template.yaml file in the root of the repository:
metadata:
  annotations:
    backstage.io/techdocs-ref: dir:.
  • Create a /docs folder in the root of your repository with at least an index.md file in it. (If you add more markdown files, make sure to update the nav in the mkdocs.yml file to get proper navigation for your documentation.) Note - Although docs is a popular directory name for storing documentation, it can be renamed to something else and can be configured by mkdocs.yml. See https://www.mkdocs.org/user-guide/configuration/#docs_dir
  1. you can add an icon to your template that will be displayed in the templates page by adding a public png URL in the template.yaml file, like:
metadata:
  mesh:
    icon: https://path.to.a.public/image.png

Step four. Registering a Use Case Template​

You can add the template using the catalog-import plugin, which you can find in the templates page by clicking on Register an existing component. Inside the page, you will need to link the committed template.yaml file; make sure to not commit the template to a branch with slashes (e.g. feature/branch-name) since the plugin will not be able to figure out the right path in this case.

Register existing component

Otherwise, you can add the template files to the catalog through static location configuration. For example:

catalog:
  locations:
    - type: url
      target: https://github.com/backstage/software-templates/blob/main/scaffolder-templates/react-ssr-template/template.yaml
      rules:
        - allow: [Template]

Attention:

  • the url you register to import the template must refer to the template yaml file and should be related to a specific branch (like in the above example)

Refer to Backstage documentation to know more about it.

Practice Shaper​

Use Case Templates are nodes of the Practice Shaper graph. Their role is to help in creating instances of system and component types.

As extensively described in the Practice Shaper documentation, every template should define a spec.generates property referencing the system type or component type of the system/component instances it generates.

warning

The spec.generates property is strongly required when the template includes a fetch:template action; otherwise, the action will not be able to complete.

fetch:template uses the value of spec.generates to automatically fill the spec.instanceOf property of the generated system or component instance