Skip to main content

Builder User Glossary

Project​

The project is the core of the Builder. It represents an aggregate of components that development teams create to ingest, transform and expose information through consumables that get published on the Market Plane as an product. The purpose of the project is driven by the business requirements of domains and they serve a specific purpose, like a Data Product, a set of data, a dashboard, an ML project, etc.

What are the phases that a Project passes through during its life?

  • The initial version of the project is 0.0.0, and since it is treated as a regular entity (of kind "system") the repository generated will contain a catalog-info.yaml file representing it.

  • At this stage, the project is "created", but is still empty, and this simply means that a repository with its definition exists.

  • Then a user can add one or more components to the newly created project. All the components are entities as well, so their repositories will contain a catalog-info.yaml with their metadata and a reference to the project they belong to.

  • Then the project as a whole can be edited through Witboost, and the versions of the various components can be updated accordingly by creating new releases, updating them, and promoting them.

  • Once the project is valid by testing it against the policies defined by Computational Governance, the project can now be deployed. The project deployment will start the underlying deployment of all of its components, in a sequence defined by the "dependsOn" relations among them.

Entity​

Inside Witboost all elements are treated as entities: systems, components, domains, releases, and even users and groups. Entities are represented by a catalog-info.yaml file stored in a repository that tells Witboost everything about that entity: type, kind, relations, metadata, annotations, etc.

Relevant entity types​

System​

The System is the main container of a data project. It derives from the defined [System Types] in the [Data Landscape] and is an aggregate of a set of [components] with data, code, metadata, infrastructure, controls, etc. All these components together are fundamental building blocks of the System, which can't exist without its parts. When a System is first created from its template inside a domain, it is just an empty set, containing only basic metadata.

Systems can take on different names, based on the Data Landscape: Data Products, Source Systems, ML Projects, etc.

Component​

Components define the pieces that compose a project and perform the technical processes inside of it. They can model different sets of physical resources like pipelines, tables, machine learning models, buckets and more.

Inside a project you can have several types of components, some of them marked as consumables that will be published to the Market Plane if present as part of the published product.

Template​

Templates are a tool that can help you create systems and components inside Witboost. They are composed of skeletons of pre-defined metadata, code, and configurations, and guide the user through the personalization of the skeleton via a form created by the Platform team. Using the user values, the skeleton content is rendered and then stored into repositories in some locations like GitLab or GitHub, allowing version control and refined access.

Each template represents a ready-to-use repository that can be created by just inserting its fundamental details. For more information on how to use templates, check out this section.

Blueprint​

Blueprints are an aggregate of templates for pre-defined use cases: they contain a system template with already-built relationships to certain components that should be created along with it. It is a very powerful and useful mechanism that will speed up the creation of well-known use cases. It guides the user during the creation of a complete data project, without the need to manually configure some of the required fields defining relationships among systems and components.

For more information on how to use blueprints to create your projects, check out this section.

Editor Wizard​

After the creation of a project containing a system and a set of components, it is possible to iterate on the definition of the metadata of these components by using the Editor Wizard, the tool Witboost provides to edit your component even after creation, through a form defined by the Platform Team allowing to modify fields, add new ones and more, following the guidelines defined on the form itself.

Descriptor​

The descriptor is the physical representation of the metadata of a project. It's a YAML file containing the definition of the system metadata along with all the component descriptors. The descriptor is built by Witboost by collecting all the components metadata plus the definition of some internal fields like the id or the project version. This is the input used by the Provisioner to perform the deployment tasks, to the Witboost Computational Governance to evaluate the compliance of the project against the set of defined policies, as well as being the metadata that gets published on the Market Plane if present.

You can always see the preview of a project's Descriptor during the lifecycle of development and evolve by creating new components, and updating existing ones via the Editor Wizard or Reverse Provisioning.

Release​

A Release is a static descriptor with a specific semantic version that represents a specific point in time during the development of the project. Witboost uses Semantic Versioning which consists of three numbers separated by a dot (<major>.<minor>.<patch>). By creating a release you're preparing the project to be deployed and published. A draft release is first created with a -SNAPSHOT suffix, which means it can still be updated as you work on your project. Then, when the project is in a state ready to go, you freeze the version by releasing it, which removes the -SNAPSHOT and means that the specific minor version can't be updated anymore, and a new minor version should be created in order to keep evolving the project.

Major version releases are considered as new, standalone projects that live in parallel with other major versions of the same project, having their own lifecycle, version control, etc.

For more information about the version management, check out the Release lifecycle documentation and the Versioning best practices.

Draft release / Snapshot​

Draft releases, also called snapshots, are releases with a -SNAPSHOT suffix and that despite being releases can still be updated. Each update on the draft release bumps the patch version and only when the draft release is upgraded to a static release, a new minor can be created.

Import - Reverse Provisioning​

The Reverse Provisioning feature provides a convenient solution to effortlessly integrate metadata from an infrastructure service like AWS into a Witboost component specification. You can find more information about it in the Reverse Provisioning documentation.