Migrating to 2.0.0

Migration Guide

Here you can find the needed steps to upgrade your current Witboost installation to v2.0.0. Please refer to the release notes for a detailed list of changes and improvements. This guide is intended for users who are currently running Witboost version 2.1.0 or earlier and wish to upgrade to version 2.0.0.

Follow these steps to migrate from the previous version to Witboost version 2.0.0:

Backup Your Data

Before starting the migration, ensure you have a complete backup of your database and configuration files.

Update Dependencies

Ensure all the dependencies in the Helm files are updated to their compatible versions for 2.0.0. Refer to the published Helm chart for details.

Upgrade Steps

Core

Changes:

The Practice Shaper has been introduced
The UI module now automatically registers resource types towards CGP based on the Practice Shaper
The UI module now hosts the marketplace proxy service as an integrated Backstage plugin
The previous marketplace proxy Scala service has been deprecated
The marketplace db schema has been refactored
UI module is now based on Node 20. The base docker image is now a Debian one. More details on the base docker image here
Component templates:
- before: when accessing a component template by clicking on the "Add" button in the components list of a Data Product's builder page, if the template had a picker field called dataproduct, it would be filled with the reference to the parent data product (e.g., system:<dp-name>)
- now: when accessing a component template by clicking on the "Add" button in the component list of a system's builder page, if the template has a picker field called parentRef, it gets filled with the reference to the parent system (e.g., system:<system-name>). If parentRef is not found, the previous behavior is retained, and the dataproduct value gets prefilled.
ComponentIdentifierPicker
- before: the component's identifier was built by concatenating the value of a picker named dataproduct (e.g., system:<dp-name>) in the same template, with the component's name
- now: the component's identifier is built by concatenating the value of a picker named parentRef (e.g., system:<parent-system-name>) in the same template, with the component's name. If parentRef is not found, the previous behavior is retained, and the dataproduct picker value is used instead

Migration:

Important! Before starting the UI with the new version, make sure to backup your current marketplace database: this version includes a critical migration.

Update the Microsoft auth configuration

auth.providers.[kind].[environemnt] -> add the sign-in resolvers

auth:
   providers:
      microsoft:
         development:
            signIn:
               resolvers:
                  - resolver: emailMatchingUserEntityAnnotation

Update the Microsoft Graph Org configuration, by replacing catalog.processors.microsoftGraphOrg.providers with:

catalog:
   providers:
      microsoftGraphOrg:
         default:
            target: https://graph.microsoft.com/v1.0
            authority: https://login.microsoftonline.com
            tenantId: <TENANT_ID>
            clientId: <CLIENT_ID>
            clientSecret: <CLIENT_SECRET>
            userGroupMember:
               filter: <same filter as before>
            group:
               filter: <same filter as before>
            schedule:
               frequency: { hours: 5 }
               timeout: { minutes: 30 }

After updating you could get some warnings while reading the users/groups. You can fix them by running the SQL script:

update catalog.refresh_state
set location_key = 'msgraph-org-provider:default'
where location_key = 'msgraph-org-provider:https://graph.microsoft.com/v1.0'

Update the LDAP auth configuration

auth.providers check that the configuration is correct:

auth:
   providers:
      simple_ldap:
         url: ldap://my.ldap.host.com
         bindDN: cn=admin,dc=my-company,dc=com
         bindCredentials: StrongAdminPassword
         searchBase: ou=users,dc=my-company,dc=com
         searchFilter: (uid={{username}})
         ... # other configurations

Update the LDAP Org configuration, by replacing catalog.processors.ldapOrg.providers with:

catalog:
   providers:
      ldapOrg:
         default:
            target: ldap://my.ldap.host.com
            bind:
               dn: cn=admin,dc=my-company,dc=com
               secret: StrongAdminPassword
            users:
               dn: ou=users,dc=my-company,dc=com
               map:
                  name: uid
                  displayName: displayName
                  email: mail
            groups:
               dn: ou=groups,dc=my-company,dc=com
               map:
                  name: cn
                  displayName: cn
                  userMembers: memberUid
            schedule:
               frequency: { hours: 5 }
               timeout: { minutes: 30 }

After updating you could get some warnings while reading the users/groups. You can fix them by running the SQL script:

update catalog.refresh_state
set location_key = 'ldap-org-provider:default'
where location_key = 'ldap-org-provider:<your-ldap-url>'

Append Taxonomy, SystemType, ComponentType, DomainType to the list of allowed entities under config catalog.rules.allow.

Copy the resource types configuration from CGP into the UI's configuration,

For example, CGP config:

resource-types = [
      {
            name = "dataproduct"
            configuration = {
               resource-name = "id"
               resource-display-name = ["domain", "name", "version"]
               resource-filter = "version"
               batch-size = 6
            }
            resolver-configuration = {
               url = "http://localhost:8088/datamesh.provisioningcoordinator"
               path = "v1/resolve"
            }
      }
   ]

maps to

practiceShaper:
   # Configuration options for the processor responsible for synchronizing resource types between Practice Shaper and Governance Platform
   resourceTypesProcessor:
      # Default resource type configuration sent to the Computational Governance Platform
      resourceTypeConfiguration:
         # Information about the shape of descriptors of this resource type
         descriptorConfiguration:
            # Field inside the resource descriptor that contains its unique identifier
            resourceNameField: 'id'
            # List of fields in the resource descriptor to be used to generate a display name
            resourceDisplayNameFields:
               - domain
               - name
               - version
            # Field inside the resource descriptor used to differentiate evaluation results for the same resource
            resourceFilterField: 'version'
         # Details about the perimeter resolver
         resolverConfiguration:
            # Base URL of the Perimeter Resolver service
            url: 'http://localhost:8088/datamesh.provisioningcoordinator'
            # Base path (without query params) to the resolve endpoint
            path: 'v1/resolve'
            # Maximum number of resources requested to the perimeter resolver per request (page size)
            batchSize: 6

New RBAC permissions have been defined. Update your assignments accordingly:

Permissions	Features
catalog.entity.read	Entity read from catalog
catalog.entity.create	Project instance creation (component and system from choose button) and import of existing from yaml
catalog.entity.refresh	Project instance refresh
catalog.entity.delete	Project instance delete
catalog.entity.create catalog.platform.create	Import of all yaml file from route “catalog-import”, template and blueprint registration
catalog.entity.refresh catalog.platform.refresh	Entity instance refresh (refresh icon button on about card)
catalog.entity.delete catalog.platform.delete	Entity instance delete (unregistered entity on context menu)
builder.software-catalog.view	View Software catalog
builder.software-catalog.view catalog.entity.create catalog.platform.create	Import of all yaml file from create button on software catalog
platform.settings.edit practice-shaper.edit	Practice Shaper tab on Settings page
platform.settings.edit practice-shaper.edit practice-shaper.import	Import practice shaper entities on practice shaper tab from button “Register new entity”

Check the updated RBAC documentation for additional details.

Custom pages migration (required only if you are using custom pages)
- Rename configuration mesh.customPages.marketplace.dataProductPage to mesh.customPages.marketplace.projectPage
- Rename configuration mesh.customPages.marketplace.outputPortPage to mesh.customPages.marketplace.consumableInterfacePage
- Refer to the documentation (Platform > Extensibility > Custom Pages) to check the updated models available to the custom page configuration, and update the value of valuePath accordingly.
  For example:
  # ... parameters: - name: 'name' valuePath: 'DataProduct.name'
  becomes:
  # ... parameters: - name: 'name' valuePath: 'name' # DataProduct object no longer available

Hasura

The first run of the updated UI project will migrate the marketplace db. Execute the following steps on Hasura to keep it aligned:

Reload all the metadata (<hasura-host>/console/settings/metadata-actions)
Delete all the inconsistent objects (<hasura-host>/console/settings/metadata-status)
Track new tables and relations in the marketplace database, marketplace schema

Coordinator

Changes:

The previous marketplace proxy microservice has been deprecated and replaced by a Backstage plugin integrated in the UI module

Migration:

Update the Coordinator configuration to make it call the new marketplace service. Base url: <UI module host>/api/marketplace. Configuration under provisioning-coordinator.marketplace-compliant-services.services (or provisioning-coordinator.depends-on)

Suggested, but not required: authenticate calls towards the witboost marketplace (which now resides in the UI module) with a server token.

Generate a base64 secret and add it to the list of accepted secrets in the UI configuration under backend.auth.keys
```
backend:
   auth:
      keys:
      - secret: ${MARKETPLACE_AUTH_SECRET} # base 64 secret
```

Configure a JWT token factory in the Coordinator:

provisioning-coordinator {
      tokens = {
         wb-marketplace = {
            type = "jwt-factory"
            jwt-factory = {
                  secret = ${MARKETPLACE_AUTH_SECRET} # base 64 secret
                  isSecretBase64Encoded = true
                  expiration = 1 hour
                  payload = "{\"sub\": \"backstage-server\"}"
            }
         }
      }
}

Configure the witboost marketplace service to authenticate with the factory above

provisioning-coordinator {
      marketplace-compliant-services.services = [
         ...
         {
            id = "witboost-marketplace"
            ...
            http-auth = {
               bearer-token-key = "wb-marketplace"
               additional-authorization-header-name = "x-witboost-user-jwt"
            }
         }
      ]
}

CGP

Breaking changes:

Configuration computational-governance-platform.resource-types is now an empty list by default (i.e., no default resource types registered)
Any previously registered resource type with name data_product has been renamed in dataproduct (without the underscore _)

Migration:

Important! This version includes a database migration that may take a few minutes to complete. Don't worry if the Coordinator/CGP instance takes some time to start on the first launch after the upgrade, and adjust any related timeouts accordingly.

If you prefer to opt for a manual database migration please reach out the Witboost development team for assistance.

If there's an explicitly configured resource type with name data_product under computational-governance-platform.resource-types, migrate its configuration from CGP to the UI module (refer to the UI section above), then remove it from the CGP configuration

Core (Practice Shaper)

Migrate to Practice Shaper by referring to the dedicated section.
Config provisioner.partialProvisioning.enabled has been deprecated. To enable the partial provisioning on Systems that are instances of a given SystemType, fill its spec.partiallyDeployableIn property. Refer to the Practice Shaper documentation for additional details

Migrating to 2.0.0

Migration Guide​

Backup Your Data​

Update Dependencies​

Upgrade Steps​

Core​

Hasura​

Coordinator​

CGP​

Core (Practice Shaper)​

Migration Guide

Backup Your Data

Update Dependencies

Upgrade Steps

Core

Hasura

Coordinator

CGP

Core (Practice Shaper)