Ingestion Entity From Providers

Each witboost deployment has a number of entity providers installed. In particular, they are responsible for fetching data from external authoritative sources (for organization data) in any way that they see fit, translating those into entity objects, and notifying the database when those entities are added or removed. If there were no entity providers, no entities would ever enter the system from external sources.

The database always keeps track of the set of entities that belong to each provider; no two providers can try to output the same entity. And when a provider signals the removal of an entity, then that leads to an eager deletion: the entity and all auxiliary data that it has led to in the database is immediately purged.

At the moment we can ingest organization data from Microsoft Graph API and from LDAP.

When you retrieve an object through one of these two mechanisms, you need to map it to an entity. The name mapping undergoes a normalization process according to this function:

normalize(name: string): string {
  let cleaned = name
    .trim()
    .toLocaleLowerCase()
    .replace(this.regex, this.replaceWith);

  // invalid to end with _
  while (cleaned.endsWith('this.replaceWith')) {
    cleaned = cleaned.substring(0, cleaned.length - 1);
  }

  // cleans up format for groups like 'my group (Reader)'
  while (cleaned.includes(`${this.replaceWith}${this.replaceWith}`)) {
    // replaceAll from node.js >= 15
    cleaned = cleaned.replace(
      `${this.replaceWith}${this.replaceWith}`,
      this.replaceWith,
    );
  }

  return cleaned;
}

It is possible to customize the regex values and the replacement values via configuration:

catalog:
  processors:
    entityNameNormalization:
      regex:  "<regex>"
      replaceWith: "<replace_with>"
    ....

The default values, if not present in configuration, are

regex: /[^a-zA-Z0-9_\-\.]/g
replaceWith: '_'

Note that the catalog has a validity check on entity names (which therefore applies to all entities and not just those retrieved via the two systems listed above). The validity check is done with this function:

export class ValidatorFunctions {
  static isValidObjectName(value: unknown): boolean {
    return (
      typeof value === 'string' &&
      value.length >= 1 &&
      /^([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$/.test(value)
    );
  }
}