--- title: File-based catalogs --- # File-based catalogs {#fbc} Operator Lifecycle Manager (OLM) v1 in OpenShift Container Platform supports *file-based catalogs* for discovering and sourcing cluster extensions, including Operators, on a cluster. ## Introduction to file-based catalogs {#olm-file-based-catalogs_fbc} File-based catalogs are the latest plain text (JSON or YAML) catalog format for Operator Lifecycle Manager (OLM). This format enables catalog editing, composability, and extensibility while remaining compatible with earlier SQLite-based catalogs. Editing : With file-based catalogs, users interacting with the contents of a catalog are able to make direct changes to the format and verify that their changes are valid. Because this format is plain text JSON or YAML, catalog maintainers can easily manipulate catalog metadata by hand or with widely known and supported JSON or YAML tooling, such as the `jq` CLI. This editability enables the following features and user-defined extensions: - Promoting an existing bundle to a new channel - Changing the default channel of a package - Custom algorithms for adding, updating, and removing upgrade paths Composability : File-based catalogs are stored in an arbitrary directory hierarchy, which enables catalog composition. For example, consider two separate file-based catalog directories: `catalogA` and `catalogB`. A catalog maintainer can create a new combined catalog by making a new directory `catalogC` and copying `catalogA` and `catalogB` into it. This composability enables decentralized catalogs. The format permits Operator authors to maintain Operator-specific catalogs, and it permits maintainers to trivially build a catalog composed of individual Operator catalogs. File-based catalogs can be composed by combining multiple other catalogs, by extracting subsets of one catalog, or a combination of both of these. > [!NOTE] > Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command returns an error if any duplicates are found. Because Operator authors are most familiar with their Operator, its dependencies, and its upgrade compatibility, they are able to maintain their own Operator-specific catalog and have direct control over its contents. With file-based catalogs, Operator authors own the task of building and maintaining their packages in a catalog. Composite catalog maintainers, however, only own the task of curating the packages in their catalog and publishing the catalog to users. Extensibility : The file-based catalog specification is a low-level representation of a catalog. While it can be maintained directly in its low-level form, catalog maintainers can build interesting extensions on top that can be used by their own custom tooling to make any number of mutations. For example, a tool could translate a high-level API, such as `(mode=semver)`, down to the low-level, file-based catalog format for upgrade paths. Or a catalog maintainer might need to customize all of the bundle metadata by adding a new property to bundles that meet a certain criteria. While this extensibility allows for additional official tooling to be developed on top of the low-level APIs for future OpenShift Container Platform releases, the major benefit is that catalog maintainers have this capability as well. > [!IMPORTANT] > As of OpenShift Container Platform 4.11, the default Red Hat-provided Operator catalog releases in the file-based catalog format. The default Red Hat-provided Operator catalogs for OpenShift Container Platform 4.6 through 4.10 released in the deprecated SQLite database format. > > The `opm` subcommands, flags, and functionality related to the SQLite database format are also deprecated and will be removed in a future release. The features are still supported and must be used for catalogs that use the deprecated SQLite database format. > > Many of the `opm` subcommands and flags for working with the SQLite database format, such as `opm index prune`, do not work with the file-based catalog format. ## Directory structure {#olm-fb-catalogs-structure_fbc} File-based catalogs can be stored and loaded from directory-based file systems. The `opm` CLI loads the catalog by walking the root directory and recursing into subdirectories. The CLI attempts to load every file it finds and fails if any errors occur. Non-catalog files can be ignored using `.indexignore` files, which have the same rules for patterns and precedence as `.gitignore` files. ```terminal {title="Example .indexignore file"} # Ignore everything except non-object .json and .yaml files **/* !*.json !*.yaml **/objects/*.json **/objects/*.yaml ``` Catalog maintainers have the flexibility to choose their layout, but it is recommended to store each package’s file-based catalog blobs in separate subdirectories. Each individual file can be either JSON or YAML; it is not necessary for every file in a catalog to use the same format. ```terminal {title="Basic recommended structure"} catalog ├── packageA │ └── index.yaml ├── packageB │ ├── .indexignore │ ├── index.yaml │ └── objects │ └── packageB.v0.1.0.clusterserviceversion.yaml └── packageC └── index.json └── deprecations.yaml ``` This recommended structure has the property that each subdirectory in the directory hierarchy is a self-contained catalog, which makes catalog composition, discovery, and navigation trivial file system operations. The catalog can also be included in a parent catalog by copying it into the parent catalog’s root directory. ## Schemas {#olm-fb-catalogs-schemas_fbc} File-based catalogs on OpenShift Container Platform use a CUE-based format with schemas that define catalog structure for Operator Lifecycle Manager (OLM). Each Operator package requires one `olm.package` blob, at least one `olm.channel` blob, and one or more `olm.bundle` blobs. ```go {title="_Meta schema"} _Meta: { // schema is required and must be a non-empty string schema: string & !="" // package is optional, but if it's defined, it must be a non-empty string package?: string & !="" // properties is optional, but if it's defined, it must be a list of 0 or more properties properties?: [... #Property] } #Property: { // type is required type: string & !="" // value is required, and it must not be null value: !=null } ``` > [!NOTE] > No CUE schemas listed in this specification should be considered exhaustive. The `opm validate` command has additional validations that are difficult or impossible to express concisely in CUE. > [!NOTE] > All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix, such as a domain that you own. **Additional resources** {._additional-resources} - [CUE language specification](https://cuelang.org/docs/references/spec/) ### olm.package schema {#olm-package-schema_fbc} The `olm.package` schema specifies package-level metadata for Operators in file-based catalogs, including name, default channel, and icon. Use this schema reference when you build or validate Operator package definitions for Operator Lifecycle Manager (OLM). :::details{title="`olm.package` schema"} ```go #Package: { schema: "olm.package" // Package name name: string & !="" // A description of the package description?: string // The package's default channel defaultChannel: string & !="" // An optional icon icon?: { base64data: string mediatype: string } } ``` ::: ### olm.channel schema {#olm-channel-schema_fbc} The `olm.channel` schema defines a channel within a package, the bundle entries that are members of the channel, and the upgrade paths for those bundles. If a bundle entry represents an edge in multiple `olm.channel` blobs, it can only appear once per channel. It is valid for an entry’s `replaces` value to reference another bundle name that cannot be found in this catalog or another catalog. However, all other channel invariants must hold true, such as a channel not having multiple heads. :::details{title="`olm.channel` schema"} ```go #Channel: { schema: "olm.channel" package: string & !="" name: string & !="" entries: [...#ChannelEntry] } #ChannelEntry: { // name is required. It is the name of an `olm.bundle` that // is present in the channel. name: string & !="" // replaces is optional. It is the name of bundle that is replaced // by this entry. It does not have to be present in the entry list. replaces?: string & !="" // skips is optional. It is a list of bundle names that are skipped by // this entry. The skipped bundles do not have to be present in the // entry list. skips?: [...string & !=""] // skipRange is optional. It is the semver range of bundle versions // that are skipped by this entry. skipRange?: string & !="" } ``` ::: > [!WARNING] > When using the `skipRange` field, the skipped Operator versions are pruned from the update graph and are longer installable by users with the `spec.startingCSV` property of `Subscription` objects. > > You can update an Operator incrementally while keeping previously installed versions available to users for future installation by using both the `skipRange` and `replaces` field. Ensure that the `replaces` field points to the immediate previous version of the Operator version in question. ### olm.bundle schema {#olm-bundle-schema_fbc} The `olm.bundle` schema defines the structure of bundle entries stored in an Operator catalog index. It specifies required fields such as package name, bundle name, image reference, and optional properties and related images. :::details{title="`olm.bundle` schema"} ```go #Bundle: { schema: "olm.bundle" package: string & !="" name: string & !="" image: string & !="" properties: [...#Property] relatedImages?: [...#RelatedImage] } #Property: { // type is required type: string & !="" // value is required, and it must not be null value: !=null } #RelatedImage: { // image is the image reference image: string & !="" // name is an optional descriptive name for an image that // helps identify its purpose in the context of the bundle name?: string & !="" } ``` ::: ### olm.deprecations schema {#olm-deprecations-schema_fbc} The optional `olm.deprecations` schema defines deprecation information for packages, bundles, and channels in an Operator catalog. When you define this schema, the web console displays warning badges and deprecation messages in the software catalog. An `olm.deprecations` schema entry contains one or more of the following `reference` types, which indicates the deprecation scope. After the Operator is installed, any specified messages can be viewed as status conditions on the related `Subscription` object. **Deprecation `reference` types** | Type | Scope | Status condition | | --- | --- | --- | | `olm.package` | Represents the entire package | `PackageDeprecated` | | `olm.channel` | Represents one channel | `ChannelDeprecated` | | `olm.bundle` | Represents one bundle version | `BundleDeprecated` | Each `reference` type has their own requirements, as detailed in the following example. ```yaml {title="Example olm.deprecations schema with each reference type"} schema: olm.deprecations package: my-operator entries: - reference: schema: olm.package message: | The 'my-operator' package is end of life. Please use the 'my-operator-new' package for support. - reference: schema: olm.channel name: alpha message: | The 'alpha' channel is no longer supported. Please switch to the 'stable' channel. - reference: schema: olm.bundle name: my-operator.v1.68.0 message: | my-operator.v1.68.0 is deprecated. Uninstall my-operator.v1.68.0 and install my-operator.v1.72.0 for support. ``` - Each deprecation schema must have a `package` value, and that package reference must be unique across the catalog. There must not be an associated `name` field. - The `olm.package` schema must not include a `name` field, because it is determined by the `package` field defined earlier in the schema. - All `message` fields, for any `reference` type, must be a non-zero length and represented as an opaque text blob. - The `name` field for the `olm.channel` schema is required. - The `name` field for the `olm.bundle` schema is required. > [!NOTE] > The deprecation feature does not consider overlapping deprecation, for example package versus channel versus bundle. Operator authors can save `olm.deprecations` schema entries as a `deprecations.yaml` file in the same directory as the package’s `index.yaml` file: ```terminal {title="Example directory structure for a catalog with deprecations"} my-catalog └── my-operator ├── index.yaml └── deprecations.yaml ``` **Additional resources** {._additional-resources} - [Updating or filtering a file-based catalog image](/openshift-docs-markdown/operators/admin/olm-managing-custom-catalogs#olm-filtering-fbc_olm-managing-custom-catalogs) ## Properties {#olm-fb-catalogs-prop_fbc} Properties are arbitrary pieces of metadata that can be attached to file-based catalog schemas. The `type` field is a string that effectively specifies the semantic and syntactic meaning of the `value` field. The value can be any arbitrary JSON or YAML. OLM defines a handful of property types, again using the reserved `olm.*` prefix. ### olm.package property {#olm-fb-catalogs-package-prop_fbc} The `olm.package` property defines the package name and version. This is a required property on bundles, and there must be exactly one of these properties. The `packageName` field must match the bundle’s first-class `package` field, and the `version` field must be a valid semantic version. ```go {title="olm.package property"} #PropertyPackage: { type: "olm.package" value: { packageName: string & !="" version: string & !="" } } ``` ### olm.gvk property {#olm-fb-catalogs-gvk-prop_fbc} The `olm.gvk` property defines the group/version/kind (GVK) of a Kubernetes API that is provided by this bundle. This property is used by OLM to resolve a bundle with this property as a dependency for other bundles that list the same GVK as a required API. The GVK must adhere to Kubernetes GVK validations. ```go {title="olm.gvk property"} #PropertyGVK: { type: "olm.gvk" value: { group: string & !="" version: string & !="" kind: string & !="" } } ``` ### olm.package.required {#olm-fb-catalogs-package-reqd-prop_fbc} The `olm.package.required` property defines the package name and version range of another package that this bundle requires. For every required package property a bundle lists, OLM ensures there is an Operator installed on the cluster for the listed package and in the required version range. The `versionRange` field must be a valid semantic version (semver) range. ```go {title="olm.package.required property"} #PropertyPackageRequired: { type: "olm.package.required" value: { packageName: string & !="" versionRange: string & !="" } } ``` ### olm.gvk.required {#olm-fb-catalogs-gvk-reqd-prop_fbc} The `olm.gvk.required` property defines the group/version/kind (GVK) of a Kubernetes API that this bundle requires. For every required GVK property a bundle lists, OLM ensures there is an Operator installed on the cluster that provides it. The GVK must adhere to Kubernetes GVK validations. ```terminal {title="olm.gvk.required property"} #PropertyGVKRequired: { type: "olm.gvk.required" value: { group: string & !="" version: string & !="" kind: string & !="" } } ``` ## Example catalog {#olm-fb-catalogs-example_fbc} With file-based catalogs, catalog maintainers can focus on Operator curation and compatibility. Because Operator authors have already produced Operator-specific catalogs for their Operators, catalog maintainers can build their catalog by rendering each Operator catalog into a subdirectory of the catalog’s root directory. There are many possible ways to build a file-based catalog; the following steps outline a simple approach: 1. Maintain a single configuration file for the catalog, containing image references for each Operator in the catalog: ```yaml {title="Example catalog configuration file"} name: community-operators repo: quay.io/community-operators/catalog tag: latest references: - name: etcd-operator image: quay.io/etcd-operator/index@sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 - name: prometheus-operator image: quay.io/prometheus-operator/index@sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317 ``` 2. Run a script that parses the configuration file and creates a new catalog from its references: ```sh {title="Example script"} name=$(yq eval '.name' catalog.yaml) mkdir "$name" yq eval '.name + "/" + .references[].name' catalog.yaml | xargs mkdir for l in $(yq e '.name as $catalog | .references[] | .image + "|" + $catalog + "/" + .name + "/index.yaml"' catalog.yaml); do image=$(echo $l | cut -d'|' -f1) file=$(echo $l | cut -d'|' -f2) opm render "$image" > "$file" done opm generate dockerfile "$name" indexImage=$(yq eval '.repo + ":" + .tag' catalog.yaml) docker build -t "$indexImage" -f "$name.Dockerfile" . docker push "$indexImage" ``` ## Guidelines {#olm-fb-catalogs-guidelines_fbc} Follow the guidelines to maintain file-based Operator catalogs. Treat bundle images and metadata as immutable. Store catalog metadata in source control as the source of truth. ### Immutable bundles {#olm-fb-catalogs-immutable_fbc} The general advice with Operator Lifecycle Manager (OLM) is that bundle images and their metadata should be treated as immutable. If a broken bundle has been pushed to a catalog, you must assume that at least one of your users has upgraded to that bundle. Based on that assumption, you must release another bundle with an upgrade path from the broken bundle to ensure users with the broken bundle installed receive an upgrade. OLM will not reinstall an installed bundle if the contents of that bundle are updated in the catalog. However, there are some cases where a change in the catalog metadata is preferred: - Channel promotion: If you already released a bundle and later decide that you want to add it to another channel, you can add an entry for your bundle in another `olm.channel` blob. - New upgrade paths: If you release a new `1.2.z` bundle version, for example `1.2.4`, but `1.3.0` is already released, you can update the catalog metadata for `1.3.0` to skip `1.2.4`. ### Source control {#olm-fb-catalogs-source-control_fbc} Catalog metadata should be stored in source control and treated as the source of truth. Updates to catalog images should include the following steps: 1. Update the source-controlled catalog directory with a new commit. 2. Build and push the catalog image. Use a consistent tagging taxonomy, such as `:latest` or `:`, so that users can receive updates to a catalog as they become available. > [!NOTE] > For more information about creating file-based catalogs by using the `opm` CLI, see "Creating a file-based catalog image". ## CLI usage {#olm-fbc-cli_fbc} To create and manage file-based catalogs, you can use the `opm` command-line interface (CLI). For instructions about creating file-based catalogs by using the `opm` CLI, see "Creating a file-based catalog image". For reference documentation about the `opm` CLI commands related to managing file-based catalogs, see "opm CLI reference". **Additional resources** {._additional-resources} - [Creating a file-based catalog image](/openshift-docs-markdown/operators/admin/olm-managing-custom-catalogs#olm-creating-fb-catalog-image_olm-managing-custom-catalogs) - [opm CLI reference](/openshift-docs-markdown/cli_reference/opm/cli-opm-ref#cli-opm-ref) ## Automation {#olm-fb-catalogs-automation_fbc} Operator authors and catalog maintainers can automate file-based catalog maintenance with CI/CD workflows. Catalog maintainers can use GitOps automation to accomplish the following example tasks: - Check that pull request (PR) authors are permitted to make the requested changes, for example by updating their package’s image reference. - Check that the catalog updates pass the `opm validate` command. - Check that the updated bundle or catalog image references exist, the catalog images run successfully in a cluster, and Operators from that package can be successfully installed. - Automatically merge PRs that pass the previous checks. - Automatically rebuild and republish the catalog image.