Metadata-Version: 2.4
Name: pkgos
Version: 151
Summary: OpenStack packaging CLI for the osbpo.debian.net repository
Author-email: Thomas Goirand <zigo@debian.org>
License-Expression: Apache-2.0
Project-URL: Homepage, https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools
Project-URL: Bug Tracker, https://bugs.debian.org/openstack-pkg-tools
Project-URL: Source Code, https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: cliff>=2.9
Requires-Dist: oslo.config>=6.4
Requires-Dist: requests>=2.7

# Table of Contents

- [What is openstack-pkg-tools?](#what-is-openstack-pkg-tools)
  * [General description](#general-description)
  * [Who initiated the project? Who maintains it?](#who-initiated-the-project-who-maintains-it)
- [Content of the package](#content-of-the-package)
  * [pkgos_func: the shell function library](#pkgos_func-the-shell-function-library)
  * [init-template: init script generation](#init-template-init-script-generation)
  * [pkgos.make: Makefile snippets for debian/rules](#pkgosmake-makefile-snippets-for-debianrules)
  * [build-tools: backport and build tools](#build-tools-backport-and-build-tools)
  * [pkgos: the Python CLI (python3-pkgos)](#pkgos-the-python-cli-python3-pkgos)
  * [misc: utility scripts](#misc-utility-scripts)
  * [etc/pkgos: configuration files](#etcpkgos-configuration-files)
  * [Debconf templates and translations](#debconf-templates-and-translations)
  * [Man pages](#man-pages)
  * [Unit tests](#unit-tests)
- [Salsa administration scripts](#salsa-administration-scripts)
- [Using openstack-pkg-tools in your own package](#using-openstack-pkg-tools-in-your-own-package)
  * [In maintainer scripts](#in-maintainer-scripts)
  * [In debian/rules](#in-debian-rules)
  * [Packaging an OpenStack service: overview](#packaging-an-openstack-service-overview)
  * [Writing debian .init.in files](#writing-debian-initin-files)
  * [Writing debian .config.in files](#writing-debian-configin-files)
  * [Database configuration (dbconfig-common)](#database-configuration-dbconfig-common)
  * [RabbitMQ configuration](#rabbitmq-configuration)
  * [keystone_authtoken configuration](#keystone_authtoken-configuration)
  * [Keystone endpoint registration](#keystone-endpoint-registration)
  * [Writing debian .postinst.in files](#writing-debian-postinstin-files)
  * [Writing debian .prerm and postrm scripts](#writing-debian-prerm-and-postrm-scripts)
- [Contributing](#contributing)
- [License](#license)


# What is openstack-pkg-tools?

### General description

openstack-pkg-tools is a set of shell scripts and helpers used to build and
maintain OpenStack packages in Debian. It contains:
- shared code for maintainer scripts (debian/*.config, debian/*.postinst, ...),
- init script templates to automatically generate init scripts for sysv-rc and
  systemd from a single, descriptive per-daemon script,
- tools to build backports using sbuild and/or Jenkins based on a gbp workflow,
- utilities to maintain the git packaging (to be included in a debian/rules).

Even though this package is maintained in order to build OpenStack packages,
it is of a general purpose, and it can be used for building any package.

Once the openstack-pkg-tools package is installed, the tools are available
in /usr/bin, and the shared libraries, Makefile snippets and templates in
/usr/share/openstack-pkg-tools.


### Who initiated the project? Who maintains it?

openstack-pkg-tools has been written by Thomas Goirand (zigo), and is
maintained within the Debian OpenStack team. It is used by all of the
OpenStack packages that the team maintains, plus several other packages
maintained elsewhere in Debian.


# Content of the package

### pkgos_func: the shell function library

/usr/share/openstack-pkg-tools/pkgos_func is a shell library that maintainer
scripts source with `. /usr/share/openstack-pkg-tools/pkgos_func`. It holds
the code needed to interact with debconf, RabbitMQ, SQL databases and
Keystone's service catalog, so that each daemon's maintainer scripts only
have to declare the few directives they need. Main functions:

Ini file and directive manipulation:
- pkgos_inifile: set, get or merge values in ini files
- pkgos_add_section, pkgos_remove_section, pkgos_extract_section
- pkgos_add_directive, pkgos_deprecate_directive, pkgos_comment_out_directive
- pkgos_search_directive_line_num, pkgos_safesed

Configuration files and admin credentials:
- pkgos_read_config, pkgos_write_new_conf
- pkgos_read_admin_creds, pkgos_write_admin_creds
- pkgos_gen_pass

RabbitMQ:
- pkgos_rabbit_read_conf, pkgos_rabbit_write_conf

Databases and debconf (dbconfig-common):
- pkgos_dbc_read_conf, pkgos_dbc_postinst
- pkgos_mysql_query
- pkgos_prerm_db_clean, pkgos_postrm_db_clean, pkgos_postrm

System users, groups and service endpoints:
- pkgos_adduser, pkgos_var_user_group
- pkgos_init, pkgos_get_id
- pkgos_register_endpoint_config, pkgos_register_endpoint_postinst


### init-template: init script generation

Your package only ships a short debian/*.init.in script holding the
descriptive part of the daemon (its description, which user runs it, which
config file it uses, ...), and openstack-pkg-tools standardizes everything
else in a single, unique script. From that single script, the
gen-init-configurations helper of pkgos.make generates both the sysv-rc init
script and the systemd unit. The pkgos-gen-systemd-unit utility (installed
in /usr/bin) does the same outside of a package build.


### pkgos.make: Makefile snippets for debian/rules

pkgos.make (and pkgos.stdpkg.make, a variant for standard packages) are
Makefile snippets to include from debian/rules:

```makefile
#!/usr/bin/make -f

include /usr/share/openstack-pkg-tools/pkgos.make

%:
	dh $@ --with python3
```

They provide ready-made targets such as:
- gen-init-configurations: generates init scripts for sysv-rc and systemd
  (automatically invoked by the dh_installsystemd and dh_installinit helpers).
- get-orig-source, gen-orig-xz, gen-orig-gz, gen-orig-bz2: upstream tarball
  helpers.
- gen-author-list, gen-upstream-changelog: documentation helpers for
  debian/copyright and debian/docs.
- fetch-upstream-remote, get-vcs-source, get-master-branch: upstream git
  helpers.
- versioninfo, regen-manifest-patch: maintenance helpers.
- display-po-stats, call-for-po-trans: translations helpers for debconf
  templates.
- override_dh_installchangelogs, override_dh_gencontrol,
  override_dh_installsystemd, override_dh_installinit: dh helpers.


### build-tools: backport and build tools

Scripts to build backports of OpenStack packages with sbuild, including the
orchestration of builds on Jenkins and the associated build infrastructure.
All of them are installed in /usr/bin:

- pkgos-bb: backports a package from a defined distribution. It is to be
  used in the Jenkins builders, which do the backports of the packages
  from Debian Unstable (or any other distribution) towards the target
  distribution of the backport repositories.
- pkgos-bop: builds a package using gbp buildpackage and writes it in
  /home/ftp. It is to be used in the Jenkins builders, which build the
  packages of the team when changes are pushed to their Git repositories.
- pkgos-bop-jenkins: same, orchestrated by Jenkins.
- pkgos-setup-sbuild: sets up the environment for using sbuild, either for
  personal use, or for a fully working Jenkins build system.
- pkgos-scan-repo: scans a Debian repository.
- pkgos-generate-snapshot: generates a snapshot of a project from its git
  repository.
- pkgos-infra-install-sbuild, pkgos-infra-build-pkg: helpers for the build
  infrastructure.
- pkgos-sbuild-lock, pkgos-keyring-package: maintenance helpers.
- pkgos-merge-templates: merges the shared debconf templates of
  openstack-pkg-tools (rabbit, endpoint, db, keystone authtoken) into your
  package's own debian/*.templates.
- pkgos-dh_auto_install, pkgos-dh_auto_test: dh helpers for Python builds.
- pkgos-add-missing-options, pkgos-readd-keystone-authtoken-missing-options:
  add missing configuration options to the packaged daemons.
- pkgos-trigger-odrm-hook: triggers the OpenStack Debian release manager
  hooks.


### pkgos: the Python CLI (python3-pkgos)

The pkgos folder holds a Python package, shipped in the python3-pkgos
binary package, providing the `pkgos` command line tool. It is built on
the OpenStack libraries: oslo.config for the option handling (including
reading /etc/pkgos/pkgos.conf as a default configuration source, with
case-insensitive option names), and cliff for the output formatting,
which brings the --format and --column options of the OpenStack clients.

It currently provides the `pkgos rmadison` command, which is like
rmadison(1), but for the osbpo.debian.net repository: for one or more
Debian packages, it displays the version published in every supported
<distribution>-<OpenStack-release>-backports and -backports-nochange
suite. The suite list is discovered dynamically from the directory index
of the repository, and restricted by default to the OpenStack releases
configured in /etc/pkgos/openstack-release.txt (the -s option overrides
that selection), and the index files of every selected suite are fetched
concurrently (threaded requests over a shared connection pool), keeping
even full-repository queries fast.

The output is sorted chronologically, using
/etc/pkgos/openstack-release.txt and /etc/pkgos/debian-release.txt (the
release order files shipped by openstack-pkg-tools) as sort hints: an
alphabetical sort would be wrong, as the OpenStack release naming looped
back at "z" with antelope, and bookworm is older than bullseye. When
these files are absent, the output falls back to alphabetical ordering.

A bash completion for the pkgos command is installed in
/usr/share/bash-completion/completions: it completes the command and
options, the -a and -f values, the release names of the order files for
-s, and the Debian package names of /etc/pkgos/all-git.

    pkgos rmadison cinder
    pkgos rmadison -s hibiscus nova
    pkgos rmadison -a amd64 python3-cinder
    pkgos rmadison -f value -c Version,Suite cinder

It lives in its own package so that its dependencies (cliff,
oslo.config, requests) are not pulled into the build environment of the
packages being built, which is where openstack-pkg-tools is a
build-dependency.

Unit tests are under pkgos/tests/unit, run with stestr (see
.stestr.conf), and as part of the package build through
override_dh_auto_test in debian/rules.

### misc: utility scripts

- pkgos-debcrate: creates a template Debian package out of a crates.io
  crate, and reports which of its Rust dependencies are missing from
  Debian or have an incompatible version.
- pkgos-debpypi: creates a template Debian package out of a PyPI package
  name.
- pkgos-choose: interactive picker using fzf, falling back to a numbered
  menu, printing the selected item.
- pkgos-reportbug: opens a new terminal window running reportbug against
  wnpp with the ITP text pre-filled, as saved by pkgos-debpypi and
  pkgos-debcrate in itp.txt.
- pkgos-parse-requirements: parses an OpenStack requirements.txt file and
  outputs the list of Debian dependencies.
- pkgos-reqsdiff: differences of requirements between two OpenStack
  releases.
- pkgos-check-changelog, pkgos-show-control-depends: maintenance helpers for
  debian/changelog and debian/control.
- pkgos-fix-config-default: fixes the default configuration of a daemon.
- pkgos-testr: runs testr from a Debian package build.
- pkgos-alioth-new-git, pkgos-alternative-bin, pkgos-fetch-fake-repo,
  pkgos-gen-completion: packaging maintenance utilities.

The misc folder also holds the openstack lintian profile (installed in
/usr/share/lintian/profiles/debian), and hotfix patches for libdpkg-perl and
sbuild (installed in /usr/share/openstack-pkg-tools).


### etc/pkgos: configuration files

The etc/pkgos folder ships its content to /etc/pkgos:
- pkgos.conf: the global configuration file, defining TARGET_DISTRO,
  TARGET_OPENSTACK_REL, SALSA_TOKEN, GROUP_ID, SUBGROUP_FILE, ALLGIT_FILE and
  more.
- debian-release.txt, openstack-release.txt: ordered lists of Debian suites
  and OpenStack releases, used by the pkgos-salsa-* scripts (for example, to
  auto-complete --debian-release / --openstack-release, or to derive the
  previous release when --repair creates a release branch).
- build-list, packages-list: package lists.
- substitute: substitution file for the packaging utilities.


### Debconf templates and translations

The debian folder holds shared debconf templates that packages merge into
their own debian/*.templates using pkgos-merge-templates:
- openstack-pkg-tools.configure-db.templates: SQL database configuration.
- openstack-pkg-tools.configure-endpoint.templates: Keystone service catalog
  endpoint registration.
- openstack-pkg-tools.configure-rabbit.templates: RabbitMQ configuration.
- openstack-pkg-tools.configure-ksat.templates: keystone authtoken
  configuration.

debian/po holds the translations for these templates.


### Man pages

man/ holds the man pages for the tools above, and salsa-scripts/man/ the ones
for the pkgos-salsa-* scripts. All of them are installed through
debian/openstack-pkg-tools.manpages.


### Unit tests

./run_tests runs all of the test_ scripts found under tests/. The
tests/osstt file (OpenStack Shell Test Tools) provides assertion functions
for shell scripts, and tests/pkgos_func tests the shell library itself.


# Salsa administration scripts

The salsa-scripts/ directory contains the pkgos-salsa-* scripts, which use
the GitLab REST API of https://salsa.debian.org to administer the Git
repositories of the Debian OpenStack team: listing all projects, creating
new ones, creating release branches when a new OpenStack release is out, and
configuring CI, IRC notifications, commit mailing lists and bug tracking
webhooks, either for a single project or for all of them at once. They also
ship bash auto-completion.

See [salsa-scripts/README.md](salsa-scripts/README.md) for the complete
documentation of these scripts.


# Using openstack-pkg-tools in your own package

Add openstack-pkg-tools to the Build-Depends of your package to use the
Makefile snippets and build helpers in debian/rules, and to the Depends if
you use the shell functions in your maintainer scripts.


### In maintainer scripts

Example from (some of) nova-common.config:

```sh
#!/bin/sh
set -e

#PKGOS-INCLUDE#

pkgos_var_user_group nova /bin/sh
pkgos_dbc_read_conf -pkg nova-common ${NOVA_CONF} database connection nova $@
pkgos_rabbit_read_conf ${NOVA_CONF} oslo_messaging_rabbit nova
pkgos_read_admin_creds ${NOVA_CONF} keystone_authtoken nova

#DEBHELPER#

exit 0
```

Here's (some of) nova-common.postinst:

```
#!/bin/sh

set -e

#PKGOS-INCLUDE#

#DEBHELPER#

if [ "$1" = "configure" ] || [ "$1" = "reconfigure" ] ; then
        . /usr/share/debconf/confmodule
        . /usr/share/dbconfig-common/dpkg/postinst

        pkgos_var_user_group nova /bin/sh
        maintain_var_lib_nova_perms $@

        # Create config files if they don't exist
        pkgos_write_new_conf nova nova.conf
        pkgos_write_new_conf nova api-paste.ini
        pkgos_write_new_conf nova logging.conf
        if [ -r /etc/nova/policy.json ] ; then
                mv /etc/nova/policy.json /etc/nova/disabled.policy.json.old
        fi

        # Tweak config files depending on debconf answers
        pkgos_rabbit_write_conf /etc/nova/nova.conf oslo_messaging_rabbit nova

        # Needed, because in some cases, it's owned by root:root,
        # which makes the nova-manage db sync fail.
        touch /var/log/nova/nova-manage.log
        chown nova:nova /var/log/nova/nova-manage.log

        db_get nova/configure_ksat
        if [ "${RET}" = "true" ] ; then
                db_get nova/ksat-admin-password
                ADMIN_PASS=${RET}
                db_get nova/ksat-admin-username
                ADMIN_USER=${RET}
                pkgos_write_admin_creds ${NOVA_CONF} keystone_authtoken nova
        fi

        db_get nova/configure_db
        if [ "$RET" = "true" ]; then
                pkgos_dbc_postinst /etc/nova/nova.conf database connection nova $@
                create_cell0_database
                run_db_sync
        fi

        db_stop

        # Fixup sudoers rights and clean old config file location
        chmod 0440 /etc/sudoers.d/nova-common
        [ -f /etc/sudoers.d/nova_sudoers ] && rm -f /etc/sudoers.d/nova_sudoers # That's the old file that we get rid of
fi

exit 0
```

### In debian/rules

Example from nova's debian/rules:

```makefile
include /usr/share/openstack-pkg-tools/pkgos.make

override_dh_auto_clean:
	rm -f debian/nova-common.config debian/nova-common.postinst \
		debian/nova-api.config debian/nova-api.postinst debian/nova-common.postrm
	rm -f debian/*.init debian/*.service debian/*.upstart
	rm -f debian/nova-api.templates debian/nova-common.templates
	rm -rf debian/po

override_dh_auto_build:
	/usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_func nova-common.config
	/usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_func nova-common.postinst
	/usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_func nova-api.config
	/usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_func nova-api.postinst
	/usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_postrm nova-common.postrm
	pkgos-merge-templates nova-api nova endpoint
	pkgos-merge-templates nova-common nova db rabbit ksat
```

The override_dh_auto_clean must remove everything that the build generates,
otherwise the generated files would end up in the source package (and be
shipped in the .diff.gz of a source upload).


### Packaging an OpenStack service: overview

The Debian OpenStack team packages the API of each OpenStack service using
a standard set of maintainer script templates. For each daemon unit (for
example glance-api, nova-api, keystone, cinder-volume, ...), the source
package ships:
- debian/<daemon>.init.in: the descriptive part of the init script,
- debian/<daemon>.config.in: the debconf questions (endpoint registration,
  ...),
- debian/<daemon>.postinst.in: what to do when the package is configured
  (registering the endpoint in Keystone, ...).

Shared configuration (the SQL database, RabbitMQ and the keystone_authtoken
section) usually goes into a <pkg>-common package shipping
debian/<pkg>-common.config.in, debian/<pkg>-common.postinst.in and
debian/<pkg>-common.postrm.in.

These .in files are not the maintainer scripts themselves: at build time,
debian/rules generates the real ones:
- /usr/share/openstack-pkg-tools/pkgos_insert_include pkgos_func
  <name>.config replaces the #PKGOS-INCLUDE# marker in
  debian/<name>.config.in by the full content of
  /usr/share/openstack-pkg-tools/pkgos_func, and pkgos_insert_include
  pkgos_postrm <name>.postrm embeds the pkgos_postrm library the same way,
  generating debian/<name>.{config,postinst,postrm}. If the package has a
  debian/po.in folder, it is first copied to debian/po.
- pkgos-merge-templates <name> <prefix> <shared templates...> builds
  debian/<name>.templates from the shared debconf templates of
  openstack-pkg-tools (see the chapters below), renaming their
  Template: ospt/ prefix to Template: <prefix>/, and merges the .po
  translations into debian/po. The <prefix> is the debconf prefix that all
  of the pkgos_func helpers use, and it is normally the project name (nova,
  glance, keystone, ...).
- The gen-init-configurations target of pkgos.make, which is automatically
  invoked by its override_dh_installsystemd and override_dh_installinit
  helpers, generates /etc/init.d/<daemon> and the systemd unit from each
  debian/*.init.in (see below).

Note that the .in files are plain shell scripts: they can define their own
functions. For example, nova-common.postinst.in defines
maintain_var_lib_nova_perms, create_cell0_database and run_db_sync, which
it calls in its postinst actions.


### Writing debian .init.in files

The .init.in file holds only the descriptive part of the daemon; everything
else is standardized. At build time, the gen-init-configurations target of
pkgos.make copies each debian/<name>.init.in to debian/<name>.init, appends
/usr/share/openstack-pkg-tools/init-script-template (the standardized body:
start/stop/status/restart using start-stop-daemon, PID file handling, and
the systemd-start verb used by the generated systemd unit), and runs
pkgos-gen-systemd-unit to generate debian/<name>.service. If a hand-written
debian/<name>.service.in exists, it is used as the systemd unit instead of
the generated one.

Variables to set in the .init.in:
- DESC: description printed by the init script messages.
- PROJECT_NAME: defines /etc/<PROJECT_NAME>, /var/lib/<PROJECT_NAME>,
  /var/log/<PROJECT_NAME>, /var/{run,lock,cache}/<PROJECT_NAME>, and the
  default system user and group.
- NAME: name of the daemon (used for the PID file and /usr/bin/<NAME>).
- CONFIG_FILE: the daemon configuration file (optional, defaults to
  /etc/<PROJECT_NAME>/<PROJECT_NAME>.conf).
- For daemons served by uwsgi: UWSGI_PORT, UWSGI_INI_PATH (the uwsgi ini
  file) and UWSGI_INI_APP (the WSGI application). The init script then runs
  the uwsgi binary of the default python3 version. TLS keys and certificates
  are picked up from /etc/<PROJECT_NAME>/ssl/{private,public,ca} if present.
  Optionally, UWSGI_BIND_CONFIG_FILE, UWSGI_BIND_CONFIG_SECTION,
  UWSGI_BIND_CONFIG_IP_DIRECTIVE, UWSGI_BIND_CONFIG_PORT_DIRECTIVE and
  UWSGI_BIND_CONFIG_WORKERS_DIRECTIVE make the init script read the bind
  address, port and number of processes from an ini file.
- Optional overrides: SYSTEM_USER and SYSTEM_GROUP (default to
  PROJECT_NAME), and any daemon specific environment variable, exported
  with export (like OS_OSLO_MESSAGING_RABBIT__PROCESSNAME below).

Example (debian/glance-api.init.in):

```sh
#!/bin/sh
### BEGIN INIT INFO
# Provides:          glance-api
# Required-Start:    $network $local_fs $remote_fs $syslog
# Required-Stop:     $remote_fs
# Should-Start:      postgresql mysql keystone ntp rabbitmq-server
# Should-Stop:       postgresql mysql keystone ntp rabbitmq-server
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Glance API server
# Description:       Frontend Glance API server
### END INIT INFO

# Author: Thomas Goirand <zigo@debian.org>
DESC="OpenStack Glance Service API (glance-api)"
PROJECT_NAME=glance
NAME=${PROJECT_NAME}-api
CONFIG_FILE=/etc/${PROJECT_NAME}/glance-api.conf
UWSGI_PORT=9292
UWSGI_INI_PATH=/etc/glance/glance-api-uwsgi.ini
UWSGI_INI_APP=/usr/lib/python3/dist-packages/glance/wsgi/api.py
export OS_OSLO_MESSAGING_RABBIT__PROCESSNAME=glance-api
```

If the daemon isn't served by uwsgi (agents like nova-compute or
neutron-dhcp-agent, for example), do not set the UWSGI_* variables: the
standardized template then runs /usr/bin/<NAME> --config-file=<CONFIG_FILE>
as the SYSTEM_USER, and logs to /var/log/<PROJECT_NAME>/<NAME>.log (or
syslog, depending on USE_LOGFILE / USE_SYSLOG in /etc/default/<NAME>).

The systemd unit is generated by pkgos-gen-systemd-unit, which is called
once per debian/*.init.in by the gen-init-configurations target of
pkgos.make: see the pkgos-gen-systemd-unit(1) man page for the reference
of the directives of the generated unit. Some parts of the unit are fixed:
- WorkingDirectory=~ (the home folder of the service user),
- RuntimeDirectory=<PROJECT_NAME> lock/<PROJECT_NAME> and
  CacheDirectory=<PROJECT_NAME>,
- ExecStart=/etc/init.d/<NAME> systemd-start (the systemd-start verb of the
  init script execs the daemon in the foreground),
- LimitNOFILE=65535,
- [Install] WantedBy=multi-user.target.

The below variables, set in the .init.in file, customize the rest of the
unit:
- SYSTEMD_TYPE: the Type= directive, which defaults to simple. Use notify
  when the daemon sends its ready notification with sd_notify(3), which
  OpenStack daemons do through oslo.service when they detect that they run
  under systemd: the unit is then only considered started once the daemon
  reports readiness. Set it to oneshot for a unit which just runs a script
  and exits: Restart= is then set to no, while all of the other unit types
  get Restart=on-failure.
- SYSTEMD_NOTIFY_ACCESS: the NotifyAccess= directive. Unset by default,
  meaning that only the main process may notify. Set it to all when child
  processes also send the ready notification, like nova-compute does.
- TIMEOUT_START: the TimeoutStartSec= directive. Unset by default; set it
  for a daemon that takes a long time to become ready.
- TIMEOUT_STOP: the TimeoutStopSec= directive. Defaults to 15 seconds; set
  it higher for a daemon that needs more time to shut down cleanly (nova
  uses 185 seconds for its agents).
- SYSTEM_USER and SYSTEM_GROUP: the User= and Group= directives, which
  default to PROJECT_NAME.
- SYSTEMD_DOC: the Documentation= directive, which defaults to
  man:<NAME>(1).
- SYSTEMD_ALIAS: an Alias= directive in the [Install] section, adding an
  extra name for the unit (octavia uses SYSTEMD_ALIAS=prometheus-proxy.service
  for its prometheus-proxy daemon).
- CUSTOM_SYSTEMD_UNIT: extra directives, verbatim, in the [Unit] section
  (swift uses it for ConditionFileNotEmpty=...). It can hold several
  directives at once.
- CUSTOM_SYSTEMD_SERVICE_DIRECTIVES: extra directives, verbatim, in the
  [Service] section (neutron-l3-agent uses KillMode=process).

The LSB headers of the .init.in file are also translated into the unit:
- the services listed in the # Should-Start: and # Required-Start: headers
  become After= and Wants= directives, each entry being suffixed by
  .service,
- in # Required-Start:, the virtual dependencies are translated: $network
  becomes network-online.target, $local_fs becomes local-fs.target, and
  $remote_fs becomes remote-fs.target,
- the services listed in the # X-Start-Before: header become a Before=
  directive.

Example (debian/nova-compute.init.in), using Type=notify, NotifyAccess=all
and a longer stop timeout:

```sh
#!/bin/sh
### BEGIN INIT INFO
# Provides:          nova-compute
# Required-Start:    $network $local_fs $remote_fs $syslog
# Required-Stop:     $remote_fs
# Should-Start:      libvirtd postgresql mysql keystone rabbitmq-server ntp
# Should-Stop:       libvirtd postgresql mysql keystone rabbitmq-server ntp
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Nova Compute server
# Description:       Provides compute server resources for the
#                    OpenStack cloud computing system
### END INIT INFO

# Author: Julien Danjou <acid@debian.org>, Thomas Goirand <zigo@debian.org>

# PATH should only include /usr/* if it runs after the mountnfs.sh script
DESC="OpenStack Nova Compute (nova-compute)"
PROJECT_NAME=nova
NAME=${PROJECT_NAME}-compute
SYSTEMD_DOC="man:${NAME}(1)"
DAEMON_ARGS=""
if [ -f '/etc/nova/nova-compute.conf' ] ; then
	DAEMON_ARGS="--config-file=/etc/nova/nova-compute.conf"
fi
SYSTEMD_TYPE=notify
SYSTEMD_NOTIFY_ACCESS=all
export OS_OSLO_MESSAGING_RABBIT__PROCESSNAME=nova-compute
TIMEOUT_STOP=185
```


### Writing debian .config.in files

The .config.in file is a debconf config script in which the
openstack-pkg-tools functions are available: pkgos_insert_include will
replace the #PKGOS-INCLUDE# marker by the pkgos_func library. The
nova-common.config.in shown above is the typical shared configuration,
asking (in order) about the SQL database, RabbitMQ and the
keystone_authtoken credentials. A unit package like glance-api usually only
needs the endpoint questions, in (debian/glance-api.config.in):

```sh
#!/bin/sh

set -e

. /usr/share/debconf/confmodule

#PKGOS-INCLUDE#

pkgos_register_endpoint_config glance

exit 0
```

The chapters below describe each helper, and which shared templates to
merge in debian/rules with pkgos-merge-templates.


### Database configuration (dbconfig-common)

debian/rules must merge the db templates:

	pkgos-merge-templates nova-common nova db

This adds a boolean question <prefix>/configure_db ("Set up a database for
this package?", defaulting to false).

In the .config.in:

	pkgos_dbc_read_conf -pkg nova-common ${NOVA_CONF} database connection nova $@

This asks configure_db and, if directed, the dbconfig-common questions (the
database server address, port, admin credentials and database type). If the
database connection directive already exists in the ini file, its values
are used to pre-fill the questions; otherwise the directive is created. The
-pkg <pkg> option works around cases when DPKG_MAINTSCRIPT_PACKAGE isn't
set in the config script.

In the .postinst.in, the database is created or upgraded by
dbconfig-common, then the schema is synchronized (this is the moment when
the user running the daemon, created by pkgos_var_user_group, matters):

```sh
if [ "$1" = "configure" ] || [ "$1" = "reconfigure" ] ; then
	. /usr/share/debconf/confmodule
	. /usr/share/dbconfig-common/dpkg/postinst

	db_get nova/configure_db
	if [ "$RET" = "true" ]; then
		pkgos_dbc_postinst ${NOVA_CONF} database connection nova $@
	fi

	db_get nova/configure_db
	if [ "$RET" = "true" ]; then
		echo "Now doing nova-manage db_sync: this may take a while..."
		su -s /bin/sh -c "nova-manage db_sync" nova || true
	fi
fi
```

The .prerm script must let dbconfig-common do its work when the package is
removed:

```sh
#!/bin/sh
set -e

if [ "${1}" = "remove" ] && [ -r /usr/share/debconf/confmodule ] \
  && [ -r /usr/share/dbconfig-common/dpkg/prerm ] ; then
	. /usr/share/debconf/confmodule
	db_get nova/configure_db
	if [ "$RET" = "true" ]; then
		. /usr/share/dbconfig-common/dpkg/prerm
		dbc_go ${DPKG_MAINTSCRIPT_PACKAGE} $@
	fi
fi

#DEBHELPER#

exit 0
```

On purge, the .postrm.in calls pkgos_dbc_postrm (from the pkgos_postrm
library, embedded with pkgos_insert_include pkgos_postrm), then removes
what is left:

```sh
if [ "${1}" = "purge" ] ; then
	pkgos_dbc_postrm nova nova-common $@
	[ -e /var/lib/nova ] && rm -rf /var/lib/nova
	[ -e /var/log/nova ] && rm -rf /var/log/nova
	# ... then remove each /etc/nova config file, and
	rmdir --ignore-fail-on-non-empty /etc/nova || true
fi
```

Arbitrary SQL can be run with pkgos_mysql_query <dbuser> <dbpass>
<dbserver> <dbport> <dbname> "<SQL>", which glance uses to convert its
tables to utf8 when upgrading from very old releases.


### RabbitMQ configuration

debian/rules must merge the rabbit templates:

	pkgos-merge-templates nova-common nova rabbit

This adds the questions <prefix>/configure_rabbit ("Configure RabbitMQ
access with debconf?"), <prefix>/rabbit-host, <prefix>/rabbit-userid and
<prefix>/rabbit-password.

In the .config.in (the section is usually oslo_messaging_rabbit; use
DEFAULT for a transport_url directive of the default section):

	pkgos_rabbit_read_conf ${NOVA_CONF} oslo_messaging_rabbit nova

This asks configure_rabbit and, if directed, the host, username and
password questions, pre-filling them from the transport_url directive
already present in the ini file (if any).

In the .postinst.in, the values are written back:

	pkgos_rabbit_write_conf ${NOVA_CONF} oslo_messaging_rabbit nova

This writes the transport_url directive (of the form
rabbit://<user>:<password>@<host>:5672/<vhost>) in the ini file.


### keystone_authtoken configuration

debian/rules must merge the ksat templates:

	pkgos-merge-templates nova-common nova ksat

This adds the questions <prefix>/configure_ksat ("Manage keystone_authtoken
with debconf?"), <prefix>/ksat-public-url, <prefix>/ksat-region,
<prefix>/ksat-create-service-user, <prefix>/ksat-admin-username,
<prefix>/ksat-admin-project-name, <prefix>/ksat-admin-password,
<prefix>/ksat-service-username, <prefix>/ksat-service-project-name and
<prefix>/ksat-service-password.

In the .config.in (the section of the ini file is keystone_authtoken):

	pkgos_read_admin_creds ${NOVA_CONF} keystone_authtoken nova

This asks configure_ksat and, if directed, the authentication server public
endpoint URL, region, and the admin and service user credentials,
pre-filling them from the existing ini file.

In the .postinst.in, the values are written back, and the service flavor of
paste_deploy is set when the daemon authenticates with Keystone:

```sh
db_get nova/configure_ksat
if [ "${RET}" = "true" ] ; then
	pkgos_write_admin_creds ${NOVA_CONF} keystone_authtoken nova
	pkgos_inifile set ${NOVA_CONF} paste_deploy flavor keystone
fi
```

When generating the reference configuration file with
oslo-config-generator, some keystone_authtoken options aren't part of any
namespace. Re-add them with (see debian/rules of glance):

	pkgos-readd-keystone-authtoken-missing-options \
		/usr/share/glance-common/glance-api.conf keystone_authtoken glance


### Keystone endpoint registration

debian/rules must merge the endpoint templates (usually in the unit
package, as the endpoint is registered only once):

	pkgos-merge-templates nova-api nova endpoint

This adds the questions <prefix>/configure_api-endpoint ("Register this
service in the Keystone endpoint catalog?"), <prefix>/api-keystone-address,
<prefix>/api-keystone-proto, <prefix>/api-keystone-admin-username,
<prefix>/api-keystone-admin-project-name,
<prefix>/api-keystone-admin-password, <prefix>/api-endpoint-address,
<prefix>/api-endpoint-proto and <prefix>/api-endpoint-region-name.

In the .config.in:

	pkgos_register_endpoint_config nova

In the .postinst.in, the endpoint is created using the openstack CLI:

	pkgos_register_endpoint_postinst nova nova compute \
		"Nova Compute Service" 8774 /v2.1

The prototype is: <prefix> <service-name> <service-type> <description>
<port> <endpoint-url-path>. For example, glance passes an empty URL path,
as it serves both /v2 and /v3:

	pkgos_register_endpoint_postinst glance glance image \
		"Glance Image Service" 9292 ""


### Writing debian .postinst.in files

The .postinst.in file is the postinst maintainer script in which the
openstack-pkg-tools functions are available through the #PKGOS-INCLUDE#
marker. Actions are guarded by configure or reconfigure, and db_stop must
be called before the end so that debconf doesn't stay blocked. Here is
(most of) debian/glance-common.postinst.in, which regroups everything
described in the chapters above:

```sh
#!/bin/sh

set -e

ETC=/etc/glance
API_CONF=${ETC}/glance-api.conf

#PKGOS-INCLUDE#

if [ "$1" = "configure" ] || [ "$1" = "reconfigure" ] ; then
	. /usr/share/debconf/confmodule
	. /usr/share/dbconfig-common/dpkg/postinst

	pkgos_var_user_group glance

	# Make sure we have a /etc/glance folder with config files in it.
	pkgos_write_new_conf glance glance-api.conf
	pkgos_write_new_conf glance glance-cache.conf
	pkgos_write_new_conf glance glance-manage.conf
	pkgos_write_new_conf glance glance-scrubber.conf
	pkgos_write_new_conf glance glance-api-paste.ini
	pkgos_write_new_conf glance glance-image-import.conf

	# Do the db creation using dbconfig-common
	db_get glance/configure_db
	if [ "$RET" = "true" ] ; then
		pkgos_dbc_postinst ${API_CONF} database connection glance $@
	fi

	pkgos_rabbit_write_conf ${API_CONF} oslo_messaging_rabbit glance

	# Upgrade or create the db if directed to do so
	db_get glance/configure_db
	if [ "$RET" = "true" ] ; then
		echo "Now doing glance-manage db_sync: this may take a while..."
		su -s /bin/sh -c "glance-manage db_sync" glance || true
	fi

	db_get glance/configure_ksat
	if [ "$RET" = "true" ] ; then
		pkgos_write_admin_creds ${API_CONF} keystone_authtoken glance
		pkgos_inifile set ${API_CONF} paste_deploy flavor keystone
	fi
	db_stop

	chown glance:adm /var/log/glance/
	chmod 0750 /var/log/glance/
	chown glance:glance /var/lib/glance/ /etc/glance/
fi

#DEBHELPER#

exit 0
```

pkgos_write_new_conf <project> <conf-file> copies a file from the package's
/usr/share/<package>/ folder to /etc/<project>/ only if it isn't already
there, so that local modifications are preserved on upgrades. It is meant
to be used for every configuration file the package ships in /usr/share.

pkgos_var_user_group <project> [<shell>] creates the system user and group
if missing, and fixes the ownership of /var/lib/<project>,
/var/log/<project> and /var/cache/<project>. The optional second argument
is the user's shell (use /bin/sh for daemons that need to run commands
with su -s /bin/sh).


### Writing debian .prerm and postrm scripts

The prerm only has to hand over to dbconfig-common when the package is
removed (see the Database configuration chapter above for the full
example).

The postrm.in is generated with pkgos_insert_include pkgos_postrm
<name>.postrm, which embeds the pkgos_postrm library: pkgos_dbc_postrm
<template-prefix> <package> $@ cleans up the debconf database and
dbconfig-common configuration on purge, after which the package removes its
own files (config files in /etc/<project>, /var/lib/<project> and
/var/log/<project>). See the Database configuration chapter above for a
full example.


# Contributing

The package is maintained at:

https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools

Merge requests are welcome. Bugs are tracked in the Debian BTS against the
openstack-pkg-tools package:

https://bugs.debian.org/openstack-pkg-tools


# License

openstack-pkg-tools is released under the Apache License, version 2.0:

http://www.apache.org/licenses/LICENSE-2.0
