Page MenuHomeVyOS Platform

equuleus: Autogenerating deb package version number for internal packages
Open, NormalPublic

Description

Currently when packages are updated, the package version number is not incremented.
That means that two ISO's released 10days apart will have could have the same version number of the package, but the package could have changed multiple times in the same timeframe. and so the package.

To make it more clear for the user (and devs) we need to increment the version number of the package for every commit done, and also show the commit id of the checked-out commit on build-time. we also need to have a solid version numbering shceme that follows the debian versioning scheme.

Generating version numbers based on git tags are quite easy to do, and can be automated when generating the package by adding this to debian/rules of the package:

override_dh_gencontrol:
        dh_gencontrol -- -v$(shell (git describe --tags --long --match 'vyos/*' --dirty || echo 0.0-no.git.tag) | sed -E -n 's%vyos/%%p')

This will give the vyos version tag as a version prepended by number of commits since the tag and the git current commit id in this format:
<tag>-<commits since tag>-g<commit id> example: 1.3.0-1-g1234567 and 1.3dev0-23-g1234567
in this example the tag name used is vyos/1.3.0
without the --long parameter the tagged commit will be named with just the tag version number, and will not get -<commits since tag>-g<commit id> appended
when the package is built from a modified source with uncomitted work, the version name will be padded with -dirty

the deb/apt version ordering scheme is not self explained, so i've generated a list of possible version names and ordered them in descending order using python apt_pkg.version_compare:

Original order               Sorted order                Upstream Version
1.3.2                      - 1.3.rolling-23-g1234567   : 1.3.rolling-23
1.3.0                      - 1.3.rolling-15-g1234567   : 1.3.rolling-15
1.3.rolling                - 1.3.rolling               : 1.3.rolling
1.3dev1                    - 1.3.dev-15-g1234567       : 1.3.dev-15
1.3                        - 1.3.dev-20.g1234567       : 1.3.dev
1.3dev5                    - 1.3.dev                   : 1.3.dev
1.3.30                     - 1.3.30                    : 1.3.30
1.3.1                      - 1.3.25                    : 1.3.25
1.3dev-10-g1234567         - 1.3.15dev                 : 1.3.15dev
1.3.dev                    - 1.3.15                    : 1.3.15
1.3dev2                    - 1.3.2                     : 1.3.2
1.3dev                     - 1.3.1                     : 1.3.1
1.3.15                     - 1.3.0                     : 1.3.0
1.3.rolling-23-g1234567    - 1.3rolling                : 1.3rolling
1.3.15dev                  - 1.3dev-10-g1234567        : 1.3dev-10
1.3rolling                 - 1.3dev5                   : 1.3dev5
1.3-dev                    - 1.3dev2                   : 1.3dev2
1.3.rolling-15-g1234567    - 1.3dev1                   : 1.3dev1
1.3.25                     - 1.3dev                    : 1.3dev
1.3.dev-15-g1234567        - 1.3-dev                   : 1.3
1.3.dev-20.g1234567        - 1.3                       : 1.3

from this list i'm proposing this naming scheme:
rolling versions will have the version number of the NEXT major version until the point that the release is frozen. at that point it increment to the next version. eg. the current rolling is 1.3 rolling, this version will be until 1.3.0 is released. at that point 1.4 rolling is released and 1.3 does not have a rolling release any more.

To keep the current development version numbers as low as possible the tag for them will be 1.3dev0, 1.4dev0 and so on. please mark there are no special characters between the version and name. this is to keep the version index under the 1.4.0 name of the first 1.4 release. this will make the third commit from initial 1.4dev release's version name to be : 1.4dev0-3-g1234567 the development version id will be helt until the next release, or until a development milestone is reached. at that point the devel version increment with an index. from 1.4dev0 to 1.4dev1

All release versions will have the same tree octet layout as found in 1.1.x and 1.2.x releases. the first 1.3 release will be named 1.3.0 and the complete package name for that version will be 1.3.0-0-g1234567 as described before, the forth part of the version number will increment once for every commit, so that will increment until version 1.3.1 is released and the counter is reset to zero.

when a package is unchanged between two major versions, it is good to add an commit to increment the targeted vyos version info for the package , and then we also have a commit to attach the version tag

the full version syntax for all releases will then be <major version>(.<minor version>|dev<dev-index>)[-<commits from tag>-g<git hash>][-dirty]

to implemenent this the following needs to be done:

  • Decide if the tagged version should have a short or long version name
  • Add a vyos/1.3dev0 tag on all internal packages (needs to be done directly on the repository)
  • add code to debian/rules in package(can be added via PR)

This is not taken into effect for now, so this is still open for discussion.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible

Event Timeline

runar triaged this task as Normal priority.Apr 10 2020, 8:46 PM
runar created this task.
runar created this object in space S1 VyOS Public.
runar updated the task description. (Show Details)Apr 10 2020, 9:06 PM
runar added a comment.Apr 10 2020, 9:12 PM

Change description since original update.
versioning of 1.3dev-3-g1234567 will count as newer then 1.3dev3-3-g1234567 this means that all dev releases needs to have a initial index. i've added it indexed from zero.
for a full version list see here

Original order               Sorted order                Upstream Version
1.3dev-0-g1234567          - 1.3.1-2-g1234567          : 1.3.1-2
1.3dev0-0-g1234567         - 1.3.1                     : 1.3.1
1.3.1-2-g1234567           - 1.3.0-7-g1234567          : 1.3.0-7
1.3.1                      - 1.3.0-3-g1234567          : 1.3.0-3
1.3dev2-8-g12345671.3.0    - 1.3dev-4-g1234567         : 1.3dev-4
1.3.0-7-g1234567           - 1.3dev-0-g1234567         : 1.3dev-0
1.3dev                     - 1.3dev2-8-g12345671.3.0   : 1.3dev2-8
1.3dev-4-g1234567          - 1.3dev2                   : 1.3dev2
1.3dev1                    - 1.3dev1-4-g1234567        : 1.3dev1-4
1.3dev2                    - 1.3dev1                   : 1.3dev1
1.3dev1-4-g1234567         - 1.3dev0-1-g1234567        : 1.3dev0-1
1.3.0-3-g1234567           - 1.3dev0-0-g1234567        : 1.3dev0-0
1.3dev0-1-g1234567         - 1.3dev                    : 1.3dev
runar updated the task description. (Show Details)Apr 10 2020, 9:21 PM
runar added a comment.Apr 10 2020, 9:24 PM

Change description since last update:

added information about using only the relese name on the tagged commit and that building from a source with uncomitted changed will result in -dirty beeing added to the version name

runar updated the task description. (Show Details)Apr 10 2020, 9:38 PM
pasik added a subscriber: pasik.Apr 11 2020, 9:31 AM
thomas-mangin added a subscriber: thomas-mangin.EditedApr 11 2020, 10:12 PM

Naming convention seems fine.

Why not alternate stable release and dev, so even numbers are stable and odd are development. The next release would be 1.4, people who are on 1.3 would know they are behind a stable and should upgrade. The next development becoming 1.5 .. etc.

equuleus would be both 1.3 and 1.4 with 1.3 becoming unsupported as soon as 1.4 is released and the development move on 1.5

All sounds good to me.

elbandi added a subscriber: elbandi.May 9 2020, 7:37 PM

you can use '+' and '~' chars in versions. plus gives newer version (1.2.3-4-5+foo32 > 1.2.3-4-5 ) but less than the new upstream (1.2.3-4-6 > 1.2.3-4-5+foo32), tilde gives older version (1.2.3-4-5~foo-2 < 1.2.3-4-5 )
i always use + for my own builds, thats gives me newer package version than ubuntu/debian version.

Hi @elbandi !

Yes, i'm aware of these modifiers. But the issue here is not to generate newer then the upstream, because we are the upstream. these changes are to make the version visible in our upstream packages. the current solution with manually versioning does not work because the Debian version is "never" incremented. (there could be hundreds of commits between each version increase..) and in the mean time it is quite hard to identify exactly what changes are made to the package in the image. for this we automatically change the version tag on build-time and adds the most resent git version tag and the number of commits since that tag visible in the upstream version tag. as for the "downstream" part of the version tag we out the git commit id and info about the "state" of the repo on build-time. those we also could identify if the package build is indeed the upstream package or a custom package by the user. :)
upstream version will be in the format of : <git-version-tag> - <commits from tag> and downstream version: -g<commit-id>(+<dirty>)? an example : 1.3dev0-4 -g1234567 and 1.3.0-0 -g1234567

se also the pr for this: https://github.com/vyos/vyos-1x/pull/406

i would leave the items in changelog, just every changelog items have git tag. if there is a new release (=new _stable_ release iso), make a new changelog item + git tag.
and for the development packages use your versioning: 1.3.0-17+dev<commitcountfromtag>-git<commit-id>
anyway, there is a "gbp dch" command can generate entries from Git commit messages. you you can fill the "dev" changelog entries with real commit messages. you can set the versionstring too.

VER=$(dpkg-parsechangelog --show-field Version)
COMMIT_COUNTS=$(git rev-list $VER.. --count)
LAST_ID=$(git rev-parse --short HEAD)
git config user.name "Vyos CI"
git config user.email ci@vyos.com
gbp dch --full -s $VER -D unstable -U low -N $VER+dev$COMMIT_COUNTS-git$LAST_ID --ignore-branch --git-author
runar added a comment.EditedMay 10 2020, 6:27 PM

VyOS dont provide the packages upstream to anyone, and a package is only installed at image create time and never upgraded. And as the changelog have up to now newer been used i dont see the point of over-complicating this. This will only make it harder to make a release image as more unnecessary (as i would call it) steps are added to the process.

As I see it the only reason for VyOS to build deb package are to make a fluent provisioning of an image using Debian and and live-build, nothing more.

I also see some issues using this script:

  • applying your script can not be done on build-time and will need a separate step by the user to apply it. this will make it error-prone and will give the wrong version information if the user does not apply it prior to building. applying this script at build-time will only work the first time as on the second run the dev changelog entry is already added and will give an incorrect VER variable and those failing all subsequent executions.
  • changing the changelog will make a "out of band" change to the repository that the user will need to revert prior to switching branches etc.
  • You change the Git configured user and email inside the script and then also changing the uses private git config. <-- but i guess this could be omitted on a user build.
  • the script does not see when the user has changed the codebase after checkout. and those the package is not marked when the user have done local changes

I you could provide a script that fixes those thee issues i'm gladly looking into it, but if not i think my patch is a fully working solution for our needs.

runar added a comment.May 10 2020, 6:33 PM

I've added an extra bulletpoint that needs to be fixed in the comment prior to this one.

For _stable_ release build, you need a fixed point in every vyos/vyatta package (vyos-1x=1.3.1, vyos-netplug=1.2.9.2-3, vyos-opennhrp=0.14.1-1, etc), so you have to release every package, something like 'Oh, this package is in this point stable enought for a stable release', and the stable iso contains this stable packages. But the development doesnt stop, some packages receive new fetaures. Until stable packages are waiting for each other, need a versioning for daily snapshot. for that, the commit count based versioning is good.

My example script is a step in a CI build: code checkout, modify changelog based on commit, build the package, put deb file somewhere. no need anyone to modify/revert the changelog file. The +dev123 versions made by the CI. Only if you make a new release for stable build, but that is a "special case". (after checkout the code to ci workdir, it's not a problem changing the user.name/email. and this changelog entry doesnt need to store, because it's generated from commit logs, anyone/anytime can be renegerated (particularly on stable release)

jjakob added a subscriber: jjakob.May 10 2020, 8:53 PM

I did something similar, including generating the changelog with name/email and releasing, here: https://github.com/jjakob/vyos-build/commit/8353f1653be76c5b200b9357ebb9d76a84d3139c
The version name generated is also one I arrived at after some consulting, what confused me the most was what the precedence of the special characters is, that wasn't very well documented in debian's docs - elbandi's suggestions are all correct and very good.