Page MenuHomePhabricator

Config load fails in ConfigTree with ValueError: Failed to parse config: lexing: empty token
Needs testing, NormalPublic

Description

A config file that loads fine in 1.2-rolling-20190623 fails to load after somewhere around 1.2-rolling-201908* with the error:

load /mnt/vyos-config/config.boot-busted
Traceback (most recent call last):
  File "/opt/vyatta/etc/config-migrate/migrate/dns-forwarding/0-to-1", line 34, in <module>
    config = ConfigTree(config_file)
  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 167, in __init__
    raise ValueError("Failed to parse config: {0}".format(msg))
ValueError: Failed to parse config: lexing: empty token
Loading configuration from '/mnt/vyos-config/config.boot-busted'
Called process error: Command '['/opt/vyatta/etc/config-migrate/migrate/dns-forwarding/0-to-1', '/tmp/tmpuyo4w_hq']' returned non-zero exit status 1..
[edit]

I first encountered this on an upgrade but it also fails when loading the saved config into the latest 1.3-rolling-201910180142.
Looking at the relevant parts of ConfigTree reveals that it's a wrapper around a compiled C library so I'm at a loss at how to debug this.
I can't provide the unredacted config for testing unfortunately so I can't provide a way to reproduce this. I have the redacted config with all IPs changed with x.x.x.x that I can provide but it's quite large so not possible to manually make usable IMO.
I tried to test it via python console but I can't get the module to work (in python3 ran >>> c = vyos.configtree.ConfigTree(""" desc "a->b" """), I get NameError: name 'vyos' is not defined)

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible

Event Timeline

jjakob created this task.Oct 19 2019, 3:42 AM
jjakob created this object in space S1 VyOS Public.
jjakob added a comment.EditedOct 19 2019, 4:18 AM

Via the process of elimination I found the culprit - comments with special characters in them, that at some point in the upgrade cycle got converted into backslash-escped decimal:

description "\197\160" # this was once Š
description "\196\140 \196\141" # Č č

\197\160 is C5 A0 which is U+0160, which was the original character.
So old versions allowed these characters in comments and worked fine with them, then at one point an upgrade converted them to backslash decimal UTF-8, which still worked until at some point they started causing the error.

pasik added a subscriber: pasik.Oct 20 2019, 1:22 PM
syncer changed the task status from Open to Needs testing.Sat, Nov 16, 11:10 PM
syncer assigned this task to zsdc.
syncer triaged this task as Normal priority.
syncer removed a project: VyOS 1.2 Crux.

The replacement of special characters with backslash decimal happened on upgrade from 1.2.0-rc11 to 1.2.0-rolling+201906231514.