Page MenuHomeVyOS Platform
Create Task
Maniphest T1787

Failed config migration from V1.2.3 to 1.2-rolling-201911030217
Closed, ResolvedPublic
Actions

Description

in reference to this forum thread: https://forum.vyos.io/t/failed-config-migration-from-v1-2-3-to-1-2-rolling-201911030217/4322/8

I just tried to upgrade a very simple VyOS 1.2.3-LTS labor instance to 1.2-rolling-201911030217. Config migration failed with this console output:

[   20.983324] vyos-router[1090]: Started watchfrr.
[   20.996641] vyos-router[1090]: Mounting VyOS Config...done.
[   21.112754] vyos-router[1090]: Starting VyOS router: migrateTraceback (most recent call last):
[   21.120689] vyos-router[1090]: File "/opt/vyatta/etc/config-migrate/migrate/dns-forwarding/0-to-1", line 38, in <module>
[   21.133234] vyos-router[1090]: config = ConfigTree(config_file)
[   21.137767] vyos-router[1090]: File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 175, in __init__
[   21.144921] vyos-router[1090]: raise ValueError("Failed to parse config: {0}".format(msg))
[   21.152688] vyos-router[1090]: ValueError: Failed to parse config: lexing: empty token
[   21.157479] vyos-router[1090]: Called process error: Command '['/opt/vyatta/etc/config-migrate/migrate/dns-forwarding/0-to-1', '/opt/vyatta/etc/config/config.boot']' returned non-zero exit status 1.
[   44.109480] vyos-router[1090]: rl-system firewall configure failed!
[   44.487986] vyos-config[1160]: Configuration error

show version before update:

Version:          VyOS 1.2.3
Built by:         Sentrium S.L.
Built on:         Fri 13 Sep 2019 11:15 UTC
Build UUID:       bb328444-2f89-4d39-8dab-068278c09194
Build Commit ID:  24f1a74bc88f3a

Architecture:     x86_64
Boot via:         installed image
System type:      Microsoft Hyper-V guest

Hardware vendor:  Microsoft Corporation
Hardware model:   Virtual Machine

DNS forwarder config before update:

dns {
    forwarding {
        allow-from 127.0.0.1/32
        cache-size 0
        domain intern {
            server 172.16.13.254
        }
        ignore-hosts-file
        listen-address 127.0.0.1
        name-server 1.1.1.1
    }

I also tried update to rolling version 201911040217, same result. The V1.2.3 config is attached as file to this ticket.

I see the same problem on another (not labor) instance of VyOS on AWS with older rolling version (201908130337) upgrade to current rolling.

dns {
    forwarding {
        cache-size 0
        domain intern {
            server 172.16.10.41
        }
        ignore-hosts-file
        listen-address 127.0.0.1
        listen-address 172.16.100.10
        name-server 172.16.100.2
    }
}

config.boot4 KBDownload

Details

Difficulty level
Unknown (require assessment)
Version
1.2-rolling-201911030217
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

Line2 created this task.Nov 4 2019, 2:52 PM
Line2 created this object in space S1 VyOS Public.
Line2 updated the task description. (Show Details)Nov 4 2019, 3:07 PM
c-po triaged this task as High priority.Nov 4 2019, 4:06 PM
c-po added a project: VyOS 1.3 Equuleus.
c-po changed Version from - to 1.2-rolling-201911030217.
c-po added a subscriber: c-po.Nov 4 2019, 4:55 PM

What is your VMs CPU/RAM configuration?

Line2 added a comment.Nov 4 2019, 4:57 PM

1vCPU, 512MB RAM

c-po added a comment.Nov 4 2019, 5:28 PM

This is weird. I can not reproduce it either with VMWare vSpher enot using Hyper-V.

I took your config and only did two litte modifications

  • remove hw-id on eth0
  • set vyos user plaintext password vyos
interfaces {
    ethernet eth0 {
        address 172.16.13.255/31
        description "Transit"
        duplex auto
        ip {
            ospf {
                authentication {
                    md5 {
                        key-id 3 {
                            md5-key key
                        }
                    }
                }
                dead-interval 40
                hello-interval 10
                network broadcast
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        smp-affinity auto
        speed auto
    }
    loopback lo {
        address 172.20.1.2/32
        ip {
            ospf {
                authentication {
                    md5 {
                        key-id 3 {
                            md5-key m3t
                        }
                    }
                }
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
    }
}
policy {
}
protocols {
    ospf {
        area 0.0.0.0 {
            authentication md5
            network 172.20.1.2/32
            network 172.16.13.254/31
        }
        log-adjacency-changes {
        }
        parameters {
            abr-type cisco
            router-id 1.0.1.2
        }
        passive-interface default
        passive-interface-exclude eth0
    }
    static {
        route 0.0.0.0/0 {
            next-hop 172.16.13.254 {
            }
        }
    }
}
service {
    dns {
        forwarding {
            allow-from 0.0.0.0/0
            cache-size 0
            domain intern {
                server 172.16.13.254
            }
            ignore-hosts-file
            listen-address 127.0.0.1
            name-server 1.1.1.1
        }
    }
    lldp {
        interface eth0 {
        }
        management-address 172.16.13.255
        snmp {
            enable
        }
    }
    snmp {
        contact "contact"
        description test
        listen-address 172.16.13.255 {
        }
        trap-target 172.16.10.45 {
        }
        v3 {
            engineid 0x80001f8880e251911b8c9e165a00000000
            group nmsreadonly {
                mode ro
                seclevel priv
                view all
            }
            user nms {
                auth {
                    encrypted-key 0x901fdcf12baab2118ef0b81eb60e110407959220
                    type sha
                }
                engineid 0x80001f8880e251911b8c9e165a00000000
                group nmsreadonly
                mode ro
                privacy {
                    encrypted-key 0x901fdcf12baab2118ef0b81eb60e1104
                    type aes
                }
            }
            view all {
                oid 1 {
                }
            }
        }
    }
    ssh {
        disable-host-validation
        loglevel VERBOSE
        port 22
    }
}
system {
    config-management {
        commit-revisions 200
    }
    domain-name m3t.ch
    host-name router2
    login {
        user vyos {
            authentication {
                plaintext-password "vyos"
            }
            level admin
        }
    }
    name-server 127.0.0.1
    ntp {
        server time.domain.intern {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
        host 172.16.10.45 {
            facility all {
                level info
            }
        }
    }
    time-zone Europe/Zurich
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack-sync@1:conntrack@1:dhcp-relay@2:dhcp-server@5:dns-forwarding@1:firewall@5:ipsec@5:l2tp@1:mdns@1:nat@4:ntp@1:pptp@1:qos@1:quagga@3:ssh@1:system@9:vrrp@2:vyos-accel-ppp@1:wanloadbalance@3:webgui@1:webproxy@1:webproxy@2:zone-policy@1" === */
/* Release version: 1.2.3 */

and then upgraded to vyos-1.2-rolling-201911030217-amd64.iso

By any chance can you pass me your complete vhd as I'm unable to reproduce this.
does it happen all the time on upgrade?

Line2 added a comment.Nov 4 2019, 5:42 PM

that's really strange. As I can reproduce the problem on both intances in 100% of the cases. But not the same config and not the same inital VyOS version.
Can I do some troubleshooting steps for you?
I can also give you the vhd, but on a private channel if possible (is it possible on slack?)

Line2 added a comment.Nov 4 2019, 5:50 PM

600MB

c-po claimed this task.Nov 4 2019, 8:01 PM
c-po changed the task status from Open to Confirmed.Nov 4 2019, 8:29 PM

Confirmed using a copy of the Hyper-V VM

pasik added a subscriber: pasik.Nov 4 2019, 10:04 PM
c-po added a comment.Nov 7 2019, 9:09 PM

The real problem is the config parser it seems as it can not deal with special characters used in a pre-login banner.

jestabro added a subscriber: jestabro.Nov 8 2019, 3:09 PM

cf. T1001 as a related issue:

https://phabricator.vyos.net/T1001

What is needed is the appropriate regex in config.py to escape those backslashes in the output of showConfig that are not part of escape sequences as recognized by lexer; I can add.

c-po moved this task from Need Triage to In Progress on the VyOS 1.3 Equuleus board.Nov 12 2019, 9:20 PM
jestabro added a subtask: T1801: Unescaped backslashes in config values cause configuration failure.Nov 12 2019, 9:56 PM
jestabro changed the status of subtask T1801: Unescaped backslashes in config values cause configuration failure from Open to In progress.Nov 12 2019, 10:01 PM
jjakob added a subscriber: jjakob.Nov 17 2019, 10:48 AM

This is related/a duplicate of T1744

c-po removed a subtask: T1801: Unescaped backslashes in config values cause configuration failure.Nov 17 2019, 10:54 AM
c-po added a parent task: T1801: Unescaped backslashes in config values cause configuration failure.
Line2 mentioned this in T1801: Unescaped backslashes in config values cause configuration failure.Nov 24 2019, 12:02 PM
c-po reassigned this task from c-po to jestabro.Nov 24 2019, 7:31 PM
jestabro closed this task as Resolved.Dec 4 2019, 4:27 PM

This is reported as resolved by Line2 in parent T1801; the related issue mentioned therein is also resolved by the revised fix in T1801.

c-po moved this task from In Progress to Finished on the VyOS 1.3 Equuleus board.Jan 30 2020, 8:22 PM
erkin set Issue type to Bug (incorrect behavior).Aug 31 2021, 6:22 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Aug 30 2022, 2:06 PM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.