Properly mount /live/persistence when booting from sdb (current)
Closed, ResolvedPublic

Description

I'm looking at the current branch, currently at 999.201608060237, installed by 'add system image http://dev.packages.vyos.net/iso/current/amd64/vyos-999.201608060237-amd64.iso'

Because of the way I've had to install VyOS on my machine, it must boot from sdb. However, something seems to be hardcoding sda, so the boot messages look like this:

Loading, please wait...
[    3.899420] sd 0:0:0:0: [sda] No Caching mode page found
[    3.905383] sd 0:0:0:0: [sda] Assuming drive cache: write through
mount: mounting /dev/sda2 on /live/persistence/ failed: No such device

Welcome to Debian GNU/Linux 8 (jessie)!

         Expecting device dev-ttyS1.device...
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Paths.
[  OK  ] Set up automount Arbitrary Executable File Formats F...utomount Point.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
...

This also means that a lot of the config files that should be persistent (such as /opt/vyatta/etc/config/scripts/vyatta-postconfig-bootup.script) are not.

Another symptom is that /opt/vyatta/sbin/vyos-persistpath prints the following:

/lib/live/mount/persistence/

Notice that $boot_device (which should appear after that final slash) is empty, because no mount matches 'upperdir=/live/persistence/[^/]*/boot'.

In fact, there is no sdb2, either. Here's the output of lsblk:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    1   7.3G  0 disk
├─sda1   8:1    1   500M  0 part
├─sda2   8:2    1   745M  0 part
└─sda3   8:3    1   6.1G  0 part
sdb      8:16   0  14.8G  0 disk
└─sdb1   8:17   0  14.8G  0 part /lib/live/mount/persistence
loop0    7:0    0 242.7M  1 loop /lib/live/mount/rootfs/999.201608060237-.squash

And here are the contents of /proc/mounts:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=809916k,mode=755 0 0
/dev/sdb1 /lib/live/mount/persistence ext4 rw,noatime,data=ordered 0 0
/dev/loop0 /lib/live/mount/rootfs/999.201608060237-.squashfs squashfs ro,noatime 0 0
tmpfs /lib/live/mount/overlay tmpfs rw,relatime 0 0
overlay / overlay rw,noatime,lowerdir=/live/rootfs/999.201608060237-.squashfs/,upperdir=/live/persistence/boot/999.201608060237-current/rw,workdir=/live/persistence/boot/999.201608060237-current/work 0 0
/dev/sdb1 /opt/vyatta/etc/config ext4 rw,noatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=10240k,nr_inodes=504408,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /opt/vyatta/config tmpfs rw,nosuid,nodev,relatime,nr_inodes=0,mode=775 0 0
/dev/sdb1 /boot ext4 rw,noatime,data=ordered 0 0
/dev/sdb1 /boot/grub ext4 rw,noatime,data=ordered 0 0
/dev/sdb1 /lib/live/mount/persistence/boot/999.201608060237-current/grub ext4 rw,noatime,data=ordered 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

I'm not quite sure how this is intended to work, at this point...

Details

Difficulty level
Normal (likely a few hours)
syncer triaged this task as Normal priority.Aug 6 2016, 10:19 AM
syncer added a project: VyOS 1.1.x (1.1.8).

Once we have device specific images, this will be solved,
but hardcoded values if they exist not great at all :)

It actually looks worse than it is!
The first message is a pain in the ass :( it is live boot trying to find persistence.. it should not be verbose, but somehow it is.

The $boot_device issue i have to check.
and sda is not hard-coded.
and your /proc/mounts looks good.

Can you maybe check the labels of you partitions "sudo blkid"
there should be one which is labeled "persistence", can you check which one that is?

Can you check if on the disk/part is a file called persistence.conf or persistence.conf.<sdxx>

Also there was a problem witch configs saving/overwriting... this has been solved from:
http://dev.packages.vyos.net/iso/current/amd64/vyos-999.201608091126-amd64.iso

so maybe you can use that image!

/opt/vyatta/sbin/vyos-persistpath Is used for an other type of install, not for add system image.

Unfortunately, I'm traveling right now, so I'll have to try out a newer image and give you the output from 'sudo blkid' in three weeks. I'll look forward to some good progress when I return!

Aha!. I've tried 999.201609070235 (current). Things look quite a bit better; /opt/vyatta/etc/config/scripts/vyatta-postconfig-bootup.script is now persisted, and things seem to start up and run quite nicely.

For reference, here is the output of blkid:

$ sudo blkid
/dev/sda1: UUID="c777e11a-9162-4dfc-9d5e-d691bd215bbf" TYPE="xfs" PARTUUID="000cd8a7-01"
/dev/sda3: UUID="a0e3f85b-6f7f-4c7b-a872-dcfa1398741e" TYPE="xfs" PARTUUID="000cd8a7-03"
/dev/sdb1: LABEL="persistence" UUID="0f1e61eb-6769-4458-872f-9522aa0c127b" TYPE="ext4" PARTUUID="c407f6b6-01"
/dev/sda2: UUID="187a6769-99c4-4433-960c-d88ed8b45283" TYPE="swap" PARTUUID="000cd8a7-02"
/dev/loop0: TYPE="squashfs"

The partitions on /dev/sda are the CentOS install that the Netgate box comes with (in its built-in flash); I haven't touched that. /dev/sdb1 is VyOS (on an mSATA flash device). This looks reasonable to me.

So, I suppose the problems I was seeing were related to the config saving/overwriting problem that has since been fixed (Is there a task or commit id I can look at for that?) and the mount: mounting /dev/sda2 on /live/persistence/ failed: No such device message was just a red herring.

Here is what is printed on boot, for the record:

  Booting 'VyOS 999.201609070235-lithium (Serial console)'

Loading, please wait...
[    3.888964] sd 0:0:0:0: [sda] No Caching mode page found
[    3.894928] sd 0:0:0:0: [sda] Assuming drive cache: write through
mount: mounting /dev/sda2 on /live/persistence/ failed: No such device

Welcome to Debian GNU/Linux 8 (jessie)!

         Expecting device dev-ttyS1.device...
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Paths.
[  OK  ] Set up automount Arbitrary Executable File Formats F...utomount Point.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Created slice system-getty.slice.
         Starting Increase datagram queue length...
         Mounting Huge Pages File System...
         Mounting POSIX Message Queue File System...
         Starting Create list of required static device nodes...rrent kernel...
         Mounting Debug File System...
         Starting live-config contains the components that co... userspace)....
         Starting udev Coldplug all Devices...
[  OK  ] Reached target Slices.
[  OK  ] Mounted Debug File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Started Increase datagram queue length.
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Listening on Syslog Socket.
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Started live-config contains the components that con...te userspace)..
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Kernel Device Manager.
         Starting Copy rules generated while the root was ro...
         Starting LSB: MD array assembly...
         Starting LSB: Set preliminary keymap...
[  OK  ] Started Copy rules generated while the root was ro.
[  OK  ] Found device /dev/ttyS1.
[  OK  ] Started LSB: MD array assembly.
[  OK  ] Started LSB: Set preliminary keymap.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /tmp...
[  OK  ] Mounted /tmp.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting LSB: Change system config parameters based ...onfiguration...
         Starting LSB: Raise network interfaces....
         Starting LSB: live-tools - System Support Scripts...
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
         Starting LSB: Prepare console...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Started LSB: live-tools - System Support Scripts.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started LSB: Prepare console.
         Starting LSB: Set console font and keymap...
[  OK  ] Started LSB: Raise network interfaces..
[  OK  ] Reached target Network.
[  OK  ] Started LSB: Set console font and keymap.
[  OK  ] Started LSB: Change system config parameters based o... configuration.
         Starting Load Kernel Modules...
[  OK  ] Started Load Kernel Modules.
         Starting Apply Kernel Variables...
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on UUID daemon activation socket.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting LSB: Cleans up pppoe peers files...
         Starting Deferred execution scheduler...
[  OK  ] Started Deferred execution scheduler.
         Starting Conntrack Daemon...
[  OK  ] Started Conntrack Daemon.
         Starting Regular background program processing daemon...
[  OK  ] Started Regular background program processing daemon.
         Starting Restore /etc/resolv.conf if the system cras...s shut down....
         Starting OpenBSD Secure Shell session cleanup...
         Starting strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf...
[  OK  ] Started strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf.
         Starting Login Service...
         Starting /etc/rc.local Compatibility...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting System Logging Service...
         Starting Permit User Sessions...
[  OK  ] Started LSB: Cleans up pppoe peers files.
[  OK  ] Started Restore /etc/resolv.conf if the system crash...was shut down..
[  OK  ] Started OpenBSD Secure Shell session cleanup.
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Login Service.
         Starting Serial Getty on ttyS1...
[  OK  ] Started Serial Getty on ttyS1.
[  OK  ] Reached target Network is Online.
         Starting LSB: start and stop the Quagga routing suite...
[  OK  ] Started System Logging Service.
[  OK  ] Started LSB: start and stop the Quagga routing suite.
         Starting VyOS Router...

Welcome to VyOS - vyos ttyS1

vyos login:

I'm going to run this version in production for a while to see if I notice any other problems.

UnicronNL closed this task as Resolved.Sep 15 2016, 8:36 AM
UnicronNL claimed this task.

This is the fix for the config issue...

https://github.com/vyos/live-boot/commit/acea7eb6cb007b5a3554d5874a25fa9c8e469a51

the message

mount: mounting /dev/sda2 on /live/persistence/ failed: No such device

should not be displayed, its live-boot doings its thing, but somehow it became verbose once i added this:

https://github.com/vyos/live-boot/commit/d35d8d1f937cbf764c67c2fc23b8dbaf18707984

And i cant figure out why....

I will close this issue, please reopen if you do not agree.