2019-02-12

Modifying boot files with SmartOS under Loader

With the advent of newboot in SmartOS/Triton, newly-installed systems will use loader as the bootloader, replacing grub. See RFD 156 for some technical background on the motivation of the switch.

It's often the case that people want to make some modification to an /etc file in subsequent SmartOS boots. As we boot from ramdisk, we can't just directly modify the files. As originally described on Keith's blog the way to get around this problem involves specifying specific files to over-ride the default.

Obviously this has changed under loader. Let's presume we want to over-ride /etc/system to set kmem_flags. First, let's take a copy of our file and edit it:

# sdc-usbkey mount
/mnt/usbkey
# mkdir -p /mnt/usbkey/etc
# cp /etc/system /mnt/usbkey/etc/    # or /mnt/usbkey/kernel/drv/dtrace.conf etc.
# echo "set kmem_flags=0xf" >>/mnt/usbkey/etc/system
Now we want loader to prepare this file as a bootfs module. In grub, we used something like "module /os/bootfs/etc/system type=file name=etc/system". For loader, it's similar:
# cd /mnt/usbkey/boot
# echo etc_system_load=YES >>loader.conf.local
# echo etc_system_type=file >>loader.conf.local
# echo etc_system_name=/etc/system >>loader.conf.local
The prefix (etc_system_) is fairly arbitrary, though often named after the module. For each file you want, you'd want a _load, _type, and _name entry specified.

Note that loader is a little more restrictive than grub: we need the path to match exactly (so no /os/bootfs sub-directory). If this all worked OK, then we should see during boot something like:

Loading /os/20190207T125627Z/platform/i86pc/kernel/amd64/unix...
Loading /os/20190207T125627Z/platform/i86pc/amd64/boot_archive...
Loading /os/20190207T125627Z/platform/i86pc/amd64/boot_archive.hash...
Loading /etc/system...
Booting...
SunOS Release 5.11 Version joyent_20190209T105033Z 64-bit
Copyright (c) 2010-2019, Joyent Inc. All rights reserved.
WARNING: High-overhead kmem debugging features enabled (kmem_flags = 0xf)...
And we should find a copy of our modified file here:
# tail /system/boot/os/etc/system 
* Use hires tick to improve some scheduling latency issues
set hires_tick=1

*
* The "buildstamp" module contains information about the git sources used to
* build the operating system image.  Once loaded, the driver will refuse to
* detach.  We load it early to ensure it is in a crash dump wherever possible.
*
forceload: drv/buildstamp
set kmem_flags=0xf