Page MenuHomePhabricator

Include only firmware we actually need
Open, NormalPublic

Description

Firmware is one of the biggest disk space hogs in VyOS images. As of 1.2.4, it takes 411 megabytes uncompressed (73 M after xz compression).

At the same time, there's a lot of files we don't need. For example, there's firmware for Yamaha's DSPs. The main problem is that individual files are small, but there's so many of them that removing what we don't need is impractical. So far the effort was focused on removing largest offenders. That brings the image size down, but it's only a band aid.

It would be much better to automatically identify the firmware we do need. It may not be practical to have a fully automated process, but I think we can make it "automated with few exceptions".

Kernel modules that need firmware usually export it using the MODULE_FIRMWARE macro. See whiteheat.c for example.

I've made a prototype of a script that finds these macros in modules enabled in the config. If it works out, we can download all firmware at build time, and copy only the files that any enabled module needs.

Here's sample output:

$ ./fwfinder.py drivers/net/ethernet/ .config 2>/dev/null
Referenced firmware: ['"xc0.bin"', '"xc1.bin"', '"xc2.bin"']
Referenced firmware: ['FIRMWARE_NAME']
Referenced firmware: ['"cis/PCMLM28.cis"', '"cis/DP83903.cis"', '"cis/LA-PCM.cis"', '"cis/PE520.cis"', '"cis/NE2K.cis"', '"cis/PE-200.cis"', '"cis/tamarack.cis"']
Referenced firmware: ['FIRMWARE_RX', 'FIRMWARE_TX']
Referenced firmware: ['SLIC_RCV_FIRMWARE_MOJAVE', 'SLIC_RCV_FIRMWARE_OASIS', 'SLIC_FIRMWARE_MOJAVE', 'SLIC_FIRMWARE_OASIS']
Referenced firmware: ['"acenic/tg1.bin"', '"acenic/tg2.bin"']
Referenced firmware: ['FW_MIPS_FILE_06', 'FW_RV2P_FILE_06', 'FW_MIPS_FILE_09', 'FW_RV2P_FILE_09', 'FW_RV2P_FILE_09_Ax']
Referenced firmware: ['FIRMWARE_TG3', 'FIRMWARE_TG3TSO', 'FIRMWARE_TG3TSO5']
Referenced firmware: ['FW_FILE_NAME_E1', 'FW_FILE_NAME_E1H', 'FW_FILE_NAME_E2']
Referenced firmware: ['CNA_FW_FILE_CT', 'CNA_FW_FILE_CT2']
Referenced firmware: ['FW_FNAME', '"cxgb3/t3b_psram-" TPSRAM_VERSION ".bin"', '"cxgb3/t3c_psram-" TPSRAM_VERSION ".bin"', 'AEL2005_OPT_EDC_NAME', 'AEL2005_TWX_EDC_NAME', 'AEL2020_TWX_EDC_NAME']
Referenced firmware: ['FW4_FNAME', 'FW5_FNAME', 'FW6_FNAME']
Referenced firmware: ['FIRMWARE_D101M', 'FIRMWARE_D101S', 'FIRMWARE_D102E']
Referenced firmware: ['"netronome/nic_AMDA0081-0001_1x40.nffw"', '"netronome/nic_AMDA0081-0001_4x10.nffw"', '"netronome/nic_AMDA0096-0001_2x10.nffw"', '"netronome/nic_AMDA0097-0001_2x40.nffw"', '"netronome/nic_AMDA0097-0001_4x10_1x40.nffw"', '"netronome/nic_AMDA0097-0001_8x10.nffw"', '"netronome/nic_AMDA0099-0001_2x10.nffw"', '"netronome/nic_AMDA0099-0001_2x25.nffw"', '"netronome/nic_AMDA0099-0001_1x10_1x25.nffw"']
Referenced firmware: ['QLCNIC_UNIFIED_ROMIMAGE_NAME']
Referenced firmware: ['NX_UNIFIED_ROMIMAGE_NAME']
Referenced firmware: ['FIRMWARE_8168D_1', 'FIRMWARE_8168D_2', 'FIRMWARE_8168E_1', 'FIRMWARE_8168E_2', 'FIRMWARE_8168E_3', 'FIRMWARE_8105E_1', 'FIRMWARE_8168F_1', 'FIRMWARE_8168F_2', 'FIRMWARE_8402_1', 'FIRMWARE_8411_1', 'FIRMWARE_8411_2', 'FIRMWARE_8106E_1', 'FIRMWARE_8106E_2', 'FIRMWARE_8168G_2', 'FIRMWARE_8168G_3', 'FIRMWARE_8168H_1', 'FIRMWARE_8168H_2', 'FIRMWARE_8107E_1', 'FIRMWARE_8107E_2']
Referenced firmware: ['FIRMWARE_NAME']
Referenced firmware: ['"sun/cassini.bin"']
Referenced firmware: ['"tehuti/bdx.bin"']

One issue is that many modules don't reference firmware file by name, but use a #define to reduce duplication. A fully working version will need to run files through a preprocessor (like gcc -I include/ -E $file -o -). Since many modules include files from arch/*/asm, we'll need to run at least make prepare first. From my quick test, it's not enough and it may be best done after a full kernel build to ensure all autogenerated files are in place.

There may theoretically be modules with undeclared firmware requirements.

But, that's a start.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible

Event Timeline

dmbaturin triaged this task as Normal priority.Thu, Jan 9, 7:27 AM
dmbaturin created this task.
Dmitry added a subscriber: Dmitry.Thu, Jan 9, 8:49 AM
pasik added a subscriber: pasik.Thu, Jan 9, 9:25 PM