Alfs Progress Bar

When building source packages and especially automated linux from scratch (ALFS) it would be nice to have a progress bar that shows a realistic estimate of how the build is progressing. But the progress bar supplied with jhalfs is disappointing.

This replacement progress_bar.sh was created to provide more useful feedback. It requires a file named pkg_sbus_list in order to be more useful, although it doesn't require that file in order to run. The progress bar is a bash script.

* UPDATE *
A newer, more efficient version is a perl script progress_bar.pl with a bash script introducer progress_bar.sh and the pair can be a drop in replacement for the original progress_bar.sh. Much of the increased efficiency is due to the fact that perl implements sleep by doing a nanosleep system call whereas bash launches a child process for each sleep.
The new progress display looks like:

 Building target 1044-expat
      0:28  0%..............100%  (0.1) 19:42:58 EST 2010/02/07

• A colored bar fills from 0% to 100%.
• Above 100%, it fills from 0% to 1000%.
• It shows the date and time of estimated completion.
• The (0.1) to the left of the date is the number of sbus.
• The 0:28 is a time counter.
• It is extremely faster than the original progress bar.
• The speed is on par with the perl module Term::StatusBar.

* UPDATE -END *

This replacement progress_bar.sh needs to be copied to the build location which is likely to be something like:
/mnt/build_dir/jhalfs
The pkg_sbus_list also needs to be there.

The pkg_sbus_list file is intended to be of the form:
001-foo 1.2
012-bar 0.5
102-etc-etc 22

The column on the left are build targets and the column on the right are estimated times in SBU units. Each package in the LFS and BLFS books gives an estimated build time in SBU units. One SBU is defined as the number of seconds required to build binutils-pass1 on the machine.

The pkg_sbus_list can have optional entries starting with 99999 which can be used as parameter data to affect the behavior of the progress bar:

99999-one-sbu [seconds] {override binutils-pass1 log or guess}
99999-sleep-time [seconds.nanoseconds]
99999-use-unix-time [y/n] {y = date as a unix timestamp}
99999-silence-date [y/n] {y = no line with date and totaltime}
99999-format-time [y/n] {n = times in seconds}
99999-time-count-down [y/n] {y = count down from est time}
99999-silence-time [y/n] {y = no times}
99999-silence-total [y/n] {y = no total time/percent}
99999-silence-percent-total [y/n] {y = no total percent}

99999-no-progress [none/date/sbu/datesbu/nobar/fancy]
=
99999-no-progress none {exit without doing anything}
99999-no-progress date {print date and exit}
99999-no-progress sbu {print est SBU and exit}
99999-no-progress datesbu {print date, SBU and exit}
99999-no-progress nobar {only percent complete and time}
99999-no-progress fancy {print arrowhead instead of dots}

Example: These settings give a quiet display only updating percentage complete every 2 seconds:
99999-no-progress nobar
99999-silence-date y
99999-silence-time y
99999-sleep-time 2

With default settings it might look like this:

sh progress_bar.sh 071-mpfr
      1.1 sbu    Wed Jan 27 11:34:54 EST 2010   (19%)    eta 10:06    (18:39:27)
 4%  [...                                             ]       0:26         0.05

The parentheses above indicate total percentage complete and total time remaining respectively. The default setings are the same as having no 99999 things, or having all 99999 things blank.

With these settings it might look like this:
99999-no-progress fancy
99999-silence-total y
99999-use-unix-time y

sh progress_bar.sh 071-mpfr
      1.1 sbu    1264610907 unix time                    eta 10:06
 9%  [=====>                                          ]       1:00         0.11

The number off to the right above (0.11) is the elapsed number of SBUs so far. That was added as an afterthought with the thinking that since SBU estimates are only estimates, this would be convenient for revising the estimates.


You can create pkg_sbus_list manually with an editor in the build directory. But, if you use jhalfs to create a Makefile in the build directory, there is an easier way.

This tool setup_progress_bar.sh is designed to read the Makefile generated by jhalfs and create the pkg_sbus_list. In order to do that, it needs to know the location of the book xml files. If you are in the jhalfs-[version] directory where you ran "make" to configure the build, it can compute the location of the book xml files. This is the directory that contains the files jhalfs and configuration. If this is the case, then the pkg_sbus_list can be generated with the command line:
sh setup_progress_bar.sh
The resulting pkg_sbus_list will appear in the build location.

Otherwise, if you are in the destination build location, you must specify the location of the book xml files by using a parameter of the form:
LFS=../lfs-6.5-xml
on the command line.

For an example, suppose that the build target location is /mnt/build_dir and I have run jhalf's "make" to configure the system, and then I had changed directory to /mnt/build_dir/jhalfs. Suppose further that the book xml files are located in /mnt/build_dir/lfs-6.5-xml. I could then generate the pkg_sbus_list by running this command line:
sh setup_progress_bar.sh LFS=../lfs-6.5-xml

If the pkg_sbus_list doesn't have any SBU numbers, that could indicate that it did not find the appropriate xml files. There can be some targets that do not have an SBU number.

The original intended operation is that the replacement progress bar would be run automatically as jhalfs builds the system. The progress bar runs "forever" until a file by the name of the target appears in the build directory at which time it exits. When jhalfs starts a target, it launches progress_bar.sh providing the target name as a parameter. When jhalfs completes a build step without errors, it "touches" a file by the name of the target and when the progress bar sees it, it exits.

The above mechanism can also be simulated with a script or manual typing. For example,
sh progress_bar.sh foo
will keep going only until a file by the name of foo appears in the current directory.

If playing with progress bar and you break out of it, cursor and
line wrap will be off. They can be turned on again with:
echo -e '\033[?25h\033[?7h'

A spinoff project is total_elapsed.sh which could indicate progress from a different point of view. It also requires the pkg_sbus_list for the computations. This program probably only works along with jhalfs because it was designed based on observations of how the jhalfs build system works. It is meant to be run in the build dir from another terminal or screen. If it is given the parameter "once", then it prints one snapshot and exits. Without the parameter "once", it refreshes every second. When a target is completed, the target shown changes and the totals are recalculated. A snapshot could look like this:

sh total_elapsed.sh once

TARGET                           TIME    RUNNING  REMAINING    ESBUS
064-linux-headers                0:55                  0:55      0.0
total                        23:09:26    4:29:59   18:39:27

In the above, the next target is 064-linux-headers, but it hasn't started building yet, so it is blank in the RUNNING column. When the target 064-linux-headers starts building, it will find out about it and will start updating the RUNNING and REMAINING columns.

Sometime, I might add more progress_bar_notes here.