Package Management by Guidelines

I'm working on some guidelines to use for package management assistance. If the guidelines are followed then the task of package management will be easier. However, the guidelines need only be followed by choice. The package management guidelines shall not be empowered to rule the system with an iron fist. On an lfs system, there is freedom to use your own guidelines, including no guidelines at all. In any case, it can be useful to know what packages are installed and what files were installed with each package.

What I would like to do with all of this is add some capability to manage what is installed on the system and yet still be able to use the commands from the LFS and BLFS books without alteration. Also, if I am thinking about removing or upgrading a package, I would like to be able to determine if there are other installed packages that might become endangered.

The guidelines I am considering involve any mixture of automated, semi-automated, and manual steps. The jhalfs system is my tool of choice for the automated and semi-automated steps. There will be no requirement to use the automated or semi-automated steps. But there are some patterns set forth by the jhalfs system that may prove to be very useful even when doing things manually. There needs to be logging of the process of building a package and there needs to be some method to keep track of which packages are installed and what versions they are. Additionally, there needs to be a means to determine which files get installed during a package installation. If the necessary information gets recorded in a log file, then a tarball can be made of the discerned files prior to removing them. Then, if an ill result is detected, the tarball can be reloaded. If there is confidence that there is no ill result, the tarball can be deleted, or it can be saved for future use. It should be noted that just because files were added or changed simultaneously with the installation of a package, it does not not necessarily mean that each of those files was part of the package. For instance, I installed the bluefish editor and 489 files got a status change. Only 52 files were really installed with bluefish. The rest got a status change because the bluefish install runs update-mime-database. So, I don't think I should be removing 489 files if I uninstall bluefish. But, if I made a tarball first, and then I did that, I could recover from the tarball if I happened to be in a position where the solution was elusive.

One version of a guidelines plan:
I. Make a script in /jhalfs/custom/config based on blfs book and the custom template.
A. Can opt to use blfs-tool as a starting point (includes dependency resolver)
B. Or manually, copy/paste to the script similarly as you would to the command line
C. Additional useful info can be recorded in a comment in the script
1. # PACKAGE NAME: {package}-{package-version}
2. # DEPENDENCIES: {from the book, a list of ones used in the build}
a. Eventually a query may be: What is installed that depends upon Z?
D. The script names are to be prepended with a sequence number
1. Then you also know in what order things were installed
2. Over time, a different install sequence may be found to be beneficial.
II. Start a new log for a package and append that script to it.
III. Record all commands outputs involved in building and installing the package.
A. Automated tools do this automatically
B. Or manually, adjust commands to append output to the log
IV. Record all discerned filesystem changes from the installation.
A. the "find before", "find after" method can avoid some problems with
1. the "find timestamp" method
2. the "installwatch" method
3. the "fakeroot" method
4. the "uninonfs" method makes complicated and don't think it can run on root "/"
5. find can differentiate directories and files
6. its not perfect either
B. It is more difficult than it sounds to identify the affected files.
1. Different "space usage" methods report different results: find, du, df, ls, stat
a. Adamant answers don't pan out -- instead, they just silence questions.
b. Df is more accurate to know actual available space
c. Du is more accurate to know space used by files (and implied dirs) in list
d. Find (can be) more accurate to find space used by just files in list
1. But this can depend on hard links
2. There is a distinction between space used and space added
e. Other factors are hard to find complete documentation
f. Space before/after install/uninstall can be helpful reality check.
2. Directories may or may not record a change due to an installation
3. Generally, you do not want to delete directories just because of change flag
4. Generally, you do not want to blindly delete possibly shared/wanted config files
5. Some files are shared/keepers even though installation caused the change flag

LOGGING INSTALLED FILES

I'm working out some scripts to help log files that changed with an installation.
makefile-custom-functions is an include file for a Makefile.
MAKEFILE_CUSTOM_FUNCTIONS is a shell script that performs similary.
The function find_them should be called once before the package build and once after the package build.
The before output should be sent to a "before" log, e.g., /var/log/find.log
The after output should be sent to an "after" log, e.g., /var/log/find.log~
The function diff_them should then be called and the output is recorded in the package's log.
Here is an example:
sh MAKEFILE_CUSTOM_FUNCTIONS find_them > /var/log/find.log
... build and install the package foo-bar
sh MAKEFILE_CUSTOM_FUNCTIONS find_them > /var/log/find.log~
sh MAKEFILE_CUSTOM_FUNCTIONS diff them /var/log/find.log /var/log/find.log~ >> foo-bar.log

The designs of the find and diff commands allow detection of files that were added, changed, or deleted. The designs also record the type of each file so that there can be queries about just files that got a status change, or just directories that got a status change. There can also be queries about the byte count of files that got a status change.

Here is an excerpt of jargon in a Makefile that is going to use find_them and diff_them during an automated build of dialog that is needed for slackware pkgtool that I want to experiment with:

5115-dialog:  5113-curl
 @$(call echo_message, Building)
 @$(call find_them) > /var/log/find.log
 @export BASHBIN=$(SHELL) && $(SHELL) progress_bar.sh $@ $$PPID &
 @echo "$(nl_)`date`$(nl_)" >logs/$@
 @$(call prt_du_cr3) >>logs/$@
 @cat custom/config/$@ >>logs/$@
 @echo "$(nl_)" >>logs/$@
 @$(call remove_existing_dirs2,dialog-1.1-20070930.tar.bz2)
 @$(call unpack2,dialog-1.1-20070930.tar.bz2)
 @$(call get_pkg_root2)
 @source envars && \
 $(crCMDSDIR)/custom-tools/$@ >>logs/$@ 2>&1 >>logs/$@
 @$(call remove_build_dirs2,dialog)
 @$(call prt_du_cr3) >>logs/$@
 @echo "$(nl_)FILE LISTING "'[+add -del !update] Name Bytes Type Inode# #links Ctime'"$(nl_)" >>logs/$@
 @$(call find_them) > /var/log/find.log~
 @$(call diff_them,/var/log/find.log,/var/log/find.log~) >>logs/$@ 
 @touch /var/lib/jhalfs/BLFS/dialog-1.1-20070930
 @$(call housekeeping)

Here are excerpts from a head and a tail of the log of that install:

Tue Sep 30 11:23:35 EDT 2008

BYTES: 2529345977 /
EXCLUDING: proc dev sys tmp var/log var/run var/cache back_ups sources home root jhalfs

#
# $Id$
#
# Any comments you wish to add
#
# PACKAGE NAME: dialog-1.1-20070930
#
# DEPENDENCIES:

PKG="dialog"
PKG_VERSION="1.1-20070930"
PKG_FILE="${PKG}-${PKG_VERSION}.tar.bz2"
#URL="ftp://anduin.linuxfromscratch.org/BLFS/conglomeration/$PKG/$PKG_FILE"
URL="http://slackware.cs.utah.edu/pub/slackware/slackware-12.1/source/a/dialog/dialog-1.1-20070930.tar.bz2"
MD5="ddcd01203d190cd6931eedb86284d34a"
for i in PATCH{1..10}; do
   unset $i
done
PATCH1=""

( cat << "xEOFx"

./configure \
  --prefix=/usr \
  --disable-static \
  --enable-nls \
  --with-ncursesw \
  --enable-widec 

make
make install

install -m 644 samples/slackware.rc /etc/dialogrc

xEOFx
) > tmp


checking for package version... 1.1
checking for package patch date... 20070930
checking for gcc... gcc
checking for C compiler default output... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for executable suffix...
...
...
..make[1]: Leaving directory `/sources/dialog-1.1-20070930'

Totalseconds: 54

BYTES: 2529843465 /
EXCLUDING: proc dev sys tmp var/log var/run var/cache back_ups sources home root jhalfs

FILE LISTING [+add -del !update] Name Bytes Type Inode# #links Ctime
! /etc 4096 d 995521 25 Tue Sep 30 11:24:33 2008
+ /etc/dialogrc 2538 f 997131 1 Tue Sep 30 11:24:33 2008
   ...
   ...
   ...
! /usr/share/man/man1 32768 d 669149 2 Tue Sep 30 11:24:33 2008
+ /usr/share/man/man1/dialog.1 40727 f 679055 1 Tue Sep 30 11:24:33 2008
! /usr/bin 20480 d 669122 2 Tue Sep 30 11:24:33 2008
+ /usr/bin/dialog 418856 f 678962 1 Tue Sep 30 11:24:33 2008

With a good log record of what was involved in the install, I can do things like ask questions about the files, or make a binary package, or remove the files. These are some scripts being tested to do some of those things based on the name of the log which might be named something like 5115-dialog in the /jhalfs/logs directory:
PACKAGE_NAME - Name and version
PACKAGE_SIZE - Installed size
PACKAGE_FILES - Files added or updated
PACKAGE_DIR_AFFECTED - Dirs that got involved
PACKAGE_TO_TAR - Make a tarball
PACKAGE_DEPENDED - Packages depending upon pkg

Maybe all I've done is to start accumulating a database of the configuration and dependency options that I chose for my installation. Maybe I've learned something from all the experiments. Maybe it will be useful and maybe it will even lead to a better way of doing it.