Need a fresh look from side at the script,which gens deplibs

Marvell Kirkwood based
Post Reply
barmalej2
Posts: 2395
Joined: Sun Apr 29, 2012 5:24 pm

Need a fresh look from side at the script,which gens deplibs

Post by barmalej2 » Sat May 09, 2015 6:55 pm

The purpose of this script is to generate list of direct dependent FFP packages for ELF files in given directory as argument for script. Possible scenarios is to use it with uncompressed ffp packages or right before creating ffp package (after make install DESTDIR=/some/tmp/dir).
Script will create two files /install/DEPENDS and /install/slack-required in that directory. /install/DEPENDS will be used for adding deps info to PACKAGES.html.
/install/slack-required can be used by slapt-get package manager, for installing dependent packages automatically. According to my early tests it works perfectly with ffp packages (although source code needs some adaptations for ffp).
http://software.jaos.org/git/slapt-get/plain/FAQ.html

My purpose is to optimize this script as much as it possible for faster processing. So any thoughts and advises, suggestions are welcome.
Example of using:
root@NSA310:/i-data/7cf371c4/testing# funpkg -x imagemagick-6.8.9.0-arm-0.txz
Extracting imagemagick-6.8.9.0-arm-0.txz ...
root@NSA310:/i-data/7cf371c4/testing# time gendeps $PWD/ffp
STATUS: looking for ELF files in /mnt/HD_a2/testing/ffp ...done! [ OK ]
STATUS: looking for shared libraries dependencies in founded ELF files ...done! [ OK ]
STATUS: saving list of dependent libraries to /tmp/gendeps.k9Uf/deplibs.lst... done! [ OK ]
STATUS: generating list of packages names with line numbers to /tmp/gendeps.k9Uf/pkgslines.lst...done! [ OK ]
STATUS: generating list of direct dependent packages to /mnt/HD_a2/testing/ffp/install/slack-required ...done! [ OK ]
real 0m 18.09s
user 0m 11.36s
sys 0m 5.67s
root@NSA310:/i-data/7cf371c4/testing# cat /mnt/HD_a2/testing/ffp/install/slack-required
bzip2
expat
fontconfig
freetype
gcc
gcc-solibs
imagemagick
lcms
libiconv
libjpeg
libpng
libtiff
libtool
libwebp
libxml2
uClibc
uClibc-solibs
xz
zlib
root@NSA310:/i-data/7cf371c4/testing# cat /mnt/HD_a2/testing/ffp/install/DEPENDS
bzip2 expat fontconfig freetype gcc gcc-solibs imagemagick lcms libiconv libjpeg libpng libtiff libtool libwebp libxml2 uClibc uClibc-solibs xz zlib
Script can be downloaded from here:
http://downloads.zyxel.nas-central.org/ ... ts/gendeps

Code: Select all

#!/ffp/bin/bash
 
# This program is free software. You can redistribute it and/or modify it under the terms 
# of the GNU General Public License as published by the Free Software Foundation
# either version 2 or any later version.
# Copyright (C) 11-30-2003 Jason Woodward 
# Copyright (C) 2015 barmalej2

tempdir=$(mktemp -d --tmpdir gendeps.XXXX)

repo1="http://www.inreto.de/ffp/0.7/arm/packages/"
repo2="http://downloads.zyxel.nas-central.org/Users/Mijzelf/FFP-Stick/packages/0.7/arm/"
repo3="http://downloads.zyxel.nas-central.org/Users/barmalej2/ffp/0.7/arm/packages/"

dir="$1"
if [ $# -ne 1 ]
then
cat << EOF

	$0 - generate list of direct dependent FFP packages for ELF files in given directory.
	Script will create files /install/slack-required for slapt-get and /install/DEPENDS
	for PACKAGES.html inside that directory.
	Usage: $0 /path/to/directory

EOF
	exit 1
fi
if [ ! -d "$dir" ]
then
	echo "$dir directory was not found!"
	echo "Check your input argument..."
	exit 1
fi
########################To be removed, just for testing
if [ -d "$dir"/install ]
then
	rm -rf "$dir"/install
fi
########################

function gen_deps_list {
# Dump required libraries info from ELF files.
deplibs="$tempdir"/deplibs.lst
#excludelist=(! -name '*.a' ! -name '*.la' ! -name '*.py*' ! -name '*.txt' ! -name '*.h' ! -name '*.ttf' \
#! -name '*.rb' ! -name '*.ko' ! -name '*.pc' ! -name '*.enc' ! -name '*.cf' ! -name '*.def' ! -name '*.rules' \
#! -name '*.cmi' ! -name  '*.mli' ! -name '*.ml' ! -name '*.cma' ! -name '*.cmx' ! -name '*.cmxa' ! -name '*.pod' \
#! -name '*.pm' ! -name '*.pl' ! -name '*.al' ! -name '*.tcl' ! -name '*.bs' ! -name '*.o' ! -name '*.png' \
#! -name '*.gif' ! -name '*.cmo' ! -name '*.cgi' ! -name '*.defs' ! -name '*.conf' ! -name '*_LOCALE' ! -name 'Compose' \
#! -name '*_OBJS' ! -name '*.msg' ! -name '*.mcopclass' ! -name '*.mcoptype' ! -name '*.po' ! -name '*.hpp' ! -name '*.ipp')
# Find ELF binaries in the specified dir.
echo -ne "STATUS: looking for ELF files in \e[1;32;20m$dir\e[0m ..."
#filelist=($(find "$dir" -type f ${excludelist[@]} -exec file {} + | grep 'ELF' | cut -f1 -d':'))
filelist=($(find "$dir" -type f | grep 'bin\/\|libexec\/\|lib\/.*\.so' | grep -v '\/$'))
#printf "%s\n" "${filelist[@]}"
elffilelist=()
for file in "${filelist[@]}"; do
	elffile="$(file "$file" | grep 'ELF' | cut -f1 -d':')"
	if	[[ -n "$elffile" ]]; then
		elffilelist+=($elffile)
	fi
done
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
#printf "%s\n" "${elffilelist[@]}"
# Get the list of dependent libs for ELF files.
if	[[ -z "${elffilelist[@]}" ]]; then
	echo "No ELF files has been found in given directory. Exiting ... "
	exit 1
else
	echo -n "STATUS: looking for shared libraries dependencies in founded ELF files ..."
	for file in "${elffilelist[@]}"; do
			objdump -x "$file" 2> /dev/null | grep NEEDED | awk '{print$2}' >> "$deplibs"
	done
	echo -e "done! \e[1;32;20m[ OK ]\e[0m"
fi
echo -n "STATUS: saving list of dependent libraries to $deplibs... "
sort -u < "$deplibs" >> "$tempdir"/deplibs.txt
mv "$tempdir"/deplibs.txt "$deplibs"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
}

function gen_pkgs_line_list {
# Download MANIFEST files from repos and combine them into the one file.
# Then generate the separate list file of package names with line number
# position in combined MANIFEST file.
manifest=/ffp/funpkg/MANIFEST
pkgslinelist="$tempdir"/pkgslines.lst
if	[ ! -f $manifest ] || [ -n "$(find $manifest -mtime +1)" ]; then
	for url in "$repo1" "$repo2" "$repo3"
	do
		echo "Downloading packages files list from $url"
		wget -q "$url"/MANIFEST.bz2 -O "$manifest".bz2
		if	[ -f "$manifest".bz2 ]; then
			echo -n "STATUS: extracting packages files list data to $manifest..."
			bunzip2 -c "$manifest".bz2 >> "$manifest"
			echo -e "done! \e[1;32;20m[ OK ]\e[0m"
			rm "$manifest".bz2
		else
			echo "Failed to download MANIFEST.bz2 from $url"
		fi
	done
fi
echo -n "STATUS: generating list of packages names with line numbers to $pkgslinelist..."
grep -n ' Package\: ' "$manifest" | awk '{print $1 $3}'| sed -re "s/\|//g" > "$tempdir"/pkgslines.txt
echo "$(wc -l < "$manifest"):$(tail -n1 < "$tempdir"/pkgslines.txt | cut -d ':' -f2)" > "$pkgslinelist"
cat "$pkgslinelist" >> "$tempdir"/pkgslines.txt
mv "$tempdir"/pkgslines.txt "$pkgslinelist"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
}

function lookup_lib {
# Find deplib line number in Manifest file and pass it for further comparison processing 
while read -r deplib
do
	liblinenumbers=($(grep -n "$deplib" "$manifest"|cut -f1 -d':'))
	for liblinenumber in "${liblinenumbers[@]}"
	do
		if	[ -n "$liblinenumber" ];then
			map_to_pkg "$liblinenumber"
		fi
	done
done < "$deplibs"
}
 
function map_to_pkg {
# Compare $liblinenumber with $pkglinenumbers. If $liblinenumber < $pkglinenumber,
# then deplib is from package with last $pkglinenumber variable value < $liblinenumber
local liblinenumber=$1
while IFS=: read -r pkglinenumber pkg
do
	#[[ ${#liblinenumber} == ${#pkglinenumber} ]] && [[ $liblinenumber == ${pkglinenumber:0:1}* ]]
	#if	[[ ${#liblinenumber} == ${#pkglinenumber} ]]; then
		if	[[ $liblinenumber -gt $pkglinenumber ]]; then
			lastpkg=$pkg
		else
			deppkgname=$(basename "$lastpkg"|sed -re "s/(.*{1,})\\-(.*[\\.\\-].*[\\.\\-].*).txz[ ]{0,}$/\1/")
			echo "$deppkgname" >> "$dir"/install/slack-required
			break
		fi
	#fi
done < "$pkgslinelist"
}
gen_deps_list "$@"
gen_pkgs_line_list
if [ ! -d "$dir"/install ]
then
    mkdir "$dir"/install
fi
echo -ne "STATUS: generating list of direct dependent packages to \e[1;32;20m$dir/install/slack-required\e[0m ..."
lookup_lib
sort -u < "$dir"/install/slack-required >> "$dir"/install/slack-required1
mv "$dir"/install/slack-required1 "$dir"/install/slack-required
xargs < "$dir"/install/slack-required >> "$dir"/install/DEPENDS
rm -rf "$tempdir"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
I was using log

Code: Select all

sh -x gendeps $PWD/ffp >> gendeps.log 2>&1 &
for analyzing how the script runs. Might be useful for someone.

Rgds,
b2

User avatar
MrDini
Posts: 651
Joined: Mon Dec 21, 2015 7:21 pm
Contact:

Re: Need a fresh look from side at the script,which gens dep

Post by MrDini » Mon Sep 05, 2016 2:36 pm

Oh, I just found this post... Really interesting!

It might be useful for my freeradius package, which has a lot of dependencies, and I don't know what are theese... But if I understood the script I have to upload the packages to my repo for dependencie checking, aren't I? Because the mc says, I have now 257 packages in txz (yes, theese are mine compiled packages), and I Can't upload all of them to my repo... :mrgreen:
Thanks nas-central the lot of help! :)

barmalej2
Posts: 2395
Joined: Sun Apr 29, 2012 5:24 pm

Re: Need a fresh look from side at the script,which gens dep

Post by barmalej2 » Tue Sep 06, 2016 5:42 am

All you need is url to MANIFEST.bz file generated from all your packages. Script will download and combine from repos1,2 and 3 MANIFEST.bz files to one file, then assign line numbers in that combined file and will compare required libs with joint MANIFEST file. That is how it finds required packages.
Bear in mind, that script will download from repos MANIFEST.bz, only if /ffp/funpkg/MANIFEST is older than one day.
If you need to force redownload, just remove /ffp/funpkg/MANIFEST.

vlmaksime
Posts: 12
Joined: Sun Apr 30, 2017 3:17 pm

Re: Need a fresh look from side at the script,which gens dep

Post by vlmaksime » Sat Jun 24, 2017 6:48 pm

Looks like in first post is old version of script.
With this script i have problems with compiling, using your scripts from here: http://downloads.zyxel.nas-central.org/ ... build/src/
Looks like latest version is here: http://downloads.zyxel.nas-central.org/ ... ts/gendeps , but I can't download it. Have 403 error, when clicking on the link

barmalej2
Posts: 2395
Joined: Sun Apr 29, 2012 5:24 pm

Re: Need a fresh look from side at the script,which gens dep

Post by barmalej2 » Sat Jun 24, 2017 7:14 pm

Corrected too much restrictive permissions, you should be able to download it now.

vlmaksime
Posts: 12
Joined: Sun Apr 30, 2017 3:17 pm

Re: Need a fresh look from side at the script,which gens dep

Post by vlmaksime » Sat Jun 24, 2017 8:30 pm

Thanks. But now I have error:

Code: Select all

STATUS: looking for ELF files in /mnt/HD/HD_a2/build/ffp .../ffp/bin/gendeps: line 65: /dev/fd/62: No such file or directory
done! [ OK ]
No ELF files has been found in given directory. Exiting ...
On this error compilation stops

barmalej2
Posts: 2395
Joined: Sun Apr 29, 2012 5:24 pm

Re: Need a fresh look from side at the script,which gens dep

Post by barmalej2 » Sat Jun 24, 2017 9:29 pm

On this error compilation stops
You can simply comment row, where gendeps script is executed.
/dev/fd/62: No such file or directory
Zyxel firmware doesn't creates required file descriptors for bash process substitution. Quote from bash manual page:
Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.
Running udev daemon can fix that auto on the fly or you can do that manually after each reboot:

Code: Select all

ln -s /proc/self/fd /dev/fd

Post Reply