General NAS-Central Forums

Welcome to the NAS community
It is currently Tue Oct 24, 2017 9:33 am

All times are UTC




Post new topic Reply to topic  [ 7 posts ] 
Author Message
PostPosted: Sat May 09, 2015 6:55 pm 
Offline

Joined: Sun Apr 29, 2012 5:24 pm
Posts: 2303
The purpose of this script is to generate list of direct dependent FFP packages for ELF files in given directory as argument for script. Possible scenarios is to use it with uncompressed ffp packages or right before creating ffp package (after make install DESTDIR=/some/tmp/dir).
Script will create two files /install/DEPENDS and /install/slack-required in that directory. /install/DEPENDS will be used for adding deps info to PACKAGES.html.
/install/slack-required can be used by slapt-get package manager, for installing dependent packages automatically. According to my early tests it works perfectly with ffp packages (although source code needs some adaptations for ffp).
http://software.jaos.org/git/slapt-get/plain/FAQ.html

My purpose is to optimize this script as much as it possible for faster processing. So any thoughts and advises, suggestions are welcome.
Example of using:
Quote:
root@NSA310:/i-data/7cf371c4/testing# funpkg -x imagemagick-6.8.9.0-arm-0.txz
Extracting imagemagick-6.8.9.0-arm-0.txz ...
root@NSA310:/i-data/7cf371c4/testing# time gendeps $PWD/ffp
STATUS: looking for ELF files in /mnt/HD_a2/testing/ffp ...done! [ OK ]
STATUS: looking for shared libraries dependencies in founded ELF files ...done! [ OK ]
STATUS: saving list of dependent libraries to /tmp/gendeps.k9Uf/deplibs.lst... done! [ OK ]
STATUS: generating list of packages names with line numbers to /tmp/gendeps.k9Uf/pkgslines.lst...done! [ OK ]
STATUS: generating list of direct dependent packages to /mnt/HD_a2/testing/ffp/install/slack-required ...done! [ OK ]
real 0m 18.09s
user 0m 11.36s
sys 0m 5.67s
root@NSA310:/i-data/7cf371c4/testing# cat /mnt/HD_a2/testing/ffp/install/slack-required
bzip2
expat
fontconfig
freetype
gcc
gcc-solibs
imagemagick
lcms
libiconv
libjpeg
libpng
libtiff
libtool
libwebp
libxml2
uClibc
uClibc-solibs
xz
zlib
root@NSA310:/i-data/7cf371c4/testing# cat /mnt/HD_a2/testing/ffp/install/DEPENDS
bzip2 expat fontconfig freetype gcc gcc-solibs imagemagick lcms libiconv libjpeg libpng libtiff libtool libwebp libxml2 uClibc uClibc-solibs xz zlib


Script can be downloaded from here:
http://downloads.zyxel.nas-central.org/Users/barmalej2/ffp/0.7/arm/scripts/gendeps

Code:
#!/ffp/bin/bash
 
# This program is free software. You can redistribute it and/or modify it under the terms
# of the GNU General Public License as published by the Free Software Foundation
# either version 2 or any later version.
# Copyright (C) 11-30-2003 Jason Woodward
# Copyright (C) 2015 barmalej2

tempdir=$(mktemp -d --tmpdir gendeps.XXXX)

repo1="http://www.inreto.de/ffp/0.7/arm/packages/"
repo2="http://downloads.zyxel.nas-central.org/Users/Mijzelf/FFP-Stick/packages/0.7/arm/"
repo3="http://downloads.zyxel.nas-central.org/Users/barmalej2/ffp/0.7/arm/packages/"

dir="$1"
if [ $# -ne 1 ]
then
cat << EOF

   $0 - generate list of direct dependent FFP packages for ELF files in given directory.
   Script will create files /install/slack-required for slapt-get and /install/DEPENDS
   for PACKAGES.html inside that directory.
   Usage: $0 /path/to/directory

EOF
   exit 1
fi
if [ ! -d "$dir" ]
then
   echo "$dir directory was not found!"
   echo "Check your input argument..."
   exit 1
fi
########################To be removed, just for testing
if [ -d "$dir"/install ]
then
   rm -rf "$dir"/install
fi
########################

function gen_deps_list {
# Dump required libraries info from ELF files.
deplibs="$tempdir"/deplibs.lst
#excludelist=(! -name '*.a' ! -name '*.la' ! -name '*.py*' ! -name '*.txt' ! -name '*.h' ! -name '*.ttf' \
#! -name '*.rb' ! -name '*.ko' ! -name '*.pc' ! -name '*.enc' ! -name '*.cf' ! -name '*.def' ! -name '*.rules' \
#! -name '*.cmi' ! -name  '*.mli' ! -name '*.ml' ! -name '*.cma' ! -name '*.cmx' ! -name '*.cmxa' ! -name '*.pod' \
#! -name '*.pm' ! -name '*.pl' ! -name '*.al' ! -name '*.tcl' ! -name '*.bs' ! -name '*.o' ! -name '*.png' \
#! -name '*.gif' ! -name '*.cmo' ! -name '*.cgi' ! -name '*.defs' ! -name '*.conf' ! -name '*_LOCALE' ! -name 'Compose' \
#! -name '*_OBJS' ! -name '*.msg' ! -name '*.mcopclass' ! -name '*.mcoptype' ! -name '*.po' ! -name '*.hpp' ! -name '*.ipp')
# Find ELF binaries in the specified dir.
echo -ne "STATUS: looking for ELF files in \e[1;32;20m$dir\e[0m ..."
#filelist=($(find "$dir" -type f ${excludelist[@]} -exec file {} + | grep 'ELF' | cut -f1 -d':'))
filelist=($(find "$dir" -type f | grep 'bin\/\|libexec\/\|lib\/.*\.so' | grep -v '\/$'))
#printf "%s\n" "${filelist[@]}"
elffilelist=()
for file in "${filelist[@]}"; do
   elffile="$(file "$file" | grep 'ELF' | cut -f1 -d':')"
   if   [[ -n "$elffile" ]]; then
      elffilelist+=($elffile)
   fi
done
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
#printf "%s\n" "${elffilelist[@]}"
# Get the list of dependent libs for ELF files.
if   [[ -z "${elffilelist[@]}" ]]; then
   echo "No ELF files has been found in given directory. Exiting ... "
   exit 1
else
   echo -n "STATUS: looking for shared libraries dependencies in founded ELF files ..."
   for file in "${elffilelist[@]}"; do
         objdump -x "$file" 2> /dev/null | grep NEEDED | awk '{print$2}' >> "$deplibs"
   done
   echo -e "done! \e[1;32;20m[ OK ]\e[0m"
fi
echo -n "STATUS: saving list of dependent libraries to $deplibs... "
sort -u < "$deplibs" >> "$tempdir"/deplibs.txt
mv "$tempdir"/deplibs.txt "$deplibs"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
}

function gen_pkgs_line_list {
# Download MANIFEST files from repos and combine them into the one file.
# Then generate the separate list file of package names with line number
# position in combined MANIFEST file.
manifest=/ffp/funpkg/MANIFEST
pkgslinelist="$tempdir"/pkgslines.lst
if   [ ! -f $manifest ] || [ -n "$(find $manifest -mtime +1)" ]; then
   for url in "$repo1" "$repo2" "$repo3"
   do
      echo "Downloading packages files list from $url"
      wget -q "$url"/MANIFEST.bz2 -O "$manifest".bz2
      if   [ -f "$manifest".bz2 ]; then
         echo -n "STATUS: extracting packages files list data to $manifest..."
         bunzip2 -c "$manifest".bz2 >> "$manifest"
         echo -e "done! \e[1;32;20m[ OK ]\e[0m"
         rm "$manifest".bz2
      else
         echo "Failed to download MANIFEST.bz2 from $url"
      fi
   done
fi
echo -n "STATUS: generating list of packages names with line numbers to $pkgslinelist..."
grep -n ' Package\: ' "$manifest" | awk '{print $1 $3}'| sed -re "s/\|//g" > "$tempdir"/pkgslines.txt
echo "$(wc -l < "$manifest"):$(tail -n1 < "$tempdir"/pkgslines.txt | cut -d ':' -f2)" > "$pkgslinelist"
cat "$pkgslinelist" >> "$tempdir"/pkgslines.txt
mv "$tempdir"/pkgslines.txt "$pkgslinelist"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"
}

function lookup_lib {
# Find deplib line number in Manifest file and pass it for further comparison processing
while read -r deplib
do
   liblinenumbers=($(grep -n "$deplib" "$manifest"|cut -f1 -d':'))
   for liblinenumber in "${liblinenumbers[@]}"
   do
      if   [ -n "$liblinenumber" ];then
         map_to_pkg "$liblinenumber"
      fi
   done
done < "$deplibs"
}
 
function map_to_pkg {
# Compare $liblinenumber with $pkglinenumbers. If $liblinenumber < $pkglinenumber,
# then deplib is from package with last $pkglinenumber variable value < $liblinenumber
local liblinenumber=$1
while IFS=: read -r pkglinenumber pkg
do
   #[[ ${#liblinenumber} == ${#pkglinenumber} ]] && [[ $liblinenumber == ${pkglinenumber:0:1}* ]]
   #if   [[ ${#liblinenumber} == ${#pkglinenumber} ]]; then
      if   [[ $liblinenumber -gt $pkglinenumber ]]; then
         lastpkg=$pkg
      else
         deppkgname=$(basename "$lastpkg"|sed -re "s/(.*{1,})\\-(.*[\\.\\-].*[\\.\\-].*).txz[ ]{0,}$/\1/")
         echo "$deppkgname" >> "$dir"/install/slack-required
         break
      fi
   #fi
done < "$pkgslinelist"
}
gen_deps_list "$@"
gen_pkgs_line_list
if [ ! -d "$dir"/install ]
then
    mkdir "$dir"/install
fi
echo -ne "STATUS: generating list of direct dependent packages to \e[1;32;20m$dir/install/slack-required\e[0m ..."
lookup_lib
sort -u < "$dir"/install/slack-required >> "$dir"/install/slack-required1
mv "$dir"/install/slack-required1 "$dir"/install/slack-required
xargs < "$dir"/install/slack-required >> "$dir"/install/DEPENDS
rm -rf "$tempdir"
echo -e "done! \e[1;32;20m[ OK ]\e[0m"

I was using log
Code:
sh -x gendeps $PWD/ffp >> gendeps.log 2>&1 &
for analyzing how the script runs. Might be useful for someone.

Rgds,
b2


Top
 Profile  
 
PostPosted: Mon Sep 05, 2016 2:36 pm 
Offline
User avatar

Joined: Mon Dec 21, 2015 7:21 pm
Posts: 597
Oh, I just found this post... Really interesting!

It might be useful for my freeradius package, which has a lot of dependencies, and I don't know what are theese... But if I understood the script I have to upload the packages to my repo for dependencie checking, aren't I? Because the mc says, I have now 257 packages in txz (yes, theese are mine compiled packages), and I Can't upload all of them to my repo... :mrgreen:

_________________
Thanks nas-central the lot of help! :)


Top
 Profile  
 
PostPosted: Tue Sep 06, 2016 5:42 am 
Offline

Joined: Sun Apr 29, 2012 5:24 pm
Posts: 2303
All you need is url to MANIFEST.bz file generated from all your packages. Script will download and combine from repos1,2 and 3 MANIFEST.bz files to one file, then assign line numbers in that combined file and will compare required libs with joint MANIFEST file. That is how it finds required packages.
Bear in mind, that script will download from repos MANIFEST.bz, only if /ffp/funpkg/MANIFEST is older than one day.
If you need to force redownload, just remove /ffp/funpkg/MANIFEST.


Top
 Profile  
 
PostPosted: Sat Jun 24, 2017 6:48 pm 
Offline

Joined: Sun Apr 30, 2017 3:17 pm
Posts: 12
Looks like in first post is old version of script.
With this script i have problems with compiling, using your scripts from here: http://downloads.zyxel.nas-central.org/ ... build/src/
Looks like latest version is here: http://downloads.zyxel.nas-central.org/ ... ts/gendeps , but I can't download it. Have 403 error, when clicking on the link


Top
 Profile  
 
PostPosted: Sat Jun 24, 2017 7:14 pm 
Offline

Joined: Sun Apr 29, 2012 5:24 pm
Posts: 2303
Corrected too much restrictive permissions, you should be able to download it now.


Top
 Profile  
 
PostPosted: Sat Jun 24, 2017 8:30 pm 
Offline

Joined: Sun Apr 30, 2017 3:17 pm
Posts: 12
Thanks. But now I have error:
Code:
STATUS: looking for ELF files in /mnt/HD/HD_a2/build/ffp .../ffp/bin/gendeps: line 65: /dev/fd/62: No such file or directory
done! [ OK ]
No ELF files has been found in given directory. Exiting ...

On this error compilation stops


Top
 Profile  
 
PostPosted: Sat Jun 24, 2017 9:29 pm 
Offline

Joined: Sun Apr 29, 2012 5:24 pm
Posts: 2303
Quote:
On this error compilation stops
You can simply comment row, where gendeps script is executed.

Quote:
/dev/fd/62: No such file or directory
Zyxel firmware doesn't creates required file descriptors for bash process substitution. Quote from bash manual page:
Quote:
Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.

Running udev daemon can fix that auto on the fly or you can do that manually after each reboot:
Code:
ln -s /proc/self/fd /dev/fd


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC


Who is online

Users browsing this forum: Google [Bot], Majestic-12 [Bot] and 77 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group