Noob Linux and NSA320 questions

Marvell Kirkwood based
voland
Posts: 4
Joined: Fri Jul 03, 2015 10:47 am

Re: Noob Linux and NSA320 questions

Post by voland » Thu May 04, 2017 12:17 am

Mijzelf wrote:/etc is on a ramdrive, and so it doesn't survive a reboot. You'll have to put your mount in a script, and put that in /usr/local/zy-pkgs/etc/init.d/. It will be called on boot with a 'startup' argument.
Please enlight me, Mijzelf, where is such place in NAS542.
I find out that reboot-resistant is /etc/zyxel/ and packages are in /i-data/.system/zy-pkgs on NAS542. But no etc/init.d in no one of that dirs, and my effort to rename/copy/modify settings from one of packages (in order to set my own startup script) was not successful (probably I did not modify some configuration file where all packages are enumerated, so system do not know about existence of my fake-package). I also locate my package to /i-data/sysvol/.PKG/ and give appropriate 755 rights to /i-data/sysvol/.PKG/<packagename>/etc/init.d/<packagename> scripts with no effect :(

In rcS1 and rcS2 I also do not found reference to place, where I can put my own startup script. Can you please tell me such place location (on NAS542)?

They completely changed mechanism of booting, disks referencing and so on, isn't they?.. Nevertheless, your FFP stick still booting well. So, you know the way :-)

I never wrote my own regular package, because I simply do not found any materials, learning how to do so (for NAS542). Can you please take a link to some tutorial?
Sorry for asking on NAS542 in NSA320 section...
Last edited by voland on Thu May 04, 2017 8:02 pm, edited 3 times in total.

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Noob Linux and NSA320 questions

Post by Mijzelf » Thu May 04, 2017 7:13 pm

voland wrote:In rcS1 and rcS2 I also do not found reference to place, where I can put my own startup script. Can you please tell me such place location (on NAS542)?
The file /etc/init.d/zypkg_controller.sh does the job, and contains this gem:

Code: Select all

	# - start to startup packages according to ${USRPKG_DEPS_START}
	while read zypkg; do
		bname=`basename ${zypkg}`
		zypkg=${ZYPKG_PKG_SRC_PATH}/${bname}
		Processed_Packages=${Processed_Packages}:${bname}
		write_log "- starting package \"${bname}\" ..."
		#ckeck ZYPKG_DEPS format
		CHKVERSION=`grep "/i-data/" ${zypkg}`
		if [ "$?" != "0" ]; then
			PKGVOLPATH=`grep ${bname} ${PKGSTATUSFILE} |grep "Installed-Rule" |awk -F":" '{print $2}' |sed 's/\/$//g' |sed 's/ //g'`
			zypkg=${PKGVOLPATH}/etc/init.d/${bname}
		fi
		if [ ! -x ${zypkg} ]; then
			write_log "---> Error: start-up program \"${zypkg}\" is not existed or not excutable"
		else
			${zypkg} startup
			if [ "$?" == "0" ]; then
				write_log "---> start \"${bname}\" successfully."
			else
				write_log "---> start \"${bname}\" failed."
			fi
		fi
	done < ${USRPKG_DEPS_START}
So if you add a startscript to ${USEPKG_DEPS_START} and that startscript is inside ${ZYPKG_PKG_SRC_PATH}, and it is executable, than it will be called with a param 'startup' on boot.
${USRPKG_DEPS_START} is /i-data/.system/zy-pkgs/USRPKG_DEPS_START. That file doesn't exist, but you can create it. ${ZYPKG_PKG_SRC_PATH} is /i-data/.system/zy-pkgs/. The script has to be in this directory, but a symlink will also do.
If you also want to call your script on shutdown, use the file /i-data/.system/zy-pkgs/USRPKG_DEPS_SHUTDOWN . Your script will be called with 'shutdown'.
I never wrote my own regular package, because I simply do not found any materials, learning how to do so (for NAS542). Can you please take a links on some tutorials?
I partly documented the case here. Actually it's no big deal. By just extracting some existing packages, it's rather easy to find how a package is build.

voland
Posts: 4
Joined: Fri Jul 03, 2015 10:47 am

Re: Noob Linux and NSA320 questions

Post by voland » Mon May 08, 2017 4:50 am

Mijzelf wrote:
voland wrote:In rcS1 and rcS2 I also do not found reference to place, where I can put my own startup script. Can you please tell me such place location (on NAS542)?
The file /etc/init.d/zypkg_controller.sh does the job. <...> So if you add a startscript to ${USEPKG_DEPS_START} and that startscript is inside ${ZYPKG_PKG_SRC_PATH}, and it is executable, than it will be called with a param 'startup' on boot.
${USRPKG_DEPS_START} is /i-data/.system/zy-pkgs/USRPKG_DEPS_START. That file doesn't exist, but you can create it. ${ZYPKG_PKG_SRC_PATH} is /i-data/.system/zy-pkgs/. The script has to be in this directory, but a symlink will also do.
If you also want to call your script on shutdown, use the file /i-data/.system/zy-pkgs/USRPKG_DEPS_SHUTDOWN . Your script will be called with 'shutdown'.
To be honest, I analyzed /etc/init.d/zypkg_controller.sh before, and (experimentally) found that script is not fully functional, at least part you mentioned above. After seen inside typos like "#ckeck ZYPKG_DEPS format" in place no dealing at all with ${ZYPKG_DEPS} (despite it is commented out) and realizing I cannot modifying this file permanently, I leave it. But your advice and respect to your recognizable work push me to analyze it more thoroughly. So, there are results:
  • Function write_log() in /etc/init.d/zypkg_controller.sh have (for speeding up reason?) commented out code for writing message to logfile, leaving only part for write message to stdout:

    Code: Select all

    write_log()
    {
    	echo $1
    	#echo "`date "+%Y-%m-%d %H:%M:%S"` [booting] $1" >> ${ZYPKG_LOG_FILE}
    }
    
    This prevents from logging booting process to file.
    Furthermore, just "[booting]" is weak identification of message source; I expecting script name there.
  • If you put only one line without <Enter> inside USRPKG_DEPS_START/USRPKG_DEPS_SHUTDOWN, it will not work at all - script just do not enter appropriate "while do" loop. And this is expectable. Just for remembering.
  • If you leave WinSCP (for those who work under Windows) at "Auto" file transfer mode, files without extension (like USRPKG_DEPS_START) will be treated as binary, so your <Enter> will be saved as <CR><LF>, and then ${zypkg} and ${bname} will contain <CR>, what brings error as you can see there, literally, with newline:

    Code: Select all

    Error: start-up program "/etc/init.d/<my-script-name>
    " is not existed or not excutable
    So please be prudent and turn on "Text" file transfer mode or use PuTTY/native Linux environment for modifying config files.
  • Function write_log() from calling zypkg_controller.sh, nor any defined there variables, cannot be used in my own script, listed in USRPKG_DEPS_START. But, for example, ${ECHO} is defined in /init and through /linuxrc somehow passed to /etc/init.d/rcS (without "export ECHO" nor "set -a"). This effectively prevents from system integration of my script.
    If such (obvious) Linux script calling behavior is used there for security reason (e.g. for preventing from modifying of system variables) or for return code receiving reason (BTW preventing also from unintentional finishing calling script by "exit 0" command in subscript), ZyXEL developers could just source it in subscript, such as

    Code: Select all

    (. ${zypkg} startup)
    instead of just calling

    Code: Select all

    ${zypkg} startup
    In way zypkg_controller.sh is actually written, I must to code my own write_log() function with specifying exactly path of system log file, or write to my own log file, or send messages somewhere in vanishing stdout :(
  • During analyzing of fragment (at line 216 and below)

    Code: Select all

    [ "${IsStarted}" == "0" ] && continue
    # start to run start commands for packages
    echo ${zypkg} | grep "^#" >/dev/null
    if [ "$?" != "0" ]; then
    	bname=`basename ${zypkg}`
    	Processed_Packages=${Processed_Packages}:${bname}
    	write_log "- starting package \"${bname}\" ..."
    	# ckeck ZYPKG_DEPS format
    	CHKVERSION=`grep "/i-data/" ${zypkg}`
    I noticed absence of redefining ${zypkg} with full path as

    Code: Select all

    zypkg=${ZYPKG_PKG_SRC_PATH}/${bname}
    after defining "bname". Without this line executing `grep "/i-data/" ${zypkg}` might end up with error "No such file or directory".
    And YES, grep failed with EVERY package. And YES, appropriate message is put to stderr every time.
    Do anybody test this script, hey, ZyXEL guys?... :shock:
  • During analyzing of fragment in next section (at line 256 and below)

    Code: Select all

    CHKVERSION=`grep "/i-data/" ${zypkg}`
    if [ "$?" != "0" ]; then
    I am confusedly thought "WTH, for just run my script, located in "${ZYPKG_PKG_SRC_PATH}/${bname}", as they demand all after all, ZyXEL devs push me to mention string "/i-data/" (without any meaningful functionality) in my code?! Why? How? For what reason? Do I must complain some rules? "
    Indeed, absence in my script line like

    Code: Select all

    # Hello ZyXEL devs, thanks for using /i-data/ in my NAS!
    will lead mentioned above "if" to be evaluated positively and so lead to attempt to parse an special file (see below) for (incorrectly written, as you can see below) attempt to find an especial path to my script (see below).
    I am able to understand speculating like "If script author will want locate his script in some non-standard place, he for sure will mention that special place in script description, and because /i-data/* is certain place to permanent storage, there with high probability will "/i-data/" string mentioned; in opposite case (script located in standard place, defined in ${ZYPKG_PKG_SRC_PATH}) author have almost no reason to state exact script location; therefore we could use this string for ensuring non-standard script placing by author, and therefore do appropriate next steps for locating it path". In this case, however, code must have "==" instead of "!=" !
    Well, but opposite way of thinking?!... Incomprehensible for me. Explainable only by developer mistake IMHO.
    Anyway, to rely on some free text inside of script... this is at least irresponsible.
    So, if you want your own startup/shutdown script (located in standard /i-data/.system/zy-pkgs/) would be loaded successfully, please do not forget to place "/i-data/" text somewhere inside, everywhere, in ever manner, no matter how :D
  • During analyzing of fragment (at line 256 and below)

    Code: Select all

    CHKVERSION=`grep "/i-data/" ${zypkg}`
    if [ "$?" != "0" ]; then
    	PKGVOLPATH=`grep ${bname} ${PKGSTATUSFILE} |grep "Installed-Rule" |awk -F":" '{print $2}' |sed 's/\/$//g' |sed 's/ //g'`
    	zypkg=${PKGVOLPATH}/etc/init.d/${bname}
    fi
    I am again confusedly thought "Well, they tried to parse ${PKGSTATUSFILE} (which is /etc/zyxel/pkgconf/status) for case of non-standard script placement. They want to find "Installed-Rule" line (where path to script is defined by script author), situated after package name in ${PKGSTATUSFILE}. But...
    1. That file is intended for packages, not for individual scripts! Or may be they suggests "There is "/i-data/" text in code => This is the package"?!
    2. WHOLE ${PKGSTATUSFILE} is walked through straight in the next part of zypkg_controller.sh, searching ALL "Installed-Rule" strings. Why to search there now, for non-package scripts?
    3. And worthe of all: executing `grep ${bname} ${PKGSTATUSFILE}` will return only ONE LINE, in which, for 120%, they WILL NOT find "Installed-Rule". Will NOT!!

    Well, it just enough to add "-A <number>" to appropriate grep for working as (probably) intended; and in our case <num> should be 7 IMHO. So, code must look like

    Code: Select all

    PKGVOLPATH=`grep -A 7 ${bname} ${PKGSTATUSFILE} |grep "Installed-Rule" |awk -F":" '{print $2}' |sed 's/\/$//g' |sed 's/ //g'`
    And, unfortunately, such code was copied from a few lines above in zypkg_controller.sh, from part of starting all packages according to ${ZYPKG_DEPS}, where it functioning incorrectly too :(
  • During analyzing of fragment in next section (at line 274)

    Code: Select all

    cat /etc/zyxel/pkg_conf/status | grep Installed-Rule | awk '{print $2}' |  while read zypkg; do
    I noticed that file /etc/zyxel/pkg_conf/status is mentioned directly instead of referencing on ${PKGSTATUSFILE} variable :?
  • The same story with /i-data/.system/zy-pkgs/ZYPKG_DEPS two lines below previous case.
  • During analyzing of fragment (at line 274 and below)

    Code: Select all

    cat /etc/zyxel/pkg_conf/status | grep Installed-Rule | awk '{print $2}' |  while read zypkg; do
    	PKGName=`echo ${zypkg} | awk -F "/" '{print $5}'`
    	cat /i-data/.system/zy-pkgs/ZYPKG_DEPS | grep $PKGName > /dev/null 2>&1
    	if [ "$?" != "0" ]; then
    		echo ${Processed_Packages} | grep ${PKGName} > /dev/null 2>&1
    		if [ "$?" != "0" ]; then
    			EXEINIFILE=${zypkg}/etc/init.d/$PKGName
    			if [ ! -x ${EXEINIFILE} ]; then
    I have found line

    Code: Select all

    EXEINIFILE=${zypkg}/etc/init.d/$PKGName
    might be

    Code: Select all

    EXEINIFILE=${zypkg}etc/init.d/$PKGName
    Actually no big deal, it is functioning as is and BTW preventing from theoretically possible mistake in package path definition (by accidental abandoning of trailing "/") in /etc/zyxel/pkg_conf/status.
  • All I told above is valid also for '"stop" )' section of zypkg_controller.sh, with the exception of 3rd part (at line 395 and below), where coders was so desperately slapdash, that forgot even replace "start-up" and "start" with "shutdown"... :oops: Thanks God, only in logging messages...
  • In one place we have

    Code: Select all

    write_log "- Failed: <...>"
    in other place

    Code: Select all

    write_log "---> Error: <...>"
    In one place we have

    Code: Select all

    write_log "Starting <...>"
    in other place

    Code: Select all

    write_log "- starting <...>"
    in third place

    Code: Select all

    write_log "---> start <...>"
    In one place log statements in /i-data/.system/zy-pkgs/tmp/zypkg.log have correct timestamp, and in other statements time is not adjusted with current timezone.

    No code culture, no rules, no common patterns... ANARCHY rules there :(
Well...
We have what we have.

Questions:
  • How is possible to modify files in /etc/* or /ram_bin/* in permanent way (i.e. for surviving reboot)?
  • How can I see a messages, sent by scripts to stdout during startup? /var/log is almost empty. Is there such a tool like dmesg?
Mijzelf wrote:
I never wrote my own regular package, because I simply do not found any materials, learning how to do so (for NAS542). Can you please take a links on some tutorials?
I partly documented the case here. Actually it's no big deal. By just extracting some existing packages, it's rather easy to find how a package is build.
Om-nom-nom, thank you for detailed tutorial for different firmwares, good job! Partly I discovered mechanism by, as you say, analyzing other packages directory structure and file content and behavior, but still some details remain unknown to me, so big thank you, Mijzelf!

P.S. If you eventually decide to enrich your tutorial with my researches above, I will have no objections.
P.P.S. Are you an ZyXEL developer, Mijzelf? Can you appeal to ZyXEL for check and correct their scripts?

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Noob Linux and NSA320 questions

Post by Mijzelf » Mon May 08, 2017 9:45 am

Wow, In-depth analysis!

You are right, the script is not very coherent, and maybe that's my fault. This part (the reading of USRPKG_DEPS_START) is new for firmware 5, and it's by default not in use. (That file doesn't exist).
I *think* ZyXEL planned to implement the possibility to install 3th party packages, by simply uploading them in the webinterface, and this is a part of the implementation, which is not mature.
That feature never made it, and maybe that is because I already ported MetaRepository to firmware 5 before they had time to finish it. And now we are left with a strange semi-manufactured scriptlet.

Anyway, you asked for a way to run your own scripts, and this is a way. OK, the script has to contain the string '/i-data/'. In most cases that string will already be there, as /i-data/ is the only sane place to put your binaries and other scripts. I guess that's the reason I never stumbled upon this 'feature'.
(To be honest, I only used this for starting my 'optware' package. When MetaRepository for fw 5.10 was ready, I had more convenient ways to get my scripts run).
The lines I quoted are not in use by the regular packages, nor by the 3th party packages installed through MetaRepository (which are 'regular', from the NAS' point of view). Their startscript /i-data/sysvol/.PKG/<package>/etc/init.d/<package> is called another way.
inside typos like "#ckeck ZYPKG_DEPS format"
Bear with them. The scripts are written by Taiwanese engineers. English is /not/ their native language.
How is possible to modify files in /etc/* or /ram_bin/* in permanent way (i.e. for surviving reboot)?
For /etc/ you can't, unless you are building your own firmware. /etc/ is inside the initramfs, which is embedded in the kernel.
For /ram_bin/ there is a possibility, but you should use it with care. /ram_bin is the mountpoint of an ext2 containing file sysdisk.img, which is mounted read-only. Further the containing partition of sysdisk.img (sda1 on fw 4, md0 on fw 5) is mounted read-only.
The file sysdisk.img itself is extracted from flash. The compressed file is part of a firmware update.
On boot the checksum of sysdisk.img is compared to a stored one, and if it doesn't fit, a fresh copy of sysdisk.img is extracted. This mechanism is used for firmware updates. So if you change anything in /ram_bin/*, it will be reverted on boot.
There is a way out. If you read /etc/init.d/rcS (containing among others the sysdisk.img extraction code), you'll see that the existence of a file /firmware/mnt/sysdisk/mount.sda1.rw.flag will skip the checksum test, and the partition md0 will be mounted rw (on /firmware/mnt/sysdisk/).
So, to be able to edit /ram_bin/*, and let it survive a reboot, you'll have to

Code: Select all

mount -o remount,rw /firmware/mnt/sysdisk/
touch /firmware/mnt/sysdisk/mount.sda1.rw.flag
reboot
After reboot you maybe have to remount /ram_bin/ rw. On fw 4.x that wasn't necessary, can't remember the status for fw 5. At least I don't see the mount.sda1.rw.flag flag handled in the mounting of sysdisk.img.

You'll have to remove the mount.sda1.rw.flag before updating the firmware. Theoretically it's possible that a new initramfs won't be compatible with an old sysdisk.img file, leaving you with a non-booting box. (In that case you can still use the 'Univeral stick' to get early telnet access, but why accept that inconvenience?)
How can I see a messages, sent by scripts to stdout during startup? /var/log is almost empty. Is there such a tool like dmesg?
Not that I'm aware of. AFAIK the stdout messages are only send to, well, stdout. You can see them, but only by looking to console, which is on the serial port.
If you eventually decide to enrich your tutorial with my researches above, I will have no objections.
Well, thanks. I don't think I will use it, as this mechanism is not really a part of the package system, and the package system is complex enough without sidely related stuff.
P.P.S. Are you an ZyXEL developer, Mijzelf?
Seeing your analysis of that script, that's almost an insult. No, I have nothing to do with ZyXEL. My
activities are just a private hobby.

Post Reply