IRC channel logs

<rickmasters>Mikaku: sorry for the delay, the offsets were zero because I made a mistake typing in the output. I was running in graphics mode.

<fossy>stikonas, pder: i think i'd prefer recreating unified-libc.c - relying on previous stages isn't the best, imo

<stikonas>fossy: perhaps we should then split libc build into a separate package?

<stikonas>we do that later with musl libc

<fossy>stikonas: yes, that is a good idea

<stikonas>pder: could you do that as part of your changes?

<fossy>that was done back when there was much less structure in the project, so everything was a bit all over the place

<stikonas>indeed

<stikonas>well, pre-bash stuff is still somewhat less structured

<stikonas>maybe if we ever get gash working, we can structure it a bit more too

<fossy>yes, kind of a nessecity tho

<fossy>yeah

<stikonas>though we might not be able to reuse the same .helpers.sh script

<stikonas>since that one relies on more binaries (coreutils, etc being present)

<stikonas>fossy: so for now I'm not looking at readline, will come back to it later. I'll first update binutils in sysa

<stikonas>We should be able to build something newer than ancient 2.14

<stikonas>fossy: interestingly, last tcc rebuild (with last musl rebuild) are important for the first stage of GCC

<fossy>stikonas: that would be good, esp for architecture support

<stikonas>for some reason without them gcc build fails (but after ./xgcc binary is built)

<fossy>odd

<stikonas>yes...

<stikonas>anyway, I just moved that final tcc rebuild before gcc

<stikonas>and moving newer autotools pre binutils now

<fossy>i finished parallelism, but ran into an issue that debug symbols for autogen, guile and gcc are not reproducibile when there is parallelism

<fossy>so i'm going back to stripping first, then parallelism irreproducibility will be eliminated

<stikonas>if we can get stripping working...

<fossy>i looked through Gentoo estrip, but it uses file anyway...

<fossy>so we need file either way

<stikonas>oh ok...

<stikonas>well, file is not a big deal

<stikonas>though we can only get it after python

<fossy>how do you mean? file is in C

<stikonas>oh, is it, ok...

<stikonas>I thought it needs python...

<stikonas>yeah, then it is no problem at all

<fossy>oh, there is a python ffi-style interface

<fossy>but its an extension of the C part, not core

<stikonas>ok, maybe that's why I was thinking of python

<stikonas>I guess we don't need that extension

<stikonas>fossy: what about gcc -s?

<stikonas>can we not use that instead of stripping afterwards

<fossy>not universally. eg, for binaries, you can (safe to strip all symbol tables & debugging); but for static libraries, that is not always safe (eg, libgcc.a). this is why different options (none vs --strip-unneeded vs --strip-debug) are used for different types of libraries

<fossy>at least that's my understanding

<stikonas>hmm, that -s command seems to be part of linking stage

<stikonas>I'm a bit surprised that it is involved for .a archives

<stikonas>those are not really linked, just an archive of .o files

<stikonas>and my autotools refactoring might also enable me to update xz

<stikonas>as I'm moving autotools much earlier, so I could build newer xz with new autotools dependencies

<stikonas>argh, the original bash implementation (pre-find) of packages does not support empty dirs

<stikonas>I guess I should move findutils back a bit earlier...

<stikonas>argh, but findutils I think depends on binutils

<stikonas>I think it's time to improve get_files from helpers.sh

<stikonas>fossy: still here?

<stikonas>fossy: do you remember why here https://github.com/fosslinux/live-bootstrap/blob/master/sysa/helpers.sh#L349 we loop and append to tar

<stikonas>rather than construct one long tar file

<stikonas>is it due to command line length?

<stikonas>anyway, I'll probably leave the loop, should be safer

<stikonas>hopefully I have a working get_files now

<stikonas>and it even ouputs nicer list, i.e. ./dir/subdir rather than ./dir/./subdir

<lilyp>Hi folks, for those of you interested in Vala (sort-of C#, but for GNOME devs), I've bootstrapped it partly over at https://gitlab.gnome.org/lilyp/vala-bootstrap – help welcome

<sam_>oh nice, thank you for sharing

<fossy>stikonas: determinstic file ordering in tars

<fossy>by default, tar is not deterministic, and before we have --sort option to tar we have to do a few hacks

<fossy>once we can make filelist.txt properly, then we use that (it's much faster)

<fossy>before that, we use get_files

<stikonas>oh, tar itself changes the order

<stikonas>anyway, I've fixed a few issues in get_files

<stikonas>doing some final testing, but I think it will even generate tarballs with the same checksum as find version

<fossy>neat!

<fossy>yeah, get_files was a bit of quick kludge haha

<fossy>i don't particularly love shell

<stikonas>the fix is fairly small

<fossy>surprised

<stikonas>I remove ./ from one echo call, moved another echo call out of esle block to run unconditionally (to print directories without files) and added --no-recursion to tar call

<stikonas>but yes, shell is a bit clunky for more complicated programs

<fossy>neat! thanks

<stikonas[m]>ok, I don't think it updated get_filels will provide the same ordering as find command but it's not critical to have that...

<stikonas>oh actually sort is available at the same time as ls

<stikonas>so I could run it past sort...

<pder>fossy, stikonas: I went with recreating unified-libc.c. I just need to work through fixing many checksums. What is the technique for doing that? --update-checksums doesnt appear to be writing updates to where I think it would be

<stikonas>pder: --update-checksums only ignores checksums

<stikonas>you should run it till completion in qemu mode

<stikonas>mount /dev/loop0p1 somewhere

<stikonas>and then cd into that mount path

<stikonas>actually cd into ${mount_path}/usr/src/repo

<stikonas>and then sha256sum *

<stikonas>and you need qemu mode to capture all checksums

<stikonas>actually early checksums are stored somewhere else....

<stikonas>hmm

<pder>I was fixing them manually and rebuilding each time, but I expect they will stop changing one we switch to musl

<stikonas>pder: so I've opened some build script and it has install ${pkg}.checksums ${srcdir}

<stikonas>yeah, once you switch to musl it will work

<stikonas>not sure which package is the last one to change

<stikonas>might be musl or even tcc after that

<stikonas>but at some point they indeed stop changing

<stikonas>but coreutils would be painful to update

<pder>Yeah, thats where I stopped yesterday

<pder>I can probably capture the log, and make a little script to update the checksum files

<stikonas>just run with --update-checksums

<stikonas>and look inside build directory

<stikonas>e.g. tmp/sysa/sysa/coreutils-5.0/coreutils-5.0.checksums

<stikonas>after it passed that stage

<stikonas>then just copy that file outside

<stikonas>otherwise you have to rebuild mes each time...

<stikonas>and that takes like 20 minutes

<pder>ok, I will try that. I thought that file was not getting updated. Do I have to use qemu mode?

<stikonas[m]>Not for early kaem checksums

<stikonas[m]>qemu mode is only needed if you want to grab *all* newer checksums including kexec-tools

<rickmasters>Mikaku: I finally got serial working. The ttyS0 login was commented out in /etc/inittab.

<rickmasters>Mikaku: I had to boot in graphics mode, edit that, then I could boot and login in serial.

<Mikaku>rickmasters: great!, now it's time to upgrade to FiwixOS 3.2 ;-)

<rickmasters>Mikaku: I've been testing the lseek hole with FiwixOS 3.2, same results. I just added a comment to the Issue.

<Mikaku>Ok, checking

<Mikaku>rickmasters: OK, it's time to keep trying on my side, thanks

<Mikaku>rickmasters: BTW, have you tested it with the lates version in GitHub?

<rickmasters>Mikaku: yes, latest testing was from current github, built from source

<Mikaku>perfect

<rickmasters>Mikaku: I noticed you boot off a floppy, are you sure the test is run on ext2?

<Mikaku>yes, I boot from floppy because is easier for me to update the kernel in the floppy than in the disk image

<Mikaku>the GRUB on floppy has root=/dev/hda2

<rickmasters>ok

<rickmasters>Mikaku: I've found something important. If I reboot, the hole is filled with zeros and stays that way.

<rickmasters>Mikaku: So, it might be related to a freshly allocated disk block that hasn't been flushed to disk or something like that.

<rickmasters>Mikaku: I didn't even re-create the file. I just rebooted and the test file was magically fixed.

<Mikaku>hmm, in my tests the file is always created for the first time because I do not reboot, I just close QEMU

<Mikaku>this doesn't corrupts the filesystem because I do not type 'sync'

<Mikaku>so when I start the system again, nothing happened and the file doesn't exist

<Mikaku>even with this mechanism, the hole is _always_ filled with zeros

<muurkha>kinky

<Mikaku>rickmasters: anyway, it looks indeed something related to buffers, or even page cache

<rickmasters>Mikaku: Maybe try putting the main code in a loop with the file name sprintf(filename, "mytest%d", i); and run 1000 times??

<rickmasters>then od -tx1 -Ax mytest*

<rickmasters>if you made testdata 16 bytes then the output should have alternating lines of zeros, but maybe it would repro

<rickmasters>Mikake: the idea being that with enough writes you'd use up any cached buffers that are preventing the repro

<rickmasters>Mikaku: or, maybe try to use a freshly created disk image

<Mikaku>rickmasters: you hit in the nail! :-)

<Mikaku>the first file (mytest1) has corruption data, the rest (999) have zeros

<rickmasters>Mikaku: well, I was expecting the opposite but ... great!

<Mikaku>yes, strange that it only happens to the first, but is not affected with only one file

<Mikaku>anyway, now I have something to work on

<Mikaku>rickmasters: thanks!

<rickmasters>Mikaku: You're welcome. Thank you for staying on it.

<Mikaku>muurkha: :-)

<muurkha>(-;

IRC channel logs

2023-02-18.log