IRC channel logs

<oriansj>in fun news, I learned stage0's bootstrapping process is a killer for microSD cards due to the extreme number of writes prior to the M2-Planet+M2libc stage

<stikonas>oh yes, we write one byte at a time...

<Hagfish>maybe worth putting a warning in a README about that

<fossy>this is not a problem i forsaw

<fossy>now that i am checksumming tarballs instead of individual files this includes docs

<fossy>and i am finding non-reproducible docs

<fossy>i will put up a draft PR shortly however with what i have done, and then it should just be not reproducible docs, and fixing up qemu mode

<doras>oriansj, stikonas: I'm using an SSD for the bootstrap. Do you expect these many writes to negatively affect it as well?

<stikonas[m]>doras: live-bootstrap is in RAM, so should be fine

<doras>I'm actually not using it in RAM.

<stikonas[m]>Oh yes...

<doras>I'm currently not doing the tmpfs.

<doras>Only /tmp inside the sysroot is mounted as tmpfs.

<stikonas[m]>Well, hard to tell, SSD might have much better cachinh

<doras>:S

<doras>So

<doras>I fixed/changed BuildStream's FUSE filesystem so it behaves more like a normal filesystem.

<doras>And the two issues that the configure script found are no longer an issue.

<doras>However... there's a third issue, which is a configure test program which fails only in the non-FUSE case.

<doras>In other words, there's nothing I can realistically do here, because it's not the BuildStream case that fails; it's the non-BuildStream case that fails.

<doras>This is the failure that I get only in the non-BuildStream case: https://paste.gnome.org/plar6yciw/sbfi5t/raw

<doras>So the existing findutils checksum depends upon it using an alternative implementation for getcwd.

<doras>So I think I'm only left with patching/forcing findutils to always use the alternative implementation, even when getcwd works fine.

<stikonas[m]>doras: yes, that's an option

<stikonas[m]>we can include such patch in live-bootstrap

<fossy>doras: i would expect it to be overall fine

<fossy>an average build probably has <10G of writes, if i had to guess

<stikonas>fossy: well, the worry is about early stages

<stikonas>i.e. hex stuff

<stikonas>which is not many bytes writte, but just as many counts

<stikonas>and in SDs even if you write one byte, the whole block is overwritten

<stikonas>well, same is true for SSDs

<stikonas>still, I've been running live-bootstrap and stage0-posix many times and have no issues

<oriansj>doras: well there is about 800K write calls in a run of stage0-posix. Now a smart kernel would buffer the writes and sync them to disk every 4K writes (or every 1MB depending on block size)

<oriansj>mind you microSDs have much fewer write cycles and no caching RAM onboard like a proper SSD. And the microSD in question has been repeatedly used in testing of stage0-posix for more than a year.

<oriansj>So a reasonable warning is if you are going to do daily builds of stage0-posix, plan on a 1-2 year replacement cycle for the drive in use.

<oriansj>(note this is a very conservative warning and this risk might not even apply to an SSD which caches writes to RAM with a write-back algorithm to the actual flash)

<oriansj>[[ideally your SSD has a supercap to provide power to flush contents of RAM to flash in the event of power loss to the drive]]

<doras>Thanks.

<doras>Regarding getcwd, I suspect there's something deeper here.

<doras>It may be related to /proc missing, as was previously suggested here.

<doras>Or something similar

<doras>Maybe I can figure out how to fix it.

<doras>Well, it's not the missing /proc

<bauen1>doras: i'm not sure if bwrap does it, but maybe it's blocking certain syscalls ?

<bauen1>or forcing the linux compatibility to be off or things like that ?

<doras>It's not supposed to

<doras>This getcwd issue is found by the configure script in all the build variations of live-boostrap

<doras>Otherwise the checksum would be different.

<doras>I suspect that somehow the use of FUSE by BuildStream works around this issue.

<doras>Interesting... the getcwd we have in the bootstrap seems to fail the test program regardless if it's run outside or inside the chroot.

<doras>If I build the test program outside the bootstrap, getcwd seems to work fine.

<doras>I'll continue digging.

<doras>stikonas: how did you choose what to import in "import-gnulib.sh"?

<doras>And what is actually the purpose of that import operation?

<stikonas>doras: import-gnulib.sh is usually the same revision as in the release tarball (if known) or else some arbitrary revision from that time that works

<stikonas>it's done because that Makefile.am file is generated by this procedure

<stikonas>and we tried to avoid using anything that is generated

<doras>I mean, there's a list of modules there: https://github.com/fosslinux/live-bootstrap/blob/master/sysa/findutils-4.2.33/import-gnulib.sh#L18

<doras>Where this this list come from?

<doras>did*

<stikonas>oh that's from the comment in the auto-generated makefile

<stikonas>if you look in tarball

<stikonas>of findutils, you'll find this list

<stikonas>look at gnulib/lib/Makefile.am

<doras>I see.

IRC channel logs

2022-01-14.log