IRC channel logs
2021-04-07.log
back to list of logs
<fossy>stikonas: how do you mean, API? <fossy>like function args and the like, not external communications to it? <stikonas>my current attempt is get_file(url, mkbuild=False, output=None): <stikonas>where url and output can be either str or list(str) <stikonas>well, we can't have dictionaries because they are not sorted <stikonas>and we need to treat the first element in a different way <fossy>yeah i just thought the same thing <fossy>stikonas: another nice rootfs feature would be to have a different package name than the basename of the tarball <fossy>then we could avoid renaming things <fossy>and another thing i am running into now is i am downloading a single .c file from alpine for a musl incompatibility <stikonas>I'm now around that part of the code right now <stikonas>ok, python thingy is starting to work... <stikonas>maybe I'll try to rework code a bit not to need that... <fossy>is self an instance of SysA? <fossy>bit hacky but you could have that instance as a global and then have a setter <stikonas>but I'm thinkinf of just getting rid of that class <stikonas>and keep functions in just module rather than class <stikonas>it's not like we will be creating many SysA objects... <fossy>as long as there is ablility for a SysB <stikonas>hmm, I'm not sure how much will be done from rootfs <stikonas>most of the processing will probably be from inside sysa <stikonas>we just need to download stuff into different target dir that is inside sysa... <fossy>i was thinking for sysb that we would just download tarballs outside and cp them to a /sysb in /sysa <stikonas>although even for sysa, I think it might be worth putting sources into subdir of /after <fossy>i think it is more useful to attribute sources to a package <stikonas>there are a lot of package directories and they are mixed with installation directories like bin/ lib/ share/ <fossy>so you mean move all the build things to a subdirectory <stikonas>anyway, I don't want to change too many things in one go <stikonas>so this time I'll just mostly convert 1-1 from bash to python <fossy>can just call destructor as a function no? <stikonas>although, I think in Python you can have that without class too <fossy>yeah you just make it into a normal function and call it at the end <stikonas>although, after sed it's just extending the list of packages, nothing new <stikonas>by the way, what example do you have of where you want different package name and basename? <stikonas>is it not what output variable was doing? <fossy>stikonas[m]: yeah that kinda is what output variable was doing. the example was however that i was downloading a raw C files <fossy>so i just wanted to be able to give it a package name <fossy>although i have said screw u to metaconfig and am moving on from trying that so not worrying about that issue anymore <bauen1>stikonas[m]: i would like to have the sources in a seperate directory (perhaps src), so that it's easier to find important / less important things <bauen1>but then again i'd also really like it if we could chroot into /after (or make it sysb), since we shouldn't be touching / anymore and everything important has been copied over <bauen1>and moving the sources in a seperate directory means i can use chkstow more easily to find "left overs" <fossy>making much better progress with cuystom makefile for perl <fossy>however old bison *is* needed, it dosen't compile with new bison <fossy>i have a backwards path for getting it but it does need old flex <fossy>so we can just not delete it <fossy>bauen1: i really prefer having the sources in a package-specific directory - it makes it very clear what package is using what and makes extractions easier <fossy>bauen1: regarding chrooting into /after, that would def be a sysb <fossy>i don't mind it for the short term but in the long term that chroot would be replaced with kernel magic <fossy>so don't rely on it being a chroot <bauen1>fossy: yes, i meant stuffing the whole e.g. bash-2.05b directory into a "source" directory <fossy>lmao i have done that twice today then <bauen1>fossy: chroot can of course also be acomplished by rebooting <fossy>but it is completely untested and not in development <bauen1>i don't really mind how it's achieved, i just want to make the "quasi" boundary into a real boundary <fossy>where are you planning to make the boundary? what package? <bauen1>fossy: the point at which we transition to /after <bauen1>fossy: that's a one liner in C to do so <fossy>you'd have to add chroot syscall support to m2-planet <fossy>we don't even have mes libc at that point <fossy>i don't think mes libc has chroot syscall either (but that's even easier to add a syscall to, i can do that if you want) <fossy>but then you'd have to wait for mes-libc-patched... <fossy>another option is using /after up until a particular point and then instating the "boundary" into real isolation through chroot <bauen1>at some point i need to make a "cut" and have everything (more or less) managed by stow / upkg so i can start with building pseudo packages in chroots <bauen1>fossy: but then you have all these things that hard code /after in their binaries <bauen1>and search pathes, which is the mess i would like to avoid <fossy>bauen1: ugh, yeah, didn't think of that <bauen1>chrooting into /after avoids that, and we can later copy over things to a new root (sysb) without recompiling things <fossy>what would be your ideal staged process for moving live-bootstrap over to upkg <fossy>the stow/upkg stuff all has to happen at once right <bauen1>but at the point where we have perl 5.6.2 we can get stow to easily build chroots for package building <fossy>particularly if we are doing that for a whole lot of packages up until stow <fossy>what exactly is the chroot building thing <bauen1>there's probably a decent-ish way of doing a recursive function in bash to do the linking, but ln also doesn't support `--relative` which i think is necessary for making stow recognise it (i'll test it) <fossy>i still don't really get what youre saying with that <fossy>do you mean like how most package managers do it <fossy>(idk about most even, but void does) <bauen1>fossy: currently "upkg-build" would make a directory, bind mount / mknod a basic /dev, copy over (or bind mount read only) /upkgs, link dependencies, copy source + package script, run package script inside chroot <bauen1>the major benefit this brings is a clear list of dependencies, so you can skip rebuilding based on hashes, or reorder things more easily <bauen1>it does *not* have any sort of dependency resolution, that's your job :D <fossy>please ensure the dependencies are versioned, whatever you do :P <fossy>because as i am sure you are aware we have many many versions of auto* <bauen1>"you want autoconf ?", "sure, maybe you'll get 1.13, 1.12 or something else ..." <fossy>saying cause most package managers only care about one version <bauen1>there is no concept of a "version" right now, mostly the convention of using "$pkgname-$pkgversion-$versionsuffix" <fossy>i would prefer that versions are handled as completely seperate packages <bauen1>calling it a package manager might be a bit of an insult towards what dpkg/apt and the likes are doing lol <bauen1>and the next package that seems to ignore DESTDIR: binutils-2.14 ... <bauen1>stikonas[m]: you've used $() instead of ${} in binutils-2.14.sh <fossy>janneke: just wanted to cehck, there is currently no progress for riscv bootstrap, right? <janneke>fossy: other than the efforts by laanwj, you mean? <fossy>rings a bell.. what did they do? <bauen1>looking at that i'm not sure if nokvm will get to the bash of interest in 2 days <stikonas>than you still need to build mes and tcc <bauen1>there's also just been another patchset on the tcc mailing list about riscv64 inline assembly, so that part is starting to get filled in too <stikonas>although, that part would have to be backported to janneke's tcc 0.9.26 too <stikonas>unless mescc can be improved to build newer tcc <bauen1>you could also try to update jannekes tcc a bit <bauen1>iirc there are some patches after 0.9.26 (and 0.9.27) that make building musl simpler <stikonas>not sure if there is more that we can simplify <stikonas>unless somebody fixes that weak symbol stuff <bauen1>0.9.27 is the latest release and i'm pretty sure ~2 years old now ... <stikonas>yeah, I just grabbed one patch from the tcc mob branch <stikonas>then there are a few patches that remove some more complicated assembly, but I don't think tcc mob will be able to deal with it anyway <bauen1>i don't remember any issues related to weak symbols but i also didn't make much use of myunix musl libc yet <bauen1>and i was using a fairly recent git commit <stikonas>well, tcc -ar had some issues with weak symbols in .a archives <stikonas>once we had binutils ar, those patches are dropped <stikonas>well, those weak symbols patches are more of a hack than proper fix <mihi>Did anyone before try to create a graph of source packages used to build build-dependencies of all Debian packages, and then run a cycle detector on it? <mihi>I wanted to try, my first attempt of manually resolving build-depends only found 41 self-cycles, but running a perl script that parses "apt-get -s build-depend" for 1000 random packages found ~30 more. However, running apt-get -s build-depend on a package takes about 10-15 seconds, so I really don't want to run it for the 28000 remaining source packages... <mihi>I understand that solving dependencies is "hard" (due to virtual packages, conflicts and version constraints), but is there really no easier way? (libapt-pkg-perl does not seem to expose this kind of action) <mihi>maybe gforce_de1977 wants to try on his server farm :-P <mihi>(next step would be to parse the output for packages that are not self-dependent and run some cycle checker algorithm on that. I assume that part would be orders of magnitude faster when using e.g. Tarjan's algorithm) <bauen1>mihi: there's #debian-bootstrap on oftc, i was also thinking of how i can eventually tie in live-bootstrap to build debian from scratch <mihi>bauen1: https://wiki.debian.org/CircularBuildDependencies suggests to run a script to find circular dependencies in essential packages. I am not sure how to interpret this. By definition, *all* packages build-depend on build-essential packages which are a superset of essential packages. So finding cycles within them is meaningless as there could be many more non-declared ones. I was interested in cycles in non-essential packages (to <mihi>find obscure compilers and stuff that may need some bootstrapping) <bauen1>oh i over read half of what you where saying, i'm sorry <mihi>so to build debian from scratch, in my opinion at least the 80 build-essential source packages need to be tried manually. <bauen1>i'm not sure how the information about source packages is distributed but dose-distcheck or similar tools parse it manually which should be *a lot* faster than invoking apt <bauen1>my idea for continueing from live-bootstrap to debian is to make the debian dependency graph non cyclic (e.g. by rewriting some packages) and to introduce some of the live-bootstrap pseudo packages as packages for the debian bootstrap <mihi>I did a manual parse (using libapt-pkg-perl) by checking all concrete build-depends of a source package against package outputs of that source package. But that totally skipped virtual packages with more than one alternative and/or and package constraints <mihi>and it seems that those are important in practice. bsdmainutils for example has no obvious build-deps on itself, but when running apt, it will install bsdmainutils to build bsdmainutils. <bauen1>mihi: it could also be faster to only use the unstable repo instead of buster + d-u + d-s <mihi>I don't believe there are that many packages in d-u or d-s that are not inside main anyway. Only more recent versions. And about unstable - how much stuff is in there that FTBFS? <bauen1>mihi: it's more about eliminating duplicated packages, and generally removing things apt has to do <mihi>even if it took 5 seconds per package instead of 10 or 15, it would run for more than 35 hours. <bauen1>so creating a graph of source packages and their build dependencies can probably be achieved by parsing the Sources manually, but you also want to do things properly (including virtual packages) and are hence using apt ? <bauen1>but yeah, the information you're trying to get would be very useful and maybe even exists somewhere ? <bauen1>or rather, i'd be very surprised if it didn't since you will hit this issue if you try to cross compile too <mihi>yeah, I wanted to avoid reimplementing the whole solver logic for dependencies, running the risk that it would be even slower than apt (and buggier). <mihi>your comment about cross compilation makes me remember that I ignored arch dependency completely. Arch:all packages (e.g compiled Java bytecode) could still create bootstrap loops but should not matter when cross-compiling. <bauen1>mihi: i'd really recommend you to talk to some of the cross compiling debian folks, i would think they already have the data that you're after (or at least know tools) <stikonas>well, at the moment debian doesn't even try to bootstrap openjdk via jamvm... <stikonas>and with jamvm you indeed have to be careful with compiled bytecode... jamvm tarball ships with it <stikonas>at some point I found that guix was not removing pre-compiled bytecode from jamvm, but luckily patch to fix that was one-liner