IRC channel logs
2024-02-13.log
back to list of logs
<fossy>janneke: perhaps I misunderstand what you are saying, but for 6+ months now live-bootstrap has had a kernel bootstrap in it? <fossy>we do not use a seed Linux kernel in live-bootstrap <fossy>(re: gnu/linux bootstrap, rather than linux bootstrap) <fossy>AlwinB: regarding Fedora bootstrap, the first step for any Linux distribution bootstrap is determining how one would make a Fedora system, completely from source, from a different Linux distribution' <fossy>e.g. can you build a Fedora ISO, or chroot, etc from source, from Debian, or some other non RPM distribution? <fossy>that's the first, and most prohibative step, usually <fossy>(when one is hooking onto the end of live-bootstrap or guix, and not bringing the entire bootstrap to it natively) <stikonas>I guess generally live-bootstrap is good enough now that steps are similar <stikonas>to bootstrap from other distro or from live-bootstrap <rickmasters>Googulator: After thinking about your report on Fiwix available user memory, I'm afraid I don't have a helpful suggestion on resolving the issue. <Googulator>One thing I tried was passing the initrd already at a high address, which lead to more user memory shown at boot, but subsequently I got malloc errors <rickmasters>Googulator: For what its worth, it's working according to the original virtual memory design by Mikaku. <rickmasters>My "hack" of relocating the initrd to upper user memory was well outside the design. <rickmasters>But in retrospect, it did use memory better for the case where there is a large initrd <rickmasters>The initrd memory mapping was handled as a special case and it was more complicated, though. <rickmasters>I was exploring making more than 2GB physical memory available within Mikaku's design. <rickmasters>User space is running out because 1.6G of that 2G is kernel memory (mostly ram drives). But if we could use 3GB it would be fine. <rickmasters>But I'm not sure how to enable that. He makes assumptions that high physical addresses are mapped differently, so I think it causes a conflict. <rickmasters>Memory is currently truncated at the bottom of mm/bios_map.c but removing that doesn't work (no surprise). <oriansj>well if one assumes PAE, it is much less of a problem <oriansj>or if we move more of the init into read-only storage <rickmasters>oriansj: there is enough physical memory without PAE. <rickmasters>The problem is that there is a simple mapping strategy that maps the low ranges into high ranges, so you run out of ranges after 2GB. <rickmasters>i.e. the initrd at physical 0x001c6000-0x501c6000 is mapped to virtual 0x801c6000-0xd01c6000 <rickmasters>There is memory available at physical 0x80000000 to 0xC0000000 but mapping it using the simple scheme would map into virtual memory used by user space. <rickmasters>Anyway, to use that physical range we need deal with it in a different way. <rickmasters>I'm probably not explaining it very well but to be honest I don't understand it as much as I'd like. <Googulator>loading srcfs from USB 2.0 now only takes about 2x as long as on SATA 2.0, vs 100x as long without the PRs <Googulator>plus, one of my motherboards would just straight up freeze without the pushad/popads - I did have those present when working on the original HW compat PR, but I removed them at some point, probably by mistake <Googulator>I chose reading 64 sectors at once because it's the largest power of 2 that's consistently supported (many BIOSes have a 127 sector limit per int 13h call, but using 127 would complicate calculations) <Mikaku>rickmasters: with the current virtual memory design, Fiwix supports initrd images of up to 2GB of memory (aprox.), do you really need more than this? <Guest47>I want to build bare metal but i don't know what <Guest47>I want to build bare metal but i don't know what? <Guest47>I want to build bare metal but i don't know how? <janneke>fossy: although i wasn't aware that live-bootstrap included a build of linux, i guess it's built as a `user-land' application, live-bootstrap itself still (implicitly) depends/runs on linux <Guest21>I want to build a bare metal but I don't know how? <Guest21>HI, I want to build on bare metal but don't know how. <Guest21>It works on qemu but it doesn't work on bare metal how start? <Guest000>hi, It works on qemu but it doesn't work on bare metal how start? <fossy>Guest000: ask your question and wait around, stop asking the same question over and over again repeatedly and leaving after a minute <fossy>janneke: yeah, but kinda even more than that. live-bootstrap is not initially booted with linux <fossy>we start with builder-hex0, rickmasters' very very minimal POSIX-compatible kernel, written in hex0 <fossy>compile up to tcc, which then builds Mikaku's Fiwix <fossy>which then builds up to gcc, which then finally builds the Linux kernel and begins running on that <fossy>that is the first time the Linux kernel is introduced to the bootstrap <janneke>fossy: wow i'll have to look into that, i wasn't aware you were this far already <fossy>it still certainly has some issues. particularly bare metal related <janneke>thanks, and sorry for misrepresenting the current state <fossy>no stress, you didn't know :D <Guest000>Guest000: I don't understand what to write in first .img <fossy>I'd recommend if you just want to give it a go doing ./rootfs.py --bare-metal, it'll tell you the path to the .img <fossy>and then write that to a USB or something <stikonas>if you have UEFI system, builder-hex0 won't work <stikonas>if you have UEFI only system then bootstrap is not complete and stage0-uefi can only get you to tcc-mes step <fossy>what is the process you have done to create, write and boot the .img? <Guest000>1 step․ ./rootfs.py -b -a x86 --cores 4 2 step․ I burned the prepared img on the drive and turned it on, but it didn't work <janneke>fossy: "Compiling an initial mes using M2-Planet. Note that this is only the Mes interpreter, not the libc or anything else." <janneke>of course, together with the mes sources, also a part of the mes c library is included in the mes-m2 build <janneke>it's not being compiled as a separate libc <janneke>but most everything that's in mes'c libc.a (as opposed to libc+tcc, libc+gnu) is being compiled, as that's what mes[-m2] needs :) <Guest000>1 step․ ./rootfs.py -b -a x86 --cores 4 2 step․ I burned the prepared img on the drive and turned it on, but it didn't work. it is not possible to move forward with other steps apart from these steps՞ <Googulator>Guest000: I've run into the same issue yesterday - some BIOSes would actually lock up halfway into loading sources, others will just be so slow as to appear to have locked up; that builder-hex0 commit should fix it <Googulator>or just use a SATA drive, which should work with the included version of builder-hex0 <janneke>fossy: i especially like the description of the early stages, with small code examples <Googulator>and yes, it does work in qemu, as well as on most bare metal setups with a conventional BIOS or CSM <Googulator>just not on USB drives on some systems - for that, you need the builder-hex0 fix I linked <fossy>janneke: good point, I'll update that bit about mes, thanks :) <fossy>Googulator: ooh, thats the problem I had on one of my systems <Googulator>fossy: yeah, and it's one that I thought I fixed already in the original HW compat work <Googulator>but accidentally deleted the fix in of the rebases <fossy>ah, lol that's annoying, i've done that before too <Googulator>it's a simple pushad/popad pair around every int 13h <Googulator>to cope with BIOSes that trash random registers after an int 13h call <fossy>Googulator: what does #436 bring init.img to? <Googulator>and that's just using existing upstream xz/lzma tarballs I could find - we can go even lower if we recompress <Googulator>I would like to also fit SSL support in a 256MiB init.img, which is gonna be harder, but hopefully doable using an alternative SSL library <fossy>yeah i think it should be possible <fossy>hm, i'm not super keen on all the lzma / xz tarballs from random mirrors though... <fossy>i guess we can just revert to the usual ones if they died... <fossy>but also seems a very easy point for source tampering.. <Googulator>I did verify that the contents are identical to the non-xz ones <Googulator>all but one decompress to the same uncompressed tarball as the one it replaces <Googulator>that one (can't remember which one) produces a different tarball, but that tarball then unpacks identically <Googulator>The tarball checksums included should catch any future tampering, unless someone manages to collide sha256 <Googulator>make-3.82 is the one where the inner tarball isn't binary identical <matrix_bridge><Andrius Štikonas> Well, one mirror tarballs have other problem <matrix_bridge><Andrius Štikonas> You are more likely to be blocked in some firewalled environments <matrix_bridge><Andrius Štikonas> Thought we have a few other one mirror tarballs but mostly out of no choice... <Googulator>We don't currently support multiple mirrors for the same tarball at all. <matrix_bridge><Andrius Štikonas> fossy: how about extending sources syntax to allow other options <Googulator>I was planning to add that; it needs a rewrite of how sources are handled <matrix_bridge><Andrius Štikonas> Well maybe switching from lzma to bz2 is most my transparent anyway <matrix_bridge><Andrius Štikonas> Should be handled all internally by helpers.sh <Googulator>and then instead of failing after any file fails to download, we go on until the end, and then check if every file listed now exists in distfiles - if not, fail <Googulator>I wonder if Python's implementation of gz/bz2/lzma/xz is guaranteed to generate the same compressed streams in different Python versions with the same parameters <Googulator>nice project, but I'm not sure if it addresses my concerns <Googulator>i.e. where the same compression parameters given to different versions/implementations of gzip produce slightly different compressed streams <civodul>Googulator: one of the thing it does is to determine compression parameters <civodul>so it can later rebuild bit-identical tarballs <civodul>it does other things too, but maybe you can extract that part <Googulator>the problem is, parameters don't entirely capture a compressor's behavior <civodul>well, it extracts parameters such that it can rebuild the same compressed file <Googulator>I don't think compression algorithms are that tightly specified <civodul>the parameters in this case are well defined: Disarchive knows how to interpret them <civodul>but it’s not subtle: it just tries a whole range of options and determines which one works <Googulator>but are the same parameters guaranteed to produce the same compressed stream on a vastly different version of gzip? <Googulator>compression is an extremely heuristic process, so to me, it's entirely plausible that a newer compressor might emit a smaller stream on the same input with the same parameters, due to improvements to the heuristics <Googulator>that's unlike decompression, which by definition, needs to be extremely deterministic <civodul>yes, it’s designed to ensure determinism <civodul>it’s very much a “black box” approach: Disarchive runs stuff and looks whether the output matches <civodul>but then it knows what to run for a given parameter <matrix_bridge><Andrius Štikonas> Googulator: but your sources proposal does not let us switch tarballs type easily. It's better than what we have now... <matrix_bridge><Andrius Štikonas> Also it might be nice to have an option of local mirror <matrix_bridge><Andrius Štikonas> E.g. download everything from http(s)://mydomain/live-bootstrap/ <Googulator>IMO the best way to handle a local mirror is to have it bypass the URLs in the sources files, and just serve & request everything by name (or even better, by hash) <Googulator>the original bz2 make tarball is a weird one; it seems to be a format intermediate between v7 tar and ustar <Googulator>which is probably why the slitaz folks chose to fully repack it, and not just recompress <oriansj>or just make an environmental variable that can override the behavior of where to fetch files from <Googulator>I was thinking passing an IP address to the host in bootstrap.cfg <oriansj>like live-bootstrap-source-mirror if set download all of the files from there; otherwise use these links <Googulator>and then having rootfs.py act as a basic HTTP server after it's done creating the image <Googulator>this file used to be part of many old versions of LFS (3.2, for example), but those got purged from LFS mirror FTPs. <deesix>In addition to that kind of copy in the Wayback Machine, archive.org allows registered users to make arbitrary uploads so we can have a directory with all those files. <deesix>As unregistered user, one can add URLs to the Wayback Machine, too. <rickmasters>Mikaku: The problem Googulator reported is that there is less memory available to user programs. <Mikaku>rickmasters: well, the ramdisk drives in Fiwix, unless Linux, use fixed memory. I mean, the kernel reserves all the memory for them instead of assign them memory dinamically <Mikaku>rickmasters: this is something that I plan to change, that is implementing RAM disk drives on top of the buffer cache <rickmasters>Mikaku: I'm not sure what that means or how it would help, sorry. The issue seems to be that only 2GB of ram is supported. <Mikaku>rickmasters: yes, there are two possible memory splits: 3/1 or 2/2 <rickmasters>The code I originally wrote for large initrd relocated it to memory that normally goes unused with Fiwix. <rickmasters>That design left more memory available to user programs. Even though Fiwix only used 1GB in the old 3/1 split, it was mostly available to user programs. <rickmasters>The question I have is why do you limit available memory to 2GB at the bottom of mm/bios_map.c (with 2/2 split)? <Mikaku>I need numbers to reproduce what you mean. What physical memory, what amount of memory for the ramdisk drive, and what statistics show Fiwix during the boot <Mikaku>I don't see an '#ifdef CONFIG_VM_SPLIT22' in that mm/bios_map.c <rickmasters>This is with a 1.3GB initrd and 256MB kexec ram drive. <Mikaku>with this configuration, user space will have no more than 450MB of memory (aprox.) <Mikaku>I remember you tested my changes on this and everything went OK <rickmasters>To be clear, your design is sound and there is no bug. <Mikaku>I also tested it and all went OK as well <rickmasters>What Googulator is pointing out is that the old implementation gave more memory to user programs. <rickmasters>He is trying to use more memory for a change to live-bootstrap he is working on. <rickmasters>Yes, 450MB of memory seems like enough to me but I guess more would be helpful. <rickmasters>So, back to the code, the mm/bios_map.c limits physical memory with: kstat.physical_pages = (GDT_BASE >> PAGE_SHIFT); <rickmasters>GDT_BASE is set in include/fiwix/linker.h based on PAGE_OFFSET and PAGE_OFFSET is set based on the CONFIG_VM_SPLIT22 <rickmasters>I was exploring how to remove that 2GB memory truncation and use 3GB of physical. <rickmasters>The problem seems to be that in page_init() there is this line: pg->data = (char *)P2V(addr); <rickmasters>I think that is pre-computing the virtual address of free memory. But that virtual address scheme only works up to 2GB. <rickmasters>The P2V macro would map physical 0x80000000 to 0xC0000000 to the same virtual addresses, but that virtual range is already mapped. <rickmasters>Because P2V maps 0x00000000 - 0x40000000 also to 0x80000000 - 0xC0000000. <Mikaku>rickmasters: I'm not following you <Mikaku>the numbers in the bottom of the URL above is like comparing apples and oranges, the first has 2GB physical and the second case with only 1GB physical you cannot define the same ramdisk size than in the first case <Mikaku>what was the cmdline in both cases? <rickmasters>In both cases there was 4GB physical in the machine. <rickmasters>The first case is 2/2 split and Fiwix limited physical to 2GB. <rickmasters>The second was the old 3/1 split and Fiwix limited physical to 1GB. <Googulator>In the 3/1 case, the initrd was relocated to be outside the 1GB normally used by Fiwix, using rickmasters' original large-initrd implementation <Googulator>Mikaku: about the Kexec memory map passing issue, it seems there are 2 (maybe 3) distinct issues at play here: <Googulator>1. Fiwix modifies the memory map for its own needs (i.e. reserving a range needed for kexec), but doesn't back up the original memory map, instead passing the modified one to the kexec guest. <Googulator>2. The actual BIOS memory map is a list of (start, length, type) records, but Fiwix prefers to instead store a (start, end, type) record list, and there seems to be some confusion about the semantics of the "end" field. <Mikaku>rickmasters: ok, both have 4GB of physical memory but different vm split, what was exactly the cmdline? I need to know how many ramdisk drives were defined and their size <Googulator>In both cases, there was a 256MiB kexec ramdisk allocated by Fiwix itself, and a 1280MiB initrd passed by kexec-fiwix at physical address range [0x1C6000, 0x51C6000) <Mikaku>so IIUC, on the rickmasters' 3/1 split with an initrd of 1.3MB and a kexec of 256MB there is more user space available than on the 2/2 split with the same initrd and kexec sizes? (assuming 4GB of physical memory in both cases) <Googulator>in fact, with rickmasters' version, the amount of userspace memory available is independent of the size of the initrd <Googulator>it's 1GiB minus the kexec space & whatever the kernel itself needs <rickmasters>I had relocated the initrd to virtual/physical 0x6000000 which was "free" in the 3/1 scheme. <Googulator>with 1.5.0-lb1, the initrd now impinges upon userspace memory - the larger the initrd, the less memory remaining for userspace <Googulator>Been looking at the deblob script; the overwhelming majority (even in the default, non-force mode) that it removes isn't actually blobs, but firmware loading machinery that's entirely harmless from a trust/security perspective if the corresponding firmware isn't installed <Googulator>In fact, I haven't been able to find a single example of a blob masquerading as source in actual C files so far <Googulator>so far, the only actual firmware that may need to be removed is in the /firmware directory, clearly separated out <Mikaku>I guess the best way is taking advantage of the memory above 1GB (in 3/1 split) and above 2GB (in 2/2 split) for ramdisk drives <Mikaku>this will complicate the current static model of reserving the whole memory for ramdisks in the case when the requested ramdisk size doesn't fit above the 1GB or 2Gb on each case <rickmasters>Mikaku: There is typically only 1GB contiguous memory available above 2GB for the 2/2 split. <Googulator>IMO it's enough to do that for the kexec ramdisk <Googulator>and also to tolerate an initrd being passed in high memory <Googulator>regular ramdisks aren't relevant for bootstrapping <Mikaku>but what if you have 2.5GB of physical memory only? <Googulator>builder-hex0 requires 3GB as an absolute bare minimum <Mikaku>the design must be generic to fit all cases <Googulator>the only potentially problematic case I see is where a user has 2GB of physical RAM, and tries to combine 2/2 split with kexec <Googulator>in which case, there's no space to place the kexec buffer in high memory <Mikaku>in that kernel the memory for ramdisk and kexec will be taken from the user space <Googulator>but even without dealing separately with kexec, 3/1 split while tolerating initrd in high memory is enough to get 768MB of userland memory <rickmasters>Googulator: Mikaku's current design pretty much assumes the initrd is contiguous after the kernel in low memory. <Googulator>multiboot already passes the initrd with an explicit address <rickmasters>Googulator: Perhaps it can be changed but it would not be trivial <Googulator>in fact, Fiwix already does boot with a high-placed initrd; it just seems to cause weird malloc failures in userspace later <stikonas>yeah, stage0-posix also needs a lot of physical RAM, probably at least 3 GB too... <stikonas>once you start building mes, those processes use a lot of RAM and have to be suspended for forks <stikonas>so you need maybe 1 GiB for program memory, 1 GiB for suspended processes... <rickmasters>Googulator: I'm surprised that works at all. See get_last_boot_addr() which is used as the foundation for locating kernel structures. <Googulator>ouch, that's indeed really naive... and exploitable for our purposes :) <Googulator>it uses the end address of the last module - as in the last one listed in the multiboot structure, not the last one in memory <Googulator>nothing stops kexec-fiwix from adding a fake, zero-length module located after the initrd in the multiboot structure, but placed immediately after the kernel image in RAM <Googulator>& extra modules are just ignored - apart from their effect on placement of kernel structures <rickmasters>I'm just saying that it's strange that you were able to boot with a high initrd given how it works now. <rickmasters>Maybe it's not that strange, but the kernel will reserve all the memory up to the end of the initrd: <Googulator>it's not that I didn't have memory, but rather malloc failed in a weird way <Googulator>probably because kernel structures overflowed into non-RAM <rickmasters>I'm a little fuzzy on virtual memory, but it's surprising that you were able to do anything in that address space, <rickmasters>like run a program at 0x0804xxxx given the whole area was kernel reserved. <rickmasters>but I'm probably just mixing up physical vs. virtual <Googulator>V2P(_last_data_addr) is last_data_addr - PAGE_OFFSET <Googulator>probably some over/underflow meant it didn't reserve correctly <rickmasters>Anyway, I see what you're getting at- use kexec to locate the initrd in a similar fashion to what was done before. <Googulator>your version copied the initrd from builder-hex0's file space to 0x1c6000, and then from there to 0x60000000 <Googulator>right now, we copy to 0x1c6000, and leave it there <rickmasters>The method that I chose was compatible with qemu's placement of the initrd although that doesn't necessarily need to be the case. <Googulator>what I want to do instead is to just pass its address in builder-hex0's file space directly to Fiwix, and then add a dummy 2nd module at 0x1c6000, to correctly place the kernel structures <Googulator>the only thing needed then is to mark the actual location of the initrd as reserved in Fiwix's copy of the e820 map <rickmasters>I shouldn't really comment until seeing the entire thing. <rickmasters>But, to be honest, the existing design plus being able to utilize more physical memory for user programs seems like it would be cleaner. <stikonas>Googulator: was it you who said there is a way to force gcc to do only 2 stage builds?