IRC channel logs

2022-05-23.log

back to list of logs

<stikonas>can anybody figure out why shellcheck is offended https://github.com/fosslinux/live-bootstrap/pull/174/checks?check_run_id=6546511171 ?
<stikonas>I guess I'll just disable it
<oriansj>well prefix as a common environmental variable name and PREFIX is close to prefix hence the error
<oriansj>just changing from PREFIX to prefix would make it shutup too
<oriansj>well we know 510 bytes is enough for FORTH and LISP (thanks to sectorForth and sectorLisp) so someone could probably do a variation of M0 in 510bytes
<oriansj>(say limited to %label and &label offsets only and %0x123 values)
<oriansj>but hex0 can be done in under 250bytes
<oriansj>and that can be done without any clever hacks
<oriansj>but there still remains the question are there any better languages that could fix in 250bytes?
<oriansj>^fix^fit^
<stikonas>if it's a better language more than that could be acceptable
<stikonas>we still need to write "hex1" and kaem in hex0 code
<drakonis>oriansj: sectorlisp?
<drakonis>oh wait
<drakonis>this one's actually 512 bytes
<oriansj>510 bytes
<oriansj> https://github.com/jart/sectorlisp
<oriansj>the last 2 bytes always have to be AA55
<stikonas>but the problem is lack of useful sectorlist programs
<stikonas>if somebody could implement e.g M0 in sectorlisp, maybe that would be useful
<oriansj>unless you made it so you are doing stosb byte es:[edi], al; push ebp
<oriansj>stikonas: well yes, alternate (ideally better roots) are encouraged for anyone who thinks they can do it better.
<oriansj>perhaps a dumb question but can sectorlisp or sectorforth self-host?
<oriansj>because I suspect they probably would not be able to
<oriansj>in which case, hex0 would be the most advanced self-hosting language that fits in 510 bytes
<drakonis>its written in C
<drakonis>oh it can be self hosted
<drakonis>provided you have another lisp implementation
<drakonis> https://github.com/jart/sectorlisp#overview
<drakonis>also sectorlisp.S
<oriansj>drakonis: self-hosting for an interpreted language would mean that it would able to run a program capable of building its own binary.
<drakonis>ah right
<drakonis>i guess not then?
<drakonis>well, check the readme
<drakonis>i don't know how to answer that
<oriansj>drakonis: depends on if a sufficiently powerful assembler can be written in its contraints
<drakonis>accurately, at least.
<drakonis>it is a lisp, it probably could, with sufficient effort.
<oriansj>looking at its implementation for car; mov(%di),%ax
<oriansj>it is limited to 16bit memory
<oriansj>you'll have to do everything as a string as it doesn't support numbers or arithmetic operations or binary operations
<oriansj>it also can't write to disk
<oriansj>only read from terminal and write to terminal
<drakonis> https://github.com/jart/sectorlisp/commit/28062acdc9b3cd7f6bda4a25bf6273b36b12b10f
<oriansj>so it is neat but not actually useful for bootstrapping
<oriansj>drakonis: look for the interrupts (that is how you write to disk leveraging the bios)
<drakonis>alright, although i'll admit that this is not a topic i should be getting into
<oriansj>only int 0x16 and int 0x10 is used
<drakonis>because it is outside of my domain of expertise
<drakonis>way out there
<drakonis>apologies.
<achaninja>The portable C implementation of sector lisp looks interesting
<achaninja>arguably if hex0 was not small, it would be a good initial seed
<achaninja>you could implement hex0 in sector lisp
<achaninja>but hex0 is smaller :D
<oriansj>achaninja: the problem with sector lisp is one can't create new binaries as it has no ability to do writes to disk
<oriansj>so one would need to hack in support for write (not easy given the number of free bytes)
<oriansj>and hack in support for a basic filesystem (possibly in the lisp code)
<stikonas>yeah, that's one of the problems of those small interpreters, they don't actually implement enough syscalls...
<oriansj>and figure out a way to encode arbitrary bytes without having support for integers, arithmetic or binary operations
<stikonas>that's why they are fairly small
<stikonas>quite a bit of code in hex0 is dealing with syscalls
<stikonas>but that's what let's us to do further bootstrapping
<oriansj>well if one hacked in support for an intersection return and was able to write arbitrary bytes into RAM, one might be able to create a function that could write to disk
<drakonis>you also have sectorforth
<oriansj> https://github.com/cesarblum/sectorforth.git
<drakonis> https://justine.lol/sectorlisp2/
<drakonis>its a different repository
<drakonis>or not really
<drakonis>hm
<oriansj>well sectorforth doesn't have an interrupt for writing to disk
<drakonis>okay same repository
<drakonis>quite impressive nonethelss
<drakonis>nonetheless
<drakonis>i found blinkenlights through it
<oriansj>sectorforth seems much more promising
<oriansj>it does have the ability to write arbitrary bytes into RAM
<oriansj>and it supports nand and add
<oriansj>no intersegment return -_-
<oriansj>and the data segment is different than the code segment, so no way to execute the bytes you write into memory either.
<oriansj>correction; the data segment is the same as the code segment
<oriansj>and the extra segment and the stack segment too
<oriansj>no dup but probably easy to hack in
<rickmasters>stikonas, orianjs: i just pushed a few small improvements to builder-hex0
<rickmasters>*oriansj
<rickmasters>it supports mkdir and access, and i fixed a bug that turned zero length writes into an infinite write
<rickmasters>anyway, mes-m2 runs hello world to completion
<rickmasters>its still under 3K, only 62 more bytes
<stikonas>oh, interesting
<rickmasters>i don't know how to setup mescc but i'd like to try when i have time, if only to discover what syscalls it attempts
<Hagfish>implementing mkdir in under 62 bytes? am i understanding that correctly?
<stikonas>rickmasters: you can try script from https://github.com/stikonas/boot2now/blob/mes/after.kaem
<rickmasters>stikonas: ok
<stikonas>though you'll have to rebase on top of your changes
<rickmasters>HagFish: it just creates a zero length file, and access just checks for an existing file, so pretty primitive
<Hagfish>interesting, thank you
<rickmasters>by supporting zero length writes, "src 0 x86/artifact" also works now, creating a fake directory, which M2-Mesoplanet looks for
<oriansj>well sectorforth doesn't support integers either (directly) so it'll be pretty hacky to get integer support in FORTH code given its primitive subset
<Hagfish>the best code is "no code", and the second best code is "just enough code" :)
<oriansj>and no dup means that one will be having to do a great deal of manual value entry
<oriansj>and its clear memory on error behavior means getting it to do anything meaningful will be an exercise in frustration
<rickmasters>BTW, i spent some time with sectorlisp and thought it was really cool, learned a lot about how lisp is implemented
<rickmasters>but sectorlisp can do very little practical stuff needed for compilers
<oriansj>no denying both sectorLisp and sectorForth are both awesome hacks
<oriansj>just getting them to do something as simple as writing an arbitrary byte into memory [and jumping to that byte later] (or onto disk) looks like an impossible dream
<rickmasters>i was thinking something internal could take a list of hex strings and write it back
<rickmasters>i spent an embarrassingly long time writing a list concat function to concat two lists, got it to work, but man it was difficult (for me)
<oriansj>rickmasters: no worries, not everyone is a good match for all languages
<oriansj>personally I spent too much time in assembly and C to flow in LISP anymore
<rickmasters>sorry, to be clear, i meant write a lisp list back to disk, not sure how you could wedge that into the interpreter (or after) though
<oriansj>it would be a very ugly hack if even possible
<oriansj> https://github.com/cesarblum/sectorforth/blob/master/examples/01-helloworld.f
<oriansj>you have to define integers
<oriansj>8bytes per 3->9 9bytes per 10->99 10bytes per 100->999
<oriansj>and you only have 4KB for everything.
<oriansj>but with creativity and a boatload of patience, it is in theory possible to use sectorForth create the bytes needed to do a disk write and execute that code
<muurkha>oriansj: yeah, I think the original BF was 256 bytes
<muurkha>wrt sectorlisp I don't want to be involved in anything that jart touches: https://en.wikipedia.org/wiki/Justine_Tunney#Political_writing
<oriansj>muurkha: you can get BF in under 100bytes if one is really willing to do very ugly hacks
<muurkha>her technical work is impressive but I don't want to get lynched for endorsing someone with her repellent political views
<muurkha>can you really?
<oriansj>muurkha: I can respect that view and those who choose to make those choices for themselves.
<oriansj>I care about the technical reality of the usefulness of people's contributions to the shared collective goal.
<muurkha>hopefully we live in a world where that is a viable strategy
<oriansj>My rejection of sectorLisp as a bootstrap language is purely on technical grounds, it simply doesn't work and it can't self-host
<oriansj>sectorForth on the other hand could be used but it takes about a half hour of perfect typing to get it to the point where you could start typing in the hex you want to write to disk
<muurkha>yeah, you need something that can edit really
<oriansj>(it takes 10 minutes of typing to do hello world)
<muurkha>speaking of technical reality I found something the other day that might be useful for hex0
<oriansj>muurkha: oh please do share
<muurkha>>>> [(ord(c) + 30) % 39 for c in '0123456789abcdef']
<muurkha>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
<muurkha>this is unfortunately very much in the vein of "a program that complies with the spec but no one knows how it works" though
<oriansj>I know how that works
<muurkha>it definitely isn't the most transparent (or robust!) way to convert a hex digit to binary
<muurkha>but there exist a number of machine languages in which it's the shortest
<oriansj>actually it isn't far off from index(c, "0123456789ABCDEF")
<muurkha>yeah, I'm not sure i386 is one of them
<oriansj>in x86 that would be add eax, 30; idiv 39 (if I am getting idiv correct)
<muurkha>you have to put the divisor in a register, don't you? or can you really use an immediate?
<muurkha>also you have to zero edx first
<oriansj>muurkha: you can use an immediate with x86
<muurkha>sweet. but it's a 32-bit immediate, isnt 'it?
<oriansj>but you might be partially right about edx; you would need to sign extend if with ctq (or something like that) if you are doing signed division
<muurkha>might be shorter to mov bl, 39 or mov cl, 39 if you have a pre-zeroed register handy. but more bug-prone
<oriansj>muurkha: F7FF is idiv 39
<muurkha>arguably it's not a good idea to include hex conversion code that silently converts uppercase A to 17 if the objective is high assurance against undetected backdoors. but it trades off against byte length
<muurkha>sweet
<muurkha>in Unicode U+F7FF is 
<muurkha>I found that in March when I was visiting my in-laws and just kept forgetting to share it
<muurkha>add al, 39 is also shorter
<muurkha>because the immediate is only one byte instead of 4
<oriansj>83c01e vs 041e; so you are right
<muurkha>it has to happen from time to time
<muurkha>I think there are no values for 30 and 39 that make this formula work for uppercase instead of lowercase
<oriansj>well it is a single bit flip to convert uppercase to lower case
<muurkha>right, or al, 32, isn't it?
<oriansj>to go from a-f to A-F one uses 0xDF
<muurkha>with AND, yes
<muurkha>i386's lack of an abjunction instruction like BIC is not an obstacle with a constant
<oriansj>so for or it would be or eax, 0x20
<oriansj>using al it is: 0c20
<midgardian[m]><oriansj> "the method for testing compile..." <- Thank you for the replies [oriansj](https://matrix.to/#/%40oriansj%3Alibera.chat), but I am curious what method you used to determine how long M2-Planet took to compile?
<oriansj>midgardian[m]: did you mean how long it takes to self-host?
<oriansj>(define (hexy a) (modulo (+ (logior a 32) 30) 39))
<oriansj>can be done in just 8bytes on x86
<muurkha>assuming edx and the non-al parts of eax are pre-zeroed
<oriansj>very true
<muurkha>I see things like that and I think that, for all the shitty bonehead moves in the 8086 design like its pseudo-segmentation, there are some fucking brilliant design features too
<muurkha>btw did you ever look at hugi-compo entries? if spending half an hour on this is your idea of fun you'd probably enjoy hugi-compo
<oriansj>muurkha: honestly never heard of that before
<muurkha>because Google is a vast wasteland of SEO spam
<oriansj>I am guessing: https://www.hugi.scene.org/compo/compoold.htm
<muurkha>yeah, https://www.hugi.scene.org/compo/ and that
<oriansj>god, I would love to see what they could do with the problem of producing the largest subset of M0 in under 510bytes
<midgardian[m]><oriansj> "midgardian: did you mean how..." <- No, sorry, in the message I replied to it is about you benchmarking how fast M2-Planet can compile 100Mloc file
<muurkha>what I think would be super cool would be an annotated disassembly of all the competition winners
<oriansj>midgardian[m]: time -v ./bin/test1000 -f testfile -o result is sufficient to get approximate runtime
<muurkha>I tried doing that for bootOS but I never finished it: https://dercuano.github.io/notes/bootos-notes.html#addtoc_4
<oriansj>muurkha: he does a very good job of covering it in his book (I bought both of his books and read them cover to cover)
<muurkha>in part because I couldn't get the ENTER command to work in QEMU, not sure what I was doing wrong
<muurkha>oh, awesome! I'd like to read his books too
<oriansj>well I can mail you copies if you would like
<muurkha>I... haven't had good luck receiving mail from overseas
<muurkha>my sister sent me a happy new year's card in December 2020
<muurkha>it arrived sometime around January 2022
<muurkha>in the page I linked above you can kind of see why I think octal is better than hex for i386
<muurkha>in octal 27x is "move immediate to 16-bit register x"
<muurkha>in hex that's b8, b9, ba, ... bf
<oriansj>I absolutely agree
<oriansj>I just used hex for x86 in stage0-posix because it was a quick hack
<muurkha>so you have to do the mapping that "be" is "move immediate to 16-bit register si" because si is 6 and e is 8 + 6
<muurkha>but you run into the problem that all the manuals and basically everyone who's familiar with i386 or 8086 code is familiar with hex
<muurkha>so I can tell you interrupt 20 (exit program) is CD20 but I have no idea what it is in octal
<muurkha>I think that in head-to-head comparisons of compiler output RISC-V with the Compressed instruction extension usually beats i386 for code density and I think even 8086
<oriansj>I just use this table: https://paste.debian.net/1241708/
<muurkha>heh
<oriansj>315 040
<muurkha>right, but it's going to take me quite a while before 315 040 is as familiar as CD 20 :)
<muurkha>(it's actually CD 21 that's the familiar one, but *that* level of mental math is within my capacities)
<oriansj>well int 040 isn't that much harder than int 0x20
<muurkha>I suspect that i386 has more opportunities for hand-tweaking because of things like the implicit operands of idiv or rep movsb and the shorter addressing modes for things like [ebp+4*esi+12]
<oriansj>muurkha: understatement of the decade
<oriansj>my x86 is the fewest instructions possible not the smallest binary possible
<muurkha>RISC-V C will generally use more instructions but (in compiler output) less bytes
<oriansj>literally just 29 DEFINES worth of opcodes (one of which is: DEFINE NULL 00000000)
<muurkha>oh, fewest distinct instructions, not shortest program counted in instructions?
<oriansj>muurkha: I was just learning x86 assembly and wanted to get done as quickly as possible as it was so much worse than knight assembly programming
<muurkha>I think probably x86 assembly is better than knight for really terrible programmers like me
<muurkha>not actually better
<muurkha>but more fun
<oriansj>I guess it depends on how one defines better
<muurkha>because (as hugi-compo demonstrates) trying to write things in fewer bytes is a never-ending challenge, and that's the kind of distraction that I'm prone to
<muurkha>I'd forgotten, there actually was a BF interpreter compo, and the winner got it down to 98 bytes: https://www.hugi.scene.org/compo/compoold.htm#compo6
<muurkha>back in the i386 and earlier days you could also try to write things in fewer *cycles*, which is the same kind of problem
<muurkha>super fun but in the larger scheme of things it's sort of wankery in the sense that the value system you need to win at it is the opposite of the value system you need to, for example, invent Python or the URL or C
<oriansj>muurkha: actually I get that appeal
<muurkha>yeah, but you also get stuff done instead of getting stuck in it :)
<oriansj>for various levels of done, I guess
<muurkha>but it's pretty great to be able to do things that other people think are impossible for anyone to do
<oriansj>muurkha: you certainly pull that off yourself
<muurkha>I do? people have definitely written Scheme compilers in Scheme before. compilers that compile all of Scheme in fact, not just part of it
<muurkha>I don't remember if I mentioned this here before, but it kind of has some of the hugi-compo energy despite being in JS: http://canonical.org/~kragen/sw/dev3/trama
<oriansj>muurkha: yes, you deliver code worth reading
<muurkha>thanks! but that's also hardly impossible for anyone to do, SICP is full of it
<muurkha>with the trama page I wanted to explore visual textures for a bilevel display that I could compute in a few instructions per pixel
<oriansj>muurkha: I know isn't impossible for anyone to do but most people don't
<muurkha>true, but most of the time it's not their objective; they're writing code because they want to run it rather than for people to read
<muurkha>I was thinking about the donut in https://www.a1k0n.net/2011/07/20/donut-math.html today
<oriansj>I have an M2-Planet version of that
<muurkha>oh really? I didn't see it!
<oriansj>just needed to add a usleep definition in M2libc but didn't get around to it
<muurkha>you could use a delay loop
<oriansj>probably, just didn't around to it
<muurkha>I thought the animation of the Lambert-shaded point cloud was visually pretty striking and it occurred to me that you could use ellipses for the points as a sort of hatching (https://en.wikipedia.org/wiki/Hatching)
<muurkha>it kind of reminded me of https://wavegrower.tumblr.com/
<muurkha>he uses ellipses that way but without the Lambert shading in a lot of his graphics
<muurkha>seems like a promising approach to non-photorealistic rendering
<oriansj>neat
<oriansj>here is my M2-Planet donut.c: https://paste.debian.net/1241711/
<oriansj>there is a divide by zero segfault and I haven't gotten the time to figure what is causing it
<oriansj>but it compiles just fine with: ./bin/M2-Mesoplanet --architecture amd64 -f /home/oriansj/donut.c -o donut
<oriansj>gcc donut.c -o donut && ./donut compiles it just fine
***roptat is now known as Guest8033
***Server sets mode: +cnt
<muurkha>neat
<doras>stikonas: it's slightly risky to resume the bootstrap like this. A build step should only be skipped if all build steps before it were skipped.
<doras>Otherwise we allow a scenario where a change in mes or in coreutils, for example, doesn't cause musl, guile or other later steps in the bootstrap to rebuild as well.
<doras>The current design in live-boostrap as far as I can tell is that all build steps that precede a given build step are considered inputs to that build step. There's no explicit dependency management. This is also why we can't do parallel builds, because we have no dependency tree.
<doras>I'm not sure yet how to achieve this functionality reliably, but it's worth thinking about.
<oriansj>doras: export step_03_complete=1 ; if sha256sum -c sha256ssums && step_03_complete
<doras>oriansj: it can work, but also requires numbering all build steps, or checking each preceding build step by name ("mes_build_complete"). It would mostly be annoying to maintain this.
<doras>I wonder if we could come up with something better than this..
<oriansj>well reverse dependency management ultimately is knowing what list of things need to be done first; So it is going to require a list regardless
<muurkha>the topological sort used as the first example algorithm in TAOCP volume 1 is something like 30 machine instructions long
<muurkha>but I guess you do need at least a list of build artifacts
<oriansj>now if we are consistent with naming, we could write a program which just reads a list file and spawns off copies of itself for its dependencies and if they all return true, the result is true else return false
<oriansj>if dependencies needs.list
<muurkha>manually sorting things into a total order is probably easier than listing the dependency of each build artifact
<oriansj>well yes: needs.list could just be the name of the previous step
<oriansj>and the sha256sum command for the previous step
<oriansj>that way the tool becomes general purpose enough such that if we later to decide to allow parallel builds, it becomes simple to convert to a planned build model like guix has.
<oriansj>or if we are willing to just make all our lives simpler longer term. Have it build the dependency tree itself from those list files let it figure out the build order
<muurkha>yeah, that's what I was saying about topological sort, and what you were saying about spawning off copies of itself
<rickmasters>oriansj: I finally got around to filing an Issue for hex1 and hex2 writing beyond declared ELF memory
<muurkha>an incremental build system is just a memoizer for your compiler and linker
<rickmasters>stikonas: could you clarify "we still need to write "hex1" and kaem in hex0 code" - aren't they written in hex0 already?
<oriansj>rickmasters: thanks
<oriansj>rickmasters: we were discussing the practical problems of using sectorLisp and sectorForth as languages for the bootstrap seed
<oriansj>and that was more about if use used them, we would have to figure out how to build those tool from them.
<rickmasters>oriansj: sure. as I said last night I explored sectorlisp for that purpose and it didn't look practical or better than the current plan
<oriansj>agree
<rickmasters>i didn't get far looking a Forth
<rickmasters>my reaction was an immediate, "yeah, uh ... nope"
<rickmasters>... the syntax makes badly written perl look good but i'm sure like anything it can sink in over time
<muurkha>haha yeah
<muurkha>I don't know, there are Forth programmers who say it does
<muurkha>but I think I've given up on that ever happening for me
<muurkha>after 27 years of writing very little Forth, but including https://github.com/kragen/stoneknifeforth
<oriansj>well FORTH as a language is fine if you can keep a stack of infinite depth in your brain
<muurkha>no, that's not the problem with Forth
<muurkha>well, it *can* be
<muurkha>but successful Forth programmers avoid doing that
<muurkha>and the stack in their brain is about the same depth as the stack you need for C
<oriansj>muurkha: 3 levels?
<muurkha>or 2 maybe
<oriansj>machine code, current code, high level plan
<muurkha>it's just that instead of writing if(NULL != head->prev) { head->prev->next = p; } p->prev = head->prev;
<muurkha>you write head @ prev @ if p @ head @ prev @ next ! then head @ prev @ p @ prev !
<muurkha>there's never more than two things on the Forth operrand stack there
<muurkha>but I just never got to the point where that was as easy for me to read as the pop infix version
<muurkha>and you can tweak it slightly, like maybe you actually want to factor out the if statement into its own subroutine, or keep head as a third item on the operand stack, or make the @ implicit in the struct access words, and maybe use VALUE instead of VARIABLE, so you can write something like p head prev next !
<muurkha>but it just never gets to the level of clarity I want
<oriansj>(if (!= NULL (-> prev head)) (:= (-> next (-> prev head)) p) (:= (-> prev p) (-> prev head)))
<muurkha>I think visually nested expression syntax makes the data flow more understandable
<muurkha>yeah, that works, though more traditionally in Lisps you make field accessors functions of their own
<muurkha>(if (head prev) (setf (next (head prev)) p)) (setf (prev p) (prev head))
<muurkha>my tentative conclusion is that Forth isn't really designed as a "programming language" in the sense of something like C, where the idea is that you sit down and write a program the way you would write an essay, and then someone else can read it
<oriansj>don't you mean (if (prev head) (setf (next (prev head))) (setf (prev p) (prev head)))
<muurkha>I do, thanks
<oriansj>well the reason I did -> is because it makes readibility easier
<oriansj>I could have also flipped the behavior so it would have been (-> head prev) and could have possibily allowed one to write (-> head prev next)
<muurkha>Forth, like bash, is designed as a sort of hybrid between an interactive command-line interface and a programming language. but it's a command-line interface for a ROM monitor or debugger, not for a filesystem
<muurkha>that is, the priority is on giving you access to the machine to try stuff out interactively
<muurkha>which is exactly the right priority if you're trying to get some experimental hardware to work on the Kitt Peak telescope
<muurkha>because you don't know enough about the hardware up front to know what you have to do
<muurkha>so in a sense Forth is more similar to GDB than it is to C
<muurkha>nowadays people often use Tcl for that kind of thing for pretty much the same reasons
<oriansj>and for good reason
<muurkha>yeah, Tcl has real drawbacks but it's a lot more readable than Forth
<muurkha>as I see it, Forth's strong point is that it gives you an interactive development environment with a debugger in 2K–16K, which is 64–1024 times less memory than you need for Tcl
<mihi>muurkha, oriansj: indeed, there is no a and b for (ORD(@"0123456789ABCDEF")+a)%b being their hex value, as b would need to be 7 and 7 is smaller than 16. However, if you allow 3 add/mod operations, you can do (ORD%32+9)%25, which as an added bonus works for both uppercase and lowercase.
<mihi>>>> [(ord(c) % 32 + 9) % 25 for c in '0123456789abcdef0123456789ABCDEF']
<mihi>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7, 8,
<mihi>9, 10, 11, 12, 13, 14, 15]
<mihi>oriansj, what do you mean that sectorForth does not support DUP? It is not a built-in word, but sp@ and @ are built-ins, and the first thing the hello world program does is define a DUP word... https://github.com/cesarblum/sectorforth/blob/master/examples/01-helloworld.f#L5
<muurkha>mihi: good point!
<muurkha>(also code written in Forth typically runs about 100× faster than code written in Tcl)
<achaninja>muurkha: I haven't seen many forth programs in practice.
<achaninja>For a lot of languages that say they are an order of magnitude better on whatever axis - there are surprisingly few things taking advantage of it to make money.
<muurkha>yeah, you probably won't find a lot of large programs written in bash or Tcl or gdbscript either, especially shrink-wrapped commercial products :)
<muurkha>or powershell either I think
<achaninja>I was also partially thinking about haskell
<achaninja>there are some interesting claims about the powers of forth around
<muurkha>yeah, and I think they're sort of misleading sometimes
<muurkha>but I think it really is true that you can be 100× more productive with Forth than with a C compiler if what you're doing is trying to figure out which registers you need to set in which order to get some piece of badly-documented hardware to work
<muurkha>and probably the biggest commercial shipment of Forth was Sun's open boot prom, which was later used in the OLPC
<muurkha>FedEx was also using Forth for their sign-here tablets their delivery drivers for quite a few years, but FedEx is just one user
<achaninja>haha, thats kind of cool
<muurkha>if what you need is to Write A Program then probably C is better than Forth, but if you want to interactively poke at device registers with a responsive scriptable environment, then Forth or Tcl is going to be a lot more productive than a C compiler
<muurkha>nowadays running Tcl or GDB on your laptop and connecting to your debug target over a serial cable is probably a better approach to that most of the time
<muurkha>and C compilers are a lot faster than they were in the 01970s or 01980s too
***Avichi is now known as Avici
<integfred>Check this out: https://github.com/publicsite/mavenCentralReconstruction_exampleOutput https://github.com/publicsite/MavenCentralReconstruction
<doras>fossy, stikonas: I submitted a PR to add a rootless bootstrap mode: https://github.com/fosslinux/live-bootstrap/pull/175
<doras>Unfortunately a recent change seems to have broke it, so it currently fails during `automake-1.11.2`'s build. I'll take a look at it later.
<doras>I'm running a qemu bootstrap in parallel to see if the failure is seen there as well.
<doras>stikonas: it seems to be related to "amhello": https://gitlab.gnome.org/-/snippets/3507/raw/main/automake-1.11.2-build.log
***j4m3s[m] is now known as tototo[m]
***ChanServ sets mode: +o janneke
<doras>Never mind, it doesn't seem to reproduce.
<doras>At all, not even in the bwrap mode.
<doras>Very odd.
<doras>The next thing I'm missing from live-bootstrap is an install mode. It's fun and all to be thrown into an interactive shell after a bootstrap success (or failure), but it's not practical for projects that want to consume live-bootstrap's final output.
<doras>Maybe an "--interactive" command line option needs to be added, so the "install mode" is the default behavior.
<stikonas[m]>doras: well, I was only adding that binary preseed thing for development purposes
<stikonas[m]>doras: but wouldn't change in mes or coreutils change checksums?
<stikonas[m]>--repo is not meant to be used together with --update-checksums
<doras>stikonas: I guess I'm imagining a future where one can make arbitrary changes, rebuild, and then only the affected steps are executed.
<doras>But it's probably going to be a long time before we can accomplish this.
<doras>stikonas: how would you approach the "install mode" for live-bootstrap? Would it be reasonable to introduce a command argument to determine if one should be dropped to an interactive shell upon bootstrap success/failure? This decision would be added to a new variable in bootstrap.cfg so we can avoid running bash interactively, and exit completely when the bootstrap ends or fails.
<doras>I actually don't need the entire "install mode" that one would imagine. No need to install the final output to a DESTDIR or similar. This is mostly because I'd expect this logic to be in rootfs.py, and in the BuildStream case I wouldn't have Python anyway.
<doras>I just need the bootstrap portion of it.
<doras>Maybe an "--exit-after-bootstrap" argument would make sense.
<doras>I'm open to ideas and name suggestions.
<Hagfish>having arguments to determine that behaviour does seem sensible, so i think you're right to suggest them
<Hagfish>i guess "exit" makes it clear, but in terms of naming, you might prefer a mention of "shell" (if that's what it affects) rather than "bootstrap" (which is just one possible use case?)
<stikonas[m]>doras: what should be the output of install mode?
<stikonas[m]>I've added after.sh hook that can be replaced
<stikonas[m]>But that's all inside chroor
<stikonas[m]> https://github.com/fosslinux/live-bootstrap/blob/master/sysc/after.sh
<stikonas>but yes, some option to rootfs.py would make sense...
<stikonas>but I don't know what's the best way to exit would be
<stikonas>1. keep installation disk image in qemu mode? But that wouldn't work in chroot mode
<stikonas>where maybe we would like to get something outside chroot?
<doras>It's probably only relevant to chroot and bwrap modes.
<stikonas>strange, my preseed PR seems to have changed quite a few package hashes
<doras>Shouldn't it keep hashes exactly the same by definition?
<stikonas>doras: I mean running "rootfs.py -c" changes hashes
<stikonas>not the packing/unpacking step
<stikonas>just running it normally
<stikonas>there are some changes to helpers.sh, but I've no idea why they cause changes in binaries
<doras>Yes, it shouldn't change those. If it does, it's probably a bug.
<stikonas>e.g. diffoscope shows https://paste.debian.net/1241789/
<stikonas>though not very comprehensible without disassembly
<doras>Also, regarding the install mode, I think the qemu mode already has a proper output in the form of a rootfs image. This is similar to the sysc rootfs directory that we end up with in the chroot and bwrap modes.
<doras>So the output is already in some form or another. The only "pressing" issue in the sense of an install mode is that we assume a human is running the bootstrap and monitoring the terminal, or expecting an interactive shell.
<stikonas>that's why I split interactive bash into after.sh file
<stikonas>so that it's easily replaceable
<stikonas>I can't understand much from sed disassembly https://paste.debian.net/1241790/ ...
<doras>I think we also have a "trap" and some other bash invocations that end up interactive.
<doras>But anyway, a CI pipeline doesn't want to end up in an interactive (i.e., blocking) shell in any form or scenario. I want to allow this use case.
<stikonas>oh that trap is only invoked on failures
<stikonas>we shouldn't commit failing commits
<stikonas>but yes, we can disable trap depending on bootstrap.cfg
<stikonas>doras: also not that traps are completely broken in sysa
<stikonas>that first bash has quite a few things broken
<stikonas>not sure if it's meslibc or compiler defines
<doras>Regarding the hash changes, I'd try to start commenting out functionality until you get the same hash. The binary changes are very odd.
<stikonas>yeah, I might try that
<doras>meslibc seems to have quite a few stubs. I wouldn't be surprised if things break.
<doras>stikonas @stikonas:libera.chat: I'd appreciate it if you could try to run my bwrap-based bootstrap when you get a chance: https://github.com/fosslinux/live-bootstrap/pull/175
<stikonas>yeah, I can try to at least kick it off
<stikonas>though I might not have time to review it all for 2 weeks or so
<stikonas>but there is also fossy to do reviews
<doras>I really wonder if it works for anyone else. It should be more reproducible than the chroot mode in theory because its environment is more confined.
<stikonas>doras: bwrap: --cap-add in setuid mode can be used only by root
<stikonas>what do I need to configure to avoid that?
<stikonas>oh, and I need to manually delete tmp/ and sysc/tmp to re-run it
<doras>You don't have user namespaces enabled in your kernel?
<doras>Well, unprivileged namespaces, more accurately.
<stikonas>I can check...
<stikonas>it does work with sudo
<stikonas>need to find it first in menuconfig
<stikonas>I think I have namespaces
<doras>I wonder why it needs the real root user though in setuid mode though. The setuid itself should give it all of root's permissions to set up the sandbox and the drop the privileges.
<stikonas>I have both CONFIG_NAMESPACES=y and individual ones like CONFIG_USER_NS=y
<doras>stikonas @stikonas:libera.chat: which distro are you using?
<stikonas>Gentoo
<stikonas>probably needs some extra configu
<stikonas>but it would be good to understand that and document it
<stikonas>and bubblewrap is build with USE=suid
<stikonas>so I think that should set setuid
<doras>I think I missed the tmp deletion part. Thanks for pointing it out.
<doras>I'm used to running with a script that manually deletes those, so I haven't noticed.
<doras>stikonas @stikonas:libera.chat: if you run the following, does it work?
<doras>unshare --user --map-root-user bash
<doras>You should "become root" without a need for sudo or a password.
<stikonas>doras: it works if I build bubblewrap without suid
<doras>Great!
<stikonas>can you document that too?
<doras>Which part of it?
<stikonas>hmm, so does your distro have bwrap as setuid?
<doras>I did mention in my commit that bubblewrap can operate in rootless mode if unprivileged namespaces are allowed.
<stikonas>unshare doesn't help
<stikonas>yes, but I think you also need no setuid on bwrap binary
<stikonas>with unshare I got bwrap: setting up uid map: Permission denied
<doras>The unshare part was just to test if the kernel supports unprivileged namespaces.
<doras>Most distros have it enabled these days. I use Fedora and Arch.
<doras>But also Ubuntu and Debian have it enabled by default.
<stikonas>oh ok
<doras>All of their derivatives also have it enabled by default, including Mint.
<stikonas>strange then...
<stikonas>anyway, I'll try without suid and see if it runs to completion
<stikonas>so no idea why I got that --cap-add in setuid mode can be used only by root
<doras>Sure.
<stikonas>might be https://bugs.gentoo.org/669210
<stikonas>well, at least some explanation
<doras>I has to use that option because we chroot between sysa and sysc as part of the bootstrap.
<doras>had*
<doras>Otherwise it's not needed
<doras>People were afraid of unprivileged namespaces until a few years ago. I think everyone realized it was quite safe around 2020, and distros that came after that had it enabled by default.
<stikonas>yeah, so it sounds like bwrap can either use user namespaces or suid but for some reason for me only the former works
<stikonas>well, I'll see if sysa->sysc transition works
<doras>Which error did you actually see in the suid mode?
<stikonas>bwrap: --cap-add in setuid mode can be used only by root
<stikonas>so coming from https://github.com/containers/bubblewrap/blob/main/bubblewrap.c#L2598
<doras>Interesting indeed. One would expect suid to allow all of root's capabilities. Strange that it doesn't.
<qyliss>the last unprivileged user namespace privilege escalation i heard about was from this year... https://seclists.org/oss-sec/2022/q1/55
<doras>qyliss: any additional functionality has an attack surface, obviously.
<doras>However, it's not a "security nightmare" as people initially thought. Or not today, anyway.
<doras>stikonas @stikonas:libera.chat: we could easily remove this capability though if we re-introduce the mode in which we go back to rootfs.py after sysa's bootstrap completes and then initiate sysc's bootstrap with an additional command.
<stikonas>hmm, maybe. Not sure what's better
<stikonas>though if we do that, only for bwrap mode
<stikonas>chroot mode should keep chrooting
<doras>The bwrap mode doesn't require entering passwords or similar upon execution, unlike the chroot mode, so the user experience wouldn't be negatively impacted.
<doras>stikonas @stikonas:libera.chat: I think we can easily avoid both the chroot to sysc and password prompts in the chroot mode, simply by running the two bootstrap executions in the context of a single elevated shell command.
<stikonas>that's true