IRC channel logs

2021-03-13.log

back to list of logs

<OriansJ>but perhaps that is just my hope to eliminate perl and autotools from the bootstrap path because none of that belongs in an environment where you know what you have and will get anything you need ahead of time.
<stikonas>well, perhaps somebody can just write makefiles for binutils and gcc...
<stikonas>although, at some point you probably still want to bootstrap perl...
<stikonas>it's still a reasonably widely used language
<OriansJ>stikonas: I figure dealing with it with GCC and Guix would be less work for us long term.
<stikonas>maybe a bit less...
<stikonas>although, I found perl work easier than bootstrapping bison...
<stikonas>(I guess because bison needed musl and libc's are trickier)
<OriansJ>or because Bison Devs assumed glibc and were lazier
<OriansJ>or IBM paid some devs to get Perl working on AIX with their wackjob libc which had knock on effects
<OriansJ>or maybe janneke never looked at building bison and skipeed the handful of functions needed in meslibc to solve it without having to do musl
<OriansJ>As creating a minimal libc is trivial
<OriansJ>but a full C89 libc is a shitload of functions
<stikonas>well, yeah, I understand the reasoning for mes libc...
<OriansJ>not sure if English or Metric shitload but it is enough to choke a llama
<stikonas>it's mostly metric system here in England.... Appart from miles...
<stikonas>fossy: this is now with green tick https://github.com/fosslinux/live-bootstrap/pull/61
<OriansJ>stikonas: sometimes referred as imperial units
<stikonas>fossy: just minor changes to address your review comments
<stikonas>I guess in US they use imperial units more...
<OriansJ>stikonas: yes with our 2liter Colas and 1KG heart-attack burgers; nothing but Imperial units
<OriansJ>it kinda depends on the industry but more often than not newer things use metric units but older things use imperial units because there are costs in changing things.
<fossy>well janneke has expressed dislike for musl so I dont think mes libc is really meant to be used as a precursor to musl
<fossy>it just happens to work well for that
<OriansJ>fossy: You are also really good at glueing together working pieces from our ad-hoc mess of bootstrap pieces.
<fossy>^-^ stikonas is very good at that too
<OriansJ>fossy: I prefer to say that stikonas is very good chasing annoying problems down.
<stikonas>well, the only reason why I chose musl was because it was the only complete enough libc easily buildable with tcc
<fossy>well there are few options that exist
<stikonas>and our goal was to remove as much pre-generated stuff as possible... So bison was necessity
<fossy>musl, uclibc, glibc, thats all the complete ish libc I can think of
<stikonas>yeah, and uclibc was not looking as easy as musl
<fossy>stikonas: only one small readme change I see for libtool then I will merge
<fossy>no
<fossy>Wasnt uclibc a glibc fork?
<fossy>oh
<fossy>no
<fossy>but incorporated code from glibc
<stikonas>fossy: ok, pushed
<fossy>Merged
<fossy>Neat I like autoclose issues
<stikonas>hmm, I'm now looking at coreutils and getdate.c
<fossy>Issue?
<stikonas>it also has getdate.y->getdate.c
<fossy>oh
<stikonas>I think the simplest way that would work
<stikonas>is to not build it and remove touch that uses it
<fossy>I swear we removed it
<fossy>or was that tar
<stikonas>(we'll later have touch back when we rebuild coreutils)
<stikonas>yeah, tar
<stikonas>so far I only found it in coreutils, nothing in between
<fossy>we use touch tho
<stikonas>catm will work...
<stikonas>and it's only in one place...
<fossy>we can patch touch probably trivally
<fossy>yeah ok
<stikonas>either patch touch
<stikonas>or replace touch with catm until yacc is ready
<stikonas>yet another option
<stikonas>is to first build heirloom tools
<fossy>1 second
<stikonas>althouth there we use true to "ignore" ranlib command
<stikonas>we can patch it out from the makefile though
<stikonas>so quite a few different options
<fossy>we dont even use touch that much tbh
<stikonas>exactly
<fossy>we can probably just not build it
<stikonas>so I only found it in bash
<stikonas>yeah, then use catm in bash
<fossy>I just checked, its like 4-5 times?
<stikonas>and after that we very quickly rebuild coreutils
<fossy>yes
<stikonas>yaeh...
<fossy>its only 10 parts away ish
<fossy>lets do that then
<stikonas>I can't remove getdate.c though...
<stikonas>we don't have rm
<stikonas>but shouldn't matter
<stikonas>we didn't remove it in tar case either
<stikonas>just don't compile it
<fossy>yeah I was gonna say
<fossy>we dont need to remove it
<fossy>just take it out of lib_src
<stikonas>at that stage we use handwritten makefiles, so full control over build process
<stikonas>yeah...
<stikonas>ok, testing now, probably won't finish today, cause some hashes might change
<fossy>yeah...
<fossy>take as long as you need
<stikonas>and some improvements from live-bootstrap are going to guix https://issues.guix.gnu.org/45962 ...
<stikonas>guix will hopefully have deterministic ar
<Hagfish>fantastic
<pabs3>OriansJ: agreed re libre hardware. but libresilicon is not enough to establish trust in hardware. have you seen bunnie's posts/talks on hardware supply chain security for eg?
<OriansJ>pabs3: yes I have
<pabs3>ah good. I especially like the RISC-V impedance mismatch one
<OriansJ>I quite enjoyed the optically verifiable circuits in the display; which logically extended to enable an optically verifiable CPU core
<OriansJ>^logically extended^could logically be extended^
<OriansJ>wouldn't allow a very powerful or complex CPU core but certainly a 32/64bit RISC-II or MIPS-I style RISC processor for bootstrapping and cross verification.
<OriansJ>RAM chips however are harder to address as we would need atleast 1GB to do a full and proper bootstrap to Linux+GCC+Guix
<stikonas>maybe can do with less than 1 GB...
<stikonas>hmm...
<stikonas>gcc itself though might need a lot
<OriansJ>stikonas: maybe but Optical Circuits would be 16KB per Chip or 262,144 chips
<stikonas>yeah, that's too many
<stikonas>even if you reduce RAM usage by some small factor
<OriansJ>even if we produced them at $1/per chip it still would cost a shitload of money
<stikonas>it wouldn't go down by orders of magnitude
<OriansJ>If we cut it down to 64MB, we are still looking at 4,096 chips
<OriansJ>which could be econommically possible to work with but not an easy thing to ask of a large population.
<stikonas>gcc source is more than 64 MB already
<stikonas>well, gcc-core is a bit smaller
<OriansJ>hence, it is the bigger half of the problem
<stikonas>well, at some point you have to switch to storage media...
<OriansJ>as one can do a 64bit processor in 20K transistors
<stikonas>store all the source code on punch cards :D
<OriansJ>but storage media these days include CPUs, ROM and RAM
<OriansJ>stikonas: here I was planning on a mile of paper-tape
<OriansJ> https://www.hermieusupply.com/en-EN/flight-strip.html
<OriansJ>As taper tape is no longer being manufactored, the closest equivelent
<OriansJ>as punch cards would be even more expensive to do (let alone distribute)
<OriansJ>Or we can engineer so that either trusted or untrusted parts can be used where solid standards exist and make it so detection of untrusted hardware changing builds can be trivially detected.
<OriansJ>Because I don't care if untrusted hardware is used to build binaries if the resulting binaries are identically to the binaries build by the trusted hardware (just slower and on more expensive hardware)
<vagrantc>and that's where reproducible builds comes in :)
<OriansJ>We also have the advantage of time; because Bootstrap pieces are small, quick to build and slow to change.
<OriansJ>vagrantc: indeed; univeral cross-reproducibility.
<fossy>yeah exactly
<OriansJ>M2-Planet+M2libc+Mescc-tools produce identical outputs regards of host operating system (or bare metal), hardware, voltages, etc
<OriansJ>There is no escaping the universal cross-reproducible builds when it even works on arbitrary hardware that hasn't even been invented yet.
<OriansJ>If one needs something with more trust like Bunny's project; then you need only trusted hardware and Bunny is doing an amazing job trying to provide that.
<OriansJ>but in terms of bootstrapping, we can leverage untrusted hardware and still not be concerned because we have a larger population of people doing the exact same calculations on arbitrary machines (any of which would detect malicious hardware tampering with our builds)
<OriansJ>(or buggy hardware which is probably much more common)
<OriansJ>and lets be honest; knight is alien enough and I'll take great pleasure getting it built into an optically verifiable chip after we get all of the software pieces is order.
<OriansJ>But we have *A LOT* of porting efforts to do.
<lle-bout>OriansJ: hey, I have a question, to bootstrap from baremetal, how can an input device like a keyboard be managed from there?
*pabs3 would suggest that input should be ignored until the final build products have been booted
<fossy>how do you mean, pabs3
<lle-bout>pabs3: but we need input to actually input the GNU Mes stuff into the baremetal env, don't we?
<pabs3>I would have assumed that the whole bootstrap would be automatic after it is started
<pabs3>so no keyboard input would be needed
<lle-bout>pabs3: well if you use GNU Mes from baremetal, you need to input something to the computer, the GNU Mes stuff can't get on there by itself
<lle-bout>You need some kind of input device to transfer GNU Mes on there
<lle-bout>If there's no kernel, no device drivers, how do you do
<lle-bout>No software for keyboards, serial ports, or others, since you explicitly want to bootstrap that
<lle-bout>There has to be some kind of hardware manipulation there to get the GNU Mes stuff on the computer
<pabs3>I thought mes was one of the later layers? but indeed the machine code seed of the bootstrap plus the source needs to be written to the device bootup storage somehow
<lle-bout>pabs3: sorry I tend to consider GNU Mes as the whole thing
<lle-bout>How do we write it to such storage?
<pabs3>I was considering that all the source would be input at the same time as the seed
<lle-bout>pabs3: yes that's probably the best thing to do, how do we do it you think?
<pabs3>and a keyboard certainly isn't useful for inputting the entirety of the gcc source :)
<pabs3>OTOH both storage and most keyboards are full of untrusted binary blobs too though, so hmmm
<lle-bout>yes probably
<pabs3>there are definitely libre keyboard firmware projects, but no libre storage firmware projects that I know of
<lle-bout>My main concern is the device driver
<lle-bout>I believe one must check with reproducibility all the binaries later on
<lle-bout>I don't believe it's possible to design malware that can infect current and future compilers in a reliable way, so that they are both themselves infected and also generate infected compilers
<lle-bout>(reproducibly)
<fossy>unfortunately a bare metal bootstrap needs an appropriate platform
<fossy>which dosent exist rn
<lle-bout>fossy: what would be the requirements?
<fossy>lle-bout: fully open design, specification and implementation; free input mechanic that can have verifiable input; running binaries directly on the CPU (no BIOS), (optional) ease of running programs without any kernel
<fossy>also a free storage system
<OriansJ>lle-bout: well you are not supposed to manage hardware when bootstrapping on bare-metal, you are supposed to leverage specified functionality to produce required results. So only the pieces such has the hex0-monitor need to concern themselves with how to deal with input from a user and the associated devices. As for Mes, it needs a kernel to work. Currently only M2-Planet+M2libc+mescc-tools (and below) can run on bare metal. (Currently
<OriansJ>specified to have one input and one output device [such as a paper tape reader and a paper tape writer])
<OriansJ>So the plan is to get a minimal POSIX kernel written in the M2-Planet C subset, then the steps leading to GCC+Linux+Guix can be done without modification.
<OriansJ>But such work is always 100% hardware specific and requires customization at the read/write functions to deal with the hardware used. (Which is why I made such work as easy as possible, rather than trying to provide a much more complex general solution, to reduce the sizes involved and the audit work required.)
<OriansJ>fossy: there are plenty of appropriate bootstrap platforms in existence already (PDP-11s, Cray-1s, etc)
<OriansJ>They are just part of the porting work required
<OriansJ>I've been leaving the more fun architectures to people who might want to do the work
<OriansJ>But if the time comes and no one else wants to do that work, I'll add it to my queue and knock it out too.
<OriansJ>As I have always said this isn't a boom done project but something we are going to have to keep working for decades as the world changes below our feet
<OriansJ>We might have short term goals and targets that we hit to keep spirits high and keep visable progress. But change is the only constant, pieces that can't keep up with it will be replaced by new ones that do.
<OriansJ>We have already what appeared impossible into a solvable problem and with the continued efforts it'll become an easier and easier problem. But we need to keep up with the world changing around us. But this is a Red Queen's race, where the better job we do the easier it will be for people to create new Architectures and get FLOSS software running on it.
<stikonas>fossy: so I think I've solved that getdate.y issue https://github.com/fosslinux/live-bootstrap/pull/62
<stikonas>had to patch it away from touch
<stikonas>since I couldn't build touch 5 with bison and touch 6 with musl
<stikonas>so for now we'll stay with touch with mes
<OriansJ>stikonas: why would touch need bison? It just sets timestamps on files/folders
<stikonas>OriansJ: well, it depends on get_date function for some reason...
<stikonas>anyway, that's a question for coreutils maintainers (at that time at least)
<stikonas>OriansJ: https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/touch.c?h=v5.0#n309
<stikonas>basically it uses bison to parse date format
<stikonas> -d, --date=STRING
<stikonas> parse STRING and use it instead of current time
<stikonas>maybe bison is a bit overkill for that...
<stikonas>but it is what it is
<stikonas>anyway, I basically "broke" that -d argument in my patch
<OriansJ>stikonas: yeah definitely overkill but probably a quick and dirty hack that stuck around because no one else wanted to do it in just plain C.
<mihi>Hello :-) https://github.com/schierlm/guile-psyntax-bootstrapping works now \o/
<mihi>and (not surprisingly) when building on 64-bit Linux, the bootstrapped psyntax-pp.scm is identical to the shipped one.
<mihi>The CI will also download guile-3.0.2, remove psyntax-pp.scm, bootstrap it and compare the hash with original.
<stikonas>nice work!
<mihi>rekado_, in case you are interested, #'(... foo) will create a syntax object that wraps a symbol foo, even if foo is an ellipsis or a bound syntax variable. Those (... ...) are used to generate an ellipsis in a generated syntax-case statement :)
<mihi>and I did not know that, so my implementation of 'syntax did not do it. That's why it was not working for me.
<OriansJ>mihi: wonderful, thank you *VERY MUCH*
<mihi>The fix for 'syntax was rather simple. https://github.com/schierlm/guile-psyntax-bootstrapping/blob/main/psyntax-bootstrap/step3.scm#L42-L43. The hardeds part was finding out in which order and when to swap the $sc-dispatch and macroexpand-hook from bootstrap impl to real so that the system does not fall apart.
<mihi>the psyntax.pp can be used for 3.0.2. to 3.0.5, but not latest git as they decided to change psyntax implementation again after a while.
<mihi>when 3.0.6 gets out we'll have to check if it is easier to bootstrap 3.0.6 from 3.0.5 or change the current bootstrap to support it.
<mihi>OriansJ, thank you very much for M2-Planet and mescc :)
<mihi>s/mescc/M2Libc/
<OriansJ>mihi: : mescc-tools but but I like your work *alot*
<mihi>OriansJ, I assume you appreciate it so much because you tried yourself hard to accomplish it. If you did not try, you don't know how hard it can be...
<mihi>but the satisfaction once it did not fail but print #<unspecified> into the REPL was worth it. :)
<fossy><OriansJ> fossy: there are plenty of appropriate bootstrap platforms in existence already (PDP-11s, Cray-1s, etc)
<fossy>how many pdp11s still exist
<OriansJ>mihi: actually I appreciate it so much because of what it represents. A full and proper bootstrap without generated files for Guix; a task which was considered an absolute impossibility not even 5 years ago. And you got the last piece of the chain done with live-bootstrap doing the pieces leading up to it.
<OriansJ>But yes, I am quite certain it was a difficult task and its completition is definitely unquestionable proof of your brilliant work.
<OriansJ>you really should announce it on #Guix because it is going to be a milestone for them too.
<stikonas>and I guess also on mailing lists: bootstrappable and guix-devel
<OriansJ>fossy: couple thousands in industrial settings, hundreds in Museums and tens in personal collections. But SIMH can be used for the development work and we only need a single working one to do the verification work.
<mihi>stikonas, fossy, OriansJ: did you check and psyntax.pp is really the last missing piece for guile? I'm not that familiar with it, and I would not be surprised if there are other generated files left...
<mihi>yet if there are more, just let's tackle the other ones too :)
<stikonas>I certainly didn't check...
<OriansJ>mihi: well last time I checked, it was Bison pregens but we bootstrapped Bison so those shouldn't be a problem but you are right; verifying isn't a bad idea.
<mihi>do you have any heuristics for finding generated files? for example some script that can parse Makefiles and find files that are both outputs and bundled in source control?
<mihi>or by grepping for keywords? or is the only way to actually review every single file?
<mihi>(or word-of-mouth here in the mailing list?)
<stikonas>hm, I sometimes grep for "generated"
<mihi>s/here/here or/
<stikonas>or something similar
<mihi>would not work for psyntax.pp :(
<stikonas>well, certainly, it only relies on people being nice and leaving a comment that file is generated...
<OriansJ>mihi: well behaved generated files try to explicitly state so but unfortunately not all programmers are so kind and thus only a manual inspection with the question can I understand this is usually the only honest method.
<stikonas>well, guile also has some lex files, but those are also bootstrapped together with bison
<OriansJ>we also have M4 and autotools are not that far off (If I remember correctly)
<stikonas>yeah, need to bootstrap newer autotools, but shouldn't be too bad
<stikonas>the harderst part I guess were perl modules that we already have
<stikonas>now just a bit of circular dependencies on automake/autoconf but those are not too hard to break with minor patching
<stikonas>(if you are unwilling to use pre-generated configure scripts)
<stikonas>I'll probably try to get newer automake working next...
<OriansJ>So mihi I am sure that all that remains can be sorted out in live-bootstrap and you took out the biggest blocker we had in guile and what remains (if anything) should be addressable.
<OriansJ>fossy: also ebay https://www.ebay.com/itm/Vintage-DIGITAL-DEC-PDP-1123-11-23-Mainframe-Computer-w-Cards-Powers-On/203250339708?epid=1820501256&hash=item2f52aa0f7c:g:tL4AAOSwgIZgA34z
<fossy>mihi: i usually do the same thing, grep for generated, and also look for any *.y *.tab.c *.lex etc etc
<fossy>OriansJ: hm, i didn't realise they were that common
<fossy>maybe i was thinking of pdp-8s that were virtually nonexistant
<OriansJ>fossy: PDP-10 and PDP-6 are the most rare only 12 and 2 made respectively.
<fossy>stikonas: are all of these pass2 checksums ones that were just missing?
<stikonas>fossy: I suspect it was some PR just before you merged checksums
<stikonas>so somehow they were missing
<fossy>stikonas: hm, sounds likely
<stikonas>so building touch is quite annoying...
<fossy>is fprintftime just another thing needed for date
<stikonas>in the end I decided to patch it...
<stikonas>yes..
<fossy>yeah thats fine
<fossy>ok
<fossy>CI then merge
<stikonas>oh, this CI run is just changed commit message...
<stikonas>I was a bit unhappy...
<stikonas>no changes since last pass
<stikonas>(there is green tick visible there)
<stikonas>if you want, we can still wait...
<stikonas>and I'm now testing "expect" changes locally...
<stikonas>to see if I can enable interactive bash
<fossy>oh i see it
<stikonas> https://cirrus-ci.com/task/5340936117944320
<fossy>yeah
<fossy>merge
<stikonas>so I hope now we got rid of all pregenerated stuff...
<stikonas>both libtool and this getdate.y are cleaned up
<stikonas>somehow they slipped through initially
<fossy>i think we will find a few slips here and there
<fossy>these things are hard to find sometimes
<fossy>i think i have found a very good way to check
<fossy>normally git checkouts *don't* have pregenerated files
<fossy>so compare a directory listing
<OriansJ>the great thing about perfection is that it is a road and not a destination.
<vagrantc>if you're lucky, some nice hikes off the beaten trail :)
<OriansJ>vagrantc: very clearly we are all entirely off the path and cutting into new jungle.
<mihi>Mailing list post: https://www.freelists.org/post/bootstrappable/Can-Guile-be-bootstrapped-from-source-without-psyntaxppscm,7
<mihi>I hope nobody is unhappy that I hijacked an old guile-user thread from 2017...
<OriansJ>with news this big I don't think they would mind.
<janneke>mihi: oh, beautiful
<janneke>this means we can probably use this bootstrapped syntax-case in mes? could even be faster than our current psyntax?
<mihi>Apart from noticing the existence of psyntax.pp in mes, I have not had a closer look at it (yet).
<mihi>or in other words if there are any Guile primitives that are not available in mes. But I guess if they are, the code can be changed to work around it. It should mostly work if you have just defmacros as well as make-syntax/syntax-expression/syntax-wrap/syntax-module
<janneke>ah okay, that means it's goint to be some work
<janneke>wondering what syntax-module might be
<mihi>I don't use it. Guile uses it to distinguish which module contained the syntax object, so that #'x on toplevel in module X does not clash with #'x on toplevel in module Y.
<stikonas>does psyntax have a lot of impact on speed of mes?
<janneke>i believe it does, however, there's no alternative to compare it to
<mihi>I just set it to '(bootstrapping bootstrapping) so that it is different from a real module, in case any syntax objects escape from the bootstrap.
<janneke>right
<janneke>i'll want to take a look after i get guile modules up in mes