<civodul>on armhf emacs is built but icecat is not <ozzloy>i'm back! thanks lf<tab> for being willing to help <davexunit>ozzloy: something is terribly wrong with your installation <rain1>how do you reply to a thread on the mailing list? I tried to reply to the compilers thread but it did not come through <rain1>I am subscribed because I had the message I wanted to reply to in my mailbox <ozzloy>davexunit, ok, i'll start over with the source code version <ozzloy>davexunit, would it be bad if i already have guile installed via apt? <rain1>I tried to subscribe and it says I already am <rain1>I just hit reply and change the address to guix-devel@gnu.org <rain1>maybe it needed moderator approval or something <mark_weaver>rain1: if you're subscribed, your messages should go through automatically <rain1>I can't work out what I did wrong oh wel <mark_weaver>it's possible that the GNU mail servers are being slow right now. <mark_weaver>they are sometimes hit with apparent denial of service attacks ***Basstard1 is now known as Basstard`
<rekado>I wonder if we can build a version of GHC with hugs, the haskell interpreter. <rekado>seems that for GHC the last version that could be built without a Haskell compiler is 4.08.2. <rekado>this can be used to build 6.8.2 (6.8.3 requires 6.4 or later) <rekado>with that we can build the last of the 6.x series (6.12.3) <rekado>this can be used to build 7.4.2 (7.6.x requires 7.0 or later) <rekado>this can be used to build 7.10.1. <rekado>and then we can finally build the latest 7.10.2 (which requires 7.6 or later) <rekado>7 steps to get to a cleanly bootstrapped GHC <efraim>how long is the build process? and what are the other inputs? <rekado>"Source releases of GHC 4.xx can be compiled up using GHC 2.10 or later." <rekado>seems like we need a couple of steps before 4.08.2 <efraim>well, at least the chain's there I guess <efraim>hopefully it doesn't need old libraries <rekado>well, I cannot seem to download the tarball for 2.10. <rekado>it would be very helpful if we could use hugs. <rekado>looks like it can be built without GHC. <rekado>the last release of hugs was in 2006 and supports more Haskell than GHC 3.02 <efraim>how does hugs get built? straight from C? <rekado>we have it in the module hugs.scm <rekado>our recipe for hugs isn't quite correct, though. <rekado>we should add yacc and a couple more things to avoid using pre-generated files. <rekado>hmm, I guess using Hugs as part of the GHC build process isn't so easy. <rekado>it's probably better to successively build GHC 3.02 -> 4.08.2 -> 6.0.1 -> 6.8.2 -> 6.12.3 -> 7.4.2 -> 7.10.1 -> 7.10.2 <rekado>hoping that GHC 3.0.2 can still be built. <rekado>I'll try to build 3.02 and report back <civodul>ooh, great that you found a build path for GHC! <rekado>with 3.02 the difficulty seems to be to figure out in which order things need to be built. <rekado>it's not just "make" in the root dir. <rekado>need to enter every subdir and install each of the tools first <efraim>how do I feed multiple substitutes to --substitute-urls? <rekado>oh, 3.02 doesn't seem to support x86_64 <rekado>maybe that's just because it's an old configure script <civodul>efraim: it's a space-separated list of URLs <efraim>also I don't think the latest tinycc release supports x86_64 <rekado>but I cannot seem to regenerate the configure script, because it uses old autoconf macros <efraim>their development over the past few years added it in apparently <efraim>we have a couple older autoconfs I thought <civodul>rekado: native i686 build, using #:system <civodul>but that won't work for other arches... <rekado>I try with the latest autoconf that can generate the configure script. <rekado>it's just config.sub that complains <rekado>says "Invalid configuration `x86_64-unknown-linuxoldld': machine `x86_64-unknown' not recognized" <rekado>"missing template: ALIGNMENT_DOUBLE", "ALIGNMENT_FLOAT", ... "LEADING_UNDERSCORE", "TYPE_TIMEZONE". <rekado>maybe I need to patch the configure.ac first <civodul>you could replace config.guess and config.sub with new ones from automake <roelj>I'm trying to build the bootstrap binaries with "guix build bootstrap-tarballs", but it fails to do so: guix build: error: build failed: derivation '/hpc/gnu-store/...-guile-bootstrap-2.0.drv' has incorrect output '/hpc/gnu-store/...-guile-bootstrap-2.0', should be '/hpc/gnu-store/...-guile-bootstrap-2.0'. All having different hashes. The store only contains ...-guile-bootstrap-2.0.drv and a .drv.lock file. <roelj>Aha! I had configured --with-store-dir with a slash at the end, which causes that problem. <rekado>roelj: you are not using /gnu/store? <rekado>bleh, the configure script also contains a platform check... <Jookia>civodul: re compilers building on LLVM- how would this help the bootstrapping issue? <Jookia>I'm not sure what 'building on LLVM' actually means- GHC uses LLVM <efraim>I'm getting lots of 504 gateway timeout errors from mirror.guixsd.org <civodul>efraim: yeah, hydra.gnu.org is overloaded again :-/ <rekado>Jookia: GHC *can* use LLVM as a backend. <rekado>but it's not implemented on top of LLVM <rekado>languages like Julia or Extempore are built with LLVM <civodul>roelj: could it be that gnu/packages/bootstrap/guile*.tar.gz is invalid? <roelj>civodul: I unpacked it, and ran ./bin/guile, which worked. <Jookia>rekado: That's kind of a good thing <Jookia>I know compilers like to bundle LLVM <rekado>(in the case of Extempore it's annoying because it needs a *patched* variant of LLVM...) <civodul>roelj: well i don't know, maybe you need to post more details to the list <roelj>civodul: Yes, sorry. First I am going to redo the whole process after reading the manual *again*. <rekado>re Haskell: seems that the tarball for 3.02 doesn't contain everything we need. An older version of happy is not included, for example. <Jookia>re bootstrapping: We're (free software engineers, not just Guix) going to have to write and maintain C->[language] compilers. I don't think there's a way around this <Jookia>Bootstrapping with older versions provides a backhaul and relies on x86 machines in a lot of cases <Jookia>Haha, I think people can settle on C for bootstrapping <rekado>we can define a grammar more easily and use the Guile VM <Jookia>I've been meaning to get in to Guile but I'm not sure about the tools and it seems a bit disorganized. Perhaps more suited for #guile :P <rekado>the substitute for boost cannot be downloaded. <wingo>heh, it is more of a note that webkitgtk is (a) alive and (b) needs a new release :) i can get around to it later if no one else does <wingo>good that webkitgtk managed to avoid an llvm dep :P <rain1>If i want the mkstemp, is there a way to search through all guix packages to see if one has it? <efraim>you can use locate to see if you already have it locally <efraim>but we currently don't have a way to search for it among the packages <rain1>I was thinking a bit about how one might build a tool for that - basically something that runs find /gnu/store > db.txt on the build farm <efraim>looking at apt-file, it looks like gnulib or nmh may have mkstemp <rain1>should I package gnulib for guix then? <civodul>Gnulib is not meant to be "packaged" <rain1>oh my bad we have mktemp binary <mark_weaver>civodul: my usual method to delete bad nginx cache items is to run "ls -lS /var/cache/nginx/nar/*/*" and find the entries that have a size only slightly larger than the result of "wget <URL>". the cached item includes the HTTP headers, which is why its bigger. you can then 'less' the binary file and see the nar URL in there, to confirmation. then just delete it. <kristofer>I reconfigured my system for a third time, spent all day waiting on it to compile only to find out that my kernel version is older yet again. I started 3 days ago with kernel 4.4.1, did a reconfigure and wound up with kernel 4.3.3, did a system reconfigure again last night after updating guix to the latest guix from git, now I'm at kernel version 4.2.6 <kristofer>why does the kernel get older everytime I system reconfigure? <mark_weaver>kristofer: if you don't run "guix pull" (or equivalent) as root, yes. <kristofer>I have guix src from git in /home/kristofer/src/guix and it's symlinked to both /root/.config/guix/latest and /home/kristofer/.config/lateset <mark_weaver>(and btw, running "guix pull" as root only affects the version of guix for root; it also needs to be run as your normal user as well, to update the version of guix for that user) <mark_weaver>oh, well, in that case, this shouldn't be happening. maybe something is wrong with those symlinks. <mark_weaver>kristofer: what is the output of: readlink /root/.config/guix/latest <kristofer>yep, symlink error. I'm officially a moron :) <mark_weaver>heh, similar things happen to all of us sometimes :) <efraim>guix refresh -t gnome gave a list of 57 updates <efraim>I almost want to update all 57 and build them all and dump the updates that don't build <efraim>use that as a start for updating gnome <roelj>Where can I find build-bootstrap-guile.sh? <roelj>Ah, I see.. it's generated in bootstrap.scm <davexunit>Kristofer: looks like you reconfigure with an older Guix each time <davexunit>so make sure you don't do that going forward. <civodul>mark_weaver: yeah i have a script for that, that i posted to guix-sysadmin ;-) <civodul>mark_weaver: i suspect that if the backend replies with 200, then closes the connection prematurely, then nginx might cache it as a zero-length reply <mark_weaver>civodul: I recently changed the nginx config to use HTTP/1.1 to talk to hydra-server, hoping that might help. <mark_weaver>with HTTP/1.0 in the absence of a content-length field, it might be hard for nginx to tell the difference between successful completion and an error, but if hydra-server uses the chunked method (forget the exact name), then the difference should be apparent to nginx. <civodul>i should do the same on the mirror, then <civodul>so nginx on hydra.gnu.org was caching empty items too? <mark_weaver>well, I don't know about empty items, but it has certainly cached many truncated items <mark_weaver>the hydra-server worker processes have a tendency to get very bloated up, and sometimes I need to kill them to bring hydra back from overload.. <Jookia>Is it a flaw of Guile that Guix has confusing error messages, or just Guix's alpha stuff? Could it be a project to make them easier to understand? <rekado>Jookia: could you give an example? <Jookia>"`(#:modules ((guix build utils)(my-module-build))" -> "ERROR: In procedure lstat: ERROR: Wrong type (expecting string): #f" <rekado>It's a bug, for example, that "guix refresh" sometimes crashes. <Jookia>Yes, but it's not really user friendly or clear <rekado>this means "lstat" was called with "#f", but it should have been a string (= a file name) <mark_weaver>Jookia: guile's error reporting could surely use some improvement, but a lot of the problem is that guix fails to preserve the source location information for code run on the build side. <mark_weaver>and unfortunately, "fixing" that would lead to large numbers of unnecessary rebuilds <Jookia> http://sprunge.us/WYAR - backtrace in question (i kinda have an inkling what's happening after testing with different inputs, not the error messages itself) <Jookia>It's a very trial and error system :\\ <mark_weaver>e.g. if you made a change somewhere earlier in the file that contains bash, that would change the line numbers of the source location information for bash, which would change its guile build script, which would change its derivation and everything that depends on bash, etc.. <mark_weaver>so unfortunately the discarding of that information is important <civodul>mark_weaver: maybe eventually we could change the derivation format to include "source maps" that would not have any effect on the hash computation <mark_weaver>unless the error message was printed by a process outside of the build container, with access to the source location information in a separate file.. sounds very hairy. <civodul>well yeah, that'd make the whole infrastructure "Guile aware" <mark_weaver>it might be better to have a mode where we preserve the source code itself, as s-expressions and without line numbers. <mark_weaver>and then guix might be able to preserve that information without it changing when the line numbers change, or when you add a comment or change formatting of a package description. <mark_weaver>although that, too, would require some non-trivial changes to guile <Jookia>civodul: They all look the same to me :P <Jookia>civodul: Just a wall of text that I ignore and try to change source code until it works <davexunit>not great to show a user in a command line tool with no other information, but great for a guile programmer to see <civodul>davexunit: i can surely parse those backtraces, but i've heard many times from others that they look obscure <civodul>so surely, there's room for improvement <civodul>but i'm far too used to it to understand what needs to be done :-) <davexunit>I guess my point is that this complaint goes for *every language* <civodul>but maybe there's a way to present things that would work better, dunno <davexunit>if you don't read the backtrace and just change stuff randomly, you aren't working efficiently. <civodul>Jookia: could you write a mock of how you would represent that backtrace? <Jookia>civodul: I wouldn't represent the backtrace, I'd give an error that the module was unable to be found to put in the store <Jookia>davexunit: Yes it's inefficient but a lot of the backtrace is noise/not understandable to me. I can kind of get what's going on but usually it's more efficient for me to just try different things <Jookia>I imagine for a Guile programmer having a backtrace when running Guix code would be good, but I'd rather see a Guix backtrace than a Gulie one. If that makes sense <Jookia>Maybe that's a flaw of all DSLs though, not sure <rekado>Jookia: you are right. Guix should not show any backtrace when using it. <rekado>when hacking on it, however, I think it's just fair to show a Guile backtrace. <rain1>nice news tar accepted a patch to help tools using it make reproducible builds! <Jookia>I wonder why e, o, t and u aren't part of derivation hashes <mark_weaver>Jookia: my guess is that the reason is that is significantly reduces the likelihood of finding something that looks like a nix-base32 hash but isn't. e and t are the two most common letters in english text, and o is quite high up there too. <mark_weaver>and identifying nix-base32 hashes is important when scanning for references and for grafting. <mark_weaver>I don't know what guix-daemon does, but the algorithm I came up with to scan for hashes on the wip-faster-grafting branch benefits greatly from the fact that long runs of valid nix-base32 characters are quite rare outside of genuine references. <Jookia>Ah, I see. I was reading through that code <mark_weaver>I scan for the '-' after the hash, and every time I find a character that's not a valid nix-base32 char, I can skip ahead 33 bytes. <Jookia>BTW I'm trying to figure out a 'solution' to statically linked binaries still pulling in glibc, gcc seems to link to gconv and other references in the program <civodul>mark_weaver: BTW, shouldn't we merge that branch? <mark_weaver>if you search for [eotu] in some random english text, you'll see that those letters are very common. <Jookia>My current one of 'remove-store-references' then using ldd to check it's not completely broken isn't inspiring confidence <mark_weaver>civodul: I have a cleaned up version of the code (lots of comments added, switched to vhashes, and some other minor things), but I haven't yet tested the new version. <mark_weaver>well, I've tested that it runs successfully, but not that it did the right thing <civodul>Jookia: did you find out why those things refer to glibc? <Jookia>civodul: No, but e2fsck/static uses this 'solution' <civodul>the gconv modules is one thing, but it's ok to use remove-store-references on them <Jookia>I put 'solution' in quotes because if another reference slips in that's needed it won't be obvious <Jookia>Maybe I should write my own 'remove gcc metadata' function <civodul>it works well in make-bootstrap.scm and other cases where it's used <civodul>because we know we're really statically linking against libc, etc. <Jookia>Oh I understand that, but it's bothersome for me when I'm trying to statically make a package statically linked but it not working for whatever reason <Jookia>Maybe I need to just check references before I try to put it in an initrd <civodul>yes, you should first try running it in a chroot, for instance <rekado>civodul: fix for CRAN updater has been posted to the list. It's just a tiny patch to the match pattern. <mark_weaver>civodul: FYI, I've recently rebased wip-loongson2f onto recent master and updated GuixSD on my Lemote Yeeloong. thanks to grafts, it is now practical for me to keep GuixSD up-to-date on this machine which I choose to avoid substitutes on. <civodul>mark_weaver: i was indeed seeing partial downloads being cached on the mirror <civodul>so i ended up using http 1.1, which should help <mark_weaver>civodul: if hydra-server uses chunked transfer encoding (or whatever it's called) to send the nars, then it should help. <civodul>so i'm guessing (hoping?) that it's always chunked <civodul>ideally someone would dive into the Perl mess to confirm ;-) <civodul>it's terrible that http provides no integrity check beyond content-length and chunks <piyo>civodul: http is delivered by tcp/ip which provides those things? <mark_weaver>and TCP connections can be reset, as opposed to closed normally, iirc. <civodul>yes but there are no HTTP-level checks <civodul>so if the connection is closed prematurely, you can't really tell if you're done or not <mark_weaver>so even without chunked transfer encoding, hydra-server should have been able to indicate the error <civodul>i think hydra-server would just eventually drop the connection, in which case its clients can't tell <mark_weaver>civodul: with chunked transfer encoding, the end is explicitly marked. a premature close will be easily detectable <mark_weaver>one thing I'm not sure about is: does hydra-server detect if the nar packing subprocess returns a non-zero status code. <mark_weaver>if there's an error in the subprocess, will it send the chunked-transfer-encoding termination making it look like everything is good? <civodul>mark_weaver: good question; it's in NixNAR.pm <mark_weaver>civodul: while investigating what would be needed to write the custom-nar-proxy for hydra, I was wondering: maybe I shouldn't bother talking to hydra-server at all for this? maybe it's easier to just use the code in guix that's used for guix publish? <civodul>i don't know what Perl does when the pipeline fails <civodul>problem is that 'guix publish' does not compress things <mark_weaver>I think it's important to strictly limit the number of concurrent packing jobs. <rain1>I had to do that for building qt becasue it froze my computer otherwise <rain1>--max-jobs=1 --cores=1 was really useful <mark_weaver>civodul: and this means that sometimes narinfos will be available, but the nar won't yet be available. <mark_weaver>civodul: would it be possible to make guix handle this case more gracefully? <mark_weaver>well, even it was possible before, this case might become more common <civodul>'guix substitute' is first invoked to say what i can substitute, and then to actually substitute things <civodul>so when it says "yes i can substitute it", then it's expected to be able to do it <mark_weaver>my work on this got blocked somewhat by the fact that guile's built-in web server doesn't seem well-suited to long-running client connections, where a lot of data needs to be sent. <mark_weaver>so I've been considering various possibilities: what would be needed to improve guile's web server; whether a simplified http server would be reasonable just for use behind nginx, etc. <taylan>mark_weaver: FYI there's a new guix-devel thread going about bootstrap compiler/binary vulnerability, in the vein of what you had mentioned some time ago <civodul>taylan: don't overload mark_weaver :-) <mark_weaver>taylan: ah yes, I saw that. thanks for digging up my old post on the subject <rain1>I just have a beginner question about developing on guix <rain1>how would you build a simple gtk+ program? <rain1>I thought I could just do: guix environment gtk+ automake pango cairo <rain1>but if i do that I have to: make PKG_CONFIG_PATH=/gnu/.. with a very long carefully built var <mark_weaver>rain1: include pkg-config in the list of packages to 'guix environment' <rekado>rain1: "guix environment --ad-hoc ..." <mark_weaver>iiuc, that should cuase PKG_CONFIG_PATH to be set automatically <rain1>I wonder if anyone has some aliases that are often used for development? <rain1>I will probably collect up mine as i make them, I saw a nice ruby on rails on gist <rekado>I usually just create a guix.scm file containing a preliminary package definition, then load it with "guix environment --pure -l guix.scm" <mark_weaver>rain1: btw, on another subject: webkitgtk-2.4.x, which midori uses, just received an unexpected upstream update with security fixes, and I pushed the update to guix master a few hours ago. <rain1>I think I might be drop midori now that I have epiphany, not sure <mark_weaver>I don't know how long upstream will continue to support that branch though. <rain1>I just needed a second browser because I can't do a lot of things with icecat <mark_weaver>rain1: I would still recommend epiphany over midori though, because it has process separation, and proper certificate checking of subresources, etc. <mark_weaver>rain1: icecat could be configured to be less strict about things. if you tell me what problem you're having with it, I might be able to help with that. <mark_weaver>the strictness customizations I know about are all undoable in the "about:config" page, and by disabling the extensions/plugins that come enabled in icecat by default. <rain1>yeah I could probably sort it if i dig around in the configuration it's just a lot of trouble <rekado>I haven't seen any way to extend epiphany/web, though. <rekado>I'd really like to write extension for it, but it looks like there is no mechanism for that. <mark_weaver>yeah, the lack of NoScript and Cookie Monster (or similar) are the main things keeping me from using Epiphany. <rekado>I wrote a mail to the epiphany list to ask about extensions but it's stuck in the moderation queue. <rain1>there is a /etc/hosts trick to get ad-blocking without an extension <rain1>although there is a slight issue with it <mark_weaver>ah, speaking of which, there's a new major stable release of webkitgtk <ng_>our urxvt / rxvt-unicode is not compiled with gtk pixbuf for transparency support by any chance? <rain1>this is a bit like how guixsd has a service to block facebook <rain1>but the list is extremely long and the linux kernel does a linear search through it, so it's not ideal <ng_>rain1: i had the idea for a unified adblocking list, but I delayed it as the task is a bit too much at the moment. <mark_weaver>ng_: I don't know, why don't you look at our package description for it and see for yourself? <ng_>i have and don't see it. which possibly means it isn't there. <rain1>It would be nice if they used a trie but I doubt they would accept a change like that <ng_>or is transparency a default? <rekado>mark_weaver: I don't even see any way in epiphany to specify/load/define extensions. <mark_weaver>ng_: sorry, I don't know if many people here use rxvt <rekado>it's good to see that webkit supports different methods of extending the browser, but I just don't know how to even start. <ng_>I am only looking into it (using it myself) to add a .desktop file for us <civodul>+ guix-prettify-mode + guix-build-log-minor-mode <mark_weaver>rekado: searching for "epiphany extensions" finds some things that suggest that extensions are supported, but i don't know the details. <mark_weaver>I only use gnome-terminal in cases where M-x shell doesn't work <mark_weaver>e.g. connecting to a 'screen' session on another machine <mark_weaver>(I know about M-x term, but I find it performs poorly in practice, and I find it awkward dealing with the key bindings) <rekado>I read that current epiphany/Web no longer supports extensions. There used to be extensions but they are no longer supported and there supposedly should be a new mechanism for extensions. <rekado>not sure if it's there already or if people are still working on it. <rekado>civodul: I also just ran into the python 2.7.10 hash mismatch when downloading a substitute. <rekado>(in an attempt to reproduce an error reported to me when installing lilypond via guix) <mark_weaver>I also run some guix builds within gnome-terminal, if they produce an enormous amount of output that would bring my emacs to its knees <civodul>ACTION drowns in a maze of gateway timeouts and hash mismatches <ng_>emacs sometimes just freezes for me. also, I guess we have transparency, if not it won't affect what I want to do. <rekado>when the exits are sealed, maybe. <mark_weaver>ng_: freezes while doing what? I've always found emacs to be quite reliable. <ng_>while being in Gnus, sometimes when starting, sometimes after hours. I have some newsservers I connect to and many big lists <mark_weaver>ng_: Gnus can seemingly hang while trying to talk to a server if there are network problems, but C-g should get you out of that. <ng_>i'll try that next time. <bavier>civodul: I haven't been able to recreate the ICE with guile's vm.c and -Os you reported the other day <civodul>bavier: i just added -Os to the CFLAGS in guile-static, in make-bootstrap.scm <ng_>would you prefer the way ratpoison.scm gets the .desktop or the way fluxbox (wm.scm) gets the .desktop file? there are 3 files on archlinux.org which would just do what I want for urxvt (urxvt,urxvtc,urxvt-abbed) <ng_>ratpoison gets them from gentoo.org, fluxbox constructs it <civodul>on my SSD laptop i reach 20-50 MiB/s when doing "guix archive --export ...texlive-texmf", according to iotop <civodul>on hydra.gnu.org, we're at 0.5-3 MiB/s <tyngdekraften>Hello, I am installing gnu/guixsd with the usb installer following the gnu.org installation manual, but I did not start cow-store on my mountpoint, is it a problem? <civodul>tyngdekraften: you may run out of memory <civodul>you can always hit ctrl-c and start again <rekado>the fontforge update from four days ago broke the lilypond build. <rekado>we haven't noticed because hydra hasn't yet tried to build lilypond. <rekado>this is going to get really uncomfortable when we make a release and the new hydra isn't available then :-( <civodul>this is already terribly uncomfortable <civodul>we're waiting for the libreboot machines... <mark_weaver>nully has built a working KGPE-D16 machine with Libreboot, but I guess thum is still having problems :-( <mark_weaver>but yeah, there have been very unfortunate delays :-( <davexunit>civodul: are we still thinking that this release will be 0.9.1? I recommend bumping to 0.10.0 due to the big changes like grafts <civodul>i was hoping to go from 0.9.x to 1.0.x... <ng_>would "Icon=terminal" work or do I need to provide paths in guix, like Exec=~a/bin/urxvt~@ <jlicht>civodul: what is the list of milestones that would allow you guys to release 1.0? <bavier>just that we shouldn't feel pressure to release 1.0 because it numerically succeeds 0.9 <civodul>jlicht: essentially making 'guix pull' trustworthy and more flexible, and having proper full-disk encryption support <civodul>also having a build farm that handles the load <civodul>davexunit: did you attend the libreboot workshop at LP? <rekado>ng_: I think it would be good not to use full paths if you can make it work without. <rekado>I found that in Xfce icons are sometimes lost upon updating and I had to manually set them to ~/.guix-profile/share... to make them reappear. <rekado>dunno, maybe I did something wrong. <rekado>anyway, I think it's worth experimenting with this. <ng_>currently I have it like this, untested and did not run it so far as I would be experimenting with this kind of code. Does it look like it would break? https://ptpb.pw/8UAL <civodul>davexunit, paroneayea: could you reply to the proposal jlicht sent on the list when you have time? <civodul>i feel you'd be in a better position than me to comment <ng_>new is the 'install-desktop phase <rekado>ng_: I cannot take a look at this right now, I'm afraid. Need to speed-package some new bioinfo thingie... <davexunit>just haven't been able to write up my feedback yet <davexunit>civodul: I am in support of this project, btw. <tyngdekraften>I restarted the installation with cow-store started and I ended up with an empty /tmp folder on my mountpoint, what could have happened? <rekado>bleh, it's true that the earliest GHC sources contain generated C files. <rekado>maybe we need gofer or some similar pre-haskell to build the first GHC <Jookia>How would cross-compiling work with the bootstrap? <rekado>here's an ancient haskell compiler from 1997 <rekado>bleh, the sources were not archived <Jookia>rekado: Building bootstrap binaries on ARM where GHC only outputs x86 <rekado>once we reach a more modern GHC we would then be able to cross-compile. <rekado>we would still depend on i686 hardware as you pointed out. <Jookia>I wonder how this would work if a previous compiler was nonfree and it was open sourced <davexunit>ideally, each compiler would come with a bootstrap mechanism, like Guile does. <davexunit>that way we won't be fighting bit rot and increasingly longer bootstrapping processes as time goes on <Jookia>Yeah, that seems to be the only way in the long term <mark_weaver>sometimes I think we should just write minimal evaluators or non-optimizing compilers for these languages in guile. things get a lot simpler when you can assume the program has no errors. <mark_weaver>although having a haskell front-end for guile might be interesting in its own right <bavier>that would be interesting indeed <Jookia>mark_weaver: it's probably the right goal <jmarciano>I hope to get Guile only frontends for Guile... please... <lfam>Two more corrupted nars, reported on guix-devel <dannym>Hmm... what's the regexp flavour that guix uses? I'm trying to match the word "plugin.py" in substitute*. Tried "<plugin.py>", "\\<plugin.py\\>", "\\\\<plugin.py\\\\>"... they don't work... <davexunit>\\\\ is to escape the escape character in guile strings <rekado>mark_weaver: I was going to suggest the same. <dannym>davexunit: yes, good point, but when I want to match only whole words? <dannym>(also, it should also match for beginning-of-line-without-whitespace) <davexunit>there's a whitespace character class, can't remember the syntax off the top of my head. <davexunit>so, guile doesn't actually document the syntad <janneke>wow, wouldn't it be nice to use gnu regexps? <mark_weaver>yes, I agree that we need a better regexp implementation <davexunit>what would be best is to use a sexp-based regexp syntax <mark_weaver>I think SRFI-115 would be a good thing to have, with an implementation that works efficiently based on Guile's internal string encoding <mark_weaver>and that in turn would benefit greatly from using UTF-8 as our internal string encoding <mark_weaver>arrays of unicode code points as integers, either 1 byte per character or 4 bytes per character, as needed. <Jookia>that's fairly sane compared to some other encodings that could be used <mark_weaver>yeah, it's a reasonable choice, and has the benefit of efficient indexing by code-point index. <mark_weaver>but I think UTF-8 is a better choice, for many reasons. for one thing, having just *one* encoding would be a big win, to avoid all of the binary operations that need four different cases <mark_weaver>and for things like string searching operations, regexps, and more generally any kind of state machine, UTF-8's special properties make it work very well <Jookia>well if it helps, indexing by code-point isn't all that's worked up to be <mark_weaver>and state machines need only 256 entries per state instead of 4 billion <mark_weaver>actually, I think it was a mistake to use the term "character" to describe code points, as opposed to graphemes. <Jookia>I think the standard was revised to change the terminology <Jookia>Generally I treat text as an opaque object to be manipulated using i18n/l10n tools <mark_weaver>sure, but what I mean is that the elements that comprise strings, which historically have been called "characters" and which are still called characters in many places, even if the recent standards make clear that they correspond to code points. <mark_weaver>when internationalizing things, for high-level purposes, I think it would have made more sense to make those elements graphemes, as perl 6 did. <mark_weaver>and for low-level purposes, bytes of UTF-8 are a good choice. <Jookia>I haven't heard much of Perl 6 besides its almost mythical status <lfam>python-pyopenssl is broken. I'm trying to update it (and python-cryptography) <Jookia>I wonder what effort would be needed to compile the entire bootstrap with tcc <Jookia>That's true, I assume right now all the bootstrap binaries could be built with a version of GCC <mark_weaver>the thing is, even with tcc, you'll still need an assembler and linker. and what architectures does tcc support? <efraim>I did some reading last night on tcc, sounded a bit depressing <mark_weaver>I had a different idea, to implement simple language implementations in Scheme, and to write a simple Scheme implementation that can run on bare metal, maybe initially as a coreboot payload. <efraim>last release supports x86, the fork and the current head supports x86_64 <lfam>efraim: What was depressing about it? <mark_weaver>there are already a *lot* of scheme implementations, so there's a lot of diversity that can be leveraged to use David Wheeler's method to detect thompson viruses <efraim>sounds like there was a bunch of politics going on with it and there hasn't been a lot of development since 2008 <mark_weaver>and with scheme, there are already some bare bones implementations, and you don't need a linker and all that other stuff. <rekado>in the case of Haskell we already have an interpreter --- but it isn't trivial to use it during the build of GHC. <rekado>writing simple language implementations in Scheme is just one step. Actually using these implementations for bootstrapping sounds like a lot of effort. <Jookia>Scheme is also somewhat future-proof is it not? I imagine you could write some simple half-implementations in Scheme then modify the final compiler to work with it <Jookia>As for a lot of effort, there's no real other options :( <ng_>i hope we get that librebooted server soon, the bottleneck with hydra is really bottleneck'ish some days. <lfam>I'm currently setting up a caching mirror for myself. That will turn 4 machines into 1 from hydra's point of view :) <Jookia>:P we're dealing with how many decades backhaul of unreproducable software? ;) <lfam>Now *that* is a depressing topic! <mark_weaver>but I guess I think it's worth a lot of effort to be able to bootstrap our OS from source code with only a small, reasonable auditable bit of machine code to start with. <ng_>if 1984 hosting wouldn't be that expensive, I would pick a faster machine w more bandwidth so I can provide that as well, but for now it's 1984 for the patches thing testing soon, then maybe later more and a dedicated server running guixsd at in-berlin some day. <mark_weaver>I've been thinking about jumping head first into this project for a while now... <Jookia>You'd also have to get upstream compiler developers to agree on keeping a certain version for bootstrapping with <lfam>mark_weaver: You're tempted to write machine code that can bootstrap the entire dependency graph? <lfam>That would be an amazing service to computer users (everyone) <ng_>enough with trying hydra to test my stuff to build today.. tomorrow is another day. <bavier>mark_weaver: I'm interested in the same, but I'm noticing that my low-level skills are still lacking at this point <rain1>mark_weaver, oh i was making notes about that with a friend - it's a really fun exercise to reduce trust down to as little as possible <mark_weaver>bavier: well, my plan is to first bootstrap a Scheme interpreter and go from there, so a large part of the work would be to build our bootstrap binaries from Scheme alone, e.g. to have things like sh, make, and a C evaluator written in pure Scheme. <lfam>I am referring to the current version of python-pyopenssl, not the old one on master <rekado>this is a very tempting project. <lfam>I also commented out the fix-tests phase <dannym>(There was actually a funny competitions that start with paper and entering opcodes directly to write an assembler, then write a tiny forth using the assembler, then write a tiny scheme in forth, etcetc to see how long it takes to get network connectivity. I wish I remembered what it was called...) <rain1>it's very fun stuff so you could probably organize people to work on lots of subproject aspects of it and make good progress <Jookia>Starting off with the assumption that you trust hardware and BIOS, you'd want a live USB that can bootstrap containing source code you verify correlates to machine code <rekado>rain1: that's actually not a bad idea. <Jookia>GNU seems to be usable as a noun for a particular community thesedays <bavier>yes, it would be great if we had a game plan (or several) with missing pieces identified <bavier>mark_weaver: the bourn-shell compiler GSoC project should help with that <Jookia>It's a little sad that after all these decades GNU is still one of the only projects the few projects that directly act for software freedom <Jookia>Has anyone audited what exactly is the minimum amount of tools needs to bootstrap? <bavier>mark_weaver: could you detail more what you meant by "first bootstrap a Scheme interpreter"? <mark_weaver>initially, all communication would be over a serial port, because those devices are *very* simple to configure and control, and some processors actually have one built in. <Jookia>mark_weaver: Wait, what level are you talking about? <rain1>reimplementing unix tools in guile helps security a lot even regardless of trust chain issues, also you could run them in a 'deterministic' language mode <Jookia>My Novena has a JTAG port if that helps :P <rekado>BTW: there's the parallax propeller, an eight core micro-controller with completely free hardware design. <Jookia>Are we now thinking of trusted hardware to bootstrap from? <mark_weaver>hardware is another thing that would need to be tackled at some point, but that's outside the scope of the project I'm thinking of taking on. <mark_weaver>except that I want this to be a portable bootstrap that can work on almost any hardware architecture with minimal differences <Jookia>mark_weaver: What are you bootstrapping from in your project, what level? <mark_weaver>but I absolutely want to avoid using *any* precompiled software code at all <rain1>it might be nice to build the OS on top of seL4 rather than linux <mark_weaver>I want to bootstrap from minimal machine code that's small enough for one person to audit, put on a boot flash chip <rain1>mark_weaver, I had a couple thoughts about assemblers yesterday if you like <mark_weaver>and I want to be able to bootstrap Guix from it without using anything precompiled at all. <civodul>Jookia: fwiw when i bootstrapped Guix (initially it was bootstrapped from Nixpkgs) i tried to reduce the set of binaries as much as possible, which was in part possible thanks to the Guile bits we use (web client, etc.) <mark_weaver>well, the interest here is encouraging. I may need to get more organized and start a mailing list or something :) <civodul>howdy, paroneayea, long time no see :-) <Jookia>paroneayea: Hey! Welcome to the beginning of mark_weaver's quest to eliminate binaries! <rain1>mark_weaver, so you know how even if you have a fixed point Compiler(Compiler)=Compiler you haven't proved that the binary has not been tampered with.. <paroneayea>civodul: maybe a bit late, but I'm writing up a blogpost encouraging people to apply on MediaGoblin and Guix GSoC, especially the 8sync one for shepherd and the "port our federation to our new standard" one <rain1>in the case of an assembler you can verify it's clean just by computing the output size - a trusting-trust attack would have to increase the binary output size <rain1>I think an assembler and tool to compute its output size would be a crucial part of a toolchain like this -- but anyway I agree with making a ML! <paroneayea>ACTION wonders if anyone in #guix is new post libreplanet talk <mark_weaver>rain1: it's a good observation, although I'm not sure how I'd make use of it, given that a similar trick could not be used for other components which I'd like to bootstrap from scratch anyway. <rekado>so, the scheme implementation would run on bare metal and talk via serial port, allowing us to avoid dealing with hard disks and drivers. <mark_weaver>or allowing those drivers to be written after some other parts of the system have been bootstrapped <rain1>mark_weaver, yes it only works for assemblers - I have some half baked ideas for more complex compilers but I need to get them solid first :) <mark_weaver>which would need to include some kind of editor and repl environment, bootstrapped concurrently. <Jookia>My Novena doesn't sport a flash chip I could boot from- however, it can boot binaries over USB <paroneayea>mark_weaver: might you bootstrap the scheme on top of guile, or use something else? Is even guile too strongly bootstrapped from C to be bootstrappable? <janneke>lfam: nothing wrong with this window <mark_weaver>I was actually thinking of starting from a Lemote Yeelong, because the hardware is so simple <mark_weaver>GRUB includes support for being the boot firmware for the Yeeloong, and it's refreshingly simple. <mark_weaver>and the processor has a serial port that can be accessed easily and quite early, before even things like raminit have been done. <Jookia>Wow that's cool. Though I'd be wary of *relying* on features most computers don't have. <mark_weaver>but of course I want to support other machines as well. <rekado>the propeller has a serial interface and forth. <mark_weaver>sure, getting early access to a hardware serial port of x86 machines should also be quite easy, and likewise should be possible before raminit <lfam>janneke: I don't want to quit this window ;) <rekado>does it really need to work on *any* hardware? <mark_weaver>I don't know off-hand about the early ARM boot process <Jookia>Early ARM boot process is dependent on chipset <rekado>could it not be enough to have a simple board that we talk to via serial interface that spits out the desired binaries? <mark_weaver>rekado: well, what I mean is that I want to emphasize portability in the design, and try to confine the non-portable parts to isolated areas. <paroneayea>mark_weaver: how can you possibly write a trustable system without static types? ;) <mark_weaver>rekado: since we don't have any hardware we can trust, we need to rely on diversity of starting points, along the lines of David Wheeler's idea. <rain1>static types don't really counterbalance GHCs 700MB of space to hide poisonous code <civodul>rain1: Rotten is an interesting exercise! <rain1>civodul, it's great isn't it! I had always known about this attack but never saw a PoC before <paroneayea>rain1: I'm trolling... I actually don't buy the "nothing is safe without static types" argument <paroneayea>the main really dangerous bugs seem to be in memory safety and bootstrapping problems <Jookia>So I guess we now know what mark_weaver's going to be doing over the next decade :P <paroneayea>most type errors result in crashes but I think very rarely (comparatively) seem to be security issues from what I've seen <mark_weaver>paroneayea: seriously though, I pondered trying to write proofs of everything along the way, but I think that might push this from a merely crazy idea to insane, and unlikely to get done by me. <mark_weaver>for now I'd be satisfied to make the code as comprehensible as reasonable, avoiding clever tricks and hacks. <rain1>They key would be to find ways that proofs can accelerate the development instead of slow things down <paroneayea>mark_weaver: I met the current emacs maintainer (John Wiegley), he's currently working for work on a "provable far into the future system" using strong proofs <paroneayea>and then even use genetic algorithms to generate your program! <rain1>the seL4 kernel has pretty much a complete security proof down to assembly code <rain1>it might be an interesting thing to study or even build on top of <mark_weaver>paroneayea: my last message was meant to be directed at you <mark_weaver>rain1: for now, the goal would be to bootstrap a GNU/Linux system, because that's the free OS that large numbers of people are actually using, but my hope is that other systems could be bootstrapped from this same tool. <mark_weaver>and specifically, I would probably focus on bootstrapping GNU Guix in particular <paroneayea>one thing that was interesting in Snowden's talk <paroneayea>he called out long-running stable systems as something we probably can't promise <Jookia>mark_weaver: At what point do you declare a system tristed? <mark_weaver>paroneayea: I came very close to diving into the world of proving as much as possible, static type systems, etc, but Sussman has recently convinced me that there are serious downsides to that approach to programming. <paroneayea>though he didn't say it specifically, I think he meant like Debian Stable and RHEL <Jookia>paroneayea: Is there some talk on this? <Jookia>mark_weaver: Well, even if you build it from sand you have to trust each piece of software too. Are we going to declare Linux trustable? At what point can we be satisfied with bootstrap binaries? <paroneayea>Jookia: there were a few, but one especially (which davexunit sent me) was very strongly emphasizing the need to have programs which are live and dynamic <paroneayea>and only very very particularly special pieces need strong proofs (like garbage collectors) <paroneayea>mark_weaver: though maybe a bootstrapped system could be considered a case where a strong proof system would help, I don't know. I agree you should aim for implementability not perfection though. <Jookia>paroneayea: I'm kind of skeptical in Haskell for the point that there's no real good hot swapping/live updating/interactive development tools <paroneayea>Jookia: yes, and it may be hard for it to happen <paroneayea>maybe if we could have proofs combined with JIT systems <paroneayea>but nobody seems to have done that research and I don't really *know* if it is possible <paroneayea>in the meanwhile, live systems seem more important to me. <Jookia>The problem I found with Haskell is that I'm not smart enough to write proofs <Jookia>That's the direction that place is going I think <mark_weaver>a bit different, but substantially overlapping ideas. <paroneayea>mark_weaver: that one is one, though the one davexunit sent me most recently is even more an emphasis (the majority of it emphasizing) the need for live, dynamic systems <mark_weaver>the thing is, even if you prove everything correct, a proof can only show that the code matches your specification. there may still be bugs in your specification. <Jookia>Then you find out you're just programming a specification <mark_weaver>code written very simply may be nearly as easy to audit as a specification, depending on the complexity of the spec. <Jookia>My 'hunch' is that most bugs/security issues can be solved with high-level DSLs in simple languages like Guile <mark_weaver>it has become clear to me that it's exceedingly difficult to write secure code in a language like C <Jookia>mark_weaver: Which is why I kind of feel a bit weird moving from programming Haskell to programming proofs with not much to gain <mark_weaver>every time you cast an integer from one type to another, or perform any arithmetic operation at all on signed integers, you must consider the possibility of overflow. <mark_weaver>recently, I've been paying more attention to this when modifying Guile's C code, and it's just a nightmare. <Jookia>I figure I can kill 90% of bugs by writing in Guile but not go that extra 10% because I'd rather be simple and lazy, and still beat most programs in security and bug count <Jookia>Rust seems to me to just be another C-like language for another niche but it's nothing revolutionary <mark_weaver>I don't remember enough about Rust to comment on it, but I remember thinking that it might well be a good way to make a well-performing browser more secure <rain1>unlike scheme, rust has a complex type system which program safety rests on <rain1>therefore the safety of programs that type needs to be proved - this is someones PhD project <mark_weaver>CompanionCube: that RFC includes this text: "Rust, at least, does not have to worry about memory safety violations, but it is still possible for overflow to lead to bugs. Moreover, Rust's safety guarantees do not apply to unsafe code, which carries the same risks as C code when it comes to overflow. Unfortunately, banning overflow outright is not feasible at this time. Detecting overflow statically is not practical, and detecting it <mark_weaver>dynamically can be costly. Therefore, we have to steer a middle ground." <CompanionCube>I'm not a rust programmer, I'm just aware that it's a thing and that they have a thing for safety and related things <mark_weaver>and that's in line with my vague recollection of Rust: they are trying their best to improve safety, but ultimately they consider high performance to be a top priority, and that's a reasonable position for them to take, given their goals: to allow them to use it to write a competitive web browser that people actually want to use. <CompanionCube>there are some other interesting projects in it - an OS and coreutils <sapienTech>I think a lot of the project ideas sound awesome, particularly the porting guix to hurd, creating a graphical package, an installer wizard, and a guile-based build tool <sapienTech>I would be really excited contributing to any of those, but I am curious what the guix group thinks is the highest priority project <sapienTech>i imagine there are differing opinions here, but let me know if there is a consensus <mark_weaver>sapienTech: it might be better to choose a project based on your interests and skills <sapienTech>i've been meaning to email you! I've been out of town for the last month and got back to my home two days ago <paroneayea>sapienTech: where are you living right now again}? <mark_weaver>sapienTech: the porting guix to hurd project is probably already locked in, since the person who has already done most of the work on that has expressed interest in doing it again this year. <sapienTech>mark_weaver: cool thanks for letting me know, i was reading some of the logs about that project and glad he is up for continuing that work <sapienTech>mark_weaver: i imagine the end goal for guix is to get more people using it, which i think would require a GUI, so that seems like a really important project <rain1>I tried installing alpine linux and the installer for that is really nice <rain1>a guix installer could be similar <rain1>(if thats what you were talking about) <sapienTech>however, for right now, would you say that since most people who use guix are techies, a gui is not as essential as something like an easy installer, or a build tool <mark_weaver>sapienTech: yeah, I'd agree that the GUI package manager is probably (IMO) a lower priority than some of the others, including the two others you mentioned, precisely because right now our audience is limited to techies for other reasons. <mark_weaver>an installer wizard is one the key blockers to allowing non-techies to use guix. <mark_weaver>but I still think it's important that whatever project you choose is well suited to your interests and skill set. <mark_weaver>all of these things will be important at some point, it's just a question of when :) <mark_weaver>so, e.g. if you would be motivated to write a GUI package manager, that would not be wasted effort. <sapienTech>mark_weaver: agreed. I think personally I am more interested in getting involved with the guix community, and working on something that can make a definite impact right away, than I am about working on a particular piece of software <sapienTech>and like you said, a definite consideration is what would suite/push my skill-set <mark_weaver>sapienTech: do you have previous experience with Scheme or Lisp? <sapienTech>i haven't worked on any large-scale projects in scheme, but I have a good amount of experience working with guile/ functional languages in general <mark_weaver>if could briefly summarize your skill set, that would help <sapienTech>I am familiar with the guile-specific features, like extending/interfacing with c programs <sapienTech>I understand many of the more advanced topics in the guile manual, and the general concepts behind scheme as a language, like latent typing, closures, macros, continuations, mutual recursion, tail recursion, etc <sapienTech>i have used guile as a problem solving language for fun, and I also have used it to write some medium sized programs, like a sudoku solver <sapienTech>additionally, i have used in the relational programming style, with conde and fresh, etc <sapienTech>but that said, i know that i still have a lot to learn, and im pretty excited to learn more <sapienTech>I don't have much experience, other than writing modules and compiling my own kernels <sapienTech>I do have experience working with c, using signals, creating my own shell, and things like that, but I still have a lot to learn there <sapienTech>so i thought this internship would be a great way to continue with guile, and learn about operating systems <mark_weaver>based on what you've said so far, I'd say that either of the two projects you mentioned (easy installer, build tool) might be a good fit for you <paroneayea>easy installer would be cool, and really is very needed too :) <mark_weaver>yeah, the easy installer would be a great thing to have. <mark_weaver>first time installation of guix is currently quite painful, and causes many people problems. <lfam>domenkozar: I see the Nix package is very out of date. I do see that it shares the Nix store. <lfam>domenkozar: Could you send a brief message to guix-devel@gnu.org asking the same question? <domenkozar>so it would be best if someone from guix community helps with that :) <lfam>domenkozar: Actually, I can send it. You might have to subscribe to send mail to that list <paroneayea>fyi, sapienTech made an android/replicant MediaGoblin app for GSoC last year for us <paroneayea>(named Goblinoid, hopefully will end up in f-droid soon) <sapienTech>paroneayea: we need to badger people at f-droid with helping the build process <paroneayea>sapienTech: btw I saw there was a commit about Goblinoid in f-droid <mark_weaver>sapienTech: I should have included "gui package manager" in that list of projects that might be a good fit. <sapienTech>they have never built a kivy app, so they are having issues <paroneayea>sapienTech: I wonder how hard it would be to package goblinoid in Guix :) <paroneayea>not to overwhelm you with other tasks, but it could be cool to have, since it isn't *really* android specific <sapienTech>paroneayea: im back now and ready to get hacking so keep hitting me with ideas :) <dannym>[I've been thinking on and off about some emacs thing which does ... forms as the GUI installer. Not a lot of dependencies (especially not X) and still can be made very user-friendly because the form could auto-update on the fly as you enter values and the documentation of what things are could be in-line. (Thinking doesn't mean I started doing it). I think many parts for it already exist, even. I think wanderlust uses these kinds of <rain1>well I really urge you to see the alpine linux installer <rain1>it is very easy and works great <rain1>just asks questions and then does it <rain1>something could easily do the same, fill in a config.scm template and go <lfam>I need to update python-pytest to, ultimately, fix letsencrypt. This will cause ~200 rebuilds. <lfam>Including libreoffice... <sapienTech>mark_weaver: nice meeting you, i'll work on the application and check back in with the group if i have questions/updates! <lfam>Any advice on this update? <rekado>CompanionCube: not for the initial GuixSD installation. <lfam>Thanks for bringing it to our attention <woodrad>In the guix talk at libreplanet, some mentioned trouble packaging Java programs. Are there some issues in the bug tracker that embody problems guix users have been having? <paroneayea>woodrad: hm, I think (tho9ugh davexunit and rekado can reply more clearly) that it's a package-by-package issue, but what community behavior encourages in java land <paroneayea>I don't know if there's anything build-system-related, though maybe others can comment <lfam>And then leaving no clues on how to reproduce those JARs <lfam>rekado recently added an ant-build-system to Guix