IRC channel logs

2015-05-25.log

back to list of logs

<paroneayea>hm, so guile used to have a swig support, but no longer?
*paroneayea would like to have xapian bindings for guile
<paroneayea>maybe I should package it
<paroneayea>oops
<paroneayea>wrong channel
***karswell` is now known as karswell
<please_help>Yup, I opened an issue about that on the swig github but they don't seem to have any guile maintainers so support is kinda dropped.
<please_help>if I have two real numbers, how do I make a guile complex number out of that?
<please_help>any better way than (+ x (* y 0+1i)) ?
<paroneayea>davexunit: maybe I should write an activitystreams 2.0 exporter for haunt
<paroneayea>I wrote a translator of atom->AS2.0!
<paroneayea>in guile
<davexunit>paroneayea: what does that do?
<paroneayea>activitystreams 1.0 is the format pump.io uses for communicating
<paroneayea>and activitystreams 2.0 is the new version being worked on in the w3c group I'm in
<davexunit>oh neat
<paroneayea>for an official federation standard
<paroneayea> http://dustycloud.org/tmp/dustycloud-as.json
<paroneayea>resulting json
<davexunit>very cool
<paroneayea>yeah I used sxml-match
<paroneayea> http://paste.lisp.org/display/148627
<davexunit>I haven't used that yet but I agree with your earlier statement about how cool it is
<paroneayea>will push to a real repo though soon
<paroneayea>davexunit: you can see the process-entry
<paroneayea>which does the whole xml->json conversion
<paroneayea>er
<paroneayea>feed->activitystream
<davexunit>that's really neat :)
<davexunit>it reminds me that I need to write a more sane JSON lib for guile, though.
<davexunit>those hash tables :(
<paroneayea>yeah
<paroneayea>main painpoint for me was that ,pp doesn't do anything with hashtables
<davexunit>yup
<davexunit>have to convert
<paroneayea>this didn't need to be that functional, though I wouldn't mind it if it could be
<paroneayea>maybe I should make a ,pp-deluxe thing
<paroneayea>for myself
<davexunit>I wonder if the hash table printer should print the key/value pairs
<paroneayea>which also expands hashtables into something that look roughly like
<davexunit>would be convenient
<paroneayea>{("foo" "bar") ("baz" "basil")}
<davexunit>it's not as convenient, but you can call (hash-table->alist table)
<paroneayea>davexunit: yeah but you have to handle doing that recursively for nested things like this
<paroneayea>I did so, it was annoying
<paroneayea>did so while debugging I mean
<davexunit>ah yes, that's true.
<paroneayea>it took me a bit to figure out how to process elements that had sub-elements I wanted but I didn't care about the order of
<davexunit>paroneayea: the most convenient thing would probably be to override the record type printer on the hash-table record type in your ~/.guile file
<paroneayea>davexunit: interesting
<davexunit>such that it renders the result of hash-table->alist
<paroneayea>probably could do that
<paroneayea>yeah
<davexunit>and then the record type printer will handle the recursion
<davexunit>if you write the code to do that I'll probably incorporate it in my .guile file
<davexunit>(srfi srfi-9 gnu) has the set-record-type-printer! procedure.
<paroneayea>aha
<paroneayea>speaking of (srfi srfi-9 gnu)
<paroneayea>immutable records with functional setters are the shit
<davexunit>yeah they are neat
<davexunit>though in practice I never use them...
<paroneayea>there was a paper I read recently advocating for "functional object oriented programming" which was pretty interesting
<paroneayea>basically saying all methods that did "mutation" should actually just return as their argument a new instance of themselves
<paroneayea>might be interesting to have a functional GOOPS? :)
<davexunit>I advocate for that when I write Ruby
<davexunit>at work
<davexunit>prefer attr_reader (getter) to attr_accessor (getter+setter)
<davexunit>return new instances of object, etc.
<davexunit>functional programming just clicked with me in a way that object oriented programming didn't.
<davexunit>problems like "I want to add a new method that operates on Array, but everyone says monkey patching is bad so what do I do?" just don't exist
<paroneayea>I like that the name of one of the companies which worked on Dylan was "Functional Objects"
<paroneayea>what a great name for a company.
<davexunit>hehe
<paroneayea>davexunit: btw, back again! on the topic of language packaging
<paroneayea>I was really surprised to find out that when I imported some modules, racket went and fetched the modules from the server and then eval'ed them
<paroneayea> http://blog.racket-lang.org/2014/12/the-racket-package-system-and-planet.html turns out there's kind of two package managers in racket, and it looks like the core stuff does this?
<paroneayea>thought you might find that curious
<davexunit>that's a strange thing
<davexunit>I can see how it would be considered convenient, but to me that's very unexpected behavior.
<paroneayea>yeah
<mark_weaver>please_help: you can write it as just +i, but better yet use 'make-rectangular'
<nalaginrut>morning guilers~
<mbuf>is there a hosting provider to install and run guile scripts?
<sneek>mbuf, you have 1 message.
<sneek>mbuf, dsmith-work says: see http://ossau.homelinux.net/~neil/ for some Guile benchmarks over git commits.
<civodul>Hello Guilers!
<paroneayea>o/
<paroneayea>hello, friends
<davexunit>hey paroneayea
<paroneayea>hi davexunit :)
<paroneayea>continuing with my activitystreams and guile experiments, I'm converting the full activitystreams vocabulary to GOOPS classes
<paroneayea>they aren't really useful / usable
<paroneayea>but I thought it would be interesting to see the whole set of properties and the inheritence system in sexps
<paroneayea>but maybe I could make them more useful, we'll see. For now, it's just an exercise.
*davexunit pokes at some procedures to read/write encrypted files.
<paroneayea>ooh
<paroneayea>davexunit: what for?
<davexunit>for my latest case of NIH syndrome: shroud https://git.dthompson.us/shroud.git
<paroneayea>davexunit: oh hey neat
<paroneayea>davexunit: I recommend looking at assword for inspiration
<davexunit>a dirt simple password manager. it's just an encrypted s-exp.
<paroneayea>it's simple (just python wrapping gpg encrypted json), works well
<paroneayea>one reallllly nice feature though
<davexunit>thanks, didn't know about that tool when I was searching.
<paroneayea>is that "assword gui" pops up a box, you enter the context, and it can dump the password into any window
<davexunit>probably should just use that instead.
<paroneayea>"assword gui" is really nice
<paroneayea>davexunit: well I have one caveat about assword
<davexunit>that's cool
<paroneayea>it's really slow to start up :\\
<davexunit>hmm wonder why
<paroneayea>so sometimes I have to wait 20 seconds to select my password
<paroneayea>probably python importing a bunch of heavy gtk libraries
<paroneayea>might be nice to have daemonized or something
<davexunit>here's the meat of the shroud ui: https://git.dthompson.us/shroud.git/blob/HEAD:/shroud/ui.scm#l140
<davexunit>load config, load db, pass both to a subcommand procedure along with command line args.
<davexunit>subcommand proc returns a db (a list of <secret>) and it gets written to disk if its different than the original.
<davexunit>secrets are just key/value pairs, but perhaps I should add more fields. does assword have useful metadata fields?
<davexunit>I was thinking that maybe a description would be useful. dunno.
<paroneayea>davexunit: it doesn't
<paroneayea>and it's annoying
<paroneayea>I often wonder the context of, what was my username here?
<paroneayea>etc
<paroneayea>davexunit: I think shroud could be useful, and I might be interested in using it if it provides those other features
<paroneayea>you may want to look at assword for inspiration regardless
<davexunit>yes, thanks.
<davexunit>perhaps I should just allow for arbitrary key/value pairs
<paroneayea>it's probably a good idea
<paroneayea>speaking of databases of sexps
<paroneayea>I am considering the maybe crazy idea of trying to see how well the SLIB relational database works
<paroneayea>for storing activitystreams stuff
<paroneayea>it seems really interesting
<paroneayea>from the docs anyway
<davexunit>neat. I know nothing about that.
<davexunit>I am going to use a database of s-exps for 'guix deploy's state file format, which will avoid an issue people have with nixops :)
<paroneayea>davexunit: what's the issue?
<davexunit>the nixops state file is a sqlite db, which can't be meaningfully shared among many systems.
<paroneayea>ah
<davexunit>so people feel compelled to use a central computer for all deploys.
<paroneayea>davexunit: would it be possible for the state structure to even be generated by hand then?
<paroneayea>maybe you wouldn't want to do it
<davexunit>for nixops?
<paroneayea>yeah
<davexunit>if you generate the proper sqlite file, sure.
<paroneayea>oh I mean for guixops
<paroneayea>sorry
<davexunit>yeah, you'd just write the sexp
<paroneayea>that's cool
<davexunit>and since it's plain text, it can be version controlled and shared amongst machines.
<paroneayea>awesome
<davexunit>so anyone could deploy from their workstation provided they have an up-to-date state file.
<davexunit>anyway, if I can get the very basics of shroud working I'll release a 0.1 and see if anyone shows up to help improve it.
<davexunit>NIIMBY - Not Invented In My Back Yard
<davexunit>guile bindings for this would be cool https://www.gnupg.org/related_software/gpgme/
<davexunit>how do I write the eof object to a port?
<davexunit>I'm trying to write to a pipe and then read the output generated
<davexunit>but I need to terminate the input somehow
<davexunit>without closing the output
<please_help>for modules, if I have root/foo/bar.scm where bar.scm has (define-module (root foo bar) ...), how do I get it to #:use-module other modules from module root, such as #:use-module (root baz bat) ?
<mark_weaver>please_help: %load-path (initialized from GUILE_LOAD_PATH) needs to include the absolute path name of a directory that contains 'root'
<mark_weaver>i.e. it needs to contain some <DIR> such that <DIR>/root/baz/bat.scm exists
<mark_weaver>please_help: also see %load-compiled-path and GUILE_LOAD_COMPILED_PATH
<mark_weaver>davexunit: the eof object is not something that can be read or written over a port
<davexunit>mark_weaver: that was my conclusion after trying various things.
<davexunit>I'm just not sure how to actually get the behavior I want and avoid a deadlock.
<mark_weaver>for a pipe, you can "send" eof by closing the write side of that pipe
<mark_weaver>terminals are unusual in that it is possible for a read to return EOF but then more input appears after that.
<davexunit>mark_weaver: I tried that method, but then ran into a problem with close-pipe because it wants to close both input/output sides of the pipe.
<mark_weaver>so in that case, it might be cleaner in some sense to consider it part of the stream, but it's actually handled in a more adhoc way. it's kind of a mess actually :-/
<davexunit>I'd still like to use close-pipe since it terminates the opened process.
<mark_weaver>davexunit: bah, I know, that's a long-standing issue that we really need to deal with :-(
<mark_weaver>where we probably == I
<davexunit>heh
<artyom-poptsov>davexunit: Hi. AFAIK one could send the EOF to a port using R6RS I/O Ports API.
<davexunit>mark_weaver: just to explicitly state my use-case, I want to write text to a pipe for gpg to read and encrypt, and then read the encrypted text into a string.
<davexunit>or write the encrypted text returned to a file.
<mark_weaver>davexunit: yes, this is a common issue. I'm sorry to say that until we add support to close half the pipe in guile, your best bet is to write either the input or the output to a temporary file instead.
<davexunit>mark_weaver: okay.
<mark_weaver>I know it's terrible. sorry about that :-(
<davexunit>thanks. if I may ask, are there complications with making close-pipe DTRT?
<davexunit>or is the blocker just free time?
<mark_weaver>well, the API would have to be extended, and it's not entirely clear to me what's the best way to do that.
<mark_weaver>the current approach of combining to two halves of the pipe into a single port is not good for a few reasons, partly that the current implementation is extremely inefficient
<mark_weaver>but more fundamentally that I really don't think input side to a subprocess is related to the output side of the subprocess. I think it's just the wrong way to think about it.
<davexunit>agreed.
<mark_weaver>so I'd prefer to add a new API procedure that returns two ports instead
<davexunit>unrelated, but for convenience it would also be nice to have call-with-{input,output,input-output}-pipe procedures that did a dynamic-wind
<mark_weaver>also, it needs to be all be very robustly thread-safe. I fixed the thread safety problems in popen a while ago but never added regression tests for that, but at this point it's important for something that daviid is doing, so we can't break it.
<mark_weaver>davexunit: if you'd like to take this on, I'd be all over that, by the way :)
<mark_weaver>but if not, no worries, I understand :)
<davexunit>mark_weaver: I'm not sure I feel up for it right now. as far as guile core hacking goes, I'd like to focus on that netlink socket patch I posted recently.
<mark_weaver>fair enough :)
<davexunit>netlink socket support on Linux will enable a pure Guile container library to be written.
<davexunit>which would hopefully become part of Guix at some point.
*mark_weaver looks again
<davexunit>the patch I posted is just a first stab at it. I'm unsure about some details, I didn't update the docs for all necessary procedures yet, and there are also no tests.
<davexunit>but I'd like to save that work until I have confirmation that the general approach is OK.
<daviid>mark_weaver: davexunit yes, just to confirm, i just shortly parsed your conversation, but popen being thread safe is absolutely essential to the work I'm doing, and it's being used by the 'my' lab clients as well
<davexunit>daviid: roger that.
<mark_weaver>davexunit: I think it's probably better to use 'memset' to clear the entire 'sockaddr_nl' structure instead of setting the 'pad' field to zero explicitly.
<davexunit>mark_weaver: okay.
<mark_weaver>e.g. see the AF_INET case
<davexunit>could you mention that in an email response so I can refer to it later?
<davexunit>much appreciated.
<mark_weaver>will do
<mark_weaver>sorry for taking so long on this
<davexunit>mark_weaver: no problem.
<davexunit>I'd like to get this ready in time for the next release, if possible.
<mark_weaver>yes, we'll make sure it gets in
<davexunit>I need to keep chipping away at what's needed to create a useful container from Guile.
<davexunit>the only other thing I noticed that Guile might be missing is a way to use the 'clone' syscall.
<davexunit>instead of calling clone, I'm using fork + unshare to achieve the same (not sure if it's 100% the same, though) result.
<mark_weaver>davexunit: remember the dynamic FFI
<davexunit>mark_weaver: the 'clone' function in glibc isn't suitable, as far as I can tell.
<davexunit>it takes a C function pointer, and a pointer to a memory block for a new C stack
<davexunit>maybe it's fine, but it doesn't seem right.
<mark_weaver>it might be that we need to do it in C, but probably not because of the two issues you just mentioned
<davexunit>okay.
<davexunit>the function would do something similar to what primitive-fork does, I imagine.
<mark_weaver>I seem to be forgetting the details right now, but for example part of our 'popen' implementation needs to be in C because of issues around threads, mutex locks, and fork.
<mark_weaver>see the comment above 'scm_open_process' in posix.c
<davexunit>okay
<davexunit>I'll see if I can write a naive 'clone' Scheme procedure and post a patch.
<mark_weaver>I think it goes something like this: programs cannot safely use threads and fork in the same program. I think it has to do with locks and the fact that the fork will copy the address space including all the lock structures, and then in the child process all of those threads are suddenly gone and may have left locks locked or in an inconsistent state.
<davexunit>thanks for the explanation.
<davexunit>that sounds familiar
<mark_weaver>but actually you can use 'fork' safely in a threaded program if you immediately 'exec' in the child
<davexunit>read something similar awhile back
<mark_weaver>where "immediately" means, more precisely, that only async-signal-safe functions can be used in the child before exec'ing.
<mark_weaver>and so we actually can't build our 'popen' module on 'fork' alone, at least not in a program that uses threads.
<mark_weaver>we need to build it on a more complex function 'scm_open_process' that handles doing both the 'fork' and 'exec'.
<mark_weaver>davexunit: and it might be that for similar reasons, you need to build something higher-level than 'clone', in C.
<mark_weaver>and that might have a side benefit, if this higher-level construct could be made portable to some other kernels besides linux.
<davexunit>mark_weaver: it might be a Linux only thing, given that 'clone' creates new Linux namespaces according to flags in a bitmask.
<davexunit>see the function declaration in 'man clone' for why I was concerned about how it might be problematic to call it from Guile.
<davexunit>fn and child_stack
<mark_weaver>davexunit: well, GNU Hurd should be able to implement something similar, in an even more natural and robust way.
<mark_weaver>and it would be great if somehow the API was somehow implementable in the Hurd.
<civodul>davexunit: re shroud, see also auth-source.el in Emacs
<mark_weaver>well, I'd be curious to hear what civodul thinks.
<civodul>i'm not sure i followed everything, but clone bindings are probably needed anyway
<civodul>then i guess the container API would provide something higher-level, which would leave room for a Hurd implementation
<civodul>that must be kept in mind
<mark_weaver>sounds good to me.
<civodul>but it's also a good thing to be pragmatic and get something working first ;-)
<mark_weaver>yeah :)
<davexunit>civodul: so a linux-only 'clone' binding?
<civodul>davexunit: yes, at least to begin with
<davexunit>do you see what concerns me in the function declaration? is it misguided fear? :)
<mark_weaver>davexunit: well, as for the C function pointer, there's 'procedure->pointer'.
<davexunit>yeah, I know. but I know that there can be tricky business when interfacing with C code.
<davexunit>and there's this whole new stack business.
<mark_weaver>and for the stack block, you could either use bytevector->pointer, or maybe call 'malloc' if the stack shouldn't be in the GC heap (I haven't really thought about it).
<davexunit>if it's totally fine to do via the FFI, I will just do that.
<mark_weaver>the main thing about the stack is that the garbage collector needs to know what's going on.
<civodul>another issue with 'clone' in libc is that it's variadic
<civodul>but yeah, like mark_weaver says, you can always give it a try and see what happens :-)
<mark_weaver>civodul: well, you can call 'pointer->procedure' dynamically.
<civodul>dynamically<?
<mark_weaver>davexunit: so, there's a bigger issue with the stack
<mark_weaver>civodul: call it once for every N, where N is the number of arguments passed to 'clone'.
<mark_weaver>or maybe even once per call to clone.
<civodul>oooh, i see
<civodul>you pre-compute bindings for several values of N, right?
<mark_weaver>civodul: the lazy way would be to call 'pointer->procedure' every time 'clone' is called, and let it be garbage collected. although I confess I don't know if the resources allocated by 'pointer->procedure' can be fully reclaimed.
<mark_weaver>the better way would be to keep a cache of those procedures.
<mark_weaver>but that has other issues, like making sure the cache is accessed in a thread-safe way
<please_help>so I have 3 libs (one per backend): libthea.so, libtheb.so and libthec.so. They're all in the same folder, which is in the PATH. If I (dynamic-link "libthea") -> OK. (dynamic-link "libtheb") -> OK. (dynamic-link "libthec") -> ERROR: file not found "libc"
<please_help>how come?
<davexunit>civodul, mark_weaver: the variadic stuff doesn't seem that useful from our perspective.
<mark_weaver>please_help: if anything at all goes wrong with 'dynamic-link', you get that same error.
<davexunit>it's just args to pass to 'fn'
<please_help>ah ok
<mark_weaver>please_help: that's because of a limitation in libltdl
<please_help>any way to tell what's failing?
<davexunit>we have lexical scope and closures, so why not just expect a thunk?
<civodul>mark_weaver: ah sure, that would work as well
<civodul>mark_weaver: sure, my concern was that libffi didn't (and so Guile doesn't) support the variadic ABI
<mark_weaver>please_help: running it in 'strace' might provide some clues
<civodul>err, davexunit ↑
<davexunit>civodul: okay
<civodul>the thing is that "in practice, it works"
<civodul>but i shouldn't say that ;-)
<mark_weaver>davexunit: so, regarding 'clone', if you intend to run Scheme code in the child process, I guess you need to make sure that Boehm GC knows what's going on, including knowing where the new stack is.
<mark_weaver>civodul: any thoughts on that? ^^
<davexunit>mark_weaver: yes, the whole point is to run Scheme code. my expectation as a user is for it to behave like primitive-fork.
<davexunit>so I was right to be suspicious of the stack stuff.
<mark_weaver>davexunit: right, but 'clone' seems to handle the stack differently. with 'fork', the stack in the child is in the same place as it was in the parent, so there's nothing to do.
<mark_weaver>davexunit: but with 'clone' it seems that a fresh stack is created, at a user-specified address.
<mark_weaver>Boehm GC needs to know where all the stacks are to do its job properly.
<davexunit>yeah. that's the bit that I'd like to hide away.
***DerGuteM1 is now known as DerGuteMoritz
<davexunit>mark_weaver: is it easy tell the GC that information using the C API?
<civodul>mark_weaver: probably, but i suspect in the case of containers, the code would probably exec right away?
<mark_weaver>civodul: davexunit intends to run scheme code in the children
<mark_weaver>davexunit: I'm sure it's not hard to tell the GC about it, but it may be that you need to consider several cases depending on what flags are passed to clone.
<mark_weaver>for example, the man page says that 'child_stack' can be 0, in which case COW semantics are arranged, like with fork.
<davexunit>this is quite the rabbit hole :)
<mark_weaver>I don't know enough about 'clone' and what it can do to fully explore this.
<davexunit>the other open question is, how much memory to allocate for the stack?
<mark_weaver>but if you intend to run scheme code in the children which inherits the parent's address space, then care will need to be taken that the GC is up to date on what's going on in the new process.
<mark_weaver>it's not clear to me that exposing the entire raw functionality of 'clone' in a single Guile API procedure is the right thing to do.
<mark_weaver>I'm not sure we shouldn't, I just don't have enough information to decide at this point.
<mark_weaver>but if it's possible to use it sanely from the dynamic FFI, then it reduces the pressure on us to add it to Guile core.
<mark_weaver>davexunit: it might be that for the case you need, passing 0 for 'child_stack' is okay, in which case we can punt on that issue.
<mark_weaver>the clone(2) man page explains the rationale for the 'child_stack' argument: Since "the child and calling process may share memory".
<mark_weaver>but for creating containers, I would think you wouldn't want them to share memory anyway.
<davexunit>mark_weaver: well, how about I just use the FFI for now in my own code with a child_stack of 0 and see how that goes.
<mark_weaver>davexunit: sounds good!
<davexunit>I'm still not sure what the difference is between using clone like this and fork+unshare
<davexunit>but I think it's better to use the intended syscall
<mark_weaver>civodul, davexunit: having thought on it a bit more, I notice that there are currently very few Linux-specific APIs in Guile. I'm becoming uncomfortable with the idea of adding several new APIs that would tempt users to write code that will not work on the HURD, especially since the HURD has its own arguably superior approach to supporting containers.
<mark_weaver>I would be much more comfortable finding an API that does what users need but abstracts away the Linux-specific parts, so that it can be implemented on the HURD later.
<civodul>mark_weaver: for Guile itself, i agree
<civodul>but i thought davexunit was talking about an FFI module outside of Guile proper?
<civodul>davexunit: one different in clone vs. fork+unshare is the PID of the child process
<mark_weaver>well, either way, it would be good not to popularize an API that is Linux-specific.
<civodul>davexunit: CLONE_NEWPID allows you to get PID 1 in the container
<civodul>mark_weaver: no, but as long as it's internal, à la (guix build syscalls), that's ok
<mark_weaver>right, internal APIs are fine
<civodul>mark_weaver: the container module that davexunit showed a while back intends to provide something higher-level anyway, like a 'with-container' form
<mark_weaver>just as long as it doesn't become popular to write code to that Linux-specific API
<civodul>nobody would want to use clone(2) in daily life ;-)
<civodul>yes, agreed
<davexunit>mark_weaver, civodul: what about netlink sockets? are those still eligible for inclusion in Guile's socket API?
<mark_weaver>davexunit: that's part of the reason I brought this up. I'm not sure how I feel about it.
<mark_weaver>I wonder if that could be done via the dynamic FFI as well.
<mark_weaver>actually, I confess that I've been wondering if we shouldn't rewrite many of our C bindings in Scheme using the FFI.
<mark_weaver>and maybe even implement ports in Scheme
<mark_weaver>but maybe that's further off
<civodul>davexunit: i see that libc defines PF_NETLINK only for the Linux port, so i'm not sure
<civodul>i hadn't noticed before
<mark_weaver>and the header file is <linux/netlink.h>
<civodul>yeah
<civodul>davexunit: perhaps it's best to have bindings to libnetlink actually
<civodul>might even be easier
<davexunit>civodul: I'll take a look at it
<mark_weaver>davexunit: please let me know if you run into difficulties
<davexunit>mark_weaver: will do.
<davexunit>I wasn't aware of libnetlink before
<davexunit>this could greatly simplify things
<davexunit>thanks for hashing this out
<mark_weaver>np, thanks for all your great work around guix and guile :)
<davexunit>hoping it will all pay off some day :)
<civodul>davexunit: i found it in "man netlink", and "man libnetlink" seems to be OK
<davexunit>civodul: yeah, the API seems like what I need.