***holomorp1 is now known as holomorph
***karswell` is now known as karswell
<wingo>hoo, guile's ports code gives me conniptions <wingo>mark_weaver: i am cooking a patch around ports in guile. are you doing anything concurrently? am i stepping on toes? <mark_weaver>my idea for making the port code not crash was to move each buffer and its associated pointers into a separate structure. <wingo>great, that is what i am doing :) <wingo>also centralizing buffering in ports.c -- right now each implementation has to re-invent how to do buffering <mark_weaver>so when a buffer was resized or switched (e.g. when switching to/from the putback buffer), the switch would happen from just changing the pointer from the main ports structure to the buffer struct <mark_weaver>which would be an atomic operation in practice on just about any architecture <wingo>i think there are still some concurrency concerns regarding the pointers that walk the buffers -- could be that a half-visible update could invalidate some invariants that two pointers into port buffers are in a certain order <mark_weaver>(I would avoid using guaranteed atomic operations, because those will be very slow) <wingo>getting that right involves compiler barriers i think <wingo>not sure about slowness of atomic ops -- keep in mind that we currently use a recursive mutex which is like the worst speed penalty <mark_weaver>the difficulty I see is when creating a new buffer structure <wingo>could be that atomics are much cheaper; dunno tho <mark_weaver>I really think that port operations like put-u8 and get-u8 should avoid any kind of synchronization at all. <mark_weaver>just recently, I was writing some code that would have been a *lot* nicer if I could have just used get-u8 and put-u8, but instead I had to avoid those because they are so slow <mark_weaver>and part of that has to do with the way that line numbers and column numbers are kept track of <wingo>i thought get-u8/put-u8 avoided line-number things <wingo>e.g. difference between scm_c_write and scm_c_lfwrite <wingo>strangely named things from the last millenium <mark_weaver>although really it would be nice is read-char and write-char were as fast as we can make them as well <wingo>my main interest in all this is concurrency <wingo>i want to be able to suspend a thread when a write would block <wingo>still not sure how the final picture looks <wingo>yeah, for something like that <wingo>but i don't want *two* kinds of ports <wingo>hence this refactoring to be able to reason about port buffers in a more specified, standard way, one which has the possibility of avoiding trampolining through c also <mark_weaver>so, from the idea about separate buffer structures that I've mentioned, the difficulty I see is when creating and destroying port buffers <mark_weaver>well, destroying is actually probably not an issue, because BDW-GC should take care of that by stopping all threads before a gC <mark_weaver>but when creating one, it's possible that another thread will see the pointer to the buffer structure written to the main port structure before the buffer structure has been initialized. <wingo>that's impossible on intel fwiw <mark_weaver>now, I think that might be addressable by relying on the fact(? needs verification) that BDW-GC arranges for newly allocatd buffers to be zeroed <mark_weaver>wingo: intel is not enough, guile needs to be portable <wingo>heh, no lecture needed, thank you :) <mark_weaver>I think the port buffer structure should store the "pointers" as integer offsets instead of actual pointers. <wingo>yes! i am doing that in my patch <mark_weaver>and that way, the worst that can happen is that there will be zeroes, which are still valid. <mark_weaver>the hope is that BDW-GC zeroes the freed memory during GC, when all threads are synchronized <wingo>i am not sure how real this concern is, relative to other kinds of allocations <wingo>i.e. allocate object X and pass it to thread Y <wingo>does Y see a fully initialized object? <wingo>you seem to be more sure about this than i am <mark_weaver>not necessarily in the absence of barriers in both threads <mark_weaver>the thread that allocates the object needs a barrier after initializing the object and before storing the pointer to it <mark_weaver>and the thread that reads it needs a barrier after reading the pointer but before reading the object <wingo>b/c this is all very system-specific <wingo>so we should be specific about which memory models we are working on <wingo>ACTION should read up on that <mark_weaver>makes me want to avoid shared-mutable-memory multi-threaded programming like the plague. <wingo>maybe you will like an ethreads-like system more ;-) <wingo>i miss a good concurrency story in guile. <davexunit>I much prefer cooperative multitasking to multithreading <mark_weaver>I know that other people want this, and we need to do our best to support it, but I myself never want to use it <wingo>we have pthreads but... hoo, difficult to say anything about it <mark_weaver>anyway, it sounds like you are covering all the bases I hoped to cover, and I'm glad to have one less thing on my TODO :) <wingo>yeah i'm happy that we're on the same page, that i'm not stepping on your toes here <mark_weaver>there are other things I want to do later to make textual I/O faster, but that's orthogonal I think <wingo>i think i won't get around yet to removing the locks <wingo>but i think this work will get us closer <mark_weaver>e.g. I want port buffers for textual I/O to be UTF-8 always <wingo>yeah! i was looking at that, but i didn't manage to understand it <wingo>like, to understand all the implications <mark_weaver>and maybe change the semantics of the line/column counters to be not the current port position, but some other position in the buffer, so they don't have to updated on every operation <wingo>anyway, there's a flag and a fast path, which is not too bad <wingo>you need a slow path anyway to call out to read(2) / write(2) / whatever the port uses <mark_weaver>right, but it would be good to minimize the number of conditional branches <mark_weaver>so, you need one conditional branch to check if the buffer is full <wingo>hmm, not sure that's the bottleneck fwiw <wingo>you have the whole scheme -> c -> type-checking -> ... <mark_weaver>well, sure, things like read-char, write-char, get-u8, put-u8, peek-char, should ideally be VM ops <wingo>i think i disagree there b/c of need to suspend calling continuation <wingo>yeah, but that can be expressed just as well in scheme... <wingo>if i build that kind of thing now it's going to be a barrier to native compilation <mark_weaver>mmm, sure, that's true. if the fast path is in scheme code, that could help a lot with the right compiler <mark_weaver>I have this idea that the representation of the port object should be optimized for the fast path, i.e. that the buffer should be the first thing you see, in a sense. <mark_weaver>it might be that this idea of a separate putback buffer that we need to switch to is just a bad idea <mark_weaver>why not just push back into the normal buffer, if there's space? <mark_weaver>maybe the port object should point directly to the buffer and then the buffer should point to the rest, dunno. <mark_weaver>I haven't really thought this through, just free thinking now <mark_weaver>bidirectional buffers is a complication here as well <mark_weaver>just recently, I write some code for guix to optimize the grafting process <wingo>yeah my patch has no separate putback buffer <mark_weaver>and I ended up reading blocks with get-bytevector-ref and then maintaining my own index and iterating over the bytevector <mark_weaver>and I was thinking to myself, it would be so much nicer if I could just use port operations directly, but I know that they are slow as hell. <wingo>does that use custom binary ports? <mark_weaver>I'd like to enable people to write natural code to read from one port and write to another. string ports also. <wingo>that would be even slower b/c scheme -> c -> scheme <mark_weaver>no, just normal file ports (or bytevector ports when grafting symlinks) <wingo>ACTION heads out for a bit, ciao! <mark_weaver>string ports are another example, which really makes me want UTF-8 as the universal port buffer representation, regardless of what encoding is used for the underlying device. <mark_weaver>hmm, actually we use string ports when grafting symlinks, but that's just because we have no way to read/create symlinks as bytevectors, although we should <dsmith-work>wingo, mark_weaver: Have you guys ever looked at the kfifo implementation in the Linux kernel? <mark_weaver>if it has some clever ideas, can you briefly summarize the ones that seem applicable? <dsmith-work>Uses offsets into the buffer instead of pointers, but lets them grow beyond the bufffer size. Always uses them by anding with the buffer size, which must be a pwoer of 2 <dsmith-work>Not sure if it applies much to Guile port buffers. Might be some interesting ideas there though. <mark_weaver>hmm, it may be that I'm not getting enough of the details to understand. <mark_weaver>there's still the issue of thread B seeing thread A's writes to the buffer pointer before seeing A's writes to the buffer itself. <dsmith-work>AH yes. Nevermind. Only applies when there is a single reader or writer. <mark_weaver>and there's also the question of how the writer avoids writing too much <mark_weaver>i.e. how it avoids going beyond the reader pointer, which it might see an old copy of. <mark_weaver>also, not sure how to increase the size of the buffer safely without barriers in the fast path. <fhmgufs>I would like to see an example project that makes use of GOOPS a lot including subclassing and so on. <mark_weaver>dsmith-work: okay, it looks like the main trick there is to avoid checking for the end of the buffer, essentially by using mathematical rings instead of unbounded integers. I've used that trick myself before linux even existed :) I don't see any mention of tricks involving thread synchronization, though. <fhmgufs>I would really like to see some real world examples of GOOPS in use. <fhmgufs>Is there something that could give me the chance to fulfill this wish? <alezost>fhmgufs: shepherd uses goops for services <fhmgufs>alezost: Ok, thanks a lot, I'll look into it. <fhmgufs>Is it okay to use (unless ...) if a procedure either returns #f or a value different from #t? <fhmgufs>I mean to test whether it hasn't returned #f. <rain1>in scheme #f is false and everything else is true <mark_weaver>right, like rain1 said, in scheme, things that expect a boolean will treat anything thats not #f as true. (unless <expr> <body> ...) will evaluate <body>... if <expr> returns #f. <mark_weaver>actually, guile adds another value that's treated as false, namely #nil, but you should avoid that. it's for the sake of guile-emacs. <wingo>paroneayea: the "web page" link for 8sync from savannah is broken <wingo>i got that first address from an internet search <paroneayea>the real "webpage" for now; I need to switch over to savannah, and also make a real webpage <wingo>do you have an idea of how 8sync relates to ethreads? <paroneayea>wingo: so, let me tell you how 8sync *currently* works. <paroneayea>wingo: right now the scheduler does smart things for suspending / resuming delimited continuations <paroneayea>wingo: and also, at the end of the function you can request that some callback is run, or pass in a request to set up a port with some events on read/write/error stuff <paroneayea>in fact the delimited continuation stuff is just a wrapper around that <paroneayea>wingo: it sets up a promise to resume the function after whatever it requested finishes <wingo>does 8sync use its own i/o operators to allow suspending? what is the relation to guile ports? <paroneayea>wingo: so right now it uses (select) on any guile ports that support it <paroneayea>I've been plannign to switch it to poll or epoll <wingo>paroneayea: but 8sync doesn't set the ports as nonblocking, does it? <wingo>and it does blocking reads, right? <wingo>after having used select() to see which ones to do i/o on <paroneayea>wingo: yes-ish, or I guess it's slightly undefined :) <wingo>is 8sync all scheme or is there some c? <wingo>ACTION too many questions, shuts up again :) <paroneayea>wingo: but I'l admit, I barely know what I'm doing. <paroneayea>wingo: it's "vulnerable" to unresumable continuations though from C code :) <paroneayea>wingo: so anyway, 8sync uses select, and that's not great. It doesn't specify what you set with fcntl or whatever, and maybe it should <paroneayea>wingo: it usually does blocking reads / writes in chunks in most demos I've done <wingo>well chunking reads and writes is the right thing <wingo>but for reliable concurrency you will want nonblocking i/o <wingo>it sounds to me like our work is complementary <paroneayea>would it be a different option than select/poll? <wingo>you can understand ethreads in 4 commits. <wingo>they are the first 4 ones on wip-ethreads <wingo>adds nonblocking i/o operations in an (ice-9 nio) module <wingo>the name is modelled after java, wheee vomit wheeee <wingo>they are nio-read, nio-write, etc <wingo>they pass a "nonblocking" flag to the operations. <wingo>it's a way to avoid interacting with ports which have a bunch of baggage <wingo>but now i am trying to reconcile them <wingo>because the end result is a parallel i/o system <wingo>which is not what we want :) <wingo>so consider this like a mushroom that blooms <wingo>eports provide buffers in scheme space <wingo>or the buffers in stdio FILE* <wingo>basically, a way of buffering reads and writes in user-space to avoid too many system calls <wingo>when the read buffer doesn't have any more data and you go to read, <wingo>or when the write buffer is full, <wingo>then it does an nio-read or nio-write <wingo>the nio-read is more like a libc read <paroneayea>ok, so it doesn't do it immediately, it queues it more or less? <wingo>it returns the number of bytes that are able to be read without blocking <wingo>yeah an eport is like a normal scheme port, but the buffering is done in scheme instead of c <wingo>the "batch your writes or reads in a buffer" strategy is very old and fundamental for perf <wingo>it's what libc's "fopen" does <wingo>if you have done c stuff, an analogy could be: <wingo>eport : file descriptor :: FILE* : file descriptor <wingo>if the nonblocking read doesn't actually read anything, <wingo>or the nonblocking write can't write, <wingo>then it calls the... /me checks <paroneayea>(ok, I have not done much C stuff, but I think I follow anyway) <wingo>this is a parameter in the scheme sense <wingo>ethreads eventually can wire this up to suspend to the ethread scheduler <wingo>but that's getting ahead of myself <wingo>so, eports: buffered non-blocking i/o with dynamic "waiter" parameters <wingo>the next commit add a wrapper for epoll <wingo>it's similar to (ice-9 poll) <wingo>in that regard it's not very interesting. <wingo>the more interesting bit is in the next commit <wingo>ethreads are a convention for how to wire up eports to delimited continuations, and a scheduler <wingo>here i suspect that 8sync is better <wingo>but ultimately something like this should land in guile eventually <wingo>but it can do so incrementally <paroneayea>ok, so it sounds like the first three things, at least <wingo>yes and i don't feel a need to maintain my own thing, in this regard <paroneayea>wingo: what would be good would be for me to get out a 0.1 release with docs <paroneayea>enough for you to be able to see how things work <wingo>i want concurrency without parallelism <wingo>(i also want it with parallelism, but that's another question) <paroneayea>wingo: right. so there are no threads in ethreads, right? :) <wingo>so ethreads and 8sync are filling this use case <wingo>but i think it would be an error to add ethreads to guile right now <wingo>because who wants the concept of an "eport" <wingo>so that's what i'm working on right now <paroneayea>wingo: so, I'm trying to think of how we can best help each other from here :) <paroneayea>wingo: I guess I could make a release of 8sync once I get back, maybe next week if all clears up on the mediagoblin release this week and I can find a little bit of time <paroneayea>but maybe then also try putting 8sync on top of nio / eports / epoll ? <wingo>well one thing to clear up would be to get an idea of what should be in guile and what should be in an add-on package <paroneayea>wingo: I'd be okay with 8sync not being independent. <wingo>because if only a little bit should go in guile, then 8sync as a brand makes a lot of sense <paroneayea>wingo: the main features 8sync have going for it as an external thing are a nice logo and hopefully catchy name ;) <wingo>but if most of it should go in guile, then we probably need a more generic name for this facility. but i am really unclear on this <wingo>paroneayea: don't forget, it has you :-)) <wingo>so i would like for once to not stand in the way of other people doing cool things ;) <wingo>ACTION does too much gatekeeping, i think <wingo>another question relates to guile 2.2 or 2.0 <rain1>dude that logo is out af this world <wingo>if you want to target 2.0, then i would say, keep 8sync as a thing, we can't work much together <wingo>this is the only thing blocking the release imo <wingo>we need to take the opportunity to get this right <paroneayea>wingo: it's easy enough to "play with" guile 2.2 now that we have a guile-next package in guix :) <wingo>or at least to lay a foundation <paroneayea>wingo: I agree that making this possible is pretty important :) <paroneayea>guile needs an answer to the non-blocking event loop question <wingo>the problem with select-then-read or s/read/write/ is that you are racing the select and the read/write <paroneayea>wingo: oh, btw, possible porkbarreling opportunity, now that you're messing with ports stuff <paroneayea>wingo: the tls-in-guile-proper stuff is being blocked by something where civodul or mark_weaver (I forget which) determined "oh no we're missing something in binary ports" <paroneayea>because a language without https, today, is pretty strange <wingo>gnome with full-screen windows is pretty great... firefox full-screen is delightful <wingo>apropos of finally having working gnome in guix :P <paroneayea>. o (we should get some screenshots on the guix website showing off gnome support) <wingo>i think this work will make custom binary i/o ports easier <wingo>er, the refactorings i am working on <wingo>but it won't magically fix it <wingo>so please bother me when/if those refactorings land <paroneayea>wingo: one more pork tossed in the barrel of this conversation! <paroneayea>wingo: so I have commit access to guile, and wip-elisp is up <paroneayea>wingo: what should I do next, or is it waiting on a review from you? <wingo>oh great, sorry my savannah credentials are elsewhere <wingo>and no gnome-keyring in guix yet so i haven't gotten around to finding them <wingo>paroneayea: lemme fetch, i can take a look now. if there are no changes to the rest of guile, there is no issue at all. if there are it will take a little more time. <paroneayea>wingo: ok, I'm also saving notes on this conversation :) <wingo>hmm, will take some more time then for the elisp review :) <wingo>yeah thanks for all your work chris :-) <rain1>I sent a release of my guile-zenity library to the mailing list <paroneayea>wingo: that's fine... yeah I saw that there's a lot of stuff changing in the actual VM type things :) <wingo>ACTION goes back to work on the port refactor <rain1>is this the best guile-irc library? <janneke>paroneayea: wow, that's interesting! <janneke>i often wonder what the `feature' is of allowing a variable to change types <paroneayea>janneke: the article on the extreme performance penalties of Typed Racket is interesting <paroneayea>I think they cite some of the code as 180 times slower than untyped racket <janneke>i work with a c++ guru at work (he learns to love scheme too) <janneke>but it's always embarassing to find an error that the compiler would have caught <paroneayea>yes, in some ways having a typed system would be nice, even though it slows down development, but it seems in the current world you have to go "all or nothing" on types <paroneayea>and if you like "REPL driven development", that probalby throws you at "nothing" <janneke>yes, i would only want to sacrifice compile time, not runtime <paroneayea>that said, sometimes when I see Elm people post the exceptions that system generates <janneke>it's okay to have to type `make' to get the compiler to help <davexunit>I actually really don't like Elm's error messages <ijp>the type diffing seemed neat <davexunit>maybe the error information itself is good, but I really don't like the super informal language <davexunit>Hey there, looks like you're trying to make a monad <ijp>at this point, presumably clippy links you to Oleg's work on Eff <ijp>I hope it becomes popular, it will be interesting to live in a post-monad-tutorial world <ijp>well, not the language Eff, but the Eff monad <ijp>we could even have a pool on when the first "left kan extension" tutorial gets released <cojy>davexunit: apart from the typing rules Eff is just the continuation monad <cojy>i implemented most of Eff the language as a macro in guile here <wingo>setvbuf is an incredibly wonky interface, blah <cojy>does guile still have call/cc as a primitive and has it ever been considered to be removed? <davexunit>call/cc is a part of standard Scheme, it couldn't be removed. <cojy>i just meant as a primitive, it could still be implemented in the library <wingo>cojy: i would like to allow it to be delimited <wingo>and, i would like to allow it to not capture the c stack, but that might not work <wingo>and i guess s/allow it to be delimited/allow the user to set the limit/ <wingo>b/c all continuations are delimited somewhere... <cojy>(define call/cc (lambda (p) (shift0 k (k (p (lambda (x) (shift0 k1 (k x)))))))) <cojy>i was playing around with this and it seems to work great <cojy>and all on the stack for the few tests i did <cojy>in racket it was having to GC like crazy <wingo>so i admit to getting lost a bit with shift and reset, somehow i understand call-with-prompt and abort-to-prompt better; a personal failing i guess ;) <cojy>samth claims this paper explains why they didnt do it but for some reason they never considered shift0 which fixes the issue.... <wingo>but the thing with call/cc is that its continuations don't compose. so you can make a call/cc that works as long as you have a prompt active "outside" your call/cc <wingo>but that's not a condition that holds for general scheme, afaiu <cojy>but it would allow you to remove call/cc as a primitive if you had that toplevel prompt in guile, right? <wingo>with the caveat that you wouldn't be able to re-instate continuations that capture the c stack <wingo>or maybe there could be some exceptions <wingo>like, reinstating the c stack within the dynamic extent of the call/cc or something <wingo>but re-instating continuations through c is becoming less important from a pure guile perspective, as we rarely recurse through c now <wingo>which used to be the case quite often <ijp>not sure if people using guile's C API care for it <wingo>you often actually don't want to be able to reinstate the c stack <cojy>i think it would be really cool to have guile to be the first scheme that implements it on top of the delimited framework <wingo>i suspect that the number of places that should have a continuation barrier but don't are much more common than actual users of reinstating the c stack <wingo>cojy: afaiu call/cc in racket is delimited, but i could be wrong <cojy>it's still it's own special thing in racket apparently, i brought this up with them a couple weeks ago and got linked that paper <cojy>wingo: is there any way to limit stack growth so runaway recursive functions don't blow up my computer? <cojy>i was never able to find how <wingo>search for call-with-stack-overflow-handler in the manual <wingo>you are the first person who has asked for that, so feedback is welcome ;) <roelj>Can I set a default value for a parameter in Scheme? <roelj>So that the parameter becomes optional <civodul>roelj: you can write (define* (foo bar #:optional (baz 42)) ...) 