IRC channel logs

2013-08-14.log

back to list of logs

<jmnoz>Hi, I'm having trouble with "syntax error: definition in expression context" when trying to use guile via Emacs (org-babel) when evaluating this code (define (add x y) (+ x y)). What am I missing?
<ijp>$2 says it's org-babel's fault
<ijp>do you have an example file?
<jmnoz>one moment
<didi>ijp: org-mode is doing this: <http://paste.kde.org/p0a233a3a/>
<mark_weaver>jmnoz: it looks like org-babel is not really set up to allow you to make definitions.
<mark_weaver>because it's wrapping your expression within (display ...)
<ijp>it must be doable
<ijp>because I'm pretty sure I've done it before
<mark_weaver>one solution is to change org-babel so that it wraps with something like (display (eval 'EXPR (current-module))) instead
<jmnoz>(my cut and paste is broken. (briefly) considering retiring from the field of computation altogether.)
<jmnoz>(that'll show'em! (shakes fist in air))
<jmnoz>mark_weaver: thank you
<mark_weaver>man, I really need to rewrite our pretty printer one of these days.
<mark_weaver>jmnoz: you're welcome!
<ijp>you can change :results from value to output
<mark_weaver>ah, that sounds promising.
<ijp>with the default ":results value" it wants to print the last expression
<ijp>which since definitions aren't expresions, is problematic
<ijp>but if you do that, you'll need to explicitly output the last value yourself (if you want that)
<mark_weaver>the (display (eval 'EXPR (current-module))) thing might be more convenient, simply because it allows both definitions and automatic output.
<ijp>and this advice turns out to be wrong in 5,4,3....
<jmnoz>I'm thinking unless this has been fixed in recent versions of org-babel maybe this would warrant a list e-mail
<mark_weaver>it would be great to get it fixed upstream.
<ijp>mark_weaver: do you fancy fixing that doc issue mentioned earlier?
<ijp>about append! and reverse!
<mark_weaver>I fixed the doc for 'append!'. Are the 'reverse!' docs also misleading?
<mark_weaver>ah yes, it is..
<ijp>cheers
<mark_weaver>ijp: thanks for the reminder
<mark_weaver>okay, this is better anyway: http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commitdiff;h=b57162c3d21fdc9e48572d7a9d4009c37f2c647a
<mark_weaver>now we say "@code{append!} is permitted, but not required, to modify the given lists to form its return." and ditto for reverse!.
<mark_weaver>ijp: btw, do you have any thoughts on my recent proposal (and preliminary patch) to improve support for R6RS exceptions in Guile?
<mark_weaver>rotty, weinholt: ditto ^^
<ijp>I'm going to be honest, I haven't read the ML in a few weeks
<mark_weaver> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14922#8
<mark_weaver>really, I'd prefer to modify Guile to use R6RS condition objects natively. our current system for reporting information about exceptions is totally ad-hoc and kind of a mess.
<ijp>I'm kind of tired, but I can look at it tomorrow
<mark_weaver>that's fine. no need to look at it at all, I just wanted to invite comments from folks who know R6RS better than I do.
<didi>jmnoz: I don't think it's a bug, although it's strange. Here is a version that works: <http://paste.kde.org/p3d3d378c>
<nalaginrut>morning guilers~
<sneek>Welcome back nalaginrut, you have 1 message.
<sneek>nalaginrut, taylanub says: C pre-processor macros like #if and #ifdef change the code that is compiled at all, so the produced executable doesn't have any of the code that was excluded, so one cannot switch between the options on-the-fly, or even by restarting. Only by recompiling.
<nalaginrut>sneek: later tell taylanub ah~thanks, I realized it
<sneek>Got it.
<nalaginrut>do we have any log server (not IRC log server), I mean a log-server written with Guile
<Chaos`Eternal>holy guilers
<Chaos`Eternal>I have one question for that whether the procedure open-process be re-written in scheme instead of C?
<wingo>Chaos`Eternal: it really can't; it has to only call async-signal-safe functions between the fork and the exec, and we can't guarantee that from scheme i don't think
<wingo>moin btw
<Chaos`Eternal>what is async-signal-safe functions?
<wingo>run "man 7 signal" in your terminal
<Chaos`Eternal>what about call call-with-blocked-asyncs ?
<wingo>that can help, but it's really tricky -- there are also vm hooks, and gc isn't async-signal-safe
<wingo>and whether a piece of code allocates or not isn't something that we can reason about currently
<Chaos`Eternal>well, do you have some code to demo the unsafy after a fork when multi-threaded?
<Chaos`Eternal>first, the child process wont harm the parent, since they seperate processes.
<wingo>it's mostly a problem in multi-threaded code
<Chaos`Eternal>second, the child process only has the context (let me say so) of the running threading which calling fork
<wingo>yes, and all other threads are killed, even if they are in critical sections...
<Chaos`Eternal>mostly the context is enough
<wingo>there is a difference between "mostly works" and "correct", and anything that does non-async-signal-safe calls after a multithreaded fork is certainly incorrect, as specified by posix
<wingo>search the mailing list archives for more, if you are interested
<Chaos`Eternal>hmm... can you give a keyword?
<wingo>e.g. http://thread.gmane.org/gmane.lisp.guile.devel/13830/focus=15466
<Chaos`Eternal>thanks a lot
<wingo>np
<nalaginrut>wingo: would you mind adding GUILE_STOP_ANNOYING env var? ;-)
<wingo>what would it do? :)
<nalaginrut>make Guile silent when it throws warnings
***chaos__ is now known as Chaos`Eternal
<nalaginrut>someone complains that
<Chaos`Eternal>no...
<Chaos`Eternal>i am not comlain about that
<Chaos`Eternal>what i am complaining is that the warning caused by some specific procedures should have the ablity to be supressed
<nalaginrut>OK, s/complain/request ;-)
<taylanub>Maybe we could have a `pragma' special-form ? :P
<sneek>Welcome back taylanub, you have 1 message.
<sneek>taylanub, nalaginrut says: ah~thanks, I realized it
<Chaos`Eternal>i am just requesting that the warning given by primitive-fork should have some way to be supressed
<wingo>from scheme you can (parameterize ((current-warning-port (make-null-port))) (primitive-fork))
<Chaos`Eternal>but the problem is that i need to enumerate each cases under which that primitive-fork is safe.
<nalaginrut>wingo: that was my answer too ;-D
<wingo>:)
<Chaos`Eternal>i don't like that.
<taylanub>Wouldn't that just suppress run-time warnings ?
<wingo>heh, understood ;)
<Chaos`Eternal>don't you guys think that there are some cases primitive-fork is safe even in multi-thread environment?
<wingo>not in general, no
<wingo>see that thread i posted: iconv locks, gc locks, etc...
<wingo>i used to be more optimistic but i am not any more
<Chaos`Eternal>understand your concern
<wingo>if you ignore the problem entirely, you get occaisional deadlocks
<wingo>so you have to deal with the problem
<wingo>but you can't, because you don't control all of the stack
<Chaos`Eternal>but in my case the fork is followed by an exec
<wingo>in that case why doesn't open-process work for you?
<Chaos`Eternal>only thing is that open-process is too simple to handle my case
<wingo>you need to do more with file descriptors or something?
<Chaos`Eternal>open-process is too pipe-oriented
<Chaos`Eternal>yes
<wingo>we should add that capability somehow
<Chaos`Eternal>so i have two choices
<Chaos`Eternal>one is make fork more safer
<Chaos`Eternal>one is enhance open-process
<Chaos`Eternal>i will try both way
<wingo>it is impossible to make fork safer :)
<Chaos`Eternal>don't agree
<wingo>dunno what to say then
<Chaos`Eternal>code talks!, haha
<nalaginrut>well, I partly agreed it's safer compared with threads
<taylanub>Let me slip in this question while we're talking about safety: Can one rely on car, set-car!, variable-ref, variable-set!, and similar things that set singular Scheme values, to be atomic even during parallelism, or could they result in partial writes and thus junk values ? In my tests they seem to behave well, but maybe it's architecture-dependent ?
<wingo>taylanub: architecture-dependent
<wingo>afaik
<nalaginrut>when someone mentioned something may has side-effects, 'safe' is gone away from my mind ;-D
<wingo>mark_weaver probably knows more
<taylanub>OK, thanks. From my understanding they compile to processor instructions that read and write single "words" of data, I guess the question boils down to the behavior of those instructions ?
<taylanub>(Well, we don't do native compilation yet, but still ..)
<wingo>the implementations should be native instructions, but really we have no guarantee about anything without synchronization primitives
<taylanub>OK. My lockless thread-safety ideas are in vain then. :P
<Chaos`Eternal>taylanub, lockless thread-safety ?
<Chaos`Eternal>you mean pure-functional?
<wingo>synchronization != locks
<nalaginrut>I think actor-model is lock free
<nalaginrut>(out of topic)
<Chaos`Eternal>but you still need lock to implement it
<Chaos`Eternal>my guile-thread-mailbox is almost finished
<Chaos`Eternal>i am moving toward guile-termite
<Chaos`Eternal>again off topic
<Chaos`Eternal>haha
<nalaginrut>based on threads is a bad idea
<Chaos`Eternal>disagree
<nalaginrut>which introduces lock again
<Chaos`Eternal>inter-process communicate is more heavy than interthread
<nalaginrut>but who mentioned IPC?
<Chaos`Eternal>inter-process communication introcuces not only locks, but also memory copies
<Chaos`Eternal>and data-[de]serializations
<Chaos`Eternal>noipc no future
<Chaos`Eternal>multi-core system is overwhelming
<Chaos`Eternal>your single process application can only utilize one core
<wingo>multi-process communication can use shared memory, but that makes it look more like a multi-thread system
<nalaginrut>yes, but I didn't say there's only one green-thread queue
<Chaos`Eternal>green-thread can still using one core
<Chaos`Eternal>multi-core means mutexes, or i learned a morden word: transactional memory
<Chaos`Eternal>haha
<nalaginrut>you are talking about shared-transaction-memory model, which is totally different thing with Actors
<Chaos`Eternal>they are mathematically equal...
<nalaginrut>everything computable things (in current machine) are mathematically, in principle
<nalaginrut>everything computable things (in current machine) are mathematically equal, in principle
<taylanub>Chaos`Eternal: Rather relying on the inherent atomicity of reading/writing singular Scheme objects (which, as just explained, is not really guaranteed), e.g. I had made this, which apparently would work fine on my platform but not necessarily all: https://github.com/TaylanUB/guile-modules/blob/master/one-to-one-q.scm
<Chaos`Eternal>The problem is , even if the mutation it's self is atomic, your whole thing is not thread-safe.
<taylanub>I'm pretty sure that it would be safe if variable-ref/set! and car/set-car! are atomic, although I can't mathematically prove it. :P
<taylanub>It's also limited to one producer and one consumer, though.
<Chaos`Eternal>let's say, when thread-a want append a new entry to list m, it must get the tail of m, then mutate the point to next entry of m
<Chaos`Eternal>let's say, when thread-a want append a new entry to list m, it must get the tail of m tail-of-m-a, then mutate the point to next entry of m
<Chaos`Eternal>at the same time, another thread thread-b also want to append an entry to m, but at this time, thread-a hasn't finished it's work
<taylanub>Hence, one producer.
<Chaos`Eternal>so, thread-b also get tail-of-m-a
<Chaos`Eternal>them they update the same position of memory
<taylanub>(Of course I don't do any checks to guarantee that no two threads can call enq!.)
<Chaos`Eternal>only the combination of operations: (set-next (get-tail-of-m) new-entry) be atomic
<Chaos`Eternal>then you can make it thread-safe
<Chaos`Eternal>but naturelly, the combination is not an atomic operation, it must be protected by mutexes
<taylanub>It must not be protected if there's only one thread doing it. :)
<Chaos`Eternal>but with only one thread, threadsafty is senseless
<Chaos`Eternal>the (ice-9 q) is enough
<taylanub>Two threads: one producer, one consumer.
<taylanub>One can only enqueue, the other only dequeue.
<Chaos`Eternal>what happens when deque and enqueue come to the same place?
<taylanub>Dequeue doesn't touch the null object at the end; enqueue prepares its contents while it still looks like a null object, then at once (via set-cdr! in this implementation) lifts its conceptual "is null" property. So if set-cdr! and cdr were atomic wrt. each other, it would work.
<Chaos`Eternal>the variable-ref in these two procedure will get the same object, but one will delete the object while another will append an object to the deleted object
<taylanub>No, enq! doesn't change the variable's value.
<Chaos`Eternal>there are still race-conditions
<taylanub>It follows the references in it to reach the null object, it can do this even if it gets an object that's then (immediately after) discarded by deq!, because this is done by setting the variable's reference to the same object which the first node was referencing.
<Chaos`Eternal>wait
<Chaos`Eternal>your enq! actually set-cdr!
<Chaos`Eternal>that is a change to the pair
<Chaos`Eternal>but the pair has already been delete by the deq
<Chaos`Eternal>s/delete/deleted/
<taylanub>No, the null object at the end cannot be deleted.
<taylanub>If it's the sole object in the chain, it means the queue is empty.
<Chaos`Eternal>your deq sets the entire pair to the next entry
<Chaos`Eternal>that's delete
<taylanub>It only does that to nodes that aren't null.
<Chaos`Eternal>yeah, i'm talk about the case when your queue has exactly one element
<taylanub>In that case deq! never mutates anything.
<Chaos`Eternal>ok, after the deq! your q still has the element?
<taylanub>Yes, that's the point of the "qnull" object.
<taylanub>It's not a singleton, it has instances that can be mutated into normal nodes.
<Chaos`Eternal>so the next call to deq will return the same element?
<taylanub>After enq! sets its contents (car), then sets its cdr to a fresh null object, deq! will now return the contents and set the variable target to the new null object.
<Chaos`Eternal>have you ever tried your code?
<taylanub>Nope, let me just test it .. should work on my platform, since my set-cdr! seems to be atomic
<Chaos`Eternal>ok, when there is one element in your q, the q would be this {variableof (value . '())}
<taylanub>It's not '(), it's a "qnull" object. Sorry for ambiguity, with "null object" I mean this special null object.
<Chaos`Eternal>{variableof (value . qnull)}
<taylanub>Yeah.
<taylanub>A qnull-terminated list. :P
<Chaos`Eternal>then the deq! , the car will get value, the cdr will get qnull
<Chaos`Eternal>right?
<taylanub>The variable's reference will be set to the qnull object that's there.
<taylanub>#<variable qnull>
<Chaos`Eternal>then , at the same time, enq! happens,
<taylanub>Yeah, it reaches the qnull, either directly through the variable (if variable-ref is executed after the variable-set! of the deq!), or through the pair (value . qnull) (if it's executed before), then prepares contents for it, then atomically (if set-cdr! is atomic) changes it so it will return #f for `qnull?' calls.
<Chaos`Eternal>after enq! finished , your q will be {variable (value . {variable (value2 . qnull)})}
<Chaos`Eternal>is that right?
<taylanub>No, only the queue itself is in a variable. It holds a qnull-terminated list.
<Chaos`Eternal>ok, you tell me when enq! finished what will q look like?
<Chaos`Eternal>so, you mean:
<taylanub>#<variable (value . qnull)>
<Chaos`Eternal>after enq! finished , your q will be {variable (value . (value2 . qnull))}
<Chaos`Eternal>this?
<taylanub>BTW I started two infinitely-looping threads, one always calling (q-enq! q 0), and the other (unless (or (q-empty? q) (= 0 (q-deq! q))) (display "panic!\\n")), and it ran fine for 10-20 seconds, after which it ran out of memory.
<taylanub>After two enq! calls, that'd be what it looks like, yes.
<Chaos`Eternal>ok,
<Chaos`Eternal>then, the deq! will finish it's work
<taylanub>It'll get value1, set the variable's reference to the pair (value2 . qnull), then return value1.
<Chaos`Eternal>no, the rest is now qnull
<taylanub>There were two values, no ?
<Chaos`Eternal>and the q will be {variable qnull}
<taylanub>deq! dequeues one item at a time ..
<Chaos`Eternal>let's run it again:
<Chaos`Eternal>given the q has one element, the q is {variable (value1 . qnull)}
<taylanub>y
<Chaos`Eternal>run deq!, the value of line 74 is value1, and rest of line 75 is qnull
<taylanub>y
<Chaos`Eternal>now enq! happens and finished. the q will be {variable (value1 . (value2 . qnull)) }
<taylanub>When did the enq happen ?
<Chaos`Eternal>the now deq! finishs its work, set q to rest, which is qnull
<taylanub>You seem to have confused something.
<Chaos`Eternal>before enq! happens and after enq! finishes.
<Chaos`Eternal>no, in multi-threaded environment, this is common
<taylanub><Chaos`Eternal> before enq! happens and after enq! finishes.
<taylanub>typo ?
<Chaos`Eternal>that's why you need lock
<Chaos`Eternal>no typo
<taylanub>No, somewhere you went wrong.
<taylanub>Let's rewind to the latest "y". :P
<taylanub>The state of the deq! call is, it's about to (variable-set! q rest) where rest holds qnull, right ?
<Chaos`Eternal>yes
<taylanub>Now deq! starts, it iterates through the linked list and finds the *same* qnull object to which the enq! state currently has a reference to, and sets its car to a value and its cdr to a new qnull object.
<taylanub>Now enq! finishes, setting the variable to the qnull object that *isn't* a qnull anymore.
<Chaos`Eternal>not starts, it just finishes
<taylanub>Yes.
<Chaos`Eternal>deq starts before enq starts and finishes after enq finishes
<taylanub>After its finished, it mutated the qnull object to which the enq! thread was holding a reference to.
<Chaos`Eternal>that's the problem
<taylanub>That's the solution. :)
<taylanub>qnull objects have identity
<Chaos`Eternal>because the cdr in line75 and set! in line 76 are seperate operations.
<taylanub>But they use the same object.
<Chaos`Eternal>but if enq! happens during the line 75 and line 76
<Chaos`Eternal>the line 76 will set thing wrong
<taylanub>No, the qnull object's contents might be mutated, but it's still the same qnull object.
<taylanub>Well, after mutated it's not a qnull object anymore, but it's the same object.
<Chaos`Eternal>line 76 is not changing the qnull, it's changing your variable
<taylanub>Yes, it sets the variable's reference to the object that was qnull when deq! started, but isn't qnull anymore after enq! ran.
<Chaos`Eternal>so you are saying that when the set in line 76 happens, the qnull is nolonger qnull?
<Chaos`Eternal>ok, lets reverse the scenario
<taylanub>Yeah, that's the point, as explained in the comments by the way. The "denull" procedure, working on the qnull objects that have identity, does the magic.
<Chaos`Eternal>lets see enq -> deq -> deq-finish -> enq-finish
<taylanub>(BTW instead of "objects that have identity" I should actually say "objects that have mutable content")
<taylanub>Chaos`Eternal: enq!'s variable-ref call will either give the old pair, or the new pair (or qnull), depending on when variable-set! interferes; in either case it will iterate through the linked list until it reaches the current qnull of the list.
<taylanub>So it's safe against intermediate deq! calls.
<Chaos`Eternal>BTW, you know, we are making jokes, both you and me can not find the simple race-conditions
<Chaos`Eternal>it must be a big joke..
<Chaos`Eternal>now i will try to point it out.
<taylanub>Eh ? There are no race conditions, except inside cdr/set-cdr!, variable-ref/variable-set!
<Chaos`Eternal>now we start at q: {variable (value . qnull)}
<Chaos`Eternal>emm, i need to give them addresses
<Chaos`Eternal> {variable (value:B . qnull:C)}:A
<taylanub>I need to go now, so please do it in your mind and ping me if you find anything.
<taylanub>(We kind of abused the channel, too.)
<Chaos`Eternal>shit, i waste my whole after noon, but you are going?
<Chaos`Eternal>just one mininute
<taylanub>Well it's you who's looking for bugs that aren't there. :P
<Chaos`Eternal>enq!: {variable (value:B . (value2:D . qnull:E):C)}A
<taylanub>You chaneg the qnull's address.
<taylanub>Oh wait, sorry.
<taylanub>Go on.
<taylanub>(Or don't. :P)
<Chaos`Eternal>in enq!: line 66, obj is C
<Chaos`Eternal>now deq, line 75 rest is also C
<Chaos`Eternal>now, enq! finished, C -> (value2:D . qnull:E)
<Chaos`Eternal>emm, seem right also
<Chaos`Eternal>what happens another enq!
<Chaos`Eternal>ok, no another enq!
<taylanub>Please just tell me if you find something wrong, I actually need to do some work now ..
<Chaos`Eternal>well, i will check it later
<Chaos`Eternal>need to go home now
<taylanub>k, bye
<ArneBab>I read about the new compiler. Can I already test it? I have a simple performance test here and would like to see the difference…
<wingo>it doesn't really work yet
<wingo>it will be ready in a few weeks i think
<ArneBab>ok, thanks for your answer!
<wingo>np, happy hacking :)
<ArneBab>another question: If I write a program in guile scheme, is there an easy way to package it as a binary for multiple platforms?
<wingo>not currently, no :/
<wingo>if you need something like that in the short term you might be better off looking at another scheme, like racket or chicken
<wingo>we'd like to do that but it will be some time before we can get all the pieces together to do that
<ArneBab>ok
<mark_weaver>taylanub: I haven't fully read the monster discussion here, but I can tell you that on most modern platforms (including Intel), one thread sees writes from other threads in possibly a different order.
<mark_weaver>so clever tricks like writing words in a carefully designed order won't work at all.
<mark_weaver>for example, suppose one thread atomically sets a pointer to a new data structure. the other thread might see the new pointer before the memory it points to has been initialized.
<mark_weaver>basically, this has to do with all the tricks that processors do to be fast. they have things like write buffers that postpone writes for a while. and there's no guarantee what order things will be written in.
<mark_weaver>unless you use synchronization primitives, which are relatively expensive.. these do things like dump all the write buffers, or force all pending reads to be done before some primitive.
<mark_weaver>IMO, the "shared memory multiprocessing" model, where you have a bunch of processes mutating a single shared memory, is fundamentally slow. what you really want to be doing is explicit message passing. and essentially that's how modern computers work anyway. but they try to use message passing to present the *illusion* of a shared memory. but in practice, the illusion isn't very good, and it's very hard to write cod
<mark_weaver>is most efficient and rock solid.
<taylanub>So when I have the sequence (set-car! qnull value) (set-cdr! qnull (make-qnull)), the other thread may actually see the set-cdr!'s effect first, then access the car before the value is set ?
<mark_weaver>yes, exactly.
<taylanub>Wow, crazy, thanks for the heads up. I'll just stop attempts at such lock-free hackery.
<mark_weaver>the only way to guarantee that the reader sees the writes in the desired order is to use some kind of synchronization primitives.
<mark_weaver>the cheapest ones are the modern atomic memory barriers, as found in C11 et al.
<mark_weaver>but they are still quite expensive (on the order of tens of cycles I believe)
<mark_weaver>the cheaper ones are more difficult to reason about though. one must be very careful to prove the relevant "happens before" relationships.
<mark_weaver>there are some clever tricks you can sometimes do. things like http://mechanitis.blogspot.com/2011/06/dissecting-disruptor-whats-so-special.html
<mark_weaver>basically, if each cache line is exclusively owned by one writer, then it can be possible to cleverly design protocols that cope with the fact that readers may see the writes out of order...
<mark_weaver>but it's very tricky.
<taylanub>Wow, there's a whole Wikipedia article on producer/consumer strategies: http://en.wikipedia.org/wiki/Producer-consumer_problem
<mark_weaver>wingo: btw, I might be able to make some more progress on wip-cps-bis if you can spare a few minutes to give me some hints about generating the offset from the current IP to the code of the procedure, so that I can generate the 'make-closure' instruction.
<wingo>heya!
<mark_weaver>greets!
<Chaos`Eternal>taylanub, still there? i am now coming to the point that your codes is really thread-safe, given the condition that only one writer and one consumer
<wingo>so make-closure takes as one of its argument an S32 or L32 or something
<Chaos`Eternal>but still a prove is needed
<wingo>that's a label, as a diff from the current IP
<wingo>so you just need to give it the label of the procedure
<wingo>and that label, in a hacky hack hack hack, happens to be the "self" of the procedure
<mark_weaver>will labels work across "begin-program" boundaries?
<wingo>so "self" is used as a label and an identifier for the closure; nasty.
<wingo>yes, labels will work across a compilation unit
<wingo>which is usually a set of programs
<mark_weaver>the 'self' label is part of the 'begin-program' directive. that's followed by a label "kentry". does it matter which one?
<wingo>good question!
<mark_weaver>okay, so they are guaranteed to not be moved apart? (or the linker will take care of it if that happens?)
<wingo>and the answer is i don't recall correctly
<wingo>"they" == multiple procedures compiled in one compilation unit?
<mark_weaver>right
<wingo>if that is the question then yes, they are all compiled into one ELF image and the labels are resolved for the whole image
<mark_weaver>one more thing: in the assembler I use the 'constant' helper procedure to get the slot index (for 'free-ref') as an immediate.. but the code generator is still emitting code to load that constant into a register. is there some mechanism handy to prevent that?
<mark_weaver>sorry, not in the assembler.. in compile-rtl.
<taylanub>Chaos`Eternal: FYI http://sprunge.us/NVLM So while the code would be correct in theory if cdr/set-cdr! etc. behaved "intuitively"; in practice, set-cdr!/cdr possibly not being atomic is not even the only problem with my code, as mark_weaver explained.
<taylanub>(I hope it was a fun brain exercise at least.)
<wingo>mark_weaver: the constant will be loaded into a register if constant-needs-allocation? returns true
<wingo>probably you need to add another case or three to constant-needs-allocation?
<mark_weaver>great, thanks for the hints! :)
<wingo>np, thanks for the hacks :)
*wingo will get to hack tonight
<mark_weaver>hopefully I can make something out of them later :)
<Chaos`Eternal>but, i must say, (ice-9 q) is the same as your code. taylanub
<mark_weaver>sounds great!
<Chaos`Eternal>so that i can remove some mutex-locks in mine code, thanks
<taylanub>Chaos`Eternal: It probably makes no claims about thread-safety.
<taylanub>If you want thread-safety with shared memory, you do need to synchronize yourself. Or just don't use shared memory.
<Chaos`Eternal>didn't get your point.
<taylanub>libdispatch: "Tasks in GCD are lightweight to create and queue; Apple states that 15 instructions are required to queue up a work unit in GCD, while creating a traditional thread could easily require several hundred instructions." I wonder what they use.
<taylanub>Chaos`Eternal: My module is very broken in practice because parallel processors are crazy. :)
<Chaos`Eternal>not sure
<taylanub>And in other words: don't ever expect thread-safety without explicitly using synchronization constructs in your code.
<Chaos`Eternal>prove it
<Chaos`Eternal>dont waste time
<taylanub>I'm just trusting mark_weaver's knowledge on that.
<taylanub>Sadly such issues are difficult to prove due to the inherently non-deterministic behavior.
<Chaos`Eternal>first, variable-set! is not necessary, you can use set-cdr! instead of it.
<taylanub>I wonder what davexunit ended up using for eir task-queue that needed to be dequeued efficiently from the main-loop thread.
<Chaos`Eternal>second, set-cdr! should be atomic
<taylanub>Chaos`Eternal: It is not guaranteed to be atomic.
<Chaos`Eternal>since it is just a pointer operation
<Chaos`Eternal>but that still depends
<taylanub>Chaos`Eternal: Did you read the paste I gave you ?
<Chaos`Eternal>i am not going waste time on that
<Chaos`Eternal>he's just talking about the issue in a general way
<Chaos`Eternal>he didn't drill into specific cases.
<taylanub>Chaos`Eternal: Just read it, it explains the issue.
<Chaos`Eternal>already read, as i've said. he's just talking about general beliefs
<taylanub>If you don't udnerstand how the issue explained there applies to my code ... think more, I guess. :P https://en.wikipedia.org/wiki/Out-of-order_execution
<taylanub>understand*
<Chaos`Eternal>the whole question left is whether set-cdr! is atomic
<Chaos`Eternal>you can say that is not atomic
<Chaos`Eternal>but your asumption is set-cdr! is atomic, remember?
<Chaos`Eternal>and it can be atomic
<Chaos`Eternal>since it may be just a pointer mutation
<taylanub>There's also the problem that when a thread does (begin (set-car! c x) (set-cdr! c y)), the processor might actually decide to set the cdr first, then the car.
<Chaos`Eternal>but they have the same effect
<taylanub>"Theoretically" (for some value of "theoretically"), the code is correct though, yes. Intuitively it seems correct, just isn't in practice.
<Chaos`Eternal>cause they have different addresses
<Chaos`Eternal>and as i have said, you actually dont need set-car
<Chaos`Eternal>you only need set-cdr!
<taylanub>Chaos`Eternal: I have stuff to do so I'll leave it to you to understand the problem explained here. Just don't use my module in "real-world" code. :D
<Chaos`Eternal>i never said that i will use your code
<taylanub>Though it all boils down to: when no explicit syncing is done, any amount of crazy optimizations are allowed that screw intuition.
*taylanub runs
<Chaos`Eternal>i said your code is the same as ice-9 q
<Chaos`Eternal>and the later is more simpler and standard in guile
<Chaos`Eternal>anyway, thanks
<ArneBab>wingo: is it possible to compile guile into a single binary with dependencies statically compiled in? Adding a script to run by default + compiling into a single binary would be pretty close to a distributable binary.
<ArneBab>mark_weaver: your message at 15:51:45 (IMO, the "shared…) was truncated after “write cod”
<taylanub>Indeed, I somehow managed to miss that.
<ArneBab>wingo: never mind, I just found the bug for that: http://lists.gnu.org/archive/html/bug-guile/2013-03/msg00002.html
<mark_weaver>ArneBab: that message ended with: but in practice, the
<mark_weaver> illusion isn't very good, and it's very hard to write code this
<mark_weaver> way that is both efficient and rock solid.
<ArneBab>ah, yes
<ArneBab>thanks!
<ArneBab>(just found that while reading)
<ArneBab>(stumbled :) )
<mark_weaver>ArneBab: if you want to read more on this subject, here are some links: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2480.html http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2010.06.07c.pdf
<mark_weaver>the first link is best if you simply want to know what you need to write safe multithreaded code using (relatively) lightweight memory barriers.
<mark_weaver>the second link is better for explaining what's going on in the hardware to cause these problems.
*ArneBab fired up the printer. mark_weaver: thanks!
<ArneBab>(this is only for curiosity, though: I like reading up deep stuff, but I’m currently not in a position where I could use it.
<ArneBab>wingo: can you alert me when there is a way to test the new compiller code for guile with a simple numeric let-loop?
<mark_weaver>I can sympathize with that :)
<mark_weaver>ArneBab: at the moment, it wouldn't quite be a fair test, because the stack VM has some assembly bits for fast fixnum arithmetic that aren't yet in the register VM.
<mark_weaver>but tuning can come after we have the basics.
<ArneBab>mark_weaver: how about a simple loop?
<mark_weaver>the compiler isn't quite there yet, but it's close.
<mark_weaver>probably on the order of days
<ArneBab>mark_weaver: nice!
<mark_weaver>anyway, going offline for a bit. ttyl!
<wingo>moo
<stis>an evening moo to you all!
<stis>wingo: nice code you wrote ...
<wingo>heya stis
<wingo>thanks :-)
<wingo>it's nasty in many ways too :)
<stis>perhasp some macros could be of use though.
<stis>to help in wthe whitespace race ;-)
*stis does not grok it in depth
<wingo>yes some macros would be useful
<wingo>like a build-cps macro for example...
<wingo>match seems OK though
<stis>Yeah, match is ok, the only drawback with match is that $ is based on position
<wingo>yes
<wingo>it works ok for some things
<stis>but (and (= acc-1 x) (= accc-2 y)) can of cause be used
<wingo>and it's nice to have positional matchers some times
<wingo>samth was telling me things about the racket matcher, it seems nicers
<wingo>*nicer
<stis>I have it in guile if you download syntax-parse
<stis>the racket matcher.
<wingo>oh i didn't know about =
<wingo>stis: nice!
<stis>I did it exersicing syntax-parse.
<stis>heh you should see the output from the macro expander of that code, 500k if I did not missremembered
<stis>if the final go
<wingo>haha
<stis>A good way to strip the syntax expression would be very helpful.
<stis>err syntax objects I mean
<wingo>for serializing source or for compiled file size?
<stis>to reduce complexity and yes the final compiled file is the target.
<wingo>you would think that for compiled files that the shared-substructure hacks would effectively compress all that
<stis>usually you can wait for the file to compile
<stis>perhasp, but 500k, what is causing that? The most obious candidate was the syntax eobjects
<stis>anyway it is impossible to see if there is any other code explosions.
<stis>let me check the typical sizes of go files ...
<stis>peval is 170k
<wingo>seems a bit large
<wingo>we will see how big the different sections are with the rtl things
<wingo>you will be able to readelf -a the compiled files
<stis>cooool, muuuu
<stis>en sweden cows says mu
<wingo>:)
<ijp>sweden needs more imaginative cows
<wingo>i think in spanish they say buuuuuu
<stis>scary :-)
<stis>anyway the man racket macher is in 526k
<stis>Typically what can cause explosions is or expressions in the matcher.
<stis>ice-9 had one of those bugs
<wingo>i suppose compiling case statements appropriately would help
<wingo>jump tables and all that
<stis>one need to bind th elambda to a var and suplly it else the expansion bangs exponentially
<stis>err bind a lambda
<wingo>sure
<wingo>that gets contified later of course
<stis>and peval would inline if it's useful
<stis>at least shold
<stis>er should
<stis>anyhow, for syntax-parse I would
<stis>1. double check the or logic for any suspicious code
<stis>2. if not found try to experiment with stript syntax objects to see if it has any effect
<stis>Is there some hints how to strip those objects the correct way?
<stis>somewhere? example code?
<wingo>i think mark_weaver does it in psyntax
<stis>anyhthing to grep for?
<stis>I got it, many thanks!!
<stis>;; strips syntax-objects down to top-wrap
<stis>is that it
<wingo>prolly, i don't really know
<mark_weaver>wingo: I haven't done anything with wip-cps-bis since my last push, so feel free to hack on it :)
<mark_weaver>stis: the code I wrote for minimizing syntax objects for psyntax-pp.scm is in module/ice-9/compile-psyntax.scm
<mark_weaver>'squeeze-syntax-object!'
<mark_weaver>note: it is not safe to run that on arbitrary code, but it's safe for psyntax.scm
<wingo>mark_weaver: cool :)
<wingo>currently adding a test suite
<wingo>wondering what to do about lone prims
<wingo>ideally at some point they should be rewritten as module-box + box-ref
<wingo>so that cse can potentially eliminate duplicate module-box instructions
<wingo>e.g. (compile 'cons #:to 'rtl)
<wingo>maybe i should add a reify-primitives pass
<wingo>that would turn primcalls to prims that don't have corresponding vm ops into normal calls
<mark_weaver>I took care of implementing branchable prims outside of test contest.
<mark_weaver>(and in test context as well)
<wingo>right, those are primcalls
<wingo>i was thinking about $prim
<mark_weaver>oh, I see what you mean now.
<wingo>and also about $primcall to unknown prims
<wingo>sound sane to you?
<mark_weaver>hmm. does it make sense to have primitives that don't have VM instructions?
<mark_weaver>I suppose it potentially allows the compiler to reason about them.
<mark_weaver>I guess it sounds sane. I haven't really thought about it much.
<wingo>yes, you would have them to allow the compiler to reason about them
<mark_weaver>makes sense.
<wingo>...conveniently. depending on what we do with modules, module variables could be reasoned about in a similar way i guess
<mark_weaver>as for the question of whether to break instructions into smaller pieces, e.g. to make the type checks explicit (so they might be optimized away)...
<mark_weaver>maybe the thing is to break them into small pieces for the optimizer, and then to coalesce them back into bigger pieces so reduce the number of dispatches? dunno.
<mark_weaver>s/so/to/
<wingo>could be
<wingo>i doubt allocation benefits from this significantly though
<wingo>if we are thinking of the make-struct example
<wingo>given that the cost of allocation is mostly in gc, and more instructions are usually retired accessing a structure than allocating it
<wingo>dunno
<mark_weaver>well, for 'make-struct' my main motivations were: (1) we could enforce immutable fields without making constructors an exception, and (2) to reduce the number of dispatches for this common operation.
<stis>thx mark_weaver: I will use it in a test to see if the stx objects may be the reason for bloat!
<mark_weaver>stis: one thing though: that code makes assumptions not only about the code being "squeezed", but also about the internals of psyntax. if you use it in code outside of guile core, it will likely break in a future version of guile.
<stis>yeah, I will just issue a bug report if there is to much fluff, or a feature request?
<mark_weaver>wingo: I continue to think that a variable-length 'make-struct' would be a win. you said before that it's like having a little interpreter in the instruction, but that's an exaggeration. it's actually just a loop, and modern processors are very good at optimizing loops.. e.g. their branch predictors handle loops specially.
<mark_weaver>that agner paper talks some about that.
<mark_weaver>stis: it's not easy to fix.
<mark_weaver>I gotta go afk for a bit.. back in about 20 minutes.
<stis>mark_weaver: ok!
<mark_weaver>stis: I actually don't see any way to fix it, even in theory, without changing the API of psyntax. in particular, the semantics of datum->syntax.
<mark_weaver>In the general case, datum->syntax requires that information about the compile-time lexical scopes be preserved.
<stis>mark_weaver: is it about datum->syntax hooking syntax objects to names of past variables?
<mark_weaver>yes, that's the issue.
<stis>and if I don't care about those names your function should be ok?
<stis>to apply
<mark_weaver>it's been a while since I thought carefully about this, but iirc, the main limitation is that 'datum->syntax' cannot be called on any of the syntax objects that are introduced by macros whose code has been "squeezed".
<stis>hmm, is this a good question on the scheme channel?
<mark_weaver>e.g. the definition of 'define' introduces a 'lambda' syntax object in the expanded expression.
<mark_weaver>well, they won't know anything about my "squeeze" code.
<stis>yeah, but psyntax and what one can do to minimize the overhead and what cost that may have
<mark_weaver>I guess it's worth a try :)
<stis>Hi all, I have a question about psyntax,
<stis>in guile the syntax objects become really huge, and it would be nice to know what your experience is about getting htem thinner
<stis>ops, wrong channel :-)
<wingo>buf, early sleepiness
<wingo>will finish in the morning
<wingo>night!
<mark_weaver>g'night wingo!
<mark_weaver>stis: the folks on #scheme might not know off-hand that guile uses psyntax. your question is really about psyntax.
<mark_weaver>the problem of large syntax objects is specific to psyntax.
<stis>ok, I made that explicit, let's wait and see, are there any logs of the scheme channel?
<mark_weaver>not that I know of, except for private logs that I keep.
<stis>oh well, I let the computer run over the night then :-)
<amk9>last guile-www release is not installable
<amk9>(hi!)
<mark_weaver>amk9: personally, I would use the new 'web' modules that come with Guile 2.
<amk9>it's missing something about mime-type
<amk9>I'm trying to port this code http://lists.gnu.org/archive/html/guile-user/2011-03/msg00061.html
<amk9>btw, https://www.gnu.org/software/guile/manual/html_node/Web-Examples.html#Web-Examples is not up-to-date
<shanecelis>Anyone know of anyone that uses macros to do code refactoring?
<amk9>for instance request-path-components doesn't exists in guile 2 web
<mark_weaver>request-path-components is defined in that example.
<mark_weaver>it has never been in guile itself.
<mark_weaver>I don't see much code at http://lists.gnu.org/archive/html/guile-user/2011-03/msg00061.html. just someone asking a question.
<amk9>I'm doing a some kind of static site generator
<amk9>to help with authoring
<mark_weaver>SXML will probably be helpful for that.
<mark_weaver>SXML is included in Guile 2.
<amk9>I mean when you write the website, you need a simple http server to serve the static files
<amk9>yes I should use it but right now I don't I generate html directly
<amk9>I more into getting the authoring syntax right
<ijp>serving a file is very simple, just return it as a string, or better yet, open the file, and return the port
<amk9>yes but why the guy from the ML want to return the mime type ?
<ijp>why wouldn't you want to return a mime type?
<amk9>I'm not sure how browser works, maybe it needs it... I try without it, thx
<ijp>the server may default to text/plain, but I'm not sure
<mark_weaver>the mime type is usually returned as part of the response headers, iirc.
<mark_weaver>right, there's supposed to be a "Content-Type: " header in the response from the server to the client.
<ijp>you are expected to return a response object, though we let you return the headers as a shorthand
<mark_weaver>when using the web stuff in Guile 2, you pass the response headers as an alist to 'build-response'.
<mark_weaver>e.g. #:headers '((content-type . (text/html (charset . "utf-8"))))
<amk9>text/plain is rendered as... plain text by the browser but I least I got something :)
<amk9>I can use file --mime-type src/sfx/index.html -b to get the mime type
<amk9>btw, I'm using run-server so there is "just" a handler
<amk9>and it expects headers and body as bytevector
<amk9>so I can't return a port
<mark_weaver>amk9: I'd like to help, but I'm very confused by what you're saying.
<ijp>you can return a port, it just isn't documented IIRC
<mark_weaver>what do you mean "just" a handler, and what does that have to do with 'run-server'?
<ijp>actually it is documented
<mark_weaver>oh, nevermind. I see now that there is more than one 'run-server' in core guile.
<ijp>"The respose and response body are run through sanitize-response, documented below. This allows the handler writer to take some convenient shortcuts: for example, instead of a <response>, the handler can simply return an alist of headers, in which case a default response object is constructed with those headers. Instead of a bytevector for the body, the handler can return a string, which will be serialized into an appropriate encoding;
<ijp>or it can return a procedure, which will be called on a port to write out the data. See the sanitize-response documentation, for more."
<ijp>except that says a procedure that's called with a port, hmm
*mark_weaver goes back to hacking the new compiler :)
*ijp goes code diving
<ijp>hmm, I was sure it allowed a port directly, but the code doesn't lie, at least not particularly well
<ijp>so, my bad