IRC channel logs

2014-02-07.log

back to list of logs

<cluck>taylanub: i drop by to snoop around every now and then :)
<Fuuzetsu>re: earlier discussion on map, I don't understand why it would stack overflow… Can someone explain? It should be perfectly doable. Oh, I suppose strictness might kill it.
<Fuuzetsu>davexunit: that's pretty much exactly how ‘map’ is implemented in Haskell (except there's an extra case for empty list but I'm sure that's implicit in your example)
<mark_weaver>Fuuzetsu: well, there are several ways to implement 'map'. One way is to create a reversed result list and then call 'reverse!'.
<Fuuzetsu>mark_weaver: I don't see why reverse has anything to do here.
<Fuuzetsu>Given (cons 1 (cons 2 nil)), what's the problem simply deconstructing the cons which is what davexunit seems to imply is a Bad Idea™
<mark_weaver>you're spoiled by haskell :) yes, strictness kills it, as you said.
<Fuuzetsu>Ah. What a terrible world you must live in where implementing ‘map’ is tricky ;P
<mark_weaver>well, there are tradeoffs.
<davexunit>Fuuzetsu: that approach will run out of stack space, but apparently not in guile master.
<mark_weaver>I've drunk the Haskell kool-aid, and it is a wonderful language in many ways, but I've come around back to Scheme for various reasons.
<Fuuzetsu>There's no kool-aid ;P Anyway, I'm not here to get into language wars, I was simply curious.
<mark_weaver>In Scheme, procedure calls in "tail position" are GOTOs with arguments, so they don't use more stack.
<Fuuzetsu>so, regular TCO?
<mark_weaver>well, I prefer to avoid the term "optimization" because that suggests that it's optional. it's a core part of the semantics in Scheme.
<mark_weaver>it's a hard guarantee.
<Fuuzetsu>tail call elimination then?
<mark_weaver>sure
<mark_weaver>however, for 'map' it's not a tail call. the result of the recursive call then has to have an (earlier) element consed onto the front of it.
<mark_weaver>so basically, a straightforward recursive 'map' implementation uses stack space proportional to the length of the list.
<mark_weaver>now actually, in Scheme, there's no stack. it's all heap. so in theory this isn't a problem, and it wouldn't be a problem with a very straightforward implementation of Scheme.
<mark_weaver>all procedure call activation frames are (semantically) heap-allocated.
<mark_weaver>however, many implementations, including Guile, use a stack where possible to reduce the amount of work the GC has to do.
<mark_weaver>anyway, it's not a problem in the master branch of Guile anymore, because the stacks are dynamically resized as needed.
<mark_weaver>Fuuzetsu: btw, I've decided not to add MVars to Guile. I intend to implement a full STM instead, hopefully in 2.0.11.
<Fuuzetsu>mark_weaver: MVars and STM are not exclusive.
<Fuuzetsu>In fact they play pretty well together
<mark_weaver>MVars seem quite limited to me.
<Fuuzetsu>Yes, they are, but so is STM. They fill in different roles which makes your decision seem strange to me.
<mark_weaver>if I knew how to implement them efficiently on top of posix threads, I might be more inclined to add them.
<mark_weaver>STM doesn't seem limited to me. it seems very general.
<mark_weaver>all of these things are easier to implement efficiently on green threads, as used in Concurrent Haskell. but then you're actually only using one core of the processor.
<mark_weaver>what can't STM do?
<Fuuzetsu>STM is in principle slower (of course you can implement some things with STM to be faster than the MVar variant but not always); STM is _not_ fair, while MVars are! You have to wake all STM threads on update unlike with MVars;
<Fuuzetsu>it basically can't do anything that involves multi-way communication between threads
<Fuuzetsu>the problem is that you can't implement fairness without sacrificing STM's compositionality but if you remove that then it's not really STM anymore
<mark_weaver>well, okay, it's a good point that STMs aren't as good for waking up other waiting threads. however, in guile we've long had something much more general than MVars that offer the same properties: condition variables.
<Fuuzetsu>Well, if you have something which is more general then sure, you might want to consider not including MVars (although I still think you should, for convenience if anything else, at least as a library). I'm simply making the case of MVars + STM + nothing vs STM + nothing
<mark_weaver>yeah, if those were the choices, I'd agree. you need something that supports fair, single-wakeup signalling.
<mark_weaver>well, we can revisit the MVar decision after 2.0.10.
<Fuuzetsu>Cool.
<mark_weaver>to be honest, I ran into unexpected behavior while writing the MVar test suite that made me wonder if the current implementation meets the required fairness properties, so I had to punt for now anyway. I don't have time to track it down before 2.0.10.
<Fuuzetsu>Interesting. Perhaps you could post that in the relevant mailing list thread for everyone to see.
<mark_weaver>yeah, I'll followup when I have a chance to investigate a bit more.
<mark_weaver>basically, when I wrapped the same experiments that I posted before, that showed nice FIFO behavior, within a 'let*' in a test, the last thread didn't get served nearly as often as all the other threads.
<mark_weaver>instead of (0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 ...), I saw (0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 ...)
<mark_weaver>I suspect it's a problem either in Guile's condition variables, or perhaps in the pthreads implementation of the machine I'm testing on.
<mark_weaver>but then, we've never made the kinds of promises about Guile's condition variables that MVars demand.
<mark_weaver>I'll look into it after 2.0.10 is released.
<Fuuzetsu>>but then, we've never made the kinds of promises about Guile's condition variables that MVars demand.
<Fuuzetsu>which is precisely why I even asked to begin with once I saw the proposal
<mark_weaver>yes, and I'm grateful that you prompted me to look more carefully at the requirements.
***vicenteH` is now known as vicenteH
<mark_weaver>Actually, Guile's mutexes and condition variables seem to have been designed to meet the same requirements specified for MVars, but we haven't documented those, and now it seems that there might be a bug somewhere.
<mark_weaver>so after 2.0.10, I think I'll try to find and fix that bug, and then maybe document the properties.
<mark_weaver>(I'll admit that I was initially annoyed that you were adding to my workload, but in retrospect I'm grateful)
<davexunit>I have a design question that concerns the coroutine scheduler for guile-2d. The way I've implemented it, all public procedures operate on the "current agenda", which can be convenient. However, I'm thinking about removing this state and exposing the procedures that need to passed an agenda object.
<davexunit>When is it appropriate to introduce the concept of a current X like with input/output ports?
<davexunit>s/need to passed/need to be passed/
<mark_weaver>Personally, I try to avoid the use of dynamically-scoped variables (which are essentially what fluids/parameters are) because they don't play well with things like lazy evaluation.
<mark_weaver>on the other hand, I must admit that there are situations where we don't know of any other tolerable solution.
<mark_weaver>I also try to avoid global variables in favor of some kind of "context", but in the case of a game library, I'm not sure it's a bad thing.
<davexunit>I've found it convenient for code to not care about what agenda it's scheduling to, but to simply declare that it wants to pause for some amount of time.
<mark_weaver>I don't see why you'd have more than one of these agendas in a single process.
<mark_weaver>yeah. an agenda of scheduled jobs seems to naturally lend itself to a global.
<davexunit>well, it could be convenient for contexti
<davexunit>oops
<davexunit>multiple agendas could be convenient for context switching.
<davexunit>like if one "scene" of the game were paused and replaced with another.
<davexunit>it would be like each emacs buffer having a scheduler of its own.
<mark_weaver>ah, so pausing essentially freezes the clock of one game. so each game would have its own notion of time, or something?
<davexunit>yeah
<davexunit>the agenda is a model for the passage of time in the game.
<davexunit>I think I'll stick with the global.
<mark_weaver>well, as with so many things, there are tradeoffs here.
<mark_weaver>the use of fluids/parameters won't play well with lazy evaluation, which seems potentially useful.
<davexunit>mark_weaver: I can imagine the problems that that will cause, but what exactly happens?
<mark_weaver>so if you rule out those, then the choices seem to be a global or explicitly pass around the agenda. so it depends on how much of an inconvenience that would be.
<mark_weaver>well, suppose I want to make a lazy list of characters read from a file.
<davexunit>an expression is forced outside of the dynamic scope?
<mark_weaver>right.
<mark_weaver>so basically, it would be very natural to say (with-input-from-file "foo" ...) where the ... reads characters and produces a stream.
<mark_weaver>but then, when something outside of that context forces the stream, the lazily-evaluated expressions are in the dynamic context of whatever asked for the next element of the stream, where the current-input-port might be something else.
<mark_weaver>more generally, dynamically-scoped variables have a way of propagating to code that you didn't know about.
<Fuuzetsu>davexunit: why not simply wrap the agenda in something like the reader monad? You get rid of the state and the passing around of the agenda is done for you. I might be misunderstanding the question though.
<mark_weaver>I have to go offline for a bit. happy hacking!
<davexunit>see ya mark_weaver
<davexunit>Fuuzetsu: in Haskell that would be what I would do
<davexunit>but it's not very Scheme-y.
<Fuuzetsu>davexunit: and why not do it in guile?
<davexunit>I think it would be better to be explicit about it.
<Fuuzetsu>but it does exactly what you want it to do + you get everything written for a monad ever
<Fuuzetsu>it seems silly not to use it
<mark_weaver>Guix uses monads to manipulate the store. I don't think it's not Schemey, really.
<mark_weaver>However, the thing is, the code that davexunit is writing is dealing with much more than just the agenda.
<Fuuzetsu>mark_weaver: Well, I don't see him saying anything about the extra bits. I answer the question I can see.
<mark_weaver>so it would have to be a much more general monad. more like an IO monad or whatever they call it in Haskell.
<davexunit>on top of the fact that I only sort of understand monads :)
<Fuuzetsu>IO isn't a much more general monad; it's just a different monad
<davexunit>sorry to keep you, mark_weaver! I think I'll just opt for the explicit approach and see how it goes. :)
<Fuuzetsu>davexunit: I think if you learned about them a bit more and then read a thing or two about monad transformers, you could roll something very useful
<mark_weaver>well, "more general" was a poor choice of words, but I mean that the IO monad supports a much larger set of operations.
<davexunit>Fuuzetsu: perhaps you're right.
<mark_weaver>anyway, gotta run....
<Fuuzetsu>mark_weaver: That's… a weird argument. IO and Reader actually support exactly the same amount of operations (those defined over Monad). There just happen to be more functions working with concrete values of ‘IO a‘ than ‘Reader a’. Anyway, I don't want to keep you.
<davexunit>he left.
<Fuuzetsu>Indeed. Well, it does not matter much, his point was silly.
<mark_weaver>Fuuzetsu: my point was silly, eh?
<mark_weaver>maybe you just didn't understand what I meant.
***haroldwu_ is now known as haroldwu
<mark_weaver>IIRC, a function that returns IO can perform _any_ imperative operation whatever, anything that some imperative program could do.
<mark_weaver>if one excludes unsafeIO (or whatever it's called), then functions whose return value doesn't return IO are much more limited in the operations they can perform. that's what I meant.
<mark_weaver>am I mistaken?
<mark_weaver>I have to go afk for a bit.
<nalaginrut>morning guilres~
<zRecursive>Does guile support srfi-46 or srfi-72 ?
<mark_weaver>2.0.10 will support srfi-46.
<mark_weaver>(the stable-2.0 branch of git does now)
<mark_weaver>we've supported tail patterns for a long time, but the custom ellipsis identifiers are new.
<mark_weaver>we don't support SRFI-72.
<mark_weaver>we use psyntax
<nalaginrut>I think you mean 43 rather than 46?
<mark_weaver>I doubt it.
<mark_weaver>both 46 and 72 have to do with macros.
<nalaginrut>46 is about syntax-rules, I think we have it, no?
<nalaginrut>ah, extension
<Fuuzetsu>mark_weaver: I don't see the ability to perform IO (rather than ‘anything that some imperative program could do’) has to do with our particular discussion though. It doesn't somehow make IO better than Reader. Also you're quite incorrect in saying that a function that returns IO can perform any imperative operation, look: f = (putStrLn "foo", putStrLn "bar") :: (IO (), IO ())
<Fuuzetsu>although that last point is just nit-picking because I think you meant something else
<Fuuzetsu>and no, the operations things can perform doesn't make IO special; the only special thing about IO is how RTS handles it once it floats up to top level in main. In any case, a careful reader (heh) should be able to quickly notice that Reader a is just an alias for ReaderT a Identity which very quickly exposes that we can simply use the provided ReaderT to make a stack with IO
<mark_weaver>please, you're being ridiculously pedantic. you know what I meant.
<Fuuzetsu>besides, we can lift absolutely anything into IO (or any other monad) with ‘return’
<Fuuzetsu>mark_weaver: Yes, I know what you meant and I'm saying it's a silly argument in this context.
<mark_weaver>yes, I mean functions that return IO where the return value floats to the top.
<mark_weaver>(or gets composed into an IO that's returned to the top)
<mark_weaver>I don't know what Reader is. It didn't exist when I last looked closely at Haskell.
<Fuuzetsu>you're correct that we would need a more general monad than just Reader for more actions, this is precisely what monad transformers let you do; I'm just trying to make a point that using IO as an example of such ‘more general’ monad is silly, it's exactly as general as Identity
<Fuuzetsu>mark_weaver: really? mtl (which houses Reader) has been around since 2006 and I'm pretty sure Reader was in a different package way before that time too.
<mark_weaver>yeah, my interest in Haskell was before that. More like the 1998-2002 timeframe.
<mark_weaver>I don't know what Identity is either.
<Fuuzetsu>well, the Reader implementation is based on a 1995 paper so I'm pretty sure it was around since then
<Fuuzetsu>Identity is the trivial structure
<Fuuzetsu>data Identity a = Identity { runIdentity :: a }
<mark_weaver>I think that Haskell has evolved too much since I last looked closely at it, for me to understand you, but it's probably not important.
<Fuuzetsu>Which part do you not understand? This is valid Haskell 98
<mark_weaver>okay, I must have forgotten too much.
<Fuuzetsu>If it helps, it's the same as doing ‘data Identity a = Identity a’ and then defining ‘runIdentity :: Identity a -> a’ with the body ‘runIdentity (Identity a) = a’.
<mark_weaver>ah, okay. That I understand.
<mark_weaver>but I don't understand how you can say that IO is exactly as general as Identity.
<mark_weaver>I think we're just talking different languages.
<mark_weaver>I'm talking about the set of imperative operations that can be performed in the real world, outside of the program, by something that returns IO () and has that return value float to the top.
<mark_weaver>and I think you've assumed that when I'm talking about operations, I'm talking about something like unit and bind or something.
<mark_weaver>or even the set of imperative operations that can be performed within the program, as in the set of locations that can be mutated.
<Fuuzetsu>Right, you said one would need ‘a more general monad’ but both IO and Identity are regular monads, they have the same operations &c. What you should have probably said is that davexunit would probably need something which can perform IO which is a completely different notion hence why I said it was silly.
<Fuuzetsu>(incidentally we can compose IO (thanks to the fact that it does happen to be a monad) with Reader to get the functionality of both and we can stack more monads together this way until we have something suitable for davexunit)
<mark_weaver>can something that returns Identity, without any IO contained within, cause a mutable variable to be mutated?
<mark_weaver>s/mutable variable/arbitrary mutable variable/
<Fuuzetsu>Something that returns ‘Identity a’, no matter what ‘a’ is, IO or not, can not update a mutable variable such as an MVar because the update has the type ‘MVar a → a → IO ()’. It is not possible (save for compiler hacks such as unsafePerformIO) to do the update _and_ return ‘Identity a’.
<mark_weaver>okay, so by the language that I'm using, it's not as general as IO.
<Fuuzetsu>you're using the wrong language
<mark_weaver>"general" is used by a lot of different people to mean a lot of different things. I reject your claim that "general" is not appropriate here.
<Fuuzetsu>I wasn't aware that admitting different operations now means ‘more general’. I could go and create a MyIdentity, give it a Monad instance, hide the constructor (which is what IO does) and you could not do anything using a function ‘something -> MyIdentity’ without in the end returning ‘MyIdentity a’. How is MyIdentity more general than anything? Is it more general than Identity? Is it more general than IO? What is the metric?
<Fuuzetsu>‘something -> MyIdentity a’*
<mark_weaver>I'm sorry, but I have no interest in continuing this conversation.
<Fuuzetsu>Uh, okay then.
<mark_weaver>zRecursive: btw, I saw the conversation on #scheme, and Guile has long supported the (... ...) syntax.
<mark_weaver>and it has several macros that expand into macro definitions.
<mark_weaver>now, with SRFI-46, it can be done in a nicer way.
<mark_weaver>but it could always be done.
<wingo>moin
<nalaginrut>heya
<civodul>Hello Guilers!
<lloda>hello
<lloda>wingo, saw your comments earlier
<wingo>howdy
*wingo going through the patchset slowly
<lloda>in the top commit I say 'arrays are not an array implementation' that might be confusing but it was the terms used in the code
<lloda>it means the underlying storage for an array
<wingo>yes i understand
<lloda>so before, that by itself could be an array
<lloda>and I removed that, because it was almost circular
<lloda>ok
<lloda>so what do you mean 'make arrays the only array implementation of rank > 1'
<lloda>?
<wingo>lloda: as it is, arrays aren't the only thing that could have rank > 1
<wingo>there could be another array implementation with those characteristics
<wingo>but the interface to add such an implementation is private, so it's a moot point
<lloda>ok, I actually don't remember that :-/
<wingo>are you around today?
<wingo>i'm going through the patches, merging as i can; probably you have seen that
<wingo>i might have questions later
<lloda>I'll be away from 10:30 to 13 or so, but I'll be available later
<wingo>cool
<mark_weaver>wingo: btw, I researched O_NONBLOCK some more, and found a very serious problem. A show-stopper, really.
<wingo>what's that?
<mark_weaver>the O_NONBLOCK flag is not specific to the file-descriptor. It effects all processes that have the underlying ofile open.
<mark_weaver>See http://cr.yp.to/unix/nonblock.html and https://www.spinics.net/lists/kernel/2007-q1/msg07191.html
<mark_weaver>"I mean: how can O_NONBLOCK _issued in a process which already exited_ have any effect whatsoever on MC or Konsole? They can't even know that it did it, right?"
<wingo>(!)
<mark_weaver>yeah, crazy, huh?
<wingo>that's wild
<civodul>that's not what http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html and http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html say though
<mark_weaver>civodul: I don't see anything in there that conflicts with the (unfortunately) reality.
<mark_weaver>s/ly//
<mark_weaver>anyway, it doesn't really matter what POSIX says. it matters what kernels actually do, of course.
<civodul>ah but that's for fcntl, and fds inherited from the parent
<civodul>ok
<wingo>or fds passed over a socket
<civodul>yeah
<mark_weaver>So, I'm at a loss of how to do portable non-blocking I/O in multiple threads with POSIX.
<wingo>i think my nio branch includes a flag in scm_t_fport indicating whether the underlying fd is in nonblocking mode
<wingo>which can be guaranteed for files you open yourself
<mark_weaver>DJB's solution was to set a timer for 10ms before calling 'read', and relying on the SIGALRM to interrupt the read. but we can't do that in the multithreaded program.
<mark_weaver>because the SIGALRM could be delivered to any thread that isn't blocking that signal.
<mark_weaver>and it only interrupts syscalls in that thread.
<wingo>mark_weaver: wouldn't dupping the fd be sufficient, somehow? is the object with the O_NONBLOCK the same object with the seek position?
<wingo>guile needs to own all of its fds anyway
<wingo>so i think that's what it does anyway
*mark_weaver looks
<wingo>mark_weaver, civodul: what do you think about making vector-ref, vector-set!, vector-length non-generic?
<mark_weaver>the flag is in 'struct file'
<wingo>currently they are primitive-generics.
<wingo>this change is in lloda's patchset
<mark_weaver>wingo: absolutely, I'm strongly in favor of that.
<wingo>ok :)
<wingo>it seems the strongest case for generics can be had in arithmetic
<wingo>still not sure if primitive-generics are the right approach, but it's not something i want to think about changing in general right now
<mark_weaver>If there was one single good type for all numbers, then I'd be in favor of using it. but sadly, there isn't.
<wingo>:)
<mark_weaver>I'm glad that we'll have generic operators for vector-like-things. but I also think we need non-generic operators for greater efficiency.
<civodul>wingo: i'm fine with that
<civodul>i'm not sure what the cost of primitive generics is
<taylanub>Does "primitive-generic" mean something more specific, or is it just any procedure implemented in C that handles multiple data types ?
<civodul>i mean, until you ensure-generic them, it shouldn't cost much no?
*civodul forgot the details
<civodul>taylanub: it means it's a procedure implemented in C that can be turned into a GOOPS generic
<wingo>civodul: the cost is not terribly high, no; otoh it makes it impossible to completely inline "vector-ref"
<civodul>right
<civodul>yes
<civodul>that alone is a good reason :-)
<mark_weaver>primitive generics dispatch to GOOPS only after the built-in cases all fail.
<wingo>because it could dispatch out to a generic and then re-join control flow
<wingo>with my optimizer hat on, i want (vector-ref foo 0) to imply that foo is a vector of length >0 :)
<mark_weaver>i.e., when we would signal a "wrong-type-arg" error, we instead dispatch to GOOPS.
<civodul>and it could be argued that there's no good use case (making a vector-ref method for vlists, for instance, feels wrong; the right thing would be to have parametrized modules)
<civodul>that said it's an incompatible change, so for 2.2?
<wingo>yes, 2.2
<civodul>good
<mark_weaver>we had generic-vector operations at one point. I actually think we should have kept them. but I guess that ship has sailed, and arrays have taken their place.
<civodul>"ra", as we used to say :-)
<wingo>bouf, that code is terrrrible
<wingo>lloda: did you pick up on that
<nalaginrut>how is the cost for catch-then-dispatch-to-GOOPS method?
<wingo>that "ra" is simply a shorthand for "array"
<wingo>arr ay
<mark_weaver>ah!
<wingo>nalaginrut: it's not a catch-then-dispatch; it's do-what-you-would-normally-do,but-if-the-type-is-wrong-call-a-goops-thing
*nalaginrut has no idea about the cost of catching exceptions
<wingo>when i realized that i felt arsonous
<jmd>nalaginrut: They are called exceptions because they should occur only in exceptional circumstances.
<nalaginrut>jmd: yes, I know what is exceptions, my question is the cost each time catching exceptions
<wingo>mark_weaver, civodul: another query. lloda has (vector? x) query specifically that X is a tc7_vect (or tc7_wvect :/ but that is another issue)
<wingo>mark thinks this is fine and i think this is fine
<wingo>then vector-ref only works on tc7 vectors.
<wingo>ok.
<mark_weaver>wingo: again, I strongly think that's the right thing.
<wingo>another related change is to uniform-vector? -- that it only returns #t for bytevectors and bitvectors, not arrays with different lower/upper bounds and increments
<wingo>so a uniform vector is a packed, 0-based vector of unboxed elements
<wingo>granted, uniform-vector? is a somewhat silly predicate, but does this change sound reasonable?
<wingo>with corresponding changes to uniform-vector-ref / uniform-vector-set!.
<wingo>basically making the specific instances of array types use specific predicates and accessors, and deferring generality to the array interface, seems to be the thrust of the patchset
<mark_weaver>I never heard of these procedures :)
<wingo>i can't be convinced to care very much about uniform-vector-ref et al; perhaps we should deprecate them in 2.0
<wingo>they straddle an uneasy middle ground
<wingo>but iirc they were useful in c
*wingo greps his source code
<mark_weaver>oh, these are SRFI-4 procedures.
<mark_weaver>so I guess we should keep them.
<civodul>wingo: with vector-ref no longer a generic, it makes sense for vector? to equate to tc7_vect, i think
<wingo>civodul: agreed
<mark_weaver>does vector-ref work on a weak vector?
<lloda>I have some time now. vector? = tc7_vect yes. I asked the list and Daniel Hartwig came against it. So this change was tentative.
<civodul>uniform-vector-read et al. were the only way to do binary i/o in 1.8
<wingo>mark_weaver: sure, but we could deprecate the C variants, and make Scheme versions that just use array-ref/array-set!
<mark_weaver>I think that (vector? obj) should return #t if and only if 'vector-ref' and 'vector-set!' can be used on 'obj'.
<wingo>uniform-vector-read is in a slightly different category; but it might make sense to deprecate it also, given that we have better primitives now
<wingo>mark_weaver: right now vector-ref can be used on weak vectors
<mark_weaver>similarly for 'uniform-vector?'
<lloda>I think uniform-vector should be srfi-4 raw vector only and not arrays. I don't think I went that far though.
<wingo>another thing preventing inlining of vector-ref, as weak-vector-ref has to take the gc lock :/
<wingo>lloda: you did, i think
<mark_weaver>I think that 'vector-ref' shouldn't work on weak vectors. I think there should be dedicated 'weak-vector?', 'weak-vector-ref' and 'weak-vector-set!'.
<lloda>there are some functions called uniform- something where I reverted the patch, will have to look
<lloda>what is a weak-vector by the way
<mark_weaver>'vector-ref' and 'vector-set!' should be inlinable and non-generic.
<lloda>+1
<mark_weaver>now, the uniform-vector-* ops, on the other hand, already need to be generics, so I don't see any benefit to limiting the types of vectors they can operate on.
<wingo>mark_weaver: we can do that now if we add weak-vector interfaces to 2.0, deprecate the use of vector* on weak vectors, and then fix in 2.0
<mark_weaver>s/vectors/vector-like-things/
<wingo>*fix in 2.2
<wingo>i think 2.0 will be around long enough for people to see the deprecation
<mark_weaver>yeah, the backward compatibility issue is a drag. but I guess people can use 'cond-expand' to deal with it.
<wingo>hum, i wonder how you make a weak vector... my guile doesn't have the documented interfaces (?)
<lloda>wingo: your patch 1fadf36
<lloda>this is circular.
<wingo>neat, there doesn't seem to be a weak vector impl in c :)
<wingo>er, in scheme
<lloda>array_handle_ref is gone later in my patchset
<mark_weaver>oh, 'uniform-vector-*' is _not_ actually part of SRFI-4. it's a Guile-specific extension to SRFI-4.
<wingo>lloda: not sure what you are saying; i have rebased your patchset in ra0 and the test suite passes
<lloda>all of it?
<mark_weaver>so we could deprecate them and eventually get rid of them, and I guess we probably should.
<wingo>there are some patches in the middle in which some parts of the test suite fails, but it passes in the end
<wingo>i haven't gotten to considering the patches in the middle; will cross that bridge when i come to it
<lloda>right
<lloda>are we getting rid of uniform-vector / uniform-array?
<wingo>so a number of projects use the scm_array_handle_uniform_writable_elements interfaces, and company; a change to make those interfaces only work for packed zero-based uniform vectors is probably a good thing...
<mark_weaver>wingo: I'd be in favor of doing as you suggest: add weak-vector interfaces to 2.0, deprecate 'vector-ref' and 'vector-set!' on weak vectors in 2.0, and remove that functionality in 2.2.
<mark_weaver>we could do that in 2.0.10
<wingo>cool
<wingo>maybe i should do that today
<mark_weaver>sounds good to me.
<mark_weaver>I don't see a compelling reason to get rid of the uniform-vector interfaces, other than to clean out cruft.
<mark_weaver>I agree that they're kind of dumb, but they don't really cause any harm either.
<mark_weaver>actually, could they just be made into aliases for the corresponding array operators?
<lloda>yes, but that's 1 step from removal.
<mark_weaver>sure, we could deprecate them too.
<lloda>I'm all for it
<mark_weaver>but we could wait until 2.4 to remove. if they're just aliases, they don't use up much space.
<wingo>so it doesn't seem that it's possible to create weak vectors from scheme
<mark_weaver>interesting.
<wingo>indeed.
<mark_weaver>are they used in guile at all?
<wingo>fwiw i think it's a good interface -- you can make weak refs using weak vectors
<wingo>lemme check
<mark_weaver>yeah
<wingo>yes they are used
<mark_weaver>how are users supposed to check whether an element has been GC'd?
<wingo>but never in a publically accessible way
<wingo>they are used in a thread's mutex set, and in the guardians impl
<wingo>in 2.0 anyway
<mark_weaver>what happens if 'vector-ref' is used to access an element in a weak vector that has been GC'd?
<wingo>neat, the constructors in C in stable-2.0 are internal
<wingo>mark_weaver: vector-ref has a different path for weak vectors
<wingo>it takes the gc lock and accesses the element within the lock
<wingo>ok, so we don't need to do anything in stable-2.0, and actually we should add the scheme-level interfaces in 2.0
<mark_weaver>sounds good, but can you answer my question? :)
<wingo>aaah, it's (ice-9 weak-vector)
*wingo sees now
<wingo>mark_weaver: i think i did?
<mark_weaver>what happens if 'vector-ref' is used to access an element in a weak vector that has been GC'd?
<wingo>it returns #f
<mark_weaver>okay
<mark_weaver>thanks :)
<lloda>returning #f is not good...
<mark_weaver>well, I suppose it won't be important to make 'vector-ref' and 'vector-set!' non-generic until we have native-code gen.
<mark_weaver>or will it?
<wingo>lloda: why?
<mark_weaver>if we don't have native code gen until 2.4, then it seems to me we could postpone making 'vector-ref' non-generic until 2.4.
<mark_weaver>I guess it just depends on how many people are using 'vector-ref' on non-vectors.
<mark_weaver>it might be more considerate to have a longer deprecation period.
<lloda>wingo: well the return for a call that worked might also be #f
<mark_weaver>hmm. why would you put #f into a weak vector?
<wingo>right, what mark_weaver said :)
<mark_weaver>if there was a use case, I suppose we could use SCM_UNDEFINED instead, and add an interface to check for it. but it would be one more gratuitous incompatibility with the past.
<wingo>i think i'll make (ice-9 weak-vector) #:replace vector-ref, vector-set!, and vector-length.
<wingo>er, humm...
<wingo>hum.
<civodul>wingo: uniform-vector-read is already deprecated actually
<mark_weaver>wingo: that might be reasonable, but if so, they should be deprecated I think.
<mark_weaver>I mean that using those names should trigger a deprecation warning.
<wingo>ah, this is what i'll do: have (ice-9 weak-vector) export weak-vector-ref et al
<wingo>and make vector-ref signal a deprecation warning.
<mark_weaver>well, hmm, it would mean that any module that imports weak-vector would have their normal vector ops be much slower.
<wingo>civodul: then that means it's gone in master, woo :)
<mark_weaver>cool! (about uniform-vector-ref)
<mark_weaver>but about making 'vector-ref' non-generic in 2.2, I'm having second thoughts.
<wingo>mark_weaver: civodul was talking about uniform-vector-read; different interface :)
<mark_weaver>oh.
<civodul>yeah, sorry for the confusion
<mark_weaver>wingo: here's my question: will there be a significant performance benefit to making 'vector-ref' non-generic in 2.2?
<mark_weaver>or will the performance benefit not be noticeable until 2.4?
<wingo>mark_weaver: i think there will be benefit in 2.2
<mark_weaver>what benefit?
<wingo>1. it's conceivable that we add low-level vm ops in 2.2. 2. non-generic vector-ref is better for type inference, alias analysis, etc
<mark_weaver>because we should compare that benefit to the grief we'll hear from users about the very late deprecation warnings that many of them won't see before 2.2 is released.
<wingo>that's ok, they don't have to switch to 2.2 directly :)
<wingo>i mean, the situation is a little bit different than in the past -- upgrades happen now when the user chooses to do so; it's our job to make it easy and attractive to do so
<wingo>and i think we're doing pretty well on both those fronts
<wingo>it's a simple procedure for people that don't want to read NEWS -- just make sure the app runs in 2.0.X without deprecation warnings, then switch over
<mark_weaver>okay, but can you help me understand the two benefits you listed? I don't know what #1 means at all. and I don't see what benefit the type inference will get us.
<mark_weaver>I can see the benefits when we have native code gen, certainly.
<mark_weaver>but I don't see the benefit before then.
<wingo>we could have raw cell operations in the vm -- take a SCM, deref it directly without checking bounds
<wingo>or type or anything
<wingo>that is 1
<wingo>that can work for other unboxed values as well
<wingo>doubles, etc
<wingo>(2) is about, let's say we have (vector-set! v 2 x) (vector-set! v 2 y)
<wingo>do we have to do the first set or not?
<wingo>does it simplify to (vector-set! v 2 y) or not?
<wingo>it's the kind of reasoning that we are starting to do now, but which is not really sound for generic operations
<wingo>dunno
<mark_weaver>okay, I see. thanks.
<mark_weaver>well, I still worry that we'll make developers angry with us.
<wingo>the other thing is that with this change the vm op will be faster because we can inline it completely
<wingo>the slow path never rejoins control flow
<mark_weaver>there was a lot of unhappiness over the 1.8 -> 2.0 transition, as you know.
<mark_weaver>and that was unavoidable of course.
<mark_weaver>but if there's unnecessary pain in the 2.0 -> 2.2 transition, I worry that we'll further damage our reputation, and that will make it less likely for developers to want to use guile, on the grounds that we're not a sufficiently stable platform.
<mark_weaver>I mean, on the one hand, there's a lot of new excitement around guile, and for good reason I think.
<mark_weaver>I dunno... just worried.
<mark_weaver>I definitely think that 'vector-ref' needs to become non-generic. as I said, I feel strongly about that.
<mark_weaver>but it's too bad that we didn't get a deprecation warning into 2.0 much sooner.
<civodul>we could poll guile-user
<civodul>"do people use vector-ref as a generic?"
<civodul>just to get a feel
<mark_weaver>sure, it would be good to have more data :)
<lloda>generic vector-ref is buggy anyways, it doesn't (didn't) work with base!=0 arrays
<lloda>and nobody wanted that fixed
<civodul>right
<civodul>that's an important argument :-)
<lloda>or it may mean that nobody uses base!=0 arrays
<civodul>yeah, that's what i was thinking
<civodul>hm
<lloda>I sure don't, I think it's a bad feature
<mark_weaver>yeah, an unnecessary complication.
<mark_weaver>I guess it's just the bounds checking that is made more complex by it.
<mark_weaver>anyway, I don't feel strongly about whether to make 'vector-ref' non-generic in 2.2 or 2.4. just voicing some concerns, that's all.
<lloda>the bases take space in the array descriptor, complicate the indexing and the checks (like if you have array-map with array of different bases, you have to check both ends instead of just the size).
<lloda>they complicate the reader too. They need special functions like array-shape different from array-dimensions.
<lloda>having bases means we can't use negative numbers to index from the end.
<mark_weaver>I would be strongly against that "index from the end" feature anyway.
<lloda>maybe. Python & J do it, it seems popular.
<lloda>the alternative is 'end' keyword which is more complicated to implement.
<lloda>it could be done in an alternative interface, not necessarily in Guile
<lloda>but if we have bases, it just can't be done
<lloda>and most of all ---the bases are just not useful. They don't serve any purpose.
<mark_weaver>well, I can imagine uses for them, but I agree that the benefits are not worth the associated costs. not even close.
<mark_weaver>I've proposed it before, and I'll do so again: maybe we should introduce a new set of interfaces, for simple multidimenstional arrays with fewer features, that could be implemented more efficiently.
<mark_weaver>and then implement the more complex legacy arrays on top of those.
<mark_weaver>the simpler ones would always have base=0, and would not support the affine transformation thing at all.
<mark_weaver>(no shared arrays)
<lloda>I disagree. You need the affine transformation to be useful to do anything with rank>1 arrays at all, even compact ones.
<lloda>s/to be useful//
<lloda>there's no space for a simpler multidimensional array type.
<lloda>e.g. if you don't have sharing, you can't even take a column from a 2D array.
<mark_weaver>well, you could copy it.
<mark_weaver>but I guess you need to be able to mutate it?
<lloda>if you cannot mutate, then you can't really use large arrays
<lloda>I mean as large as your memory
<mark_weaver>but you're saying that you need to be able to mutate a column of a 2D array, with the code written as if it's mutating a 1D array?
<lloda>it doesn't cost more to handle a shared array. For 1D array you can save the step size, but for rank>1 there's just no advantage.
<lloda>and yes, I think it doesn't make sense to need different code to handle those situations
<lloda>because the code is going to be similar even at a low level, once you go to rank>1
<mark_weaver>okay
<lloda>hm, sorry for the rant
<mark_weaver>no, not at all. I didn't take it as a rant.
<lloda>ok
<lloda>I think I've written about this before. Many libraries that take 2D arrays take only particular forms.
<lloda>For example they require the last stride to be 1.
<lloda>This is very common.
<mark_weaver>However, I think that your claim that is doesn't cost more only holds true if you can optimize and entire nested loop structure, so that you can convert the individual index computations into loops where much of the index computation is hoisted out of the loop.
<mark_weaver>sorry, that was awkwardly written. I'm overtired.
<lloda>No, I think I see what you mean, but I don't think it's true.
<mark_weaver>and I wonder how often we'll be able to actually accomplish that.
<lloda>ok.
<lloda>to give an example, you have a 2D array.
<lloda>you don't support shared arrays, so the array is compact.
<lloda>you don't need a stride for the last axis, because the array is compact, so you know this is one.
<lloda>However you still need a stride for the other axis, because the number of columns is a runtime variable.
<lloda>so the only thing you've saved for general indexing is the last stride.
<lloda>for a 2d array that may matter. (I don't think it will in practice, but I accept there's a difference).
<lloda>for a general n-d array, the difference becomes trivial if it wasn't already.
<lloda>in exchange, you can't do any array views without copying.
<lloda>can't delay array operations, like transpose.
<lloda>can't do any broadcasting without explicitly copying the array or writing nested loops for each specific case of broadcasting.
<lloda>also can't handle Fortran-order and C-order transparently.
<mark_weaver>well, let's see. for a non-shared compact 3D array with dimensions (a b c), I think index (x y z) becomes offset (((x*b)+y)*c)+z. is that right?
<lloda>yes.
<mark_weaver>so what's the formula for an arbitrary shared 3D array?
<lloda>m*x + n*y + o*z
<lloda>you've saved one product.
*wingo pushed the weak-vector thing to stable-2.0, adding weak-vector-ref et al, deprecating vector-ref on weak vectors
<lloda>ok, b + m*x + n*y + o*z
<mark_weaver>right :)
<lloda>which is the definition ---'an affine transform'
<mark_weaver>okay, so you save one product and one sum.
<lloda>right.
<mark_weaver>for a 2D array, that means twice as many products and twice as many sums :)
<mark_weaver>a fairly common case.
<lloda>You really think it's worth it? having a separate type and interface for this? and the cost of that sum & product will be hidden beneath whatever else you are actually accessing the array for.
<wingo>the memory access costs would dwarf arithmetic ops.
<mark_weaver>I'm not coming to any conclusions here. just observing.
<mark_weaver>well, okay, it's not worth it.
<mark_weaver>I'll defer to your judgment lloda.
<mark_weaver>As long as I have cheap 1D vectors, I'll be happy.
<wingo>and the ability to build whatever n-dimensional interfaces on top in scheme, whee :)
<mark_weaver>indeed
<lloda>mark_weaver: thanks. it's good to have it challenged, though, because I do have prejudices.
<lloda>wingo: yes to that. Having arrays done in Scheme would be good for peval, type inference
<lloda>I think
<mark_weaver>lloda: I've been challenged before, and it's not pleasant. sorry about that.
<lloda>not at all ;p
<mark_weaver>it's too bad about the base!=0 thing though.
<mark_weaver>as you say.
<lloda>yes :( worst is that I don't think this is actually used (or widely used).
<lloda>there's another behavior that I would ask to have changed. It's about array-map! and array-copy!.
<mark_weaver>maybe we should poll guile-user about base!=0, and then maybe deprecate base!=0.
<lloda>I'm afraid that you won't hear a peep in the ml, but then lots of bug reports at release
<mark_weaver>well, we could always undeprecate if there's too much of an uproar.
<mark_weaver>lloda: so what about array-map! and array-copy! ?
<lloda>yes. So they don't require the argument sizes to match.
<lloda>but they have opposite behavior.
<lloda>for array-map! the src must cover the range of the dst. So src must be equal or larger.
<lloda>for array-copy! it's the other way around, dst must be larger or equal than src.
<mark_weaver>why not improve both so that it doesn't matter which one is larger?
<mark_weaver>seems cheap enough to do.
<lloda>that's one way.
<mark_weaver>what would you recommend?
<lloda>require the ranges to be exactly the same.
<mark_weaver>why break existing code without a compelling reason?
<lloda>hm. good question.
<lloda>I actually never used the different-sizes feature. I only found about it when I saw that half my array wasn't set and there was no error.
<mark_weaver>fwiw, R7RS, SRFI-1, and SRFI-43 all allow unequal sizes for these kinds of ops.
<lloda>that's true...
<mark_weaver>well, for 'map' anyway.
<lloda>for array-map! and array-copy! though, you need to either 1) construct a shared array to the view on either side, or 2) have a just-for-this-purpose array traversal op that handles the extra bounds.
<lloda>2) is what is done in ramap.c, iirc
<lloda>so to handle the difference btw array-map! and array-copy!, the arguments are exchanged in the call to that traversal function.
<lloda>b/c logically array-copy! is a subset of array-map!, so they are both written in terms of that traversal operation.
<lloda>it would simplify the traversal op to do 1)
<mark_weaver>well, it's not just one shared array, right? array-map! accepts any number of sources, and they all could be different, or no?
<mark_weaver>the docs make it sound like all the arrays could be different shapes.
<lloda>yes, this is true.
<lloda>so iiuc you're saying that a traversal op that runs over the sources using the bounds of the destination makes sense.
<lloda>that's exactly how array-map! is implemented, in fact.
<mark_weaver>well, actually I'm going a bit further and saying that rather than using the bounds of the destination, it could use bounds that are the minimums of all the bounds.
<mark_weaver>or the intersections of all the bounds, I guess.
<lloda>that would make sense.
<lloda>actually the why it's now also makes sense in a way.
<lloda>b/c if you do (array-copy! src dst), you kind of expect all of src to be found somewhere in dst. Or I least I think that was the idea.
<mark_weaver>the case where all the shapes are the same could be handled specially, if it's more efficient to do so (I guess it would be)
<mark_weaver>but I really can't say whether any of this is worthwhile.
<mark_weaver>I only thought of it because I assumed you didn't like the asymmetry of how those two procedures deal with src vs dst.
<mark_weaver>but handling the special case might be worthwhile, regardless.
<lloda>it's true I didn't like that.
<lloda>ok, I accept that I don't have a good case.
<mark_weaver>if I were writing this code, I would probably include a few special cases.
<mark_weaver>or at least experiment with them and see if they were worthwhile.
<lloda>I rewrote that traversal and added some more cases where the nested loop can be linearized (to do the op in a single loop). The difference between array-map! and array-copy! makes this harder because inside the traversal routine I don't know which argument is the destination.
<mark_weaver>well, I suppose that the code of calling 'proc' probably dwarfs the other things.
<mark_weaver>so 'array-copy!' might be the only one where special cases are worthwhile.
<mark_weaver>s/code/cost/
<mark_weaver>yeah, now that I think about it, I'd guess that special cases in 'array-map!' are not worthwhile, but that 'array-copy!' should have its own implementation with a few special cases.
<wingo>civodul: wdyt about adding a setvbuf pointer to the scm_t_ptob_descriptor in 2.2?
<wingo>seems like the right thing
<mark_weaver>wingo: did you see my patch for that?
<wingo>mark_weaver: no, i didn't; is it for 2.0 or 2.2?
<mark_weaver>for 2.2
<wingo>neat
<mark_weaver>wingo: I CC'd you. Date: Tue, 21 Jan 2014 02:41:05 -0500 Subject: Re: Making custom binary input ports unbuffered
<wingo>i see it now
<mark_weaver>civodul raised some questions and we kind of stalled.
*mark_weaver goes to take a nap
<civodul>wingo: agreed of course, and mark_weaver was working on it
<civodul>i think we're pretty much there
<wingo>hrm
<civodul>or maybe i was just being slightly boring as usual ;-)
<wingo>i need a merge from stable-2.0 so i can continue with lloda's thing, i think
*wingo grumbles
<civodul>ah
<civodul>have a nap synchronized with that of mark_weaver maybe? :-)
<civodul>so the only issue was about the way scm_setvbuf would be exposed
<civodul>i mean, scm_set_port_setvbuf
<civodul>maybe we can go ahead with what mark_weaver had, and refine things eventually
<wingo>cool; seems to me i should just merge it myself; the changes needed are really trivial, and we can fix other things later; i don't need to be blocking on that discussion
<civodul>apparently merging things as is from 2.0 to master wasn't trivial
<civodul>which is why mark_weaver came up with an actual "port" of the functionality
<civodul>dunno
<wingo>heya joolean :)
<joolean>hey wingo!
<wingo>civodul: probably it was an easier merge for me because i was the one who refactored the ports code in the first place
<civodul>ok
<wingo>whew, merged stable-2.0 to master
<davexunit>yay!
<wingo>civodul: wdyt about deprecating uniform-vector-ref, sound ok to you?
<wingo>the replacement would be array-length
<civodul>array-ref
<wingo>and array-ref
<wingo>and such
<civodul>yes, sounds good
<wingo>cool, will do
<civodul>i didn't know it existed :-)
<wingo>:)
<civodul>so we had both generatlized-vector-* and uniform-vector-*
<civodul>woow
<civodul>:-)
<civodul>*generalized-vector
<wingo>so much generality :)
*wingo steps out, biab
***Guest25985 is now known as micro
<wingo>civodul: do you have a brain cell for a quick question?
<wingo>n/m i think i have the answer
<mark_weaver>wingo: another array question?
<wingo>mark_weaver: yes, but it was a silly question :)
<mark_weaver>okay
<mark_weaver>thanks for merging 2.0 into master! I very much wanted that done, and was dreading that job :)
<wingo>no problem :)
<wingo>i went ahead and did it without your patch, as you saw; we can fix up the interface in master as we like
<mark_weaver>*nod* that's probably best. as you said, it was easier for you because you refactored ports.
<mark_weaver>wingo: the 'subr' argument to 'scm_out_of_range' in 'scm_c_weak_vector_ref' really ought to be non-NULL.
<mark_weaver>ditto for 'scm_c_weak_vector_set_x'. no?
<wingo>mark_weaver: indeed!
<wingo>i have to head out for the evening
<mark_weaver>also, you mentioned before that a mutex had to be locked. why not here?
<mark_weaver>okay, ttyl!
<wingo>regarding the mutex, yes it needs to be locked here
<wingo>the current code does not do it in stable-2.0
<wingo>it does in master
<mark_weaver>should we fix that in 2.0?
<wingo>as you like
<mark_weaver>heh, okay :)
<wingo>:)
<mark_weaver>sneek: later tell wingo: Regarding O_NONBLOCK, you asked "wouldn't dupping the fd be sufficient, somehow? is the object with the O_NONBLOCK the same object with the seek position?"
<mark_weaver>hmm, where's sneek?