IRC channel logs
2013-10-27.log
back to list of logs
<dje42>does it make sense for a random access port to have a non-zero seek value for the first byte? <dje42>If it does, it's not easy to seek to the beginning. <dje42>I ask because I'm trying to decide semantics of a gdb+guile memory port that has a restricted memory range. Is the first byte at 0 or at the address of the first byte ? <mark_weaver>dje42: I think seeking to 0 should probably go to the beginning of the accessible range. <mark_weaver>maybe it could work the other way, but it sounds dicey to me. <dje42>The problem I'm trying to solve is then how to have, e.g., the disassembler provide the real address. <dje42>I'll need to provide some out-of-band means. <mark_weaver>well, if the disassembler takes the port as an argument, could it not also take the starting address as the argument as well? <dje42>Hmmm, which is actually something I need already, e.g., if I want to disassemble code in a bytevector that "comes" from a particular address <mark_weaver>I dunno, if you find it makes the code awkward, you could try making the seek values be the same as the addresses. maybe it'll be fine. <ijp>are prots really the right thing for what you are doing? <mark_weaver>the main thing is, there's a possibility that some guile code will read some bytes starting at 0 before you get a chance to seek. even if the current guile doesn't do it, maybe some future guile will. <mark_weaver>hmm, well, actually I take that back. we really shouldn't be doing that for arbitrary ports. <dje42>There are lots of reasons to access memory, and I'm not thinking ports will be the only way. But, <mark_weaver>although seekable ports are generally treated a bit differently, in general. I suspect that there's an assumption in a lot of code that seekable ports are not something you have to worry about doing speculative reads on. <dje42>for the disassembler, there are lots of sources of input (memory, code already in guile via some random means), files, or whatever. <mark_weaver>and it's virtually unheard of to have a port where you can't read the beginning but you can read from some later position. <dje42>Providing support in the disassembler for using ports, provides a means to handle lots of cases without adding complexity in the disassembler itself. <mark_weaver>dje42: this is for inferior memory, right? (I might not be using the term correctly) <dje42>correct, and that's a perfectly fine term <mark_weaver>okay. out of curiosity, what will your custom binary port have to do to read the inferior memory? <mark_weaver>okay. I was just wondering if there was a better way to do this, but I guess not. <dje42>Note that the inferior could be running on a different machine. e.g., gdb<->gdbserver <mark_weaver>dje42: one thing to be aware of is that guile ports have a buffering system that sits on top of your custom binary port. so the reads done from scheme will not map directly to calls to your custom port. <mark_weaver>in particular, guile will generally try to read a whole buffer at a time. <mark_weaver>if you need to, you can disable buffering, but that will be quite inefficient. <mark_weaver>it's okay to return less than was asked for, though. (not sure how exactly that works in the R6RS custom port API). <dje42>I was thinking of providing an API to the user to set (and reset) the buffer size. <mark_weaver>and of course that's how file ports work without causing unnecessary blocking from things like terminals or sockets. <ijp>mark_weaver: iirc you are supposed to return the amount of bytes you wrote to the supplied bytevector <ijp>I haven't written a custom binary port in a few years <mark_weaver>(I use the term "file ports" for anything that uses a POSIX file descriptor, not just actual files) <dje42>GDB has its own built in buffering (the "dcache") that does a similar thing. E.g. you want to minimize target memory reads when doing a backtrace. <ijp>I think if you look in the mosh repository there are test cases in there for custom ports that I wrote <mark_weaver>dje42: especially when one considers debugging over a serial link, it's good that gdb has a cache. but it also makes sense for guile have a buffer that's native to it, because that allows it to look at that buffer efficiently. <dje42>Not quite sure yet how it will all play out. Until there's a working prototype it's hard to collect real data. :) <tupi>mark_weaver: hello, any thought on the (ice-9 popen) / thread unsafe bug i submited? <mark_weaver>tupi: it weighs heavily on my mind from time to time. thanks for reminding me. <mark_weaver>tupi: what makes it all the harder to fix is that none of us have any machines that are capable of reproducing the problem. so all I can do is think it through carefully and try to get it right so you dont have to ask your client to test too many things. <mark_weaver>a while ago, I made a patch to fix the problem for good, and then ran into deep problems with asyncs, which I posted about on guile-devel, but no one responded to it. <tupi>mark_weaver: yes I understand that. and to make it worse, I also have extreme difficulties to have access to the server .. I need to take a plane, ask an employee to authorize the machine out of the lab [-> in his house] and then I can access internet , try the patch ... [the customer's lab has total restriction on accessing internet in and out ... <mark_weaver>tupi: the async part of the code is the part that closes pipes that are no longer accessible (as detected by GC). <tupi>i must confess that i am very concerned, since I recommended guile as the solution and route to follow to control processes ... but i guess it will eventially work ... <mark_weaver>if you are sure that you close all pipes manually, then you could perhaps simply disable that part of the code entirely for now. <mark_weaver>another option would be to spawn a separate thread whose sole job is to periodically free up pipes that are no longer accesible. <tupi>i guess i will have to do that untill a solution emrge. i actualy do close all pipe manually, because I need to status id to check if it did work ... <mark_weaver>of course, if an exception is thrown while a pipe is open, it might not get closed. <tupi>the later i don't understand, how you do that? <tupi>anyway, I hope you'll [and developers] find a solution <mark_weaver>well, do you use 'catch' anywhere in your code to catch exceptions? (you needn't, just curious) <tupi>I think that for the time beeing, I'll not close pipes anymore unless in debug mode [so that i can still close them while working on lower speedy machines and get the status [it would be difficult not to while debugging ...] <mark_weaver>tupi: there are two global objects in (ice-9 popen) that need to be protected by mutual exclusion: (1) port/pid-table, and (2) the guardian. <tupi>mark_weaver: if you look at the code [bug 15683 related email] you'll see exactly what's going on <mark_weaver>it occurred to me recently that we don't need a global port/pid-table at all anymore, because now ports have private alists, so we can put the pid there. and then there's no mutex needed. <tupi>i only use the mutyex to right to my log port <mark_weaver>unfortunately, port/pid-table is exported from the module, so we probably can't get rid of it in the stable-2.0 series, but you could get rid of it in your private version. <tupi>i don't use it to close pipes ... should I ? <tupi>mark_weaver: yes, anythink that get rids of the bug will do: we could even create a special git branch ... <mark_weaver>the guardian is much trickier. it also needs to be protected by a mutex, but a mutex isn't even enough, because it will also get used within the async that gets queued as part of the after-gc-hook. <mark_weaver>tupi: anyway, this is a lot of technical detail, but here's the summary version: I could give you a patch fairly soon that should fix the problem if you are willing to give up the auto-reaping of pipes that are never closed. <tupi>i don't understand the very last part of the sentence, sorry <mark_weaver>right now, if pipes become inaccessible (i.e. there are not referenced anywhere, and thus the garbage collector decides they are garbage), they will be automatically closed and freed. <tupi>i close all my pipes: you mean 'other pipes' that would not have been closed .. that's fine by me, they would be pipes opened by gule itself ... did i get it rioght? <mark_weaver>if you are willing to give up that up, then I can provide a fix for you fairly soon. <mark_weaver>tupi: I think it's not something to worry about for you. <mark_weaver>tupi: it's not even all pipes, just pipes created by the (ice-9 popen) module. <mark_weaver>dsmith: what got entered into his database, and how do we remove it? <tupi>mark_weaver: i need to close the pipes manually and get the status:exit-val, because the user could make errors [like passing wrong image type ...] in which case i raise an error [later i would like to catch these in a better way but ...] <tupi>if your patch let's me do that, it is perfect by me <mark_weaver>we'll also get it fixed in core guile-2.0 as well, but that will be a harder fix. but I don't want to keep you waiting any longer. <ijp>it seems to me it was a mistake to expose the pid table anyway <ijp>not that that module is a shining example of guile <sneek>Its been said that if you is willing to give up that up, then I can provide a fix for you fairly soon. <tupi>mark_weaver: if you prefer to get it fixed in guile-2.0 and think you can get a solution before november the 17th, it will be fine [i did ask a month to solve it ...] if that avoids you spending 'twice' the time ... <mark_weaver>tupi: can you send me the popen-thread-safety patch that you're using on the big server? the one that doesn't quite fix the problem? <dsmith>I think my fresh master make check is hung in the pitfalls test <mark_weaver>if I'm not mistaken, it might be enough to simply comment out the line that registers the 'after-gc-hook'. <dsmith>sneek, forget nalaginrut: It seems, in-terms of libraries, the rule of thumb <tupi>mark_weaver: you did push it at some point, because I am [just] using git stable branch ... am i mistaken here ? <mark_weaver>tupi: I never pushed it. I knew it wasn't quite sufficient. <dsmith>sneek, forget Basically if things <tupi>mark_weaver: let me check <dsmith>sneek, forget That reminds me. Just got an offer for a qube-3. This <mark_weaver>tupi: the one remaining problem was the auto-reaping. <mark_weaver>I'm pretty sure that asyncs are a fundamentally broken idea. <tupi>mark_weaver: if i paste the (ice-9 popen) that all machines use, would that be good enaough? i can not find the patch per se anymore <ijp>so asyncs are just fancy hooks <mark_weaver>hooks are executed at well defined times. asyncs are not. <mark_weaver>sorry, I should have been more specific. everything I've said about "asyncs" are really about "system asyncs". <ijp>well there are other problems with that analogy <ijp>mark_weaver: my hook analogy <dsmith>Sorry for the sneek spam. Should have been doing that in a /query or something <mark_weaver>tupi: ah, well, that doesn't even contain the partial fix I sent you a while back. that's *completely* thread-unsafe :) <tupi>my mistake then! i was convinced that you did push [and did not even check] oh man ... <mark_weaver>tupi: sorry, it was perhaps a reasonable expectation that I would have done so by now. <tupi>let me try to find the patch <mark_weaver>the (ice-9 popen) module has long been an embarrassment, in more ways than one :-( <tupi>mark_weaver: i need to know if you think you'll get a patch by the 17th or if i start not to close pípe [unless in debug mode ...] <mark_weaver>tupi: you'll definitely get a patch by then. probably long before then actually. <mark_weaver>and not closing the pipe would not fix the problem anyway. <mark_weaver>if you don't close it, the auto-reaper will try to close it, and run into the same problem. <tupi>ok, i will stand-by then. i will try to get some funding for your patch, keep track of your hours [but it is not a promisse, i am not the boss...] <mark_weaver>no promise needed, and donations always most welcome :) <cky>mark_weaver: Agree about (ice-9 popen) being an embarrassment. I once saw a post on codereview.stackexchange.com asking about that, and I wanted to write some comments for improvement, but couldn't write one that didn't seem subpar. Then I realised that the problem was in (ice-9 popen) itself. <cky>I'll try to devote some time this afternoon/evening towards committing some of my macro hygiene changes (the longest part of preparing commits, whether at work or for personal stuff, is writing the ChangeLog/commit message). <ijp>take a shot for every misuse of dynamic wind <cky>ijp: What would you see as a misuse of dynamic-wind? <cky>I mean, unless you and I are talking about different things, I'm not getting drunk any time soon. ;-) <ijp>dynamic wind is not for managing resources <ijp>it's basically worthless <cky>Next you'll be telling me that call/cc is basically worthless. ;-) <ijp>I can tell you that if you'd like <ijp>I'll even do it with a straight face <mark_weaver>cky: I know of problems with popen, but I don't see one that's demonstrated with that question. <ijp>cky: if you save a continuation in the body of that dynamic wind, and reenter you need to create a whole new pipe <cky>mark_weaver: No, I mean that I want to refactor that code into something that is more reliable, but (ice-9 popen) doesn't provide very many facilities to make that happen. <ijp>you've lost all the context that was available when you stored the continuation <ijp>since the whole point of dynamic wind is to do that, it's an abuse <ijp>actually, it's worse, because it doesn't even create a new pipe <mark_weaver>cky: 'with-input-from-port' is probably what's needed there. <ijp>you reenter, then all that's going to happen is half a second later you get an read error <ijp>unless you handled your eofs properly <cky>mark_weaver: For example, I'd like to make a call-with-input-pipe first, then have the OP's with-input-from-pipe reuse that (while setting the current-input-port parameter/fluid). <mark_weaver>but it would be noted that the 'with-*' and 'call-with-' port procedures do *not* close the port on errors, for better or worse, but that's a longstanding API decision in scheme (not just guile) <cky>Wow, that's news to me. Holy cow. <mark_weaver>they only close the port if the inner procedure returns normally. <ijp>there have been at least 3 different articles at various times amounting to "dynamic-wind is not unwind-protect" <cky>ijp: I know that Riastradh wrote such a one. <ijp>IIRC the pregexp author did one, and I know one common lisper has <mark_weaver>cky: but if you want to implement something like that, I don't see why the guile popen API doesn't allow it. <cky>ijp: So now both call/cc and dynamic-wind are worthless, in your view. Now what? :-P <ijp>do you want the list? <mark_weaver>apart from the lack of thread safety, the main flaw I know of in (ice-9 popen) has to do with creating bidirectional pipes. <add^_>ijp: :-D I want to see that list! <cky>mark_weaver: Bidirectional pipes were always a little flaky, even in shells. <cky>mark_weaver: It seems few people know how to use them correctly. <mark_weaver>really, there are two separate pipes, but (ice-9 popen) melds them together into a single port object. <ijp>basically call/cc exists to implement dynamic continuations, and dynamic-wind exists to implement paramters, and that's it <ijp>s/dynamic/delimited/ <mark_weaver>there are efficiency issues, because legacy guile soft ports (which ice-9/popen uses to meld the ports together) don't allow for efficient implementation <cky>call/cc can implement delimited continuations? Hallelujah! <ijp>cky: yes .... if you have sate <mark_weaver>but more seriously, there's no way to close the pipe that feeds input to the process before closing the pipe that reads output from the process. <mark_weaver>it means that you cannot, for example, pipe something through grep and have it work. <cky>Right, and that's why making them a single port is a flaky idea. <cky>Some, like djb (IIRC), have argued the same for sockets. <cky>Namely there should be separate fds for the read and write sides. <cky>That would eliminate the need for shutdown(2). <mark_weaver>quite possibly, the idea of read and write sides of _any_ port being melded together is a mistake. <cky>Even for files, I'd argue that the two should be separate. <cky>Then you don't need separate get/put positions. <cky>(Seek positions, I mean.) <mark_weaver>well, for files the get/put positions shouldn't be separate anyway. <ijp>I hadn't realised they were (or I had, and forgot) <mark_weaver>the idea is that you can issue an arbitrary sequences of operations including read, write, and seek. <mark_weaver>I know it's all a bit messy, but seekable ports are somehow different than streams, it seems to me. <mark_weaver>admittedly, this is probably not a popular conception of file I/O among functional programming circles, but it can be dramatically more efficient in some important cases. <mark_weaver>consider what you need to write an efficient database that's backed by a single file or device, for example. <cky>.oO(This is probably why I do not maintain SQLite. ;-)) <mark_weaver>or a filesystem for that matter. if we had a scheme machine running on the bare hardware, what would it do with the hard drive? <mark_weaver>at some level, one has to deal with the realities of the physical world, which is mutable :) <mark_weaver>unless you want to buy a new 1TB hard drive for every 1TB written :) <ijp>hard to say, we're so biased by current choices <gzg>mark_weaver: Did Movitz based project ever implement a FS? <ijp>I think it would be nice if our view of a file was more structured, kind of like the sqlite author says "our job is not to replace oracle or mysql, it's to replace fopen" <cky>mark_weaver: Datomic certainly assumes a world where storage is totally elastic. Everything in Datomic is immutable (think git, but applied to a graph database). <mark_weaver>ijp: I agree that we could eliminate some of it -- for example, we could not have a traditional filesystem at all, and just have a big heap on disk. <cky>mark_weaver: So you can look at the state of the data at a given transaction, for example (much like you can "git checkout" a specific revision and see the code as it was then). <mark_weaver>but I think that if you really thought about how to build a purely functional computer from sand, designing everything fresh, you'd still run into problems. <ijp>a lot of technology wants to move towards the flat model for user files <ijp>flat hierarchy that is (if you'll excuse the oxymoron) <mark_weaver>for one thing, at some level you need mutable memory. I'm sorry, but that's just a physical reality. <mark_weaver>at least as long as we're dealing with physical objects made of matter, assuming that the matter is not all disposable. <cky>Well, of course; othewise people wouldn't devote as much time as they do trying to create fast and reliable GC algorithms. <cky>A GC works with mutable memory to create an abstraction of disposable memory. <mark_weaver>ijp: if you have even a vague idea of how to get around that, I'd like to hear it. <ijp>what am I supposed to be advocating for? <mark_weaver>ijp: well, you wrote "hard to say, we're so biased by current choices", presumably in response to my previous message "at some level, one has to deal with the realities of the physical world, which is mutable :)" <ijp>ah, I was responding to the comment before <cky>Anyway, I'm planning to create and push to a cky-hygienic-macros branch, I hope nobody minds. I'll start trickling commits in as I write commit comments for them. <mark_weaver>cky: cool! it would be good if it's not one monster commit. I think it's better if it's broken down into fairly small commits. <cky>And whoever wants to can just cherry-pick in whichever commits look decent enouggh to integraate. <cky>Right, that's the idea. <cky>In fact, I'll push the first one now. <cky>I never work off of master. <cky>Anyway, first commit for your viewing enjoyment: 98a1e95e9997b8a9b38693bd772fb0e20e88a935 <cky>I originally used datum->syntax for that change, but Eli Barzilay made a very good point that syntax parameters are much more appropriate for this scenario. <cky>(Basically: if you're building identifiers out of something the macro caller specifies, then datum->syntax is fine. Otherwise, if you're just injecting a fixed identifier, syntax parameters is better.) <cky>Oops, I pushed to the wrong remote. <cky>(For safety, I set up the Savannah repo to be "upstream", rather than "origin". So in this case, I accidentally pushed to "origin" (which is my GitHub mirror) at first.) <mark_weaver>if that macro were part of our API, we wouldn't be able to change 'push' from a procedure to a macro, but in this case it's fine. <mark_weaver>BTW, I recommend putting "push = HEAD" in the [remote "origin"] (or "upstream") section of guile/.git/config <cky>Yep, but no part of our API should really be injecting fixed identifiers anyway. ;-) <cky>What does "push = HEAD" do? <mark_weaver>if you don't do that, then "git push" pushes *all* branches by default. <ijp>I was bitten by that recently <cky>Oh, I always manually specify the branch to push, for that reason. <mark_weaver>with it, "git push" pushes only the branch you're currently in. <cky>It's an ingrained habit after being bitten by that a few times (at work). <mark_weaver>probably a better solution, actually, in case I forget to add the bit to some other git repo config :) <cky>Luckily at work we have push hooks for everything, so non-compliant commits get rejected straight away. <cky>At work we use a "push first, review later" model. (This works because only very select people can push to master, and our push hooks ensure that only reviewed tickets can be merged into the master branch---yes, it integrates with Redmine (which is the ticketing system we use).) <cky>so commits that fail review cannot get merged into master (whether accidentally or deliberately). <cky>Next commit coming up soon (it's another of the "less-straightforward" translations in (ice-9 i18n), so I had to run all the tests (including installing all the locales the test uses, so that I'm getting all PASSes, and no UNRESOLVEDs) just to double-check). <cky>mark_weaver: Would it break ABI compatibility to change a function from one that used rest arguments to one that used a case-lambda? <ijp>I think it would be fine, though define* might be preferable <cky>This particular use case is not amenable to define*, I think. <ijp>what's the function? <cky>So, in (ice-9 i18n), there's a number of places that are basically (lambda (foo bar . locale) ... (apply nl-langinfo ... locale)). <cky>I basically turned that into a case-lambda to handle the with-locale and without-locale separately, to avoid letting people pass in more than one argument.