IRC channel logs
2025-06-19.log
back to list of logs
<dariqq>turns out I was dumb yesterday, I can just dump the data from my port to the input of the pipeline. It works great for small data but the archive i tested with was a bit to large. I dont know yet why this causes waitpid to wait forever. Smaller archives are almost instant <dsmith>dariqq, You can deadlock if you are reading from the pipe while the pipe is waiting for input. Your smaller sizes are prob within the pipe buffer size. <dsmith>Also, your input to the pipe might need to be closed so the pipe sees an EOF <dsmith>The default pipe buffer size is usually 64K on Linux (16 pages, a page is usually 4K) <dariqq>dsmith: yeah it looks like it gets stuck writing things to the input pipe. I am reading from my port with the data in 16k chunks and then dump those into the pipe <dariqq>but some of the things that work are like 5M big but for others the input pipe dies after 1-2M <dsmith>Also, it can get stuck the other way around. The pipe output might have a full buffer and won't do any more reads until it can finish it's write. <dariqq>dsmith: You mean that i should start reading from the pipe output sooner? <dsmith>dariqq, Could be. If you are decompressing, the output should be larger than your input. <dariqq>that seems to work (i am just draining it into the void for now) but a lot more annoying because now I need to juggle 4 ports around to benefit of the output. Ideally i would want to connect the pipe input to my input port automatically so I dont have to keep track of all of the data I am writing into the pipe and only need the pipe output and the pid <dariqq>i tried redirect-port but then xz is seeing that the input is closed and aborts. so i would need to set this up before and hten I am back at the very manual filtered-port from (guix utils) <dsmith>Pipes are tricky when you have one process both feeding the input and consuming the output. In a "normal" pipeline (in the shell) they are separate processes, and so not a problem <dariqq>dsmith: I cleaned up the procedure from a guix a bit (mostly dup2 -> redirect-port to keep the offset in the port) which makes it a lot less scary to look at and i dont have to manage the data between my input port and the pipe input. Thanks for your help on this <rlb>dariqq: in general, another traditional way to handle that is to use a separate thread for either the input side or the output side, but then you have to be careful wrt concurrency -- how careful depends on the language/platform, etc. <dsmith>Was considering mentioning threads, but then it's a "now you have two problems" thing. <rlb>Hah, I thought about including "now you have two problems" :) <dariqq>threads would definitly be overkill. I am just trying to reimplement the rpm2cpio.sh script in guile and I have to say i am not a fan of the format. <ieure>Does Guile have something like ignore-errors? <ieure>I need to delete-file something that might not exist, and don't care if that's the case. <dariqq>something like false-if-exception <rlb>Can guile optimize away code/checks based on const optional arg values, e.g. if you have (define* (foo x #:optional want) (case y ((#:rainbow) ...) ...)) and then calls like (foo ... #:rainbow)? <adamhume>try ,opt (let () (define* (foo x #:optional y) (case y ((#:rainbow) x))) (foo 1 #:rainbow)) <adamhume>this simple case will be optimized, and constant 1 is residualized <adamhume>TIPS: If you want to test how your function will be optimized, wrap it in a let body. If you just define it in top-level, Guile will not try to optimize it