IRC channel logs
2016-08-29.log
back to list of logs
<sapientech>hi all, just created a guile package that I want to add to guix. Was wondering what the best approach to compiling and installing it would be. i see some packages that simply copy hte scheme files over, and others that use a makefile + autoconf <sapientech>there arent to many exampels online for using make with pure guile packages, and most are 5+ years old <wingo>yeah i wish we had something simpler <amz3`>there should be better story for pure guile packages in guix than autotools <amz3`>this piece of software is genius <amz3`>it works very well, and is kind of fast! <amz3`>I mean it's much faster than my parser combinator implementation <amz3`>davexunit: héllo, did you push somewhere the code powering your blog, I looking for an example code to have code highlight wiht guile-commonmark <amz3`>OrangeShark: do you use davexunit code highlighter? <amz3`>I don't even know how to fold a tree <amz3`>so it's a rather complicated mission to get it working <OrangeShark>amz3`: ya, there is SXML transforming and tree fold in the manual but I haven't really looked at it yet. The idea would be to find each code block and check for the class with "language-langname" <amz3`>I'll work on the documentation proper and fix that highlighting when dave will be back <OrangeShark>we should also try to add more languages to the code highlighter <amz3`>OrangeShark: you have two new issues on guile-commonmark :) <OrangeShark>amz3`: the link you provided for how it is rendered for markdown-toc is to your localhost <davexunit>I use the pre-post-order procedure from the sxml library to rewrite the sxml tree that guile-commonmark produces <davexunit>the two hacks I do are syntax highlighting, which you obviously know about, and pretending that markdown can handle video tags ;) <davexunit>and sxml makes both hacks astonishingly easy <OrangeShark>davexunit: oh nice, so if it ends it a .webm, you automatically convert it to a video <paroneayea>sapientech: I recommend the autotools route... snarf from Haunt/Sly/Guix/Pubstrate, particularly the pre-inst-env.in, configure.ac, and Makefile.am files <paroneayea>strutucrally, autotools feels not great to me, and I don't like that it generates big macro expanded shell I can't expand, but I have to admit it Just Works (TM) once you set it up <paroneayea>sapientech: good luck getting it into Guix... what's the project? :) <davexunit>I need to reboot my 'guild init' project that will automatically generate boilerplate for new guile projects, including automake/autoconf stuff <amz3`>let's write this documentation <davexunit>I think it will be increasingly important to help newcomers make libraries the right way <OrangeShark>I know I was confused at how to set up a project for guile, so I looked at various projects to see how they did it. <OrangeShark>would definitely be nice to have a way to generate it <paroneayea>looks like I can sign about 500k documents / second with this libgcrypt using ffi code, not bad! <paroneayea>and if I serialize to base64, I can do about 10k per second ;) <paroneayea>I think I see ways to speed up the base64 encoding/decoding that I borrowed from Guix (which borrowed from that other library) but I'll worry about that later. <davexunit>paroneayea: check out the disassembly on guile 2.1 sometime :) <paroneayea>davexunit: will do... the main thing, according to statprof, that's slow on guile 2.0.X, is that using put-char wraps via with-i/o-port-error and going through the custom-throw-handler is slow <paroneayea>and it does it writes by character to the port of the finalized string <paroneayea>if it wrote it all at once to the port, that's one way to speed it up... <davexunit>yeah certainly sounds like there's an opportunity for improvement there <paroneayea>but right now, more important to focus on getting a working demo out there :) <paroneayea>I think I can get the cookie signing finished today, and have a nice working sessions module. <amz3`>ACTION needs a replacement for github <amz3`>hey OrangeShark what are you doing the next few years? <rekado_>with guile-next splitting up the request body at the boundary is now much faster. It's down to 5 seconds for 170MiB. <rekado_>that's not great, but it's a lot better than before. <rekado_>I also replaced "equal?" with "eqv?" <dsmith-work>amz3`: seems like lots of people are moving to gitlab now that gitorious has gone <dsmith-work>mark_weaver: I know you are very picky about these things. <rekado_>paroneayea: I think I'll need cookie signing for my project as well. That's needed for keeping track of user sessions, right? <davexunit>I've thought about writing at least a gitweb replacement in guile <random-nick>is it possible for a github replacement to support git pull requests? <random-nick>yeah, but they are not compatible, I don't think you can't send git request-pull requests to github <OrangeShark>you have to send request-pull to someone who has access to the repo <OrangeShark>it just github only does it between projects on github <paroneayea>rekado_: good to know there are other users out there. I'll work on breaking out guile-webutils this week :) <paroneayea>I'll also be working on a "form" library shortly, using immutable objects + generic methods + sxml <paroneayea>something like the nice form libraries every MVC web framework has thesedays, but with less state management :) <amz3`>I had several times a look at libgit2 but there is so much to bind that it can not be done manually <amz3`>luckily there is an xml spec <paroneayea>davexunit: so, signed data in the session needs to be serialized to something. <paroneayea> - de/serialize just using read/write. I'm still a bit unclear about how safe this is <davexunit>why use json when you can use an sexpression? <paroneayea>anyway, it would be hard to do something malicious: <paroneayea>before (read) would ever be called, you'd have to have a legit signature first. <davexunit>it's just reading in an s-expression, not evaluating it, so it's all good. <paroneayea>davexunit: I remember reading in Land of Lisp that (read) there might not be generally safe because of reader macros <davexunit>1) aren't you in control of that data that is written/read? <paroneayea>2) no, but I don't know much about if there's any edge cases where guile does by defaul <davexunit>are you saying that users can insert arbitrary scheme expressions somehow? <davexunit>or is this an implementation detail on the server side? <paroneayea>so the data is put in the cookie, along with a sig <paroneayea>(but unless you can tamper with the signature it shouldn't matter) <paroneayea>so I'd be reading stuff from the browser passing back a cookie <davexunit>I'm failing to see where the user could insert an expression that 'read' could potentially evaluate due to exploiting a reader macro <davexunit>but that stuff from the browser is an arbitrary s-expression? <paroneayea>davexunit: well you want structured data, so yes it would be... basically, the server will set the data in the session, then tell the browser to set it as the new cookie <paroneayea>and then the browser will pass it back with every request <paroneayea>so it's a string that will be decoded with (read) <paroneayea>davexunit: thanks for talking it through with me! I just didn't want to miss something and then regret it <random-nick>and even if they can sneak in a reader macro you are in control of them <paroneayea>it's too bad that the read/write options are globals <paroneayea>that does mean that importing a library could change your read/write options, it looks like <wingo>i suppose that doc update doesn't touch security at all tho... <paroneayea>wingo: yeah... srfi-10 seems to be one way where (read) can get scary <paroneayea>wingo: really makes me wish I could specify a scope where I *know* how read behaves <wingo>nah, i think a port option or something is what you want <paroneayea>or even a module that provides (read) and (write), splatting over the default ones unless you namespace it :) <dsmith-work>There is also read-hash-extend (I think srfi-10 uses that) <paroneayea>wingo: I saw that there was some mention of being able to control how read works in the port... but it looked like the *data* read by read was able to change read behavior <wingo>how could the data change the behavior? only via #. <wingo>no one in their right mind turns that one on :) <wingo>there's also #!r6rs i think... <paroneayea>wingo: I think this paragraph is not clear to me: <wingo>that changes some parameters, but they are static parameters, and maybe only in certain situations <wingo>anyway not like a danger or anything <paroneayea>wingo: nothing in that section mentions how # gets turned on to enable that <paroneayea>it sounds like *any* port that runs across #!fold-case will start doing that to me.. <paroneayea>but maybe I'm just not understandgin while reading <paroneayea>it doesn't seem like those are dangerous options <wingo>when #!fold-case is in a file that's certainly what you want <wingo>if an attacker can inject #!fold-case in some thing... <wingo>it's an interesting idea, i hadn't thought about that <wingo>mark_weaver will like that one :) <paroneayea>well, the question is whether (read) is "safe" when you get arbitrary incoming data <wingo>we probably need to have a port option to disable these things i guess <paroneayea>in the case I'm in, a signature is checked before it ever hits (read), but could read/write be as safe as json? <wingo>paroneayea: what does "safe" mean in this context? threat model missing :) <paroneayea>wingo: think "reading data off a REST style API" <paroneayea>eg, you're probably at no risk parsing that data through json <wingo>arbitrary code execution is only possible via read-eval (the #. thing) <wingo>that should be off globally. <wingo>you can get some code execution via srfi-10 or read-hash-extend <wingo>so you have to know whether the extensions that are active in your program are safe against your threat model <wingo>a syntax extension which turns #'foo into (syntax foo) <paroneayea>though say I decided "to hell with json, my REST API is going to use sexps! Wheee!" <paroneayea>and then I import a module that itself imports srfi-10 <wingo>yeah, not full #. code execution but yes <wingo>but that same argument applies if a module enables #. <wingo>so yeah, it's a property of your whole program <paroneayea>wingo: basically I'm saying, I don't want either of those to happen <paroneayea>because I don't know much about my libraries, necessarily <wingo>what code can be run by `read' <wingo>yeah i know what you mean. i solve this by being an asshole and never using other people's code :/ not a great solution <dsmith-work>paroneayea: You could purge srfi-10 from systems you control? <wingo>if you want to control the property globally, yeah like dsmith-work says we can add global kill switches <wingo>we could also add per-port kill switches but that's less convenient in some ways <wingo>heh, at that point just write your own `read' :) <wingo>there's also some memory dangers <davexunit>I don't know read is implemented, so I don't know how feasible it would be to say "don't use any reader macros for this read call" <wingo>read is implemented in c right now. for options it's better to attach them to the port. <paroneayea>so, this conversation is making me feel like: don't use read for any data that comes over the wire that you don't really trust <paroneayea>so, it's still safe with the signed sessions, because an attacker would need your key to get you to do something evil with read <paroneayea>but #e10e10000, srfi-10, and #. are all worrying possible attacks against using vanilla read for much data heading over the wire that you don't control <wingo>i think there is another attack, which is (((((((((((((((((((((((((( <wingo>the c stack keeps recursing, eventually segfault i think <paroneayea>wingq: I guess the json parser I'm using is also semi-vulnerable to that, but it wouldn't be in C <wingo>ACTION currently trying the ((((( attack <wingo>do you have a limit on your message size? <paroneayea>wingo: for this particular case (sessions), the limit is 4kb, but I'm not worried about that. Probably for the json-ld stuff coming over the wire... I haven't set any limit :) <wingo>scheme@(guile-user)> (call-with-input-string (make-string 200000 #\\() read) <paroneayea>ACTION saves this conversation in eir notes.org file :) <dsmith-work>paroneayea: or whatever you are planning on read'ing <dsmith-work>scheme@(guile-user)> (call-with-input-string (make-string 4096 #\\() read) <dsmith-work>ERROR: In procedure scm_i_lreadparen: #<unknown port>:1:4097: end of file <paroneayea>or if not according to the http spec, according to browser conventions :) <paroneayea>not sure if this is from any spec, or just convention <paroneayea>dsmith-work: in my particular case, for sessions, it's still not risky because again, someone would need to forge the key... <paroneayea>I'm still interested in understanding in general how "safe" read / write are in various scenarios <dsmith-work>What would a json parser do on something like (make-string reallylarge-N #\\{) ? <DeeEff[m]>Ideally a JSON parser would quit and tell you to stop sending nested structures without matching parens <DeeEff[m]>in reality it would probably return an object at depth of "reallylarge-N" <stis>I just made that document for your pleasure ;-)