<unknown_lamer>hrm, I wonder if an array-contents* that returned a copy of the array storage if it happened to be a shared array would be Evil (tm) or not <unknown_lamer>(I totally brainfarted, and finally realized I can use the array functions on uniform vectors, so I'm doing a lot of make-shared-array to slurp columns, transposing here and there, etc.) <Onslauth>Hi, sorry to bother you, I have run into a bit of a problem with the guile configure script <Onslauth>It works great on Solaris, and Ubuntu, however on SLES the values --with-gmp-prefix are being ignore, and no CPPFLAGS are being set there after. <Onslauth>I don't suppose anyone has any ideas or suggestions? <nalaginrut>Onslauth: how do you confirm that it was ignored? <Onslauth>if you examine the config.log on a working system versus on SLES <Onslauth>the CPPFLAGS coming in from the environment will be empty <Onslauth>and after processing, CPPFLAGS will include -I for all the values specified in --with-XXX-prefix <nalaginrut>Onslauth: ok, I'll give a test, but it may take time ;-) <Onslauth>Is there anything I can do to help? or anything else you would like me to generate? <nalaginrut>ok, I'll take some time for building the environment <Onslauth>For the configure, I specify the following *nalaginrut is finding a machine for testing <Onslauth>--with-libltdl-prefix, --with-libgmp-prefix, --with-libiconv-prefix --with-libunistring-prefix --with-libreadline-prefix --with-threads <Onslauth>CONFIG_SHELL, BDW_GC_FLAGS, BDW_GC_LIBS, LIBFFI_CFLAGS, LIBFFI_LIBS <Onslauth>everything is pretty nailed down and specified, so there shouldn't be much to search for *nalaginrut recorded these <Onslauth>config shell is set to bash, and its the version that came with SP3, which is 3.2.51 <Onslauth>If I set CPPFLAGS directly, then I can get configure to run properly. <Onslauth>Ok, the repository is a bit outdated and hard to use <nalaginrut>Onslauth: although I can't promise to help you solve it, but I happen to have the disk for a test <nalaginrut>I see, much packages needed to compiled manually <Onslauth>Yeah, their repository is way outdated, and barely contains anything <Onslauth>If you try to use the opensuse repository, you might get lucky, but it generally leads to conflicts with all the packages already installed <Onslauth>Heh, you can get access to the SLES repo free for 60 days <Onslauth>but its pointless, everything is old and outdated <Onslauth>and there are no development packages there <Onslauth>Ok, but there aren't even bzip2 headers? <Onslauth>so how do you build things in the first place? <Onslauth>Its been a painful experience trying to get guile running on SLES <Onslauth>and this configure is the last problem I was hoping to solve. I don't like that it doesn't seem to work at all on SLES. <nalaginrut>yes, when you choose a server/enterprise edition distro, you can't expect new things <nalaginrut>but for SLES-12 someday, it has guile-2.0.9 or higher <Onslauth>Infact our entire release includes all the prerequisites. We package everything together. <nalaginrut>wingo_: hmm...why it's wingo_ rather than wingo? ;-P <Onslauth>SLES was just a requirement for one project, but the configure is killing me. ***wingo_ is now known as wingo
<Onslauth>I also tried to autoreconf the configure, but SLES is missing the gettext AM_GNU_GETTEXT macros. <Onslauth>Um, currently we are also having another problem with Guile on another project <Onslauth>if the number of entries in a let block exceeds a certain number, it breaks <Onslauth>That number being somewhere past the 800 mark, but I am not the one developing that, I just received a basic breakdown of it <wingo>humm, if you can reduce that to a bug report that would be great <Onslauth>Sure, I will speak to the guy dealing with that at the moment <nalaginrut>he's using a guile-2.0.9 release tarball, so maybe it's a little outdated <Onslauth>If you don't mind me asking, what was the limit with the number of entries in the let block before? <wingo>it wasn't supposed to be there at all :) <wingo>there were two ways of addressing let-scoped vars <wingo>one with one byte and one with two or three i think <wingo>but the wider address scheme wasn't tested <wingo>it's possible though that the number of let vars doesn't correspond to the source program <Onslauth>Ok, I think thats the one I ran into a few months ago and spoke to you about <wingo>so maybe you hit the 256 limit <Onslauth>I believe the guy is currently running up to about 800 before it breaks <Onslauth>He's already nailed down the exact number <Onslauth>But I am not on that project so don't know the specific details. <Onslauth>And just confirmed, using the 2.0.9 tar ball from the ftp server. <Onslauth>I just spoke to the guy, he should be coming online now <Onslauth>He says "It breaks at ~890 declarations in the let* and ~890 statements." <Onslauth>I believe he has example code to cause the problem <wingo>Onslauth: can you post it to paste.lisp.org? <Onslauth>The guy says he will try get to it later, depending on time <Onslauth>paste.lisp.org cannot handle the paste size <Onslauth>Is there somewhere else you would like me to paste it to? <Onslauth>That code apparently breaks guile 2.0.9 tarball release <Onslauth>If you remove the last 50 or so statements at the end of the file paste it seems to work fine. <Onslauth>According to the guy, he says it sends the compiler into a loop and terminates with a stack-overflow <wingo>Onslauth: if it's a stack overflow thing, set GUILE_STACK_SIZE in the environment before you compile <wingo>we fixed that in master, mostly, but i don't think we can fix it automagically in 2.0 <wingo>perhaps we could use mmap instead of gc_malloc to allocate the stack, and map a larger stack... <Onslauth>What is the current master version going to be? <Onslauth>2.0.10, or will it be a new major branch? <wingo>i don't recommend you use it in production <Onslauth>didn't complain about not finding gmp.h ? <nalaginrut>Onslauth: it's continuing, waiting for the crash <nalaginrut>actually, you may no have to specify gmp prefix if you just install it in default <Onslauth>wingo - can you recommend a value for GUILE_STACK_SIZE? <Onslauth>nalaginrut - yeah, we don't install it in default location <wingo>Onslauth: the default is 65536, double that if you like <Onslauth>everything we build or require we have brought in and installed to a custom location, so that we always make sure we link against known versions <nalaginrut>Onslauth: maybe it's conflict to the one installed by sles? <Onslauth>but it wouldn't complain about not finding it then? <nalaginrut>I mean it's found before check the prefix you specified <Onslauth>and then errors out with "cannot find gmp.h" <Onslauth>wingo - sorry, I seem to be playing postman here, the guy asked what the following does "(debug-set! stack 4000000)" <wingo>Onslauth: that changes a stack limit for C code <wingo>GUILE_STACK_SIZE is for scheme code <Onslauth>Thanks wingo, the GUILE_STACK_SIZE solved the problem <Onslauth>You have the same problem with the configure? <Onslauth>wingo - The two guys you just helped say if you ever in South Africa, they'll treat you to beers + dinner <Onslauth>They've been fighting with this problem for a week now, trying to find a work around. <Onslauth>nalaginrut - must I give you a copy of the config.log <Onslauth>trying to put it on gist, might be a bit too big for it too :( <nalaginrut>well, there'd be some place to put something like log <lloda>unknown_lamer: array-contents in 2.0.9 has a bug where it returns non-contiguous arrays in some cases even if you pass #t for the strict argument. E.g. (array-contents (make-shared-array #f64(1 2 3 4) (lambda (i) (list (* i 2))) 2) #t) -> should be #f, but returns #f64(1.0 3.0). This is fixed in my array patches. <wingo>the ones that i'm totally going to look at today! *wingo repeats apologies, feels like a liar <lloda>really, the longer it takes you, the worse you feel, which leads to doing things, in the end. <lloda>about array-contents*, NumPy has a bunch of functions that returns array with different storage. Like, C-order, Fortran-order, guaranteed new copy or not, etc. You need those because a lot of C libraries only take specific types of arrays. <lloda>For example, FFTW takes *any* Guile array as is, including any number of dimensions or strides. It's great. <lloda>But OpenGL only accepts some kinds of strides, so if your Guile array is general, you may need to copy. <lloda>So we should have something that does the work of array-contents*, but more general, or a bunch of functions of this sort like NumPy has. <gjanssens>I've run into another gnucash with guile 2 mystery <gjanssens>We have a test file to test some of our functionality <gjanssens>This test file runs fine if GUILE_AUTO_COMPILE=0 *wingo fishes out a link for that... <gjanssens>ERROR: In procedure #<syntax-transformer N_>: <gjanssens>ERROR: Wrong type to apply: #<syntax-transformer N_> <wingo>let me finish one thing, brb <gjanssens>If needed, here's the full test output (with full backtrace): <wingo>when test-load-module is expanded, the definition of N_ is not available to it <wingo>because the expander doesn't see through gnc:load-modules <wingo>so you need to do an eval-when, or begin-for-syntax, or something <wingo>the problem with load is the same as the problem with gnc:load-modules <gjanssens>ok... trying to apply this to my real world case *ponder* *ponder* <gjanssens>So I should use the begin-for-syntax example you gave on the user list ? <gjanssens>wingo: am I supposed to at the cond-expand construct litterally, or should form ... be replaced with something ? <gjanssens>I have added the code litterally, but I still get the exact same error. <gjanssens>Perhaps the issue is two levels of indirection <gjanssens>gnc:module-load loads a C module, which in turn eventually loads another scm module (gnucash app-utils) <gjanssens>Would the second module be fully parsed when evaluating ? <wingo>i would use begin-for-syntax <wingo>so you define it at the top of your file <wingo>then wrap anything you need at expansion time in begin-for-syntax <gjanssens>wingo: did you look at my last pastebin snippet ? <wingo>gjanssens: you have to use it at the top level i think <wingo>also are you sure that module-system-init doesn't define N_ ? ***Fuuzetsu is now known as Guest42540
<gjanssens>wingo: gnc:module-system-init doesn't define N_ <wingo>then do the begin-for-syntax outside of the if <gjanssens>If I have to use begin-for-syntax at the top level, how can I get to the return value of the gnc:module-load call ? <wingo>you can (begin-for-syntax (define foo blah)) <wingo>generally when you see this error the cause is something like this <wingo>when guile expanded a program's macros and syntax, it didn't know that (in this case) N_ was a macro <wingo>so it reified a call to an unknown function instead <wingo>but at runtime, we find that N_ is not a function after all... <gjanssens>No doubt there will still be a few cases like this in the gnucash code <gjanssens>With a couple more fixes like the above, all gnucash tests run successfully with guile 2 now <gjanssens>he, gnucash does have a very slow release cycle <gjanssens>the current stable release is still supporting guile 1.6... <gjanssens>that was rather ugly if we wanted to support guile 2 as well <gjanssens>so the next stable release (2.6, due end december) will drop guile 1.6, but support 2.0 instead (in addition to 1.8) <dsmith-work>At the rate wingo has been going lately, 2.2 will be out not too long after that. <wingo>2.2 would still be in mid-2014 i think <wingo>anyway, the change from 1.6 to 1.8 to 2.0 was the largest change <gjanssens>My very subjective feeling is that there are still several apps lagging behind in their guile support <gjanssens>I know that lilypond (at least until recently) still used guile 1.6 on Windows <gjanssens>(causing library load conflicts for gnucash at the same time :) <gjanssens>I had to spend quite some time getting it properly fixed for 2.0 <davexunit>yeah, many applications haven't upgraded to 2.0 <mark_weaver>The change from an interpreter with lazily expanded macros to AOT compiler was the most difficult change for guile-using projects to adapt to. I doubt we'll see anything that difficult again. <gjanssens>it does take some tweaking in the exising code <gjanssens>which is not always obvious for someone not familiar with the guile internals <gjanssens>but I see the benefit: guile 2 is clearly a lot faster in gnucash <wingo>i'm glad that is the case at least! <mark_weaver>gjanssens: thanks very much for getting gnucash working with guile 2 :) <ArneBab_>gjanssens: do you have some test results which show that in a way non-specialists can understand? <zxq9>gnucash is on guile2 now?!? <gjanssens>ArneBab_: no I didn't take any formal measurements <ArneBab_>gjanssens: damn - that would make a great news story. <gjanssens>But I have run the gnucash test suite about a hundred times the last couple of days <ArneBab_>can you just run it ~10 times with 1.8 and 10 times with 2.0 using time? <gjanssens>When just looking at some of the slower tests processing, I'd say guile 2 woud be about a third faster <gjanssens>ArneBab_ I can check that and come back on that later <gjanssens>Of course the above is a subjective estimate :) <ArneBab_>do you have a suite of tests where all tests use guile? <gjanssens>Perhaps that's an even better test candidate <ArneBab_>I’d rather see current reallife tests of a new development which can be done with ease right now than ideal tests which likely won’t manifest <gjanssens>swig has an extensive test suite, but very little run-time tests <gjanssens>most of the tests are compile/link time tests <gjanssens>The best performance test I can think of would probably be the report tests in GnuCash <ArneBab_>→ it would be cool if you could just run your test suite a few times in each configuration with time. <wingo>gjanssens: startup time could be a nice measurement, if guile is loaded at startup <gjanssens>That's a set of 3 tests that render 3 real life gnucash reports and compare the output to a reference output <wingo>and assuming you are compiling some scheme files <gjanssens>wingo: we currently rely on auto-compilation still <gjanssens>So I'd first run everthing to get it compiled <ArneBab_>gjanssens: can you highlight me once you have the results? <dsmith-work>wingo: So it's now possible to do a parallel build of guile? (make -jN) <wingo>dsmith-work: that has always been possible! <wingo>if you haven't done that and have multiple cores, you were missing out <mark_weaver>wingo: I've frequently had problems building recent master (since RTL) with 'make -j4' <wingo>i have occasionally had similar thing <wingo>i don't know how, but it seems gc could be missing a reference somewhere <wingo>but it works after all is compiled <dsmith-work>wingo: I suspected it was something with building psyntax, and your recent commit seems to address that. *dsmith-work is trying now... <mark_weaver>wingo: might there be a problem with the new loader, such that it sometimes does the wrong thing if a new .go appears just as it's looking to load it? <gjanssens>- guile 1.8: real: 3m59s user: 2m40s sys: 0m9s <gjanssens>- guile 2.0: real: 2m48s user: 1m45s sys: 0m11s <gjanssens>Startup time (wall clock time to show main window, test run 3 times at least) <gjanssens>The tests come indeed close to my 1/3 faster estimate <ArneBab_>gjanssens: can you give me the standard deviation, too? (or just all numbers)? <wingo>mark_weaver: that could be, yes -- would be strange, but I guess it's possible <ArneBab_>gjanssens: the standard deviation gives an indication whether the results are significant. <mark_weaver>wingo: given the nature of the parallelism during the GUILEC portion of the build, I can't imagine what else it would be. <wingo>mark_weaver: yes, though i wonder if it's some gc issue, like maybe the root set is off by a word or something <wingo>restarting the build has "fixed" it for me when I have seen it <mark_weaver>same here. but I've never seen the problem with building without -j* <mark_weaver>why would the gc problem only show up when multiple build processes are working? <mark_weaver>(though I don't mean to assert certainty that the problem never happens without -j*; maybe I've just gotten lucky all of those times :) <mark_weaver>(usually when I've restarted the build, I've done so without the -j*) <wingo>so i was thinking it could be an ordering issue <wingo>with -j you can get different compile orders <wingo>depending on what finishes when <wingo>so the set of e.g. symbols that are live could depend on... well i don't know really :) <mark_weaver>I think I may have saved a copy of at least one of those backtraces. <mark_weaver>wingo: in the backtrace I have, the error is "Unbound variable: emit-static-set!" <wingo>mark_weaver: i think we might need to collect more, and possibly a core file also <wingo>a core file would include the mappings, no? *mark_weaver doesn't know either <gjanssens>ArneBab_: that's all the numbers I have... :( <gjanssens>The average time per test run is the number above divided by 20 <gjanssens>that is, it was less than measurable on a wall clock ***Fuuzetsu is now known as Guest22602
***Guest22602 is now known as Fuuzetsu`
***Fuuzetsu` is now known as Fuuzetsu
<ArneBab_>gjanssens: no probs - thanks for the info! It just doesn’t allow doing a barplot with errorbars ☺ <ArneBab_>that’s something which could be put in a small news-note ☺ <DeeEff>is there a function in guile that will let me take a 1D list and reshape it into a 2D list? <ijp>presumably you mean a list of lists, but what criteria do you want to use to reshape it? <ijp>chunks of n? partitioning according to an equivalence predicate? randomly apportion to n lists? <DeeEff>ijp: I'm likely just going to pass in specific dimensions to the list <DeeEff>or rather, I'm trying to read in a file line-by-line, and place each of those lines in a list of lists <ijp>okay, but why do you need a list of lists, as opposed to a list of strings? <DeeEff>the file itself is full of numbers, so I'm making a matrix, effectively <DeeEff>but I'm not looking to make a full matrix type, just something that can store the numbers <ijp>there are (multi-dimensional) arrays in guile already <ijp>although, I'll grant you they are quite horrid <DeeEff>can you read into them from a file? <ijp>unpleasant enough to use that 19/20 I'll use a vector of vectors <DeeEff>okay, well I'm still learning scheme in general, so I haven't quite gotten to vectors and arrays yet, but I'll look into those in more detail later. <DeeEff>In the meantime, if there's a function that can re-dimension a list of lists I would like to know about it. <ijp>there is no such function in guile <ijp>it isn't a particularly common thing to do <ijp>I have one for chunking a list into a list of lists of length n, which is about half what you want <DeeEff>hrm... well, I suppose I can write it myself. That is effectively similar to what I want though. <ijp>give me a second, I'll break out all the dependencies <DeeEff>This is a very rudimentary way to do it <DeeEff>Assuming you have any list that has n x m elements <DeeEff>you get a list of m lists with n elements <DeeEff>I should probably check to make sure the number of elements in L are divisible by n though <ijp>right, but in general you never want to append to the end of a list <ijp>(append '(1 2 3) (list 4)) -> (cons 1 (cons 2 (cons 3 (list 4)))) <ijp>now imagine the list is length 1000 <ijp>and you are doing that every time round the loop <DeeEff>so there's a massive performance hit then <ijp>building in reverse, then reversing, is a silly idiom but you get used to it <DeeEff>strange. I would have thought reversing the list would give you just as much of a performance hit <ijp>but you only do it once <DeeEff>In this case, I would end up doing it m times <DeeEff>'m' being the number of 'n' length lists in my list <ijp>I wonder when I wrote that 'chunk', it feels like it must have been years ago <DeeEff>It seems like you added more helper functions than what I ended up using, but I'm not sure if mine is as stringent about edge cases and errors as yours is <ijp>The reason it is written that way, is that the built-in split-at was too strict <ijp>so I had to go back and fix the original version <ijp>and fitting two loops into one usually runs the risk of being unreadable later ***sneek_ is now known as sneek