IRC channel logs

2013-11-12.log

back to list of logs

<davexunit>has anyone here had any experience trying to reduce the number of GC runs when writing a realtime application?
<davexunit>I having a little graphical demo that animates smoothly, but experiences a hiccup about once per second due to the GC running.
***fangism is now known as fangism-ctrl-Z
<nalaginrut>morning guilers~
<madsy>Morning all
<zacts>lo
<nalaginrut>heya
<madsy>Anyone have an idea why ",q" terminates my whole application even when scm_shell() runs in its own thread?
<madsy>Logic says it should just terminate the thread and return to scm_join_thread()
<madsy>s/says/dictates
<nalaginrut>no idea, any test code?
<madsy>nalaginrut: I'll dish up some later this afternoon :)
<wingo>moin
<nalaginrut>wingo: heya
<unknown_lamer>scheme@(figl gl)> (make-shader* (version-2-0 vertex-shader) "void main(void) {
<unknown_lamer> gl_Position = ftransform();
<unknown_lamer>}")
<unknown_lamer>$5 = 9
<unknown_lamer>finally!
<wingo>nice :)
<wingo>now we need a compiler from scheme to shade language
<wingo>maybe not ;)
<wingo>*shader
<unknown_lamer>that's the next step
<unknown_lamer>well, the first next step is writing something that uses vertex/fragment shaders instead of the fixed pipeline and shows something on the screen
<wingo>easiest to start with fragment shaders
<unknown_lamer>I am trying to use the subset of gl that works with 2.1 and 3.x, mostly
<unknown_lamer>so no standard perspective matrix/frustrum whatever
<unknown_lamer> http://www.arcsynthesis.org/gltut/index.html this was the first thing about opengl I read that made any sense
<unknown_lamer>out with the old, in with the new
<unknown_lamer>I'd have just updated figl against opengl 3.3, but my dang laptop graphics don't support anything past 2.1
<wingo>opengl lollerskates
<wingo>your best bet is indeed the common subset of 2.1 and 3.x
<unknown_lamer>I think a few possible ffi extensions will appear from this too
<unknown_lamer>working with OUT arguments is a huge pain in the ass
<unknown_lamer>also creating arrays of pointers in guile
<unknown_lamer>bytevectors upon bytevectors
<unknown_lamer>CFFI has a few bits that help with hiding the awful truth, I think I'm going to end up adapting those once I bind more of the evil parts of the GL api
<wingo>we need better macros, that's for sure
<unknown_lamer>(define (make-pointer-array . pointers)
<unknown_lamer> (uint-list->bytevector (map (compose pointer-address (@@ (figl gl types) coerce-array-pointer)) pointers)
<unknown_lamer>that works... for now. With my single function heh
<wingo>@@ evil
<wingo>:)
<unknown_lamer>that whole function is evil
<unknown_lamer>I am pretty sure I should have it take unwrap as an argument
<unknown_lamer>good enough for 4 a.m.
<madsy>unknown_lamer: You're the author behind figl?
<unknown_lamer>nope
<unknown_lamer>I hear wingo hacked it into existence
<madsy>ah
<madsy>I made my own OpenGL setup, which is a bit more high level
<unknown_lamer>orly
<madsy>But I'm embedding scheme, not extending it
<unknown_lamer>ah
<unknown_lamer>C must always be subservient to a real language ;)
<unknown_lamer>until one day there is no more C
<madsy>I don't think we'll see such a day :)
<madsy>Later today I'll add my bone animation system to my demotool. It will be awesome
<unknown_lamer>what a depressing world
<unknown_lamer>nice
<wingo>nice!
<madsy>The standalone example is already on github. Download and use it for whatever you like
<madsy>It uses Assimp
<wingo>link?
<madsy> https://github.com/Madsy/Assimp-GL-Wrapper
<madsy>I didn't compile it as a library, but its usage is pretty simple
<nalaginrut>seems the repo of guile-emacs is unstable, many times it throws "early EOF"
<madsy>You can ignore the GLFW and libpng dependencies. The actual model loading only requires libassimp
*wingo will check it out later
<madsy>unknown_lamer: Here's the beginning of my OpenGL-based engine: https://gist.github.com/Madsy/7427888
<civodul>Hello Guilers!
<wingo>moin :)
<madsy>Yay, I aced the job interview :)
<wingo>congrats :)
<davexunit>yay!
***haroldwu_irssi is now known as haroldwu
<madsy>wingo: thanks :)
<dsmith-work>madsy: Yey! Congrats!
<madsy>Thanks :)
<madsy>Starting in two weeks
<wingo>doing what? :)
<madsy>Java development
<madsy>Not my fav but it beats working at a grocery store
<wingo>indeed :)
<davexunit>"will hack guile for food"
*civodul dreams of guile-jobs.com
<gzg>civodul: fossguile-jobs.com* :^P
<mark_weaver>madsy: I answered your question about ",q" a few days ago, but I guess you didn't see it: https://gnunet.org/bot/log/guile/2013-11-09#T274635
*gzg wonders if there's generally an equivalent to fossjobs.com, out there.
<gzg>Not real domain, fyi.
<gzg>Not a*
<TaylanUB>gzg: Don't get my hopes up like that. :D
*TaylanUB is looking for a different job, iOS development is unethical and often very boring ...
<davexunit>TaylanUB: I have avoided iOS development, though my currently employer has started projects to develop iOS applications.
<davexunit>applications that I have 0 interest in maintaining and hopefully won't have to.
<mario-goulart>TaylanUB: edw announced a job opportunity at actionx.com some days ago on #scheme
<mario-goulart>(NYC)
<davexunit> https://hacks.mozilla.org/2013/11/live-editing-webgl-shaders-with-firefox-developer-tools/
<davexunit>I should add this feature to guile-2d
***fangism-ctrl-Z is now known as fangism
<madsy>mark_weaver: Thanks :)
<unknown_lamer>davexunit: https://github.com/cbaggers/cepl
<unknown_lamer>sort of what I'm trying to copy for Guile
<unknown_lamer>getting good error messages out of the GLSL compiler looks ... impossible
<davexunit>unknown_lamer: oh wow I had no idea about this.
<davexunit>wow.
<unknown_lamer>it requires GL 3.3 at least though, so I haven't messed with it too much
<unknown_lamer>since only my workstation/server/tv support that
<unknown_lamer>I did manage to get virtualgl working from it though
<unknown_lamer>also all of the documentation is rtfs or watch these very long screencasts, and I'm not a huge fan of screencasts
<davexunit>me either.
<unknown_lamer>call me old fashioned, but I just want a web page with screenshots of the effects of commands inline
<unknown_lamer>partly because I hear "skimming" and "scanning" are good ways to prepare yourself for digesting information
<unknown_lamer>the videos I did skip through made it look cool though
<davexunit>I hate screencasts because I can't move at my own pace
<unknown_lamer>I think I can actually get something pretty similar with figl up ... today?
<davexunit>I use figl for guile-2d.
<unknown_lamer>if I can hook into the repl to run the glutMainLoopNextEvent thing after every evaluation...
<unknown_lamer>then you should be able to get basic hacking going by just evaluating forms within (with-window ...)
<unknown_lamer>granted, only painted after each evaluation
<unknown_lamer>I think?
<unknown_lamer>maybe I will be surprised
<unknown_lamer>I really hate how the GL folks didn't go with the magical everything-is-immutable-and-threadsafe OpenGL 3.0 >:O
<davexunit>good luck with that.
<davexunit>it sounds like our work overlaps in some ways.
<davexunit>cepl's repl interaction is better than guile-2d's right now.
<unknown_lamer>when it works ;)
<unknown_lamer>my fifteen minutes with it had it crashing sbcl here and there
<unknown_lamer>I love how OpenGL can do that to a program
<davexunit>haha
<davexunit>hard to escape segfaults
<davexunit>unknown_lamer: have you done much opengl programming in guile yet?
<unknown_lamer>davexunit: not really
<davexunit>ah okay.
<unknown_lamer>I hacked on the vbo.scm demo to use GL_POINT instead of GL_QUAD, and made it do some funky things
<unknown_lamer>drew a few triangles
<unknown_lamer>then got stuck in FFI hell ;)
<davexunit>haha
<davexunit>I want more Guile graphics programmers to help me defeat the GC and achieve stutter-free realtime graphics programs. :)
<unknown_lamer>ah yes, the gc blocking
<unknown_lamer>that's gonna be fun, vbo's frame rate output showed considerable stutter
<unknown_lamer>it kept 60fps, but each frame render is not evenly timed...
<kurohin>In my quest to learn guile/scheme(one week in) I have now started to feel that my pinkies are somewhat strained, does anyone have any advice on how to mitigate the often use of shift and maybe dash.
<unknown_lamer>not too noticeable in a particle system, but make the rendering more realistic and suddenly...
<davexunit>unknown_lamer: if you run that with GC_PRINT_STATS=1 you will see all the GC runs.
<madsy>davexunit: Easy. Just disable the GC ;-)
<unknown_lamer>kurohin: I swap [] and ()
<unknown_lamer>kurohin: also use paredit-mode!
<davexunit>the GC is a good thing, but I need to make some optimizations to make it run less.
<unknown_lamer>hrm, doesn't libgc have some kind of non-blocking realtime collector mode
<davexunit>unknown_lamer: I could be doing it wrong, but running in incremental mode didn't help when I tried.
<davexunit>I think the solution will have to be some combination of: C extensions, GL shaders, more mutable data structures, and perhaps interning of certain immutable types.
<madsy>davexunit: Replace some scm_gc_mallocs with malloc? :D
<madsy>davexunit: If it knows about less memory it will run less, right?
<davexunit>madsy: pure guile currently. no C code.
<davexunit>the GC isn't a big issue for me until my game has a large number of objects being updated each frame. which is common because I like to write bullet hell games that involve hundreds to thousands of particles on screen.
<unknown_lamer>hrm
<kurohin>unknown_lamer: thanks
<unknown_lamer>davexunit: gc is triggered from gc_malloc, right?
<unknown_lamer>davexunit: thought about implementing bulletml for guile ;)
<davexunit>I'm not sure of the inner workings.
<davexunit>unknown_lamer: I hate bulletml.
<unknown_lamer>there's the old realtime trick of "don't allocate any memory" :(
<unknown_lamer>but unfortunately it's not easy to know what in guile will cons and what will not
<davexunit>I really like kentacho and his games, but bulletml is terrible.
<unknown_lamer>ah ok
<unknown_lamer>I hadn't looked into it much
<unknown_lamer>implement something cooler so I can steal it
<davexunit>unknown_lamer: floating point numbers are allocated on the heap.
<davexunit>unknown_lamer: bulletml is XML, bleh.
<unknown_lamer>well, that's no good
<unknown_lamer>at least I hear uniform vectors store things like that packed
<unknown_lamer>so, actually, hrm...
<davexunit>the "issue" has been discussed here before about the floats.
<unknown_lamer>you could do something heinous
<unknown_lamer>allocate an f32vector statically, and then mutate the cells within ... (maybe a few quick accessors and a set! expander)
<unknown_lamer>gl wants packed floats anyway
<davexunit>it isn't a trivial problem to solve, and the use case of realtime graphical applications simulating tons of particles is not a very good reason to significantly alter guile.
<unknown_lamer>'tho I'm not quite sure how guile's ffi converts/doesn
<unknown_lamer>t immediates
<davexunit>unknown_lamer: that might help, but you still need to create the float to stuff in there.
<unknown_lamer>dangit
<davexunit>figl's packed structs use the bytevectors already. I use them.
<unknown_lamer>some C might be appropriate...
<unknown_lamer>bah
<unknown_lamer>making the gc never block is a better approach
<unknown_lamer>I hear it can totally be done
<davexunit>unknown_lamer: yeah I think C is the answer, which will have the side effect of making my library more difficult to install and use.
<davexunit>but I could write quite the blazing fast particle simulation in C.
<unknown_lamer>' All collections initially run uninterrupted until a predetermined amount of time (50 msecs by default) has expired'
<unknown_lamer>there we go
<unknown_lamer>now, can you use an env var to tune that down to, say, 1ms
<unknown_lamer>and will the gc still function at all at that point
<unknown_lamer>it seems to imply that it can do incremental sweeping
<unknown_lamer>the problem being that if it can't get *any* work done...
<TaylanUB>Your total throughput will decrease as you make it more incremental, AFAIK.
<davexunit>people urge that I don't add a C extension to guile-2d, but it's the only way I see to get performance where it needs to be.
<TaylanUB>davexunit: Any idea what the bottleneck/s is/are ?
<davexunit>TaylanUB: from what I can tell, when doing a lot of floating point math per frame.
<unknown_lamer>TaylanUB: perils of realtime processing, basically
<TaylanUB>OK. I reckon mark_weaver had a patch for unboxed floats, dunno what happened with it.
<davexunit>animating and rendering a few sprites: no issues. animating and rendering a thousand sprites: a lot of GC.
<TaylanUB>(It was unclear whether it's an improvement *in general*, or only in specific cases, etc., IIRC.)
<davexunit>TaylanUB: yeah he made a really cool patch, but from what I understand it will not be applied.
<davexunit>it sacrifices too much.
<TaylanUB>Ah OK, too bad.
<davexunit>I should probably still try it out and see if things work better, though...
<davexunit>so it's basically: write C or wait for native compilation ;)
<unknown_lamer>GC_PAUSE_TIME_TARGET - Set the desired garbage collector pause time in msecs.
<TaylanUB>davexunit: Was there discussion on whether it could be made optional ? I imagine it might have high maintainance burden for little gain if it were just a compile-time option; no idea how possible it'd be as a run-time option.
<davexunit>TaylanUB: I thought that it would have to be a compile-time option, which isn't worth it imo.
<TaylanUB>Yeah, I agree, it would be just nasty if you end up with a library that requires a special Guile build and doesn't just work with people's out-of-the-box Guile from their favorite OS's package-manager. :\\
<davexunit>there's a lua game engine called love, it is written mostly in C for performance.
<davexunit>I don't think that I will have to write much C, but for certain things I think it will be unavoidable.
<unknown_lamer>hrm, setting pause time target has no effect on frame stutter in vbo.scm
<unknown_lamer>incremental does seem to drop the stutter from 30ms to 20ms though
<unknown_lamer>still a lot higher than human perception
<unknown_lamer>I kind of miss the old GC, for all of its evil... it had (with-deferred-interrupts ...)
<unknown_lamer>i.e. no gc until I'm done
<unknown_lamer>for realtime tasks, I could see a simple interface... (with-delayed-gc ...) + (gc #:optional max-run-time)
<unknown_lamer>caveat: don
<unknown_lamer>'t allocate faster than you can free...
<davexunit>perhaps we could tell the GC to wait longer before running, which will make less runs, but each GC run will take longer...
<unknown_lamer>Collection 76 reclaimed 8810960 bytes
<unknown_lamer>--> Marking for collection 77 after 8810176 allocated bytes
<unknown_lamer>so... that's a ton of garbage for a single frame :(
<unknown_lamer>sometimes more than once per frame!
<davexunit>unknown_lamer: whoa, 8MB of garbage?
<davexunit>unknown_lamer: for my particle test, the GC runs less, about once per second.
<unknown_lamer>this is modifying 5000 particles per frame
<unknown_lamer>and using multiple intermediate floats for some calculations
<unknown_lamer>I am amazed it runs at all now :)
<davexunit>got to go. :) ttyl