<OriansJ>dddddd: well I figure we give janneke something to play with; as the other pieces are just the bare minimal set. <OriansJ>The fully complete and perfect release can of course happen when you feel happy with it dddddd ***pgreco_ is now known as pgreco
<fossy>cant even kill the pid with kill -9 <fossy>it wouldnt die with kill though... <OriansJ>fossy: well gash-utils are currently unfortunately slow; SIGKILL doesn't work on processes that have a handle for that signal the way you expect. Essentially guile tries to clean everything up prior to termination if I remember correctly. <fossy>but I left this for half an hour on the one sed, which normally takes under 30 seconds <OriansJ>fossy: did you check the memory usage? It could have hit your swap <fossy>Why the actual f&*@ is it so hard to get something working everywhere and not reference anything external <janneke>fossy: in our 3rd implementation of portable lilypond binary pack scripting (GUB), we added a LD_PRELOAD hack to ascertain the [cross] build not to reference /usr <janneke>that proved near-impossible and we needed to relax some usage of / <janneke>which made the problem "smaller", but still leaky <janneke>or if you want to see it black/white: incorrect <janneke>guix is the first [cross-]build system done right, imo <OriansJ>fossy: the reason is because dynamic linking infected everything; especially when and where it never made sense. <OriansJ>you have to go out of your way to statically compile the binary and then suddenly alot of issues "magically" go away. <OriansJ>now that disk storage is cheap and nix/guix exist, I can not imagine a reason why we finally give up that terriable idea. <dddddd>indeed, disk is not the only concern <OriansJ>xentrac: I haven't seen any study proving that is actually a real benefit in measurable terms. <OriansJ>dddddd: another one is security where static binaries bring numerous benefits. <dddddd>at the cost of rebuilding the world for every vuln? I don't think so. <OriansJ>dddddd: not the whole world, just the programs that depend upon the library. <dddddd>Not everyone has 4TB+16GB and multi Ghz setups <OriansJ>eg a X11 lib vulnerability would not require a bash recompile <OriansJ>and dddddd guix has substitutes for those people <OriansJ>not everyone has to compile from source, so there is no build advantages involved in that sort of argument for most users. Only build farms and people like me end up having to waste cycles. <OriansJ>but if you look closely at M2-Planet's build process, notice one can build libraries to assembly source once if one wishes to save compile cycles. <xentrac>yeah, I haven't seen measurements of it either <OriansJ>and compiling from entirely source allows the compiler to do even more optimizations for better performance than with binary linking. <OriansJ>xentrac: indeed, any claim asserted without evidence can be rejected immediately without evidence. <xentrac>it's harder to quantify the benefits of xclock not being 6.3 megabytes instead of 40 kilobytes in terms of L1 cache pressure. how often does xclock bring that code into cache? surely there are large parts of libXmu xclock isn't actually using at all. and probably xclock only wakes up to evict other programs' copies of libc and Xlib from the cache hierarchy rarely, perhaps once a minute <OriansJ>xentrac: not to mention C compilers can do dead code elimination to remove most of those 6.3MB that belongs to Xlib which are not being used. <xentrac>but it seems clear that 6.3 megabytes is a factor of 150 larger, so it would be surprising if the systemwide effect of eliminating sharing of cache lines between processes were insignificant <xentrac>how big is your statically linked xclock? I don't have one here <xentrac>I was just adding up the sizes of the libraries <xentrac>static linking does improve performance generally speaking <OriansJ>xentrac: never bothered to compile xclock; never actually used it. <OriansJ>But I do know M2-Planet binaries well. <dddddd>If one does not build she downloads the updates, not a free resource also. <dddddd>(download the patched lib v. all the programs affected --if you can id them) <OriansJ>not discounting the programs who used the lib but not the vulnerable function(s) which were dead-code eliminated anyway. <OriansJ>a staticly built M2-Planet with GCC is 845,416bytes but statically self-built is only 147,029bytes with no size optimizations at all <OriansJ>which means GCC is not dead-code eliminating anything from glibc which M2-Planet isn't using. <OriansJ>which mean our existing build tools leave lots of room for enhancements and waste elimination. <OriansJ>now xentrac I will not argue that people who want dynamic linking shouldn't have it but rather in bootstrapping and bootstrap programs; only static linking should be used. <xentrac>oh, in that case you are clearly correct <xentrac>I thought you were making a much broader and more dubious claim <xentrac>that dynamic linking in general was obsolete <xentrac>which I doubt, although I suppose it's possible <OriansJ>xentrac: it is a technology, where it might be useful but I have yet to see where it has a real measurable advantage. <OriansJ>especially in the light of all the work nix/guix do to work around its flaws. <xentrac>yeah, I certainly agree that dynamic linking is a pain in the ass <OriansJ>it previously had an advantage in storage constrained systems but now that we literally are having systems with dozens or more copies of the same library on the same system. Such a benefit seems more dubious than ever. <xentrac>yeah, agreed; it should be possible to use Nix or Guix to take those measurements <OriansJ>especially with certain case that most programs only use a subset of the library's features in most cases. <xentrac>usually not a statically determinable subset though <OriansJ>xentrac: I am not so certain of that, given you know what functions are called by main. <xentrac>my intuition is that since modern systems are bottlenecked on main memory bandwidth, the order or two of magnitude of space reduction of dynamic linking will make a big difference <OriansJ>simply recurse down the source call graph and any function not called can be eliminated. <xentrac>and a bigger difference, as dddddd points out, in your modem bandwidth, although maybe we can improve that with better delta coding <xentrac>clearly, and if you put each function in a separate file that's what a standard linker does <OriansJ>xentrac: personally I think each group should have a build server they collectively trust which would only need download source diffs from git. and their internal bandwidth be limited by what they choose to setup internally. <OriansJ>distribute trust, so that no single compromise impacts the security of the community. <OriansJ>a single 56KB internet connection is more than enough for a large group of people with a properly configured mesh network with internal services (such as a youtube-dl caching service) <OriansJ>16TB of hard drive cost spread across 100 people is $2.3/YEAR (assuming you want fault tolerance and backups of data) <OriansJ>which covers a great deal of educational materials, source code and entertainment. ***dddddd_ is now known as dddddd
<xentrac>yeah, if you can manage high internal bandwidth <OriansJ>well USBv3.0 has considerable bandwidth, microsd to a less degree and not to mention the net bandwidth of carrier pigeons <OriansJ>high internal bandwidth is generally quite easy and cheap if one is willing to cooperate with others (scaling laws apply heavily) <OriansJ>10s -> KB range easy; 100s -> MB range easy; 1000s -> GB range easy <OriansJ>range is from others is also a major constraint problem. (next door neighbor just run some cat6; across a few miles or more fiber optic or directional antenna may be required) <xentrac>there's also the coercion problem; some friends who set up a neighborhood Ethernet LAN in the 90s in SF had to put signs on all the cables to warn the cable TV folks that those weren't their cables <xentrac>right now if I were to go outside to run cable to my neighbor's house I'd quite plausibly get arrested <xentrac>this was one of the original major drawbacks of the electric telegraph relative to competing systems like Chappe's optical telegraph: the electric telegraph had to run a cable along the ground that could be cut at any point between the endpoints, while the optical telegraph used much more easily defensible stone towers <xentrac>(but Wikipedia tells me "on two occasions [the Chappes'] apparatus at Place de l'Étoile, Paris was destroyed by mobs who thought they were communicating with royalist forces" <fossy>well OriansJ I still plan to produce static binaries throughout the bootstrap. and luckily gnu programs seem to mostly (*hem hem* guile) honour --disable-shared --enable-static <fossy>I would really appreciate thoughts on how to go from here <fossy>1. Compile everything in VMs/containers. Pros: certinaly isolates everything. ensures programs do not interact with any host binaries. Cons: added overhead. <fossy>2. Attempt to use LD_PRELOAD to restrict interactions with the host. Pros: works wellish and dosent require added overhead. Cons: hacky and as janneke said nearly impossible <xentrac>why nearly impossible? I must have missed that part <fossy>3. Make the build systems never interact with the host. Pros: best in the long run. Cons: no matter which way it is compiled it is often extremely difficult to ensure it restricts the host. we have no idea what environment we are operating in - yet we want a certain environment. <fossy><janneke> fossy: in our 3rd implementation of portable lilypond binary pack scripting (GUB), we added a LD_PRELOAD hack to ascertain the [cross] build not to reference /usr <fossy><janneke> that proved near-impossible and we needed to relax some usage of / <xentrac>I think #1 is obviously the right solution in theory, but how practical it is in practice depends on your context. I'd think Guix already goes pretty far towards #1 <fossy>xentrac: my goal is to produce a toolchain that any linux distro can use to bootstrap <fossy>or, that any project can use <fossy>the guix bootstrap is great if you use guix <fossy>im extending mescc-tools-seed to gcc essentially <janneke>xentrac: in short: we didn't build a whole system, like guix <xentrac>can't you bootstrap any linux distro from guix? <janneke>we used a number of shortcuts, like the shell etc <janneke>when you build everything from scratch, like with guix, it should work <OriansJ>xentrac: agreed on the range issue, I however am confused on the arrested for home improvement aspect you expressed. (unless you are renting property in which case completely reasonable but surprising if they don't implement network infrastruture themselves) <OriansJ>fossy: well I guess we need to make guile comply with proper behavior. <OriansJ>as chroots and VMs are just duct tape and rubber bands around badly behaved software. <OriansJ>one can of course use kaem's nightmare-mode to strip out every bit of environment that you pass to programs; then in theory the only things making a program behave badly will be source code and arguments (and the files expressed by those arguments recursively) <OriansJ>xentrac: also note that until we have a POSIX kernel that M2-Planet can bootstrap we have a potential vector for a trusting trust attack in the kernels upon which the guix bootstrap runs. <xentrac>since Thursday night there's a country-wide lockdown where people are only allowed to go out of their houses for one of 34 approved reasons, most of which require written permission from your employer <xentrac>originally there were 24 but they forgot some. in particular they forgot to include "operating nuclear plants", which I'm fairly sure was an unintentional omission <xentrac>building a neighborhood LAN is not among them <xentrac>yesterday they arrested 300 people for violating the quarantine order, so in absolute terms the risk is low <OriansJ>xentrac: a bit of an excessive over reaction to the current virus outbreak. <xentrac>well, we'll see. we're at 158 cases today, so probably when we see the results of this lockdown order in 10 days, we'll have around 1500 <OriansJ>for a under 5% mortality rate when the average yearly flu is 3-4% <xentrac>and it's not going to be a 5% mortality rate when the hospitals fill up <xentrac>since it's guaranteed to have no impact on contagion by the end of the month it's also guaranteed that they'll extend it <xentrac>I just came back from the grocery store a block away with about 12 days' worth of food <lfam>The mortality rate would skyrocket if we run out of hospital space <OriansJ>well my Mormon neighbors certainly are prepared, they are religiously required to have a 1year food supply for everyone in their homes <xentrac>although ventilators really only cut the mortality rate a little <xentrac>yes, the Mormons will be in a much better position after this <lfam>The real issue is they will have to stop treating other kinds of cases, so people will die from things that would normally be treatable <xentrac>unfortunately I can't go stay with the Mormons because I don't know any and also I can't travel without getting arrested and likely deported <OriansJ>lfam: well triage is about doing the actions with the highest likihood of maximizing lives saved <lfam>In Italy they stopped treating anybody over a certain age or with more than one condition <lfam>And it still was not enough <lfam>Hopefully we can avoid that <OriansJ>lfam: well I haven't followed the details in Italy nor know anyone personally there to give me an honest perspective but I would be surprised if doctors gave up on people with a greater than 30% chance for survival unless limited by supply shortages.