IRC channel logs

2022-08-29.log

back to list of logs

<Hagfish>here's an interesting thought experiment...
<Hagfish>suppose that e.g. google make an AI for doing code review / security research, and it's so good that it can find every security flaw in any piece of open source software
<Hagfish>there may already be research projects that try to do this, i don't know what the state of the art is
<Hagfish>because there are plenty of examples of commits which fix bugs, and a smaller number of such bug fixes which are deemed relevant enough for a CVE
<Hagfish>so maybe an AI could learn what sort of bugs are vulnerabilities
<Hagfish>so yeah, imagine that google send out automatically generated patches for every vulnerability in every piece of open source software
<Hagfish>is it conceivable that there would never be another vulnerable piece of software ever released?
<blockhead>or would the ai introduce vulnerabilities to its eventual advantage?
<Hagfish>or would there be a cat-and-mouse game of hackers trying to make AIs that are good at creating underhanded code
<Hagfish>right
<Hagfish>i'm assuming for now that the AI is smart enough to do this task, but not smart/agenty enough to deceive humans
<blockhead>ok
<Hagfish>it's possible that an AI could find a security flaw, and provide a fix that contains an even more subtle flaw
<Hagfish>but i'm not sure if that's really a viable form of attack
<Hagfish>the proposed patch would still have to be reviewed by a human (ideally multiple humans)
<Hagfish>this AI would have no knowledge about the outside world, and have only been trained on source code (although i'm sure there is a lot of world knowledge embedded in code comments)
<oriansj>well that is literally one of the tech researched in the game singularity
<oriansj>but with copilot use actually showing up in production environments, we are running into potentially much bigger problems
<Hagfish>yeah, it's a centralised attack vector into multiple repos that are ostensibly independent
<oriansj>and as time goes on programmers will only get lazier and trust the machine to do the right thing more and more until they stop understanding the thing they are instructing the computer to do
<Hagfish>if they can't understand the code they are being suggested, then i'm not sure what value they are providing
<Hagfish>maybe they're programming in a higher level language
<Hagfish>"prompt engineering"
<Hagfish>i could see that happening, yeah
<Hagfish>but it's always been the case that the value of a software engineer isn't their ability to memorise and type syntax
<oriansj>Hagfish: you forgot checkbox engineering is a growing software profession
<Hagfish>heh, yeah
<Hagfish>people have analogised large corporations as being a type of AI
<Hagfish>or perhaps better Artificial Stupidity
<oriansj>I like to think of Governments and Companies as Meat based AIs that are slowly improving themselves to solve their ultimate objective
<oriansj>and I uploaded some basic system setup procedures https://git.sr.ht/~oriansj/System_setup for anyone interested.
<Hagfish>looks good
<Hagfish>echo -e "o\nn\np\n1\n\n\na\nw" | fdisk /dev/sda
<Hagfish>heh, i've not seen that done before
<oriansj>makes for a rather quick format of /dev/sda now doesn't it
<Hagfish>i feel like fdisk should accept a slightly nicer syntax, but combining commands with a pipe is a very normal unix thing to do
<oriansj>well if it does, now it is easy to send me a patch to fix it and let everyone else know you did it with a commit ^_^
<oriansj>and if you notice it is the only procedure for having an encrypted debian /boot that I know of
<Hagfish>yeah, the first step is getting something that works, and making it accessible for other people to review/improve
<Hagfish>it's not like the solution introduces some weird dependency that will make life difficult going forwards
<oriansj>well minimal dependencies right now
<vagrantc>need to brush up on bootstrappable paths ... might be giving a talk focusing on that and reproducible builds soon
<vagrantc>~october
<Hagfish>that would great!
<Hagfish>will the talk be recorded?
<oriansj>vagrantc: stage0-posix -> live-bootstrap (minor gap) Guix
<oriansj>we have a POSIX kernel written in hex0 (builder-hex0) capable of running all the steps needed to bootstrap TCC and then it is just work to get a kernel TCC can build that is also good enough to bootstrap Linux and Guix to finish the chain
<Hagfish>it would be great if that could be announced at the talk in october, but i've no idea how quickly that work is progressing
<oriansj>Hagfish: well rickmasters announced on August 25th, 2022 the successful building of tcc-0.9.26 on builder-hex0 (but no code yet)
<Hagfish>yeah, i think i saw that announcement
<oriansj>but we can't announce that to the larger community without working code
<Hagfish>is building a kernel with TCC "just" a matter of writing a few patches to a 2.x linux release, do you think?
<oriansj>but the current code does work for all of the steps in stage0-posix so we can definitely announce that in vagrantc's talk
<Hagfish>true
<vagrantc>and there seem to be an EFI implementation of some things in the works?
<oriansj>now the question of how big the gap between guix and live-bootstrap is a question best left to guix developers to answer
<oriansj>vagrantc: yes a port of stage0 to UEFI and stikonas is doing a wonderful job of it
<vagrantc>Hagfish: i think it would be recorded (if my talk is actually selected)
<stikonas[m]>vagrantc: yes, I'm porting stage0-posix to run in uefi
<stikonas[m]>Reached hex2 and catm
<vagrantc>bsidespdx.org will be the conference ... october 7th, looks like
<vagrantc>stikonas[m]: nice!
<oriansj>unfortunately my stage0 talk to Michigan Cyber Summit was rejected as "there was a large number of quality presentation proposals received"
<Hagfish>that's a pity
<Hagfish>don't they realise that all their work ultimately depends on yours? :D
<oriansj>not yet
<Hagfish>ironically that's one of the reasons for giving such a talk: to raise awareness
<oriansj>but it does put my approval/submission average to 1/64 so I guess that is nice
<Hagfish>no way. i didn't realise you'd made so many attempts
<oriansj>Hagfish: that is the secret to my success. I fail a shitload
<Hagfish>i think most success stories probably have a history of information-gathering failures behind them
<Hagfish>isn't that the story behind the name WD-40?
<oriansj>I don't know
<Hagfish>the story i heard was that it was the 40th attempt to create such a chemical
<oriansj>but most failures tend to be forgotten by others unless they are seriously harmed by your failure.
<Hagfish>yeah, making systems "fail safe" is an important design consideration
<oriansj>stage0 is littered with failures and false starts
<Hagfish>exploring the solution space
<oriansj>not to mention all of the things that can be improved
<Hagfish>and of course you know the old rule "to get the correct answer on the internet, you just have to provide the wrong answer and wait for people to correct you" :)
<oriansj>surprising hard to do when breaking ground technically
<Hagfish>speaking of october, apparently "ARPANET’s first host-to-host message was sent at 10:30 p.m. on October 29, 1969" https://www.icann.org/en/blogs/details/the-first-message-transmission-29-10-2019-en
<Hagfish>yeah, bootstrapping is a question that few people ever even thought to ask, let alone answer
<oriansj>well when you are told it is too hard, most people never try
<Hagfish>for the right sort of personality, though, that just makes the problem irresistible
<Hagfish>and all the more rewarding with each step of success
<oriansj>Oh, I just had to do bootstrapping work for my own mental health and I am amazed by how much we have grown and how quickly we have got this far.
<Hagfish>i think we shouldn't under-estimate the extent to which (some) people are motivated to do work which has longterm benefits for the rest of society
<Hagfish>it relies on a people having a combination of skills, long-term thinking, and pro-social thinking, but those are by no means mutually exclusive traits
<Hagfish>just rare traits
<Hagfish>and yeah, in some cases, that motivation manifests itself in terms of a desire to avoid mentally harmful (e.g. meaningless or immoral) alternative endeavours
<oriansj>well motivation only gets one started on a journey. only discipline gets one to the end.
<Hagfish>but we do live in a society that tries to hack people's reward functions to make them focus on selfish accumulation of wealth, which can limit talented people's desires to focus on pro-social activities
<Hagfish>yeah, i guess i saw discipline as a subset of skill, but you're right, it can be helpful to separate them
<oriansj>stikonas: you may wish to comment on this: https://github.com/oriansj/stage0/issues/26#issuecomment-1229524740
<oriansj>hmmm it is hard to trim lines to 80 columns when the sed command is 131 chars long
<Hagfish>i'm not sure what the recommended style/lint rules are for sed
<Hagfish>you can put the sed command in its own file, right?
<oriansj>now the real question would this be kosher: https://paste.debian.net/1252008/
<Hagfish>hmm, tricky
<Hagfish>i feel like the data section ("base udev autodetect...") should be separated out from the command part, but that's a personal preference
<Hagfish>are those values in some particular order?
<Hagfish>and would it be more clear as a for-loop?
<Hagfish>maybe not, as that would mean a less clear separation between sed "code" and shell code
<oriansj>yes the order matters
<Hagfish>oh, interesting
<Hagfish>is it clear from the context to a new developer where they would need to add a new value to that list?
<oriansj>well the single command it would be obvious in the iterated form less so
<Hagfish> https://paste.debian.net/hidden/77a1f641/
<Hagfish>i think that's what comes to my mind
<Hagfish>i haven't tested it, so it might not be viable :)
<oriansj>well I made a test file, lets give it a try
<blockhead>wait, give it a chance to study ;)
<oriansj>minus a little bit of extra whitespace, I must say it does work Hagfish
<oriansj>and here is my test file: https://paste.debian.net/1252011/ so you can test too
<oriansj>it has 4 test cases, unpopulated HOOKS= line, populated HOOKS= line, unpopulated HOOKS= in comment and populated HOOKS= line in comment
<Hagfish>that looks pretty comprehensive, yeah
<Hagfish>i think there might be some usability research which shows that vertical lists are easier for humans to parse (for some use cases) than running/horizontal lists
<Hagfish>for alphabetically sorted lists i'm sure there's a big difference, but i can't think of a proof for that being true of non-sorted lists
<Hagfish>the trade-off is probably the amount of scrolling someone has to do to navigate around in a file
<Hagfish>if we were talking hundreds of entries, it would probably be worth pull them in a separate file, even though that's an extra level of indirection
<oriansj>well if anything gets that complicated, I would have clearly lost my mind
<oriansj>now the new column size is 69
<Hagfish>that sounds more reasonable
<oriansj>and now arch is up with the rest: https://git.sr.ht/~oriansj/System_setup/tree
<oriansj>I probably should do one for gentoo but I know there are a few people here much better than me with gentoo
<Hagfish>169 lines of clear, well-commented code/config
<Hagfish>very nice
<oriansj>and the install guides for all the systems leaves the user at the exact same UI experience
<oriansj>on a fully encrypted /boot+/ system
<oriansj>I am however missing the bits needed to remove the double prompt for luks password in guix
<oriansj>which is thus far fortunately the only defect
<Hagfish>it tries to decrypt the drive twice?
<oriansj>well it prompts the user for luks password once as grub and once as linux
<Hagfish>ah, tricky
<oriansj>probably, just haven't had time to sort that out yet
<Hagfish>yeah, presumably there's some standard way to make that work, but it's not something i'm familiar with
<oriansj>probably should do a matching setup with puppet and ansible
<oriansj>and then everyone would have a universal path to an oriansj devel environment
<oriansj>not that it would suit most people as is it quite minimal
<oriansj>and there are several bugs I haven't hit/eliminated yet
***ChanServ sets mode: +o janneke_
<Gooberpatrol66>oriansj: if you ever figure out how to fix that password double-prompt, let me know lol
***Spawns is now known as Spawns_carpeting
<Hagfish>if there's an open bug filed about it, maybe the link to that can be included as a comment in the config so that people can keep track of the status