IRC channel logs

<kriskowal>attn cwebber tsyesika my good friend and guile-knower rrix tells me that empty list and empty alist are not distinguishable. i can see this not being a problem from guile to guile, which has me worried about round-trips for js {} and [] https://github.com/ocapn/ocapn/discussions/89

<kriskowal>when i was working on thrift at uber, my team made a fuzzer that would generate random messages and bounce them off an echo handler, over the cross product of supported languages. we made a cool docker-based orchestrator for those tests https://github.com/crossdock/crossdock

<kriskowal>the neat thing about crossdock was that engineers working on each language implementation didn’t need to have the tooling for every supported language to run the tests. something like that might be useful.

<tsyesika>kriskowal: it's true though at least with the `guile-json` library it represents json arrays as vectors. In goblins we'd probably convert `{}` struct like objects to our own ghash and arrays to lists

<tsyesika>I don't think this woul dbe a problem for us to be able to round trip properly

<kriskowal>oh, that’s a relief

<kriskowal>is guile-json able to serialize and deserialize ghash as well? i’m sure that’s not a requirement, but it’s consistent with the requirement on the agoric side that markm mentoined, regarding the ability to treat JSON data as a subset of ocapn data.

<kriskowal>that is, we can lift JSON, send it over captp, and write the same JSON back on the far side.

<kriskowal>this is evidently the only reason we needed a distinguishable null.

<tsyesika>no, guile-json has no idea about ghashs, if we choose to implement with that library we'd write something to convert to and from the format guile-json expects data to be in to something we'd want to use within goblins

<kriskowal>The convention of 'null for that purpose in Guile is interesting as well.

<kriskowal>kk, good to know.

<kriskowal>JavaScript’s entanglement with JSON is understandably much tighter.

<tsyesika>though I think both dthompson and cwebber have their own json libraries due to how guile-json approaches some things so I'm not sure what we'd use

<tsyesika>btw I've not pushed it anywhere but I did implement encoding to smallcaps which is neat

<kriskowal>i’ll pass that along!

<kriskowal>could you riff with me how to represent a ghash in guile, for the rosetta stone?

<tsyesika>ghash is just a hash map implementation with some specific qualities we want, I'd just write something like <ghash>

<dthompson>tsyesika: our ghash thing is pretty internal to goblins right? I didn't think it was something that would get exposed over captp.

<tsyesika>no ghash wouldn't get exposed over CapTP

<dthompson>I'm a little confused by the current conversation :)

<tsyesika>it's just internally how we'd probably represent the map / structy thing in smallcaps

<tsyesika>though since they can only have string keys maybe it's over kill (ghashes have some special equality stuff for refrs)

<tsyesika>not sure! but either way however we decide we'd to represent them, there shouldn't be a conflict between json `[]` and `{}`

<kriskowal>i'm now confused indeed

<dthompson>the simplest thing using only core guile data types is that json arrays use vectors and json objects use hash tables.

<dthompson>I second tsyesika that there are no round tripping concerns

<tsyesika>I'm confused to the confusion :) I was just trying to clarify how `{}` and `[]` could be differenciated in guile and there's no need to worry

<kriskowal>so, in goblins, suppose you have a reference to remote object bob. bob implements a method 'echo. echo accepts either a hash table or a list. how would that look in guile?

<kriskowal>in js, this would look like E(bob).echo([]) and E(bob).echo({})

<kriskowal>I presume it would look something like (bob 'echo '()) and something else.

<kriskowal>or possibly like (bob 'echo #()) and something else

<dthompson>if using vectors and hash tables (just one possibility, not the only) then (-> bob 'echo #()) and (-> bob 'echo (make-hash-table))

<tsyesika>probably something like this: be `(<- bob 'echo (list))` and `(<- bob 'echo (make-ghash))`

<dthompson>no literal syntax for hash tables in guile

<tsyesika>I don't believe we currently have serialization for the inbuilt guile hash tables but it could be added with a marshaller

<kriskowal>cool, and i gather the specifics are still in the air. that i didn’t know either. thank you!

<kriskowal>so i can write in TBD hash table

<tsyesika>yeah one of the guile hash tables implementations :) I suspect ghashs because of the `eq?` equality stuff for refrs but it could be something else too

<dthompson>yeah imo the specifics aren't that important. just know that we can do key/value pairs just like anyone else. ;)

<tsyesika>btw, going back to my implementation of smallcaps. If I understood the JS code correctly, you pass regular integers through to their JSON representations and JS big ints as the smallcaps encoded big int format. this might cause some round tripping issues since both would be decoded to regular guile ints which are big ints (all ints are just big ints).

<tsyesika>when I implemented the encoding I just made all ints be represented as a smallcaps big ints

<dthompson>yeah I found it kind of strange to have two integer types because what constitutes "big" depends on the word size of a particular cpu.

<dthompson>and also because json makes no such distinction :)

<kriskowal>how does guile decide whether to represent a json number as a bigint vs real? presence of a decimal?

<kriskowal>tsyesika you are correct wrt smallcaps serializing js number as json number and js bigint as json string

<tsyesika>dthompson probably knows better, but I'd assume it uses something like the`inexact?` function

<kriskowal>and we’re not concerned with roundtripping bigint with to/from json over captp.

<tsyesika>to determine int vs float

<dthompson>kriskowal: yeah the decimal point determines if it's an exact integer or inexact rational (float)

<kriskowal>that’s unfortunate. this comes up in a number of other contexts in other ways.

<tsyesika>oh interesting `(exact? 2.0) ;; #f` I didn't know that

<kriskowal>the answer in “go” is that the receiver gets to decide using a go struct definition

<dthompson>guile doesn't come with a json parser, though, so I'm just explaining a choice I made once.

<tsyesika>yeah the guile-json library is third party and there are others

<dthompson>it could be that all numbers are decoded as floats

<kriskowal>ah. the choice that is most consistent with js is to use a real always, and rely on the fact that 53 bits of integer precision are extractable.

<dthompson>I don't see why it's an issue though?

<kriskowal>the go decision is awful for different reasons

<kriskowal>go’s interpretation of JSON allows for the serialization of int64 with full precision as a json number.

<dthompson>like on the encoding side scheme 1 would be encoded as 1 in json, scheme 1.0 would be encoded as 1.0. seems fine.

<dthompson>or it could always include a decimal point. still the same number.

<kriskowal>javascript’s JSON.parse just loses precision. terrible for nanosecond resolution timestamps, which are common enough in go.

<kriskowal>we’re using string for bigint in smallcaps precisely because you can’t reliably round-trip more than 53 bits of integer precision between JSON implementations.

<tsyesika>it's probably the time of night where I am but I think I'm a bit confused, did you mean represent all numbers as floats, I think I probably just misunderstood

<kriskowal>in any case, this is likely to not be germane to guile, except insofar as that you might get passable data from javascript consisting of reals, bools, lists, hash tables, and some distinguished value for null. we’d never expect to serialize a captp bigint / guile int in JSON.

<dthompson>so you're saying that there are json implementations out there that don't handle arbitrary sized integer values?

<kriskowal>in JS, all numbers are floats (except some operators coerce and truncate them to 32 bit integers)

<dthompson>because afaik the json spec does not impose a limit on integer size

<kriskowal>dthompson, yes. Specifically, JS doesn’t parse arbitrary precision numbers out of JSON.

<tsyesika>so to be clear if the JSON number reaches guile as the number 5 and we round trip that as "5+" (the stringy smallcaps encoding), it's be considered a success?

<kriskowal>and you are correct that the JSON spec is silent on the topic.

<tsyesika>*it'd

<kriskowal>ECMA 402 defines JSON and JS bindings that are more specific about how to serialize double floats into JSON without loss of precision, which the IEEE JSON spec is also silent on.

<dthompson>regarding ints vs. bigints, I would expect there to be a canonical form: numbers under 2^n are encoded as fixnums, greater are bigints

<kriskowal>optimizations of that kind exist in every js vm but are not observable to the language.

<kriskowal>that is, if the vm’s internal representation of a 32 bit integer overflows, the internal representation gets silently lifted to a double float.

<kriskowal>tsyesika, i would want to be more specific about what the round-trip is, to answer your question…

<kriskowal>i think it would be delightful but also unnecessary for Alice (JS) to read a JSON file with JavaScript’s JSON.parse then send the resulting value to Bob (Guile) and for them to write out an identical JSON file. Then, for Bob to do the reverse and for Alice to have the same JSON file again.

<kriskowal>Just for JS<->Guile over OCapN round-trips (no JSON involved), we need JavaScript number to be Guile real, so if Alice (JS) parses cap data and sends it over the wire to Bob (Guile), all the numbers are going to be reals.

<kriskowal>For the whole trip from JSON at rest to JSON at rest and back again to work, that suggests that Guile would need a JSON library that writes Guile-real as JSON-number and always reads JSON-number as real.

<kriskowal>The JSON library would also have to use the same distinguished `null` value.

<kriskowal>I do think that you should do this, but it is not germane to OCapN. For OCapN, it’s sufficient (to Agoric) to be able to round trip passable data just over OCapN (and also for JavaScript to be able to read JSON, echo it over OCapN, and write identical JSON back). We place no constraints on how Guile drops and lifts JSON locally.

<kriskowal>That said, for completely orthogonal reasons, if you find yourself using JSON as a shared medium with other languages for any reason at all, you really need to represent JSON number as Guile real. For example, Protobuf 3 has a well-defined mapping to and from JSON that, like smallcaps, uses decimal strings for any integer with more than 32 bits of precision.

<dthompson>ACTION is going to read all of that... just in a meeting atm

<dthompson>regular numbers always being floats would be fine I think

<dthompson>a lot of parallels in this discussion to my current main focus of web assembly right now...

<dthompson>because we are dealing with Scheme <-> JS via WASM so we have the same numeric representation problems

<dthompson>in our Scheme <-> JS WASM bridge, Number is a Scheme flonum (rational) and BigInt is a Scheme int.

<dthompson>so yeah this all makes sense to me now

<cwebber><tsyesika> though I think both dthompson and cwebber have their own json libraries due to how guile-json approaches some things so I'm not

<cwebber>it's the same library alsmost ;)

<cwebber>I just made some modifications

IRC channel logs

2023-09-28.log