UTF-16 surrogate pairs decoding

[vshabanov]

Hi, UTF-16 surrogate pairs are incorrectly decoded:

λ> Json.decode $ Bytes.fromByteString "\"\\ud800\\udc00\""
Right (String "\65533\65533")
-- should be
λ> Aeson.decode "\"\\ud800\\udc00\"" :: Maybe Aeson.Value
Just (String "\65536")

And invalid sequences are emitted in some cases:

λ> J.decode $ Bytes.fromByteString "\"\\ud800\\udfff\""
Right (String "*** Exception: recoverDecode: invalid argument (invalid byte sequence)
-- should be
λ> Aeson.decode "\"\\ud800\\udfff\"" :: Maybe Aeson.Value
Just (String "\66559")

[andrewthad]

Thanks for the report with good examples. I'll figure out how to address this soon.

[mwotton]

any luck with this? this is a blocking bug for me now so I'm happy to put some time in, but I'm by no means a unicode expert.

[andrewthad]

I have not made any attempts to fix this. If you are able to fix it, though, I would welcome a PR.

[mwotton]

alright, i'll give it a go.