Illegal Characters/Escape-Necessary characters for JSON

Question

I am getting data from one API and POSTing into another API but some records are ending up in error. I have figured out that there are illegal characters in the fields of the source API.

Unfortunately there are limitations on my destination API due which I can not use URLEncode.encode, escaping etc or any other content-type but application/json and also due to other reasons my only option is to manually find and remove these characters with predetermined replacement characters like space/hyphen. My client has agreed to this approach as well. Now I don't know all of these characters.

according to me the following characters are causing problems:

‘
”
\
{
}
[
]

is the above list complete and correct

p.s. Data Sample 1

{
"name": "JSW-13867",
"description": "Hi,

    It would be extremely useful
if it was possible to assign dates in the scrum board columns.

#We already configure the sprint end date,
so that could be displayed in the last column.\\\\
So,
in our
case,
currently the last column is named "Done".
What we 'd like to see is "Done (by 08/Jul/16)"\\\\
Although this information is 1 click away,
so it wouldn 't be that big of an improvement (though easy to implement). But it makes sense if #2 below is implemented too.\\\\#
What would be important in our
case is the one - to - last column,
which is "Test Done".\\\\
That would become "Test Done (by 01/Jul/16)".\\\\
What would be probably the correct / concise way,
is a configuration on a board level,
of an X amount of days before the sprint end date,
for any column;which,
ifset,
will be displayed as per my examples(or similar).

The last sentence of #2 is my recommendation / feature request...

Thanks for a great product :) \\
\\
----
\\
original post: https://answers.atlassian.com/questions/39228431/answers/39228618","fields": {"field": [{"dataType": "ENUM_TYPE","id": "1716","name": "KFH-Issue-Issue Details:Issue Rating","hasChanged": true,"enumValue": {"name": "Very Low"}},{"dataType": "ENUM_TYPE","id": "530","name": "OPSS-Iss:Status","hasChanged": false,"enumValue": {"name": "Open"}},{"dataType": "ENUM_TYPE","id": "1717","name": "KFH-Issue-Issue Details:Issue Source","hasChanged": false,"enumValue": {"id": "4333","name": "IAD","index": 2,"hidden": false}}]},    "typeDefinitionId": "24",    "primaryParentId": "26982"}

p.s . Data Sample 2

{
"name": "JSW-13840",
"description": "  h3. Summary   If we set the *Incoming Authentication* on Fisheye side towards connection from JIRA and have the  Executed As  field specified by a user, the trigger creation will be mentioned that there is a problem with the configuration and mention that Fisheye status is  Not Working     h3. Steps to Reproduce   # Integrate JIRA with Fisheye.  # Make sure that Fisheye is connected to a repo and at least One commit mentioning a JIRA ticket.  # Edit OAuth  Incoming Authentication  on Fisheye side for connection with JIRA and set the  Execute As  field.  # Update the configuration.  # Navigate to JIRA and add a trigger to a specific workflow transition    h3. Expected Result   - The Diagnostics will state that the connection is  Working     h3. Actual Result   - The Diagnostics state that Fisheye connection is  Not Working     h3. Notes   - Using JIRA 7.1.7 and FerCru 4.0.4  - Tested this with 2LOi and it mention that there is no problem with the configuration. But, if we added a user on that particular field, it will back to  Not Working  after refreshing the page.  - Samse as 2LOi, 2LO without any additional settings also did not have any issue.  - Result of the diagnostic with  Execute as    {code}{ appLinkDiagnosticResults  [{ name   Julians-MacBook-Pro , type   FishEye / Crucible , icon   http //localhost 8717/jira_717/s/en_US-6hktwl/71011/b6b48b2829824b869586ac216d119363/5.0.7/_/download/resources/com.atlassian.applinks.applinks-plugin applinks-images/images/types/16fecru.png , supportedVersion  true, accessible  true, inboundStatus  null, outboundStatus   NOT_AUTHENTICATED , local2LOConfigured   OK , remote2LOConfigured   NOT_TESTED , working  false}], dvcsDiagnosticResult  { dvcsAccountDiagnosticResults  []}, devToolsDocoUrl   https //docs.atlassian.com/jira/jcore-docs-071/Integrating+JIRA+with+Code+Development+Tools   {code}  - Result of the diagnostic for aynthing without  Execute as    {code}  { appLinkDiagnosticResults  [{ name   Julians-MacBook-Pro , type   FishEye / Crucible , icon   http //localhost 8717/jira_717/s/en_US-6hktwl/71011/b6b48b2829824b869586ac216d119363/5.0.7/_/download/resources/com.atlassian.applinks.applinks-plugin applinks-images/images/types/16fecru.png , supportedVersion  true, accessible  true, inboundStatus   WORKING , outboundStatus   WORKING , local2LOConfigured   OK , remote2LOConfigured   OK , working  true}], dvcsDiagnosticResult  { dvcsAccountDiagnosticResults  []}, devToolsDocoUrl   https //docs.atlassian.com/jira/jcore-docs-071/Integrating+JIRA+with+Code+Development+Tools }  {code}",
"fields": {
    "field": [{
        "dataType": "ENUM_TYPE",
        "id": "1716",
        "name": "KFH-Issue-Issue Details:Issue Rating",
        "hasChanged": true,
        "enumValue": {
            "name": "Low"
        }
    }, {
        "dataType": "ENUM_TYPE",
        "id": "530",
        "name": "OPSS-Iss:Status",
        "hasChanged": false,
        "enumValue": {
            "name": "Open"
        }
    }, {
        "dataType": "ENUM_TYPE",
        "id": "1717",
        "name": "KFH-Issue-Issue Details:Issue Source",
        "hasChanged": false,
        "enumValue": {
            "id": "4333",
            "name": "IAD",
            "index": 2,
            "hidden": false
        }
    }]
},
"typeDefinitionId": "24",
"primaryParentId": "26982"
}

Why don't you escape them properly like everyone else? json escape != url-encode -> http://stackoverflow.com/questions/3020094/how-should-i-escape-strings-in-json — zapl, Jun 27 '16 at 17:26
i tried that, my destination API only supports `application/json` content. So the encode strings end up , well you know what encoded string look like. Escaping is not practical t all, I wiil have to parse every key of every object and escape. Do-able not practical. — Moon, Jun 27 '16 at 17:42
the way to escape is defined by `application/json`, when the API actually supports that format, it supports escaping as well. edit: is your code building that sample data or is that coming from somewhere else and you can't change it? — zapl, Jun 27 '16 at 17:44
Look at the [JSON spec](http://www.json.org/). You only have to escape `"`, ``\``, and control characters (e.g. `\r`, `\n`, `\t`, ...). Using a JSON library to build JSON is the easiest way to accomplish that. — Andreas, Jun 27 '16 at 17:54

madatx · Accepted Answer · 2016-06-27T18:22:14.640

0

I suggest you to use a Java JSON library to properly json encode data for destination API. If you would provide data example we could help you more.

UPDATE: Here is the code that works with your data

public static void main(String[] args) {
    String desc = "\"description\": \"";
    String fields = "\",\"fields\"";
    int indexStart = njson.indexOf(desc) + desc.length() ;
    int indexEnd = njson.indexOf(fields);

    String ex = njson.substring(indexStart, indexEnd);

    String str = JSONObject.quote(ex);

    String begin = njson.substring(0, indexStart - 1);
    String end = njson.substring(indexEnd + 1);

    String validJSon = begin + str + end;

    JSONObject obj = new JSONObject(validJSon);
}

njson is string from your example

edited Jun 27 '16 at 18:22

answered Jun 27 '16 at 17:42

madatx

81
5

example posted above – Moon Jun 27 '16 at 17:45
@Moon yes, this is not a valid JSON. In the "description" field you have double quotes. From here, you can take a substring from "description" to "fields" and simply JSON encode this substring. After that you will have the valid JSON object. – madatx Jun 27 '16 at 17:56
it worked. Also the aforementioned method replacing and escaping special characters. there is just 1 out of my 50 records which go through by both thecniques. Please see Data Sample 2 – Moon Jun 27 '16 at 19:04
the only possible error field is only 'description' because rest are from Dropdowns or System Generated – Moon Jun 27 '16 at 19:06
The problem with your example is that the code from Data Sample 2 does not have a space before `fields`. To make this work replace `fields = "\",\"fields\"";` to `fields = "\"fields\"";` , then delete the last quote from substring `int lastQuoteIndex = ex.lastIndexOf("\""); ex = ex.substring(0, lastQuoteIndex);` and after `JSONObject.quote(ex)` add comma if needed `if (!end.trim().startsWith(",")){ str += ',';}` – madatx Jun 27 '16 at 19:54
For more common situations you can use regex to retrieve data from `description` field – madatx Jun 27 '16 at 20:01

Illegal Characters/Escape-Necessary characters for JSON

1 Answers1