Access DOM content of browser within Android runtime

Question

Say we have a standard login page like the one below:

We can access the HTML elements in the DOM using jQuery or plain JavaScript like this:

In other words, the way to get the pixel location of an element in a web page is quite simply by using element.getBoundingClientRect():

var rect = element.getBoundingClientRect();
console.log(rect.top, rect.right, rect.bottom, rect.left);

So we can do this from the console or programmatically from a web app.

Now, say we have an Android browser (Chrome/Mozilla/WebView) in the foreground at any time. I can retrieve the URL of the web page in the browser. In this case:

https://login.microsoftonline.com/

So my question is, given the URL of a login page, how do I similarly get access to the same input field on an Android browser?

I need to be able to access the HTML elements of a web page in an Android browser, and calculate its pixel location. As input, I have the URL of a web page in any Android browser.

I am talking about doing this from an Android app, within the Android runtime, i.e. programmatically using Java/JS code.

In case someone needs the DOM structure of the page as text, it can be obtained programmatically with the following (Java) code:

URL url;
HttpURLConnection urlConnection;
String DOMContent = null;

try {
    url = new URL("https://login.microsoftonline.com/");
    urlConnection = (HttpURLConnection) url.openConnection();
    int responseCode = urlConnection.getResponseCode();
    if(responseCode == HttpURLConnection.HTTP_OK){
        DOMContent = readStream(urlConnection.getInputStream());
    }

} catch (MalformedURLException e) {
    e.printStackTrace();
} catch (IOException e) {
    e.printStackTrace();
}

I need access to the HTML elements of a mobile web page within the Android runtime, just as we would in a web app or extension in a desktop browser. Or in other words, I need to be able to access/manipulate the DOM content of a mobile browser from an Android app.

How can this be done?

Update:

JavaScriptBridge looks promising. DocumentBuilder could help us convert the DOM into Java objects which may then be accessed/manipulated natively from Android.

References:

1. How to execute JavaScript on Android?

2. Calling JavaScript functions in WebView

3. How to run Javascript code in a background Service on Android

4. Is there any way to get access to DOM structure in Android's WebView?

5. Android webview Access the DOM

6. In Android Webview, am I able to modify a webpage's DOM?

7. Android WebViews and the JavaScript to Java bridge

8. Using Javascript bridge in android

9. Alternative way for communication between WebView and native

"and calculate its pixel location" -- it has no pixel location, as it is not on the screen. — CommonsWare, Aug 03 '19 at 12:17
@CommonsWare I do not know what you mean by "it is not on the screen". It is not on the screen as a native Android element, and that I know. But surely it is present physically on the device screen as part of a web page. If we can calculate its pixel location on a desktop browser, we should likewise be able to do so on an Android device, at least in theory. — Y.S, Aug 03 '19 at 12:20
Not with the code that you are showing in your question. You are just making an HTTP request. There is no Web browser or other Web rendering engine, let alone a screen. — CommonsWare, Aug 03 '19 at 12:24
@CommonsWare: I included the code as a suggestion. If someone needs the DOM content, we can get it using the URL of the web page. What I have as input is the URL. — Y.S, Aug 03 '19 at 12:26
Unless you are rendering the Web content in your own process (e.g., via `WebView`), you have no way of knowing much about how it is going to be rendered. The pixel location of HTML elements depends on lots of things: CSS, JavaScript, the actual Web rendering implementation, the size of the rendering window, the scroll position of the content within that rendering window, etc. — CommonsWare, Aug 03 '19 at 12:30
You first need to render the page to know the element locations. But you can inject your own content into html you are rendering (including JS content). Into this content you inject your JavaScript which calls native Android method via JavaScript bridge and there you pass a value of element.getBoundungrectangle. this has more info https://stackoverflow.com/questions/38796319/injecting-javascript-bridge-in-webview — Igor Čordaš, Aug 04 '19 at 11:27
`I am talking about doing this from an Android app, within the Android runtime, i.e. programmatically using Java/JS code.` that's impossible unless you render the page. The position of an element is decided by many factors. Just using Java isn't enough. — Zun, Aug 07 '19 at 14:02
I'll donate €50 to a charity of your choice if someone is able to answer `I need access to the HTML elements of a mobile web page within the Android runtime, just as we would in a web app for a desktop browser. Or in other words, I need to be able to access/manipulate the DOM content from an Android app, given the URL of the web page.` without rendering the page first or using something like JSoup :) — Zun, Aug 07 '19 at 14:12
@Zun: The page is definitely rendered. It is in the foreground on the screen. And I need those €50 myself :) — Y.S, Aug 07 '19 at 14:21
Please edit your post because it seems like you have way too many points that contradict themselves. **IF USING WEBVIEW IS ALLOWED, WHICH YOUR SUBMISSIONS SUGGESTS IT IS NOT**, you can run a JavaScript command using `evaluateJavascript` which gets the "physical" location of an element — Zun, Aug 07 '19 at 14:29
To get DOM from webView :- var dom = document.getElementsByTagName('html')[0].outerHTML; — Amol Gharpure, Sep 30 '19 at 07:03

score 1 · Answer 1 · answered Aug 07 '19 at 14:33

1

Use the following code after the page has been loaded (implement a custom WebViewClient and check the onPageFinished)

String query = "document.getElementById(\"WhateverElement\").getBoundingClientRect();"    

webView.evaluateJavascript(query, new ValueCallback<String>() {
        @Override
        public void onReceiveValue(String s) {
            Log.d("LogName", s); // s has the getBoundingClientRect
        }
    });

answered Aug 07 '19 at 14:33

Zun

1,407
3
14
22

This answer is NOT a solution to the question asked. The question is how to get access to the DOM of a mobile browser programmatically from the Android runtime. Just like we do from a web app or extension in a desktop browser. – Y.S Aug 23 '19 at 12:15
@Y.S that's impossible. Have a nice day – Zun Aug 23 '19 at 12:24

Sahil Manchanda · Answer 2 · 2019-09-30T09:48:24.130

The webpage needs to be rendered first in order to get the pixel position of the input field. and to manipulate dom content in android, the best way is to go with JS Interface. The way mentioned by ZUN is also helpful. You can execute arbitrary JavaScript at runtime either way.

However, there is one particular library that can catch your attention i.e. jsoup

From the jsoup website:

jsoup is a Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods. jsoup implements the WHATWG HTML5 specification and parses HTML to the same DOM as modern browsers do.

Side note: if you don't want to show webpage to the user till all of your calculations are done, you might be interested in playing with Visibility of WebView

Access DOM content of browser within Android runtime

2 Answers2