PMs and RDs, Look Here: Starting from Scratch to Develop or Create an App Browser!

Introduction

This article will explore the direction of developing or creating an app browser solution,

providing references for collaboration between project management (PM) and research and development teams (RD), ensuring a smooth development process and high-quality results through preliminary research.

1. Requirements
   • Want to create an app browser that includes most of the features available in mainstream browsers, such as search, tabs, downloads, favorites, sharing, etc.
   • Support for URL filtering, site blocking, etc.

<li><b>Compatibility Testing:</b>
  <ul>
    <li>
    Developing your own browser,

    so you need to ensure that most of the browser's features are present,

    to avoid the embarrassment of other browsers having features that yours lacks...

    </li>

    <li>Developing a web browser might have compatibility issues with various web pages. You can refer to <a href="https://blog.user.today/inapp-browser-webview-problem" target="_blank" rel="noopener noreferrer">this article</a> for more information.</li>

    <li>Need to test whether your self-developed browser supports HTML tags:
      <ul>
        <li><a href="http://beta.html5test.com/" target="_blank" rel="noopener noreferrer">HTML5 Feature Detection</a></li>
      </ul>
    </li>
  </ul>
</li>

<li><b>Confirming Requirements for Self-Developed Browser</b>
  <ul>
    <li>Since development hasn't started yet, first confirm the target level for your browser.

      If there is no clear goal for the browser,

      you can research several third-party browsers,

      and decide whether to start from WebView or modify some open-source projects.

      Of course, if you use open-source projects, you also need to pay attention to licensing issues.

    </li>

    <li>
    This table lists the differences between two third-party browsers and a browser developed from WebView.

    It provides references for subsequent requirements and discussions on the actual features needed,

    and helps in deciding whether to use third-party sources or develop independently.

    <img src="/images/browser/001.png" alt="safe browser 01" / class="prose-img">

    </li>
  </ul>
</li>

URL Content Filtering

  such as restricting access to certain rated websites to make your browser safer...

  you need to filter URLs.

• Implementing content filtering solutions,

  mainly divided into integrating third-party SDKs and self-implementation,

• Both options need to be considered
  • Developing browser content filtering on your own is simpler. If you want to restrict all browsers within the system, you need to consider whether to use auxiliary permission schemes (the following schemes list the implementation methods of competitors for reference). Some vendors use proxy or DNS to solve this issue. However, developing auxiliary permissions or proxies will require **more time**, which should also be taken into account.
  • When considering these methods, you need to consider whether they are available on both platforms.
• **Integrate third-party SDK**:
  • Different third-party SDKs provide different types of threats. If some types are not provided, you will need to maintain them yourself.
  • If using Google's **Web Risk API** (available on both platforms when connecting to the network request API), you need to review the following to understand its rules:
    1. Web Risk Service Level Agreement
    2. Quotas and Limits
    3. Threat Types
  • If integrating **Trend Micro HNS SDK**, you need to understand whether the SDK provides the data we target.
    • Since this SDK requires contacting the company's customer service, and the development documents are not displayed on the webpage, you can only understand the approximate range provided through the following images:
    • Scope and details provided
    
    • System requirements and the range of dual-platform support
    
    • Or you can refer to the analysis of Trend Micro's app below to speculate that the content behavior might be similar to that app.
  • If integrating **CleanBrowsing**, note the following:
    1. Supports Android 9.0 and above. Set DNS through the **DNS-over-TLS (DOT) protocol** in the settings: Method (Since this implementation method uses the function within Android, it is speculated that iOS might not support it)
    2. You also need to contact the company's customer service. Here is the filtering range provided on the website:
    
• Develop a content-filtering browser on your own
  You can use some third-party open-source resources, such as `PhishTank` (actual examples are listed below), which provides open-source data that might be phishing websites.
  Additionally, you need to consider whether the target types provided are as expected. If not, how do you want to maintain them, etc.

• Integrate existing SDKs in the market, such as Google's Safe Browsing API, Web Risk API, or Trend Micro's HNS SDK, etc.

      <a href="https://cleanbrowsing.org/filters/" target="_blank" rel="noopener noreferrer">CleanBrowsing</a> can perform filtering by setting DNS.
  
      Below, we list various methods for developers to reference.
  
      <ul>
          <li>
              <b>Safe Browsing API</b> (Non-commercial) vs <b>Web Risk API</b> (This solution requires payment)
              <ul>
                  <li>
                      Safe Browsing API is for non-commercial use only. For commercial use, you need to use
                      <a href="https://cloud.google.com/web-risk?hl=zh-tw" target="_blank" rel="noopener noreferrer">Web Risk API</a>.
                  </li>
                  <li>
                      <a href="https://cloud.google.com/web-risk/docs/apis" target="_blank" rel="noopener noreferrer">Web Risk Rest API Documentation</a>
                       : This API checks and filters URLs to see if they are of a threat type, implemented via HTTPS requests during actual development.
                  </li>
                  <li>
                      Types of APIs provided by Web Risk
                      <ol>
                          <li>
                              <a href="https://cloud.google.com/web-risk/docs/update-api?hl=zh-cn" target="_blank" rel="noopener noreferrer">Lookup API</a>
                              : After specifying <b>threatTypes</b> and target URL, the backend queries and returns the threat type and the last valid time.
                          </li>
  
                          <li>
                              <a href="https://cloud.google.com/web-risk/docs/update-api?hl=zh-cn" target="_blank" rel="noopener noreferrer">Update API</a>
                              : Downloads the web risk hash list, stores it locally, allowing developers to check locally and use other APIs to confirm if a match is found.
                          </li>
  
                          <li>
                              <a href="https://cloud.google.com/web-risk/docs/evaluate-api?hl=zh-cn" target="_blank" rel="noopener noreferrer">Evaluate API</a>
                              : After specifying <b>threatTypes</b> and target URL, the backend queries and returns the threat type and trust level.
                          </li>
  
                          <li>
                              <a href="https://cloud.google.com/web-risk/docs/submission-api?hl=zh-cn" target="_blank" rel="noopener noreferrer">Submission API</a>
                              : Reports URLs considered risky to the server.
                          </li>
  
                      </ol>
                  </li>
                  <li>
                      Types of threats provided by Web Risk API
                      
  
                      <img src="/images/browser/006.png" alt="safe browser 06"  class="prose-img">
                  </li>
              </ul>
          </li>
          <li>
              <b>HNS SDK</b> This platform does not provide SDK development documentation. If using this solution, you need to contact them for more information (However, after researching the company's security products, it is speculated that this SDK should meet these requirements).
          </li>
          <li>
              CleanBrowsing If using this solution, you may need to understand if it can be set via code for Encrypted DNS – DNS over TLS support.
              <ul>
                  <li>
                      Check the product, it also has auxiliary permissions and admin:
                      <a href="https://cleanbrowsing.org/help/docs/cleanbrowsing-on-android/#step-2-give-accessibility-control-to-app" target="_blank" rel="noopener noreferrer">Process Documentation</a>
                  </li>
              </ul>
          </li>
      </ul>
  </li>
  <li>
      <b>Internal Development</b>
      <ul>
          <li>
              URL blacklist, content parsing and filtering data sources
              <ul>
                  <li>
                      There is an open-source maintained third-party phishing site list <a href="https://www.phishtank.com/phish_search.php?verified=u&active=y" target="_blank" rel="noopener noreferrer">PhishTank</a>
                      It appears to be continuously maintained, providing about 70,000 confirmed active phishing sites, and also provides suspected phishing sites.
                      
  
                      <img src="/images/browser/007.png" alt="safe browser 07"  class="prose-img">
  

<img src="/images/browser/008.png" alt="safe browser 08"  class="prose-img">

Browser Feature Development Reference

• This solution mainly involves,

  using the library provided by Google itself,

  by sending an intent,

  to open the specified URL in Chrome.

  If the device does not have Chrome or a browser that supports custom tabs,

  it will still send an intent to invoke the system browser (usually the default browser).

  Additionally, the page using custom tabs will have fixed menu options,

  one of which allows opening in Chrome.

  The customization scope of the page is also limited within Chrome.

  reference

• When setting the webViewClient in WebView,

  if a URL does not meet the restrictions, it will redirect to another page.
  
   <pre>
   public boolean shouldOverrideUrlLoading(WebView view, String url) {
       if (!isUrlValid(url)) {
           view.loadUrl("file:///android_asset/error.html");
           return true;
       }
       return false; // Returning false means the URL will load normally
   }
   </pre>
  

• Set the userAgentString of the WebView,

  and adapt according to the version (but the server of the URL must also support it).
  
   <pre>
   currentWebView!!.settings.userAgentString = resources.getStringArray(R.array.user_agent_data)[2]
   currentWebView!!.reload()
   </pre>
  

• The default browser agent settings in user_agent_data

  <img src="/images/browser/016.png" alt="safe browser 16"  class="prose-img">
  

• If some web components do not display, check if the component requires JavaScript.

  <pre>
  currentWebView!!.settings.javaScriptEnabled = true // or false
  </pre>
  

• To support WebView search,

  you can input the corresponding URL for the target search engine and modify the value at the end.
  
  <pre>
  Example using Google search:
  https://www.google.com/search?q=search content
  </pre>
  Then provide the user with the option to set the default search engine in the app.
  
  <img src="/images/browser/017.png" alt="safe browser 17"  class="prose-img">
  

If you can directly modify their projects,

it can save a lot of time.

However, you need the capability to make those modifications,

and when formulating a plan, you need to ensure the open-source license allows commercial use.

Below are a few I have researched for your reference.

1. lynket

• License: GNU v3 Public License.

2. duckduckgo

• Currently, the most complete one is duckduckgo.

  Implemented purely in Kotlin, source code is easy to read.

• License: Apache 2.0 license.

3. foss browser

• License: AGPL-3.0
• Due to license issues, not considered for now.
• It is FOSS, developed purely in Java.
• Has the features of open source.

4. Yuzu browser

• Apache License 2.0
• Implemented purely in Kotlin,

  but when I downloaded the official code to build,

  I encountered some issues with the mac version.

  If anyone else encounters this, adjustments may be needed.