PMs and RDs, Look Here: Starting from Scratch to Develop or Create an App Browser!

1. Requirements
   • Want to create an app browser that includes most of the features available in mainstream browsers, such as search, tabs, downloads, favorites, sharing, etc.
   • Support for URL filtering, site blocking, etc.


2. Compatibility Testing:
   • Developing your own browser,
     so you need to ensure that most of the browser's features are present,
     to avoid the embarrassment of other browsers having features that yours lacks...
     
   
   
   • Developing a web browser might have compatibility issues with various web pages. You can refer to this article for more information.
   
   
   • Need to test whether your self-developed browser supports HTML tags:
     • HTML5 Feature Detection


3. Confirming Requirements for Self-Developed Browser
   • Since development hasn't started yet, first confirm the target level for your browser.
     If there is no clear goal for the browser,
     you can research several third-party browsers,
     and decide whether to start from WebView or modify some open-source projects.
     Of course, if you use open-source projects, you also need to pay attention to licensing issues.
     
   
   
   • This table lists the differences between two third-party browsers and a browser developed from WebView.
     It provides references for subsequent requirements and discussions on the actual features needed,
     and helps in deciding whether to use third-party sources or develop independently.
     

• Implementing content filtering solutions,
  mainly divided into integrating third-party SDKs and self-implementation,
  
• Both options need to be considered
  • Developing browser content filtering on your own is simpler. If you want to restrict all browsers within the system, you need to consider whether to use auxiliary permission schemes (the following schemes list the implementation methods of competitors for reference). Some vendors use proxy or DNS to solve this issue. However, developing auxiliary permissions or proxies will require more time, which should also be taken into account.
  • When considering these methods, you need to consider whether they are available on both platforms.
• Integrate third-party SDK:
  • Different third-party SDKs provide different types of threats. If some types are not provided, you will need to maintain them yourself.
  • If using Google's Web Risk API (available on both platforms when connecting to the network request API), you need to review the following to understand its rules:
    1. Web Risk Service Level Agreement
    2. Quotas and Limits
    3. Threat Types
  • If integrating Trend Micro HNS SDK, you need to understand whether the SDK provides the data we target.
    • Since this SDK requires contacting the company's customer service, and the development documents are not displayed on the webpage, you can only understand the approximate range provided through the following images:
    • Scope and details provided
    
    • System requirements and the range of dual-platform support
    
    • Or you can refer to the analysis of Trend Micro's app below to speculate that the content behavior might be similar to that app.
  • If integrating CleanBrowsing, note the following:
    1. Supports Android 9.0 and above. Set DNS through the DNS-over-TLS (DOT) protocol in the settings: Method (Since this implementation method uses the function within Android, it is speculated that iOS might not support it)
    2. You also need to contact the company's customer service. Here is the filtering range provided on the website:
    
• Develop a content-filtering browser on your own
  You can use some third-party open-source resources, such as `PhishTank` (actual examples are listed below), which provides open-source data that might be phishing websites.
  Additionally, you need to consider whether the target types provided are as expected. If not, how do you want to maintain them, etc.

• Integrate existing SDKs in the market, such as Google's Safe Browsing API, Web Risk API, or Trend Micro's HNS SDK, etc.
  CleanBrowsing can perform filtering by setting DNS.
  Below, we list various methods for developers to reference.
  
  • Safe Browsing API (Non-commercial) vs Web Risk API (This solution requires payment)
    • Safe Browsing API is for non-commercial use only. For commercial use, you need to use Web Risk API.
    • Web Risk Rest API Documentation : This API checks and filters URLs to see if they are of a threat type, implemented via HTTPS requests during actual development.
    • Types of APIs provided by Web Risk
      1. Lookup API : After specifying threatTypes and target URL, the backend queries and returns the threat type and the last valid time.
      
      
      2. Update API : Downloads the web risk hash list, stores it locally, allowing developers to check locally and use other APIs to confirm if a match is found.
      
      
      3. Evaluate API : After specifying threatTypes and target URL, the backend queries and returns the threat type and trust level.
      
      
      4. Submission API : Reports URLs considered risky to the server.
      
      
    • Types of threats provided by Web Risk API
      
  • HNS SDK This platform does not provide SDK development documentation. If using this solution, you need to contact them for more information (However, after researching the company's security products, it is speculated that this SDK should meet these requirements).
  • CleanBrowsing If using this solution, you may need to understand if it can be set via code for Encrypted DNS – DNS over TLS support.
    • Check the product, it also has auxiliary permissions and admin: Process Documentation
• Internal Development
  • URL blacklist, content parsing and filtering data sources
    • There is an open-source maintained third-party phishing site list PhishTank It appears to be continuously maintained, providing about 70,000 confirmed active phishing sites, and also provides suspected phishing sites.
      
  • The official website FAQ states that it can be used commercially
    
  • Refer to competitors to develop URL blacklists and content parsing filters
    • Later, I checked the Trend Micro Mobile Security app which also provides browser protection features
      
    • It is speculated that it also uses accessibility permissions to obtain the browser URL for corresponding processing
      The place where the app enables accessibility permissions also mentions this method
      
    • Here is a demo to capture the Chrome URL
      First, create Accessibility, then add the target package name in the declaration XML
      Then, in the code, retrieve the EditText to get the URL on the screen:
      
      
    
    
    • Log of the actual URL obtained
      
    
    
    • Additionally,
      it seems to handle commonly used apps,
      if it is not a commonly used app or difficult to capture the package name,
      the competitor provides an option to select the app to protect,
      and then connects through a VPN (presumably with a proxy or DNS filtering)
      
    
    
    • The app also performs website type filtering,
      it appears to subdivide websites by category for filtering
      
      
    
    

• This solution mainly involves,
  using the library provided by Google itself,
  by sending an intent,
  to open the specified URL in Chrome.
  If the device does not have Chrome or a browser that supports custom tabs,
  
  it will still send an intent to invoke the system browser (usually the default browser).
  Additionally, the page using custom tabs will have fixed menu options,
  one of which allows opening in Chrome.
  The customization scope of the page is also limited within Chrome.
  reference

• When setting the webViewClient in WebView,
  if a URL does not meet the restrictions, it will redirect to another page.
  

       public boolean shouldOverrideUrlLoading(WebView view, String url) {
           if (!isUrlValid(url)) {
               view.loadUrl("file:///android_asset/error.html");
               return true;
           }
           return false; // Returning false means the URL will load normally
       }
       

• Set the userAgentString of the WebView,
  and adapt according to the version (but the server of the URL must also support it).
  

       currentWebView!!.settings.userAgentString = resources.getStringArray(R.array.user_agent_data)[2]
       currentWebView!!.reload()
       

• The default browser agent settings in user_agent_data
  
• If some web components do not display, check if the component requires JavaScript.
  

      currentWebView!!.settings.javaScriptEnabled = true // or false
      

• To support WebView search,
  you can input the corresponding URL for the target search engine and modify the value at the end.
  

      Example using Google search:
      https://www.google.com/search?q=search content
      

  Then provide the user with the option to set the default search engine in the app.
  

1. lynket

• License: GNU v3 Public License.

2. duckduckgo

• Currently, the most complete one is duckduckgo.
  Implemented purely in Kotlin, source code is easy to read.
• License: Apache 2.0 license.

3. foss browser

• License: AGPL-3.0
• Due to license issues, not considered for now.
• It is FOSS, developed purely in Java.
• Has the features of open source.

4. Yuzu browser

• Apache License 2.0
• Implemented purely in Kotlin,
  but when I downloaded the official code to build,
  I encountered some issues with the mac version.
  If anyone else encounters this, adjustments may be needed.