An Intro to the Systemless Interface of Rooting Android Phones
Table of Contents
What you can learn from this article:
- What is meant by rooting an Android phone
- Steps to take before proceeding to root your phone
- Google’s SafetyNet check
- The preferred root method in today’s world – Magisk – and an outline on how it works
- Things you can do after flashing Magisk – the utility of rooting your phone
- Things (I feel) you should avoid doing after rooting systemless-ly
- Uninstalling Magisk
- What to do in case the phone is bricked
What this article will not teach you:
Specific methods to root your phone or flash custom recoveries. Such methods vary greatly from one phone to another. For example, in Samsung phones, you can flash files with software called Odin, which is specifically written for Samsung phones. On other brands, you have to use some other software, if they exist, or use Fastboot. I have not included these topics in this article. My attempt has been to keep the content of the article as general as possible so that it is applicable to a large number of different devices rather than cater to a specific device.
Disclaimer:
This article is meant to provide information to the readers regarding the Android rooting process. Please do not hold me or Physics Forums responsible if your device is bricked before, during or after the rooting process.
You need to have a clear understanding of the working of the Android system before you try to root your phone. Remember that rooting your phone will bestow superpowers upon you. If you do not know how to utilize these powers and safeguard them, you might make your device vulnerable to threats that can, among other things, steal your personal information and empty your bank account.
You must clearly understand that once you root your phone, unlock the bootloader, or flash a custom recovery, the warranty of your phone is void.
As Android is an open-source project, modifications to the stock ROM are considered legal in most countries. However, you should enquire into the laws of your country and ensure that you are not doing anything illegal.
Don’t say that we didn’t warn you!
What is meant by rooting an Android phone?
If you have used Windows, you must be familiar with “administrator permissions”. On Linux, there is the concept of super user (SU) privileges. Something similar exists in Android. The process of gaining super user privileges on an Android device is commonly called rooting. The system super user account is called root, hence the term rooting. This account has permissions over all files and programs on a UNIX-based system. It has full control over the operating system.
Rooting is done using an SU interface that supervises the root permissions granted to different applications. The process of flashing the SU application is generally done through Android Debug Bridge (ADB) from PC, or from the recovery, which is a minimal Android boot image that provides maintenance functions.
Before proceeding to root your phone…
Backup your stock ROM and data
Before you root your phone, you must ensure that in case of any mishap during or after the rooting process, you should be able to restore your phone to the state that it was in before the mishap. This means that you should backup the important sectors of the phone memory, which can be restored later if the situation demands. In addition to backing up your media, documents, etc., you also have to backup /system, /boot, /data and /recovery partitions.
/systemhas the files that make up your Android framework./bootcontains files that are executed when the Android system boots./dataconsists of the data of all user-installed applications which are present in the internal storage./recoveryhouses the image of the recovery console, which is an independent interface that can be triggered by certain hardware key combinations. Recovery works when the Android OS has not loaded, so you can make changes to partitions like/systemand/bootfrom it. These partitions are otherwise mounted read-only. Recovery will work even if the Android system cannot boot, thereby providing you with an opportunity to detect and rectify the errors that are preventing the system from booting normally.
Backing up /system and /boot constitutes backing up your stock ROM. Along with that, /data is also backed up, so that the app data is not lost.
There is another directory, /vendor. According to Android Open Source Project (AOSP) documentations,
The
/vendorpartition contains any binary that is not distributable to the Android Open Source Project (AOSP). If there is no proprietary information, this partition may be omitted.
/vendor is not a very important directory, so you can ignore it while backing up.
Your phone comes preloaded with a stock recovery. The problem is, this recovery has very limited functions, as most phone companies do not want users to tweak their stock ROM. Important functions like backup and flashing files are not available in the stock recovery. In order to carry out such functions, you have to flash a custom recovery like TWRP (Team Win Recovery Project). TWRP fulfills all the requirements of a good custom recovery. On the downside, once you flash TWRP, you won’t be able to backup the stock recovery.
There are some ways to backup the stock recovery. For example, you can download the Flashify app from Play Store, which has an option to back up the stock recovery. But for almost all such methods available, your phone has to be rooted. If you (somehow) root your phone without using the recovery, you will lose the stock /system and /boot partitions, which is not acceptable at any cost. So, don’t lament over losing the stock recovery. In addition, TWRP is so useful that there will hardly be an instance when you would want to return to the stock recovery.
Once you have flashed TWRP, it is easy to backup your ROM. Instructions for backing up from TWRP are widely available, so I am not mentioning them here.
One point to remember is that TWRP does not back up your media files. Be sure to back those up separately before rooting.
Unlock the bootloader
Referring to the AOSP documentations once again,
A bootloader is a vendor-proprietary image responsible for bringing up the kernel on a device.
Unlocking the bootloader means you are allowing modifications in your system.
Different phones have different ways of unlocking the bootloader. For example, in Samsung, simply toggling a button in Developer Options in settings will unlock the bootloader. In some others, you have to run a program called fastboot on your PC and through it, you can unlock your bootloader. Some devices might not need this step at all; you can directly proceed to flashing a custom ROM in those phones. Search Google for the method applicable to your device.
Google’s SafetyNet check:
Towards the end of the year 2014, several Android users with rooted phones found that Android Pay was no longer working on their phones. This is because Google had introduced its latest invention – SafetyNet – in Play Services and Android Pay was making use of that.
SafetyNet tells an app whether the phone is in a tampered state or not. This feature is now included in Google Play Services by default. There are many parameters based on which Google issues the SafetyNet results. SafetyNet has two parts: basicIntegrity and ctsProfileMatch. As per the Android Developers’ Guide on SafetyNet, CTS (Compatibility Test Suite) Profile matching is a stricter test compared to basicIntegrity. Most apps making use of SafetyNet will work if the latter is satisfied, while some (like banking apps) demand that the phone should pass the former as well.
Among other parameters, SafetyNet checks whether the /system partition has been tampered with. Once it finds tampering, the ctsProfileMatch test will start yielding a negative result. (How SafetyNet finds this tampering is out of the scope of this article; refer to the Further Reading section for some idea.)
Keep in mind that SafetyNet is a necessary evil. Say you are a layman, and you do not know anything about the Android framework or rooting. You give your phone to a person for repairing something. That person roots your phone installs malware and returns it to you, but you don’t notice that. He aims at stealing your personal data. But once you start using your phone, Google detects the tampering, prevents any vulnerable app (like banking apps or mobile wallets) from running on your phone, and informs you that your phone is unsafe. In fact, even an OTA (Over-The-Air) upgrade to your stock ROM may have a bug that can leave your system insecure. SafetyNet is a boon to users in these situations.
The preferred root interface in the modern world – Magisk – and an outline on how it works
There are two types of root interfaces – one that modifies the /system partition directly (the traditional way), and one that does not do so. The latter interface is termed “systemless”. Traditional interfaces like SuperSU cannot bypass SafetyNet.
Some experienced members at XDA Developers realized that, among other things, if the interface doesn’t directly tamper with /system, then SafetyNet can be fooled. But there seemed to be never-ending lists of bugs when such systemless interfaces were tested. Finally, John Wu (XDA username: topjohnwu) devised Magisk, which could bypass the SafetyNet check (both ctsProfileMatch and basicIntegrity). Magisk is open-source; the main repository is hosted on GitHub.
If you look at the Android booting process, you will find that at one stage, the kernel calls the init process. Magisk replaces this init with magiskinit. This replacement is unavoidable, and changes are made in /boot partition. But /system partition is left unaltered.
Magisk provides root access by providing a working binary with the path /sbin/.magisk/su. Any application that tries to run this will bring up Magisk to grant them root access, which is, in turn, managed and maintained by the Magisk Manager application.
Magisk uses the well-known Linux technique bind mount. Bind mount is an alternate view of the file directory, but any changes done in bind mount are not reflected in the original file system. In essence, it is a property of the live system. So, if you make modifications in /system respecting the guidelines of systemless rooting, you will see (using a root file browser) that /system has been modified. But in reality, the changes are made in the bind mount. The original /system is available at /sbin/.magisk/mirror/system. Magisk can unmount /sbin and the bind mounts to hide all modifications easily.
Another important and interesting feature of Magisk is MagiskHide, which can hide the root from apps that tend to malfunction when they detect that the device is rooted (even if SafetyNet is bypassed by the root interface). This is done by making the bind mounts invisible to that app.
As a side note, Magisk has no dedicated website, and all updates are announced at XDA Developers. So, download Magisk only from the official thread on XDA or from GitHub.
Instructions for flashing Magisk using TWRP are easily available on the internet; a simple Google search should help you.
Things you must do after flashing Magisk:
At this level, I assume that you have flashed Magisk and have Magisk Manager installed.
Switch on Magisk Hide:
From Magisk v20.4 onwards, MagiskHide is no longer switched on by default. From the homepage in Magisk Manager, follow the steps below and switch on MagiskHide for the apps that do not normally require SU permissions:
Checking SafteyNet Status, and what to do if the test fails:
The next thing to check is whether your device passes SafteyNet. You can do this in the Magisk Manager app:
In most cases, Magisk will be able to fool SafetyNet. If, however, the test fails, you have to take the help of the MagiskHide Props Config module developed by XDA member didgeridoohan.
Things you can do after flashing Magisk – the utility of rooting your phone systemless-ly
Remove Ads completely — AdAway
AdAway is an open-source ad blocker for Android using the hosts file approach. It is hosted on GitHub. Download the apk, install it, and then allow the necessary permissions to the app. The app will download hosts from pre-defined sources. You can also define your own whitelists/blacklists. The downloaded hosts will be merged with your sources and the hosts.txt file on your device will be rebuilt. The result: advertisements will be completely removed from your device.
Consider an app that displays ads. It will not be able to display them simply because it will fail to connect to the ad servers. The same holds for websites visited from web browsers.
Unleash the power of Magisk modules
Modules are programs that will modify your system under the hood of Magisk. They are accessible from Magisk Manager:
You must reboot after installing/uninstalling modules for changes to take effect.
Some modules that I find interesting:
- Busybox for Android NDK (the one developed by XDA member osm0sis). Coupled with a terminal emulator app like Termux, you will be able to make changes to your device/manage some modules from the terminal.
- YouTube Vanced Magisk Repo (watch YouTube videos without interrupting ads, play with screen off, etc.)
- ADB and Fastboot for Android NDK (installs the Android Debug Bridge server; helps in controlling other devices connected via USB OTG)
- Terminal systemizer (convert user-installed apps to system apps)
- Terminal debloater (systemless-ly uninstall system apps; won’t free up space, but will silence irritating stock bloatware that cannot be disabled)
Before flashing a module, read its official thread (mostly on XDA) and the README.md file on Github to make sure you know what you are doing. Not all modules that you see listed will be compatible with your phone. Some of them might need other modules for working, so check that out too. Your phone should not have more than one kernel-tweaking modules working together, otherwise either both will shut down, or your device will malfunction.
View your file system using a root file browser
If you are interested in the technicalities of how things work in Android, you may download a root file browser and use it to view the files that were hidden from your view to date. Study the directories to see how the Android file system works. Don’t modify anything unless you know what you are doing, otherwise, you may brick your phone.
Flash custom ROMs
Once you have a rooted phone with a custom recovery, you can try custom ROMs. These ROMs are often much better than the stock ROMs. One famous custom ROM is LineageOS. Installation guides of custom ROMs are widely available on the net. Before flashing them, make sure to check the following:
- The known issues, and how they will affect you.
- You have downloaded the official build of the ROM.
- The version is specifically meant for your phone and is compatible with the CPU architecture of your device.
- Your preferred root method works with that ROM.
- The apps that you are using currently are compatible with that ROM.
- Whether you need to separately flash Google Play Services like OpenGApps.
Checking these is important. For example, there is no official LineageOS build for Samsung Galaxy On7. Somebody on XDA developed an unofficial build, but that has issues with the camera and VoLTE, so it is kind of useless to flash such a ROM.
Things (I feel) you should avoid doing after rooting systemless-ly
There are some applications available in Play Store, like Titanium Backup, which will give you options to natively modify the /system partition. Refrain from using such applications, because those will break the systemless interface of Magisk. Always respect the systemless technology of Magisk. For direct modifications of /system and other partitions, you are welcome to use traditional root interfaces like SuperSU.
Uninstalling Magisk
Uninstalling Magisk is very easy. It is possible via the Magisk Manager app:
or using TWRP: Go to the release page under the Magisk Github repo, find the version of Magisk installed on your device, download the uninstaller zip file for that version, and flash it using TWRP. (The uninstaller is version-specific, so make sure you download the one corresponding to the version installed on your device). /sbin and all bind mounts will be unmounted.
What to do in case the phone is bricked
When your phone cannot load the files of the Android OS and boot normally, it is said to be bricked. One of the most common ways of bricking the phone is through a bootloop, where the bootloader tries to execute the boot scripts, fails to do so, and recursively keeps on trying and failing. Bootloop is often encountered while rooting a device or when a custom ROM is flashed.
Different phones show different indications when they are bricked. Some phones, for example, Samsung, will keep on showing the company’s logo during booting, but never actually finish booting. If you leave it in that condition, it will keep on trying to boot until the battery is completely drained.
If you find your phone in a bootloop, do not panic. It is very common to brick devices during rooting or flashing custom ROMs. When I had tried to root my phone for the first time, I had also ended up with a bricked device. And guess what – I had not made a backup of the stock ROM, so I was in a greater mess. If you have followed the article properly, you should have already backed up the stock ROM using TWRP.
When the device is in a bootloop, pull out the battery to shut it down completely. Then boot into recovery. First, uninstall Magisk by flashing the uninstaller zip (see above). Then restore the backup that you had made using TWRP before rooting your phone. This should restore your phone to the state it was in before rooting.
If in a rare case your phone still does not work properly, try searching Google for the stock ROM of your phone, and flash that. That will surely remove all the problems.
Final words
The source codes of Magisk, its modules, etc. are available publicly on Github or other websites, as stated in this article. However, directly reading these files may become difficult and time-consuming for a person who is not experienced in this field. In this article, I have tried to compile together whatever I have learned to date about Magisk and systemless rooting in a less technical language compared to the original source files so that it can appeal to a larger audience. Some things might change with time, and I will try to keep the article updated as far as possible. If I come to learn something more on these topics, I will add them too.
I sincerely thank Greg Bernhardt, the admin of Physics Forums, for providing this opportunity to me. Without his efforts, this article would have never become a reality.
Further Reading:
- Android device partitions and file systems
- Android boot process (on XDA developers)
- Wikipedia page on rooting
- A blog post on SafetyNet
- A guide to flashing Magisk using TWRP
- Some details of SafetyNet at Android Developer website
- Magisk developer details, as provided by John Wu on Github
- Magisk documentations on Github
- An answer on Android StackExchange on how Magisk works
- Joshua J. Drake et al. – Android Hacker’s Handbook (The book does not include systemless rooting, as Magisk was written after that)
- A collection of Magisk modules at XDA. Not all the modules will work on your phone, so be aware before flashing.
- A thread on XDA discussing how to install and uninstall Magisk modules through TWRP
- What is ADB & Fastboot?
- Download ADB and Fastboot without installing Android SDK
Studying physics at a college in Kolkata, India. Interested in accelerator physics, beam instrumentation and diagnostics, and particle physics detector design. Also passionate in electronics and programming in Java, Android, MATLAB and Python.
The hosts file is saved in
/system/etc. This means that in order to modify that file, the malware will ask for your permission to use the Magisk SU binary to get root access. If you deny permissions to any app that you don't recognize, you should be safe.You need to have really stable hands and infinite patience while tearing down a smartphone. A salute to you and anyone else who is experienced in this.
There are other benefits as well. For example, Magisk has a module Energized Protection (I have mentioned it in the updated version of the article). This module has a file of blacklisted hosts, and it simply blocks all requests to those hosts. Thus, it blocks ads and malicious websites not only on the web browsers, but also on apps. The root browser app that I have complains that it cannot connect to the Google Ad hosts. Such strict ad blocking is not possible without rooting the phone.
Updated the article.
I'm suggesting that the article should say that prominently. Laymen could be fooled into doing something they don't understand. And if someone approaches you and says "do what that Insights article says" to your phone so that you can use this neat new feature, you should run away.
Since this article focuses on systemless rooting, the greatest benefit (without changing to a custom ROM) is the power of modules. I have written on that briefly, because if I started writing about each and every module that I use currently, it would become another separate article. Anyways, I will add a few points.
No, she shouldn't. Let's be frank about it: rooting is not for a layman. Just like you would refrain from teaching perturbative QCD to a person having insufficient knowledge in physics, I would advise people who have no idea on the Android system to keep away from rooting, because this thing is no joke. For example, I would not root my parents' phones, because if some problem suddenly crops up, I might know how to handle that, but they won't be able to do anything without my support. And I won't be available all the time to tackle these problems.
But that does not always mean that you cannot root if you are a layman. I was a layman too when I started out with these things, but I have learnt and am still learning on these topics. If you are interested in these things, knowledge will come to you automatically, and once you are confident enough, you can proceed to root your phones. You will make mistakes, like @Rive said, and you have to be prepared for that.
For example, I see that an app developer with many phones available may benefit from rooting.
But I also fear that consumers with limited technical skills, motivated by questionable benefits of rooting, and playing with their one and only digital device, may be ill advised to play with rooting. There may also be bad actors advising vulnerable people to root their phones to gain something attractive.
"View your file system using a root file browser" that is not a utility that an ordinary consumer considers a benefit. In the USA, we like to use the Aunt Millie metaphor. Is rooting something that your great aunt Millie could do? should do?
I considered rooting only once about 10 years ago. I wanted to make my phone a WiFi hotspot. I feared doing it because of inexperience, and because I had no local support group of friends who could rescue me if I bricked my phone. Also, that was the only phone my wife and I owned, and we were cruising on remote islands with no access to the cell provider's stores or repair centers. But before I could decide, an update from the provider added wifi hotspot as a standard feature.
The author can edit Insight articles. I suggest that you may want to add some paragraphs to the article, and not just reply in this comment thread.
That's why I always prefer phones which have removable batteries.
Yes, primarily because the backup you make from TWRP cannot be compiled into a .tar file easily. Otherwise by now XDA would have flooded the internet with the stock ROMs of all available devices.