Postmortem Incident Report and Path Forward

As part of our continuing commitment of full transparency, we wanted to provide background and full post mortem report on the recent hacking incident that took place that affected multiple Loopring Smart Wallet users as well as share the steps taken since and the now pivoted roadmap Loopring will take to build toward a stronger future.

The incident that took place recently was actually a mix of two separate but connected incidents that ultimately led to the loss of user funds for some of the affected Loopring Smart Wallets.

TLDR;

• In April there was an attack on the Loopring relayer leading to an internal loss of team assets
• In June there was a follow up attack on Loopring Wallet addresses, which resulted in 58 user addresses losing funds in the process
• While ultimately end users are responsible for their own self-custodial security to ensure safety from central, singular points of failure — the Loopring team is continually open to working with more law enforcement and security teams to help users recover lost assets
• In light of this incident, we have shifted our resources and core focus to the Multi-Network Loopring L3 products (Loopring DeFi)
• We are very optimistic that we have a strong path forward with our roadmap of scaling the Loopring ecosystem to a much wider audience through our Multi-Network Loopring Layer 3 future with the launch of Loopring DeFi across many networks

Incident #1: attack on Loopring assets

When:

The incident happened on April 24,2024.

What was the impact:

Loopring’s own assets were lost due to this attack.

Summary of the attack:

The hacker compromised some Loopring L1 operator accounts and Loopring L2 operator accounts. For the L1 operator accounts, the hacker managed to transfer assets out of the impacted accounts directly. For the L2 operator accounts, since the hacker only had access to the EDDSA key of the accounts, they were unable to transfer assets directly. Instead, the hacker used their own account to place orders at inflated prices within the order book, subsequently manipulating Loopring operator accounts to execute trades. This method allowed the hacker to siphon assets from the compromised operator accounts.

Our Analysis:

The compromise affected Loopring L1 and L2 Operators in distinctly different ways. For L1 Operators, the hacker directly transferred assets. However, for L2 Operators, the hacker was forced to manipulate accounts to sell assets at significantly low prices to other accounts, rather than transferring them directly. These behaviors led the team to believe that the hacker had access to the ECDSA key for L1 Operators but only gained access to the EDDSA key for L2 Operators. This conclusion aligns with the fact that the secretManager holds the ECDSA keys for L1 Operators and only the EDDSA keys for L2 Operators. Since the secretManager module is the only one with access to these keys, further investigation of the logs revealed unauthorized access to the secretManager from an unknown IP address, reinforcing this conclusion.

Upon further investigation, the team discovered the existence of another key with administrative privileges. Possessing this key allowed full access to the secretManager. At the time, this key was not strictly managed, and several internal members had access to it. This led the team to believe that the breach may have been caused by key exposure and potentially initiated internally, instead of due to the hacker having already exploited our backend environment, and may have led to further damages later on.

After resetting the environment, replacing keys, and implementing stricter access controls, the team mistakenly assumed this type of breach could not happen again, and did not conduct a thorough review of the backend environment. Two months later, when the Loopring 2FA database was compromised, after a thorough check, we found out at that our servers had already been exploited and some malware had been planted.

Incident #2: attack on user assets

When:

The incident happened on June 9,2024.

What was the impact:

Out of 40,000+ Loopring Smart Wallets currently deployed, 58 Loopring Smart Wallet addresses lost assets due to this attack.

Technical Background:

Loopring offers Loopring Smart Wallet, a smart contract-based wallet deployed on the Ethereum network. Each smart wallet is owned by an account that has full control over its transactions. However, the security of these wallets depends on Guardians, which are trusted accounts used to authorize security-sensitive actions, such as resetting ownership through a Recovery Operations.

If a user loses access to their device, they can initiate a Recovery operation to reset the smart wallet’s ownership. In cases where the Loopring Official Guardian (an account controlled by the Loopring team) was the only active Guardian, these accounts became vulnerable.

The Exploitation:

On June 9th, the Loopring Two-Factor Authentication (2FA) server was compromised by a malicious attacker. The attacker gained full read/write access to the server and temporarily replaced affected users’ 2FA data with their own email addresses. This allowed the attacker to initiate a social recovery procedure on behalf of 58 affected users.

Due to the manipulation of 2FA information, when the Loopring relayer responded to the social recovery request, it sent the verification code to the attacker’s email. Once the attacker entered the correct verification code, they were verified as the owner of the wallet. Since the Loopring smart wallet operates purely through smart contract logic, after Loopring signed “YES” to the social recovery request, the wallet was recovered under the attacker’s control. The attacker then gained full access to the affected wallets and drained their assets.

Most impacted wallets relied solely on the default Loopring Official Guardian, which only uses users’ 2FA information for verification. Four wallets with more than one guardian were also compromised because the second guardian (another Loopring Smart Wallet) had already been compromised, also relying on the default Loopring Official Guardian only.

Below are the related transactions regarding moving the asset out of users’ accounts:

Post-Attack Asset Movement:

Once the attacker gained control of the compromised wallets, they transferred the assets to their own addresses. Transactions involving the stolen assets have been traced, and we are actively collaborating with exchanges like Binance to track the movement of these funds as they reach traceable destinations.

The hacker has consolidated all the exploited assets into the following address: 0x44f887cfbd667cb2042dd55ab1d8951c94bb0102. They have also converted non-ETH assets (such as LRC) into ETH. Please note that some of the stolen assets were converted to ETH directly on Loopring Layer 2 via compromised accounts.

Since the assets were moved to this address, they have remained static. We are continuing to monitor this wallet closely for any future transactions or movements.

Our Analysis:

This incident highlights critical vulnerabilities in relying on a single Guardian for wallet security or relying on any vulnerable Guardian. Since the Loopring Official Guardian was compromised, all wallets solely dependent on it were at risk. Moving forward, it is essential for users to implement multiple Guardians to enhance the security of their smart wallets.

It’s important to highlight that the Loopring wallet is only considered secure when users opt for multiple trusted guardians (minimum 3, the more guardians the more secure it is) and should not rely solely on the default Loopring Official Guardian.

Post-Mortem Action:

Upon detecting the abnormal activity, the Loopring team paused the social recovery and Layer 2 services for the affected wallets. An internal investigation revealed that the incident stemmed from a compromise of Loopring’s 2FA service.

Loopring immediately contacted SlowMist, a professional blockchain security group. Initially, the team suspected an internal breach due to two key factors:

1. Previous Incident in April: In an earlier attack, the attacker compromised Loopring’s DeFi operator accounts, manipulating trades to steal assets. This incident required both deep knowledge of Loopring’s Layer 2 and access to operator account keys, raising the suspicion of an internal breach. Loopring responded by resetting operator accounts and tightening access controls but did not involve an external security firm for a comprehensive review.
2. 2FA Data Handling: The 2FA data was not stored in plain text and required serialization. Only someone with detailed knowledge of the 2FA database logic could have replaced users’ 2FA information with the attacker’s email address and then restored it afterward.

Collaboration with SlowMist revealed that Loopring’s AWS servers were infected with malware. The first traceable activity related to this malware was found on April 19th, before the initial Loopring operator compromise. It is likely that the same attacker was involved in both incidents. The first incident provided more traceable information, including asset withdrawal and deposit transactions with centralized exchanges like Binance, which could help identify the attacker through KYC information.

After the Smart Wallet Compromise incident, Loopring reported the matter to the Singapore police and hired a lawyer to manage the legal process. However, the Singapore police did not accept the report because:

1. There were no Singaporean victims
2. The compromised server was located in the U.S. (AWS)

Without Singaporean law enforcement’s support, Loopring was not able to obtain the KYC information from the first incident, which would have been crucial for tracing the attacker behind the Smart Wallet compromise.

Loopring is continually open to working with more law enforcement and professional security teams to track down the perpetrator and help users retrieve their lost funds. We will continue to provide updates if there is any further progress in the investigation.

Who lost assets during this process?

Overall, out of over 40K smart wallets, only 58 wallet addresses lost assets in this attack.

Users who did not utilize the intended functionality and add additional guardians to their wallets (only relying on the centralized 2FA service known as the Loopring Guardian) were the vast majority of affected wallets that had assets drained by the hacker.

There was also an edge case where a small amount of users had additional guardians set up, but one of the guardians was a Loopring Smart Wallet that only had the default Loopring Official Guardian. Unfortunately, since this guardian was compromised during the incident, the combination of a compromised 2FA server and a compromised guardian left this user vulnerable. In other words, the root of the guardian trust setup was not satisfied for the affected users. Out of 58 compromised accounts, 3 were compromised due to this.

Is there potential to recoup lost assets?

As is the reality with self-custodial crypto, hackers continue to target users without proper security. Without taking extra steps to guard against centralized or single points of failure, user asset loss can occur. 

Although there is nothing Loopring can do to help offset these users losses, the team is, as mentioned above, continually open to working with more law enforcement and professional security teams to track down the perpetrator to try to help recover their assets.

If there is any success any recouping assets for users, these assets will be sent back to the users that lost funds during the attack.

How does this impact my wallet?

A vast majority of Loopring Smart Wallet users were unaffected by this incident.

Moving forward, it is very import to ensure that your guardians are secure. For example, if your other guardian is a Loopring Smart Wallet, make sure it has more than just the default guardian setup. If the other guardian is your EOA wallet, take care not to lose the private key.

What is Loopring doing to prevent this from happening again?

Improving our security has always been a top priority.

After the incident, Loopring prioritized upgrading security protocols based on the learnings. Working with SlowMist, AWS, and Alibaba Cloud, the team conducted a thorough security audit of all test and production environments, identifying and addressing potential vulnerabilities. The team also made significant efforts to upgrade the software environment to mitigate these risks.

Additionally, Loopring has introduced human oversight for all social recovery operations. All requests will now be reviewed by authorized personnel to ensure there are no issues before approval.

Are other Loopring products safe to use?

As mentioned above, a vast majority of Loopring Smart Wallet users were unaffected. Proper guardian/key management can protect users from similar situations, which have become far too common in the crypto industry.

Other, completely separate Loopring products, like the Loopring Protocol (that runs Loopring L2 + L3), which is completely trustless and non-custodial, remain secure and unaffected. The Loopring Protocol has been battle tested since 2017 and is the basis of our Layer 3, multi-network expansion.

Users deposited to Loopring L2 or Loopring L3 remain unaffected by this. These will continue to operate as normal now and into the future.

On the protocol side, we are also working with the L2beat team to constantly have the best security practices in place as well as increase our Stages.

Decentralizing the Loopring Smart Wallet

This incident has prompted Loopring to reconsider the interplay between Web2 and Web3 elements in its ecosystem. While the Loopring Protocol and Smart Wallet are deployed on a blockchain — a trustless environment — some critical services, such as the Loopring Official Guardian, still rely on traditional centralized systems. For example, social recovery, a key feature of smart wallets, can rely on these centralized services, especially when only the default Loopring Official Guardian is used.

Despite having explicit risk disclosures and warnings in place for users who set up only one guardian, it is evident that this approach is not sufficient. To implement a more secure solution, Loopring must focus more on decentralization and trustlessness.

For the wallet, we are now building out trustless solutions for current wallet users to always be able to interact with the blockchain as well as always have access to their assets in the wallet regardless of any centralized Loopring services.

Meanwhile, we have also concentrated our core focus to Loopring Layer 3 and the Loopring DeFi DApp suite of products, which offer a promising path to better scale Loopring’s (and the LRC token’s) future across many networks.

The Smart Wallet, and more importantly the mobile app, will still be available to users but the mobile focus moving forward will be to build out a better mobile-optimized way to experience the DeFi suite of products in the new DeFi DApp on many networks across Ethereum.

To be clear, the Loopring Wallet will continue to be available for current users, as we plan to maintain the wallet in our mobile roadmap for the Loopring app, but the development efforts on the mobile side will be shifted to making the Loopring DeFi DApp optimized for mobile users.

Suggested Actions for Affected Users

We are committed to transparency, which is why we are sharing this post-mortem and providing detailed information to assist you in filing a case with your local law enforcement.

Loopring’s Path Forward

As previously mentioned, the focus has now shifted to Loopring Layer 3, which is our multi-network DeFi DApp rollout across as many networks as possible.

This new focus will better scale Loopring’s future by introducing unique, new Loopring products to millions of new users across other popular networks, where users already have funds to deploy into DApps like Loopring DeFi (no bridging required!).

Much like how Uniswap or Aave are DApps built on top of Ethereum and can be deployed across any network on Ethereum, now, so can Loopring — through the new Loopring DeFi DApp.

The main difference between DApps like Uniswap or Aave and Loopring DeFi is that, instead of relying solely on smart contracts, Loopring DeFi operates through the Loopring protocol via our DeFi Port functionality. This approach allows us to add new features without modifying the smart contracts, as the core smart contract for the trade-optimized, app-specific ZK-Rollup has already been fully tested.

In practice, this means that to deploy Loopring DeFi on top of other networks, we simply just have to deploy the same well-proven ZK-Rollup protocol (as a Layer 3 — even though this is hidden in the user experience) on other networks.

A big benefit of this approach is that by deploying our protocol as a L3 infrastructure on other networks (like Arbitrum, Base, Taiko), we can create a highly optimized app-chain experience for the end-user. We can add functionality and products easily, improve the user-experience and work to optimize + lower fees for users, more-so than another DApp that is based on smart contracts, deployed directly on the L2.

We believe this new, focused roadmap is also better aligned to help grow and improve the tokenomics of the LRC token.

Because we are deploying our ZK-Rollup protocol everywhere we go, this allows us to collect protocol fees everywhere we go.

As the products inside the new DeFi DApp get adopted and help users earn more APYs, profits and execute better trades, some of this profit can then be passed along and shared with LRC stakers, on top of the protocol fees being generated as well.

All-in-all, we are very encouraged by the potential for these unique products across popular networks. We believe they have the potential to garner up some major adoption and usage, which in turn can benefit the whole ecosystem in a big way.

Source : Loopring Protocol - Oct 11, 2024