At Kleros, we take smart contract security very seriously. Back in the day, there weren’t many automated tools available to help you catch vulnerabilities in your smart contracts. We developed an extensive internal process, and we can proudly say Kleros contracts have never had a vulnerability in production.

Recently, the tooling ecosystem around smart contract security has developed enough for us to take an interest in bringing some of the tools into our processes. We see these tools as a first pass linter to ease the burden on our human reviewers so that they can focus on the real hard to catch vulnerabilities instead of going back and forth on lesser issues.

Kleros team member, Danil Dmitrikov, conducted an analysis in March 2020 of the most popular available tools and tested them on our recruitment test contracts. We think this is a valuable resource to share with the community. In total, there were 11 vulnerabilities to find. Here is the result of his analysis, followed by a section on visualization-only tools and test coverage. The tools are in no particular order, and you can skip to the end if you just want to read our conclusions and recommendations.

Static/Dynamic Analysis Tools

Slither

It’s quick and easy to use. It has about 40 security checks it applies to contracts and shows if any of them are positive. It sometimes displays false positives, although not many, such as possible reentrancy with transfers or sends, etc. Out of all the tests, it only correctly solved the 9th one, which has a low-level call and reentrancy.

Crytic.io

It works on Slither’s engine but has a larger number of checks, so it’s like a better and more elaborate version of Slither. Overall it has similar issues, such as marking the transfer function as not reentrancy safe. For some reason, despite having a lot more checks, it doesn’t have a simple underflow/overflow check (the ones it has only apply to ERC-standard functions). It correctly solved the 9th and 11th test. Also, just like Slither, despite having occasional false positives, it gives some useful suggestions such as “change public to external,” “the contract stores ETH but doesn’t have a withdrawal option,” etc. However, still, it didn’t provide correct answers to most of the tests.

Manticore

A relatively slow tool. It takes 5-10 minutes to test a simple contract, and in the case of a more complex contract, it just froze and stopped responding. While testing, it creates a folder with many analysis files and one global summary file. It correctly solved the 5th (overflow) and 9th tests. It doesn’t give many false positives. The global summary report is usually empty.

Mythril

Also relatively slow but more useful than Manticore. It correctly solved around half of the tests except the 4th, 7th, 8th, 10th, and 11th. Froze and stopped responding when analyzing more complex contracts. It gives occasional false positives highlighting a line that doesn’t have any issues. Overall it’s a less advanced version of MythX, and their GitHub page suggests using the latter.

MythX

The best tool among all checked. It requires you to register on the MythX platform, where you generate an API key. You can use the key either with VSCode, Truffle, or Remix plugins. It also works directly with the MythX CLI. I tried all four options. The analysis is relatively quick, and results show on the mythx.io dashboard.

There is also an option to perform in-depth analysis, but it’s only available if you upgrade your subscription. According to the description, an in-depth analysis can take from 30 to 90 minutes to finish. Overall it correctly solved all tests except the 7th, 8th, and 10th. When analyzing our more complex contracts, it pointed out a potential out-of-gas problem because of looping, which seems reasonable.

Contract Library

It can be potentially useful. It has some vulnerability checks, which can be used as a filter to find examples of the particular vulnerability among deployed contracts. It applies to contracts on test networks as well. You can also select an already deployed contract to see if any of the checks apply to it. The downside is that the number of checks it uses is relatively small.

Securify

An online analysis tool. You upload the contract and get the analysis results. It’s swift, but the number of checks seems very limited. For each analyzed contract, it gave almost the same type of report, such as “the contract locks ether” or “fields that can be modified by the user must be inspected,” etc. Sometimes gives false positives. It correctly solved only the 9th test (reentrancy). There was also an option to request an audit for the contract, but I didn’t use it.

SmartCheck

An improved version of Securify. It has a larger number of checks (but for some reason doesn’t have an overflow check). The checks are more descriptive, too, and also provide some examples and recommendations. This tool correctly solved the 4th, 9th, and 11th tests and pointed out a potential out-of-gas problem in the more complex contracts. It also has an option to request an audit for the contract.

Echidna

I didn’t manage to make it work, but it works similar to Truffle testing according to the GitHub page. You have to write a solidity file with checks you want the target contract to undergo, and Echidna will create various inputs trying to break these checks. Overall it doesn’t seem very practical to use.

Oyente

I briefly checked this tool, and it only works for compilers below 0.4.17. The GitHub repo also seems abandoned. Considering how outdated it was, I decided to leave it.

Octopus

This tool provides some low-level analysis, and I don’t think it can check smart contracts for vulnerabilities. At least I didn’t find a way to do that.

sFuzz

sFuzz is also an online tool. You paste the contract to analyze and get the report. Each time I tried to use it, the analysis stopped at 95%. The one time it didn’t stop was when I uploaded an almost empty contract. I tried it on different browsers and had the same result.

Visualization Tools

Solidity Visual Auditor

Solidity Visual Auditor is an extension for Visual Studio Code. I checked this tool a year ago, and it hasn’t changed much since then. It highlights certain variables, shows the line they are at, etc., which I don’t find particularly useful.

Surya

Surya is a tool that’s integrated into Solidity Visual Auditor. It’s straightforward to use. After installing the extension, you get some tools you can use with the contract, such as parsing, drawing a graph, showing function signatures, etc. All of them are quick and easy to use. It has a flattener, which can combine the contract with many dependencies into one and can be useful upon deploying. However, the flattening only works if the dependencies are local, and errors if imports reference an NPM package, like “import "@kleros/kleros-interaction/contracts/libraries/CappedMath.sol". You have to reference the contract in your local node_modules folder for it to work.

Solgraph

Too simplistic to be useful. All it does is just draw graphs. The description says that it also highlights potential vulnerabilities, but it doesn’t provide any security analysis. If it finds a vulnerability in a function, it only marks it as “UNTRUSTED” without giving any reason, and that’s it, which is questionable. Their GitHub repo seems outdated as well.

Piet

Web application with functionality similar to Surya. Also draws graphs, extracts documentation from the contract, parses, etc. It can be used to interact with deployed contracts as well. It’s pretty easy to use, and overall it looks well-made, but the interface is a little unintuitive, and the tool’s description says that it’s still in demo mode and can contain bugs. I don’t see many use cases for it.

EVM Lab

EVM Lab provides some interactions with the EVM, but I didn’t manage to make it work. There is a screenshot of the tool on its GitHub page, which shows a complex analysis, but there is not much info on using or installing it. Their wiki page doesn’t say much.

Ethereum Graph Debugger

I also didn’t manage to run it. Simple NPM installation results in an error that says that the problem is probably with the package itself. I tried it on Windows and Linux with the same result.

It’s just a tool strictly for debugging, and even its GitHub page suggests alternatives such as Remix, etc.

Test Coverage

Solidity Coverage

A pretty useful tool to check the percentage of code lines covered in your test files. The coverage rate and traces it gives seem pretty accurate.

Conclusion & Recommendations

Out of all the static and dynamic analysis tools we tested, MythX performed the best by far. We plan to do further testing to see if the paid version is worth the price.

The results make sense because MythX uses all the approaches of the other tools combined. It conducts static analysis, symbolic execution, and fuzzy testing.

So yeah, we’ll be trying MythX in our security process going forward. We also recommend developing with Solidity Visual Auditor and Surya when dealing with complex contract interactions and ensuring your tests cover your contracts’ entirety using Solidity Coverage.

We know first hand how time-consuming it can be to sort through every new tool to decide what to adopt. When it comes to smart contract security, the process is even more time-consuming because the types of instruments involved take time to set up, and your choices can have very tangible financial consequences. We hope this lightens your workload a little bit the next time you go shopping for smart contract security tools.

Where Can I Find Out More?

Join the community chat on Telegram.

Visit our website.

Follow us on Twitter.

Join our Slack for developer conversations.

Contribute on Github.

Download our Book