Getting Started

Welcome! This guide will help you get OpenTitan up and running.

Workflow Options

An important preliminary note: to run OpenTitan software, you will need to not only build the software but somehow simulate the hardware it runs on. As shown in the diagram below, we currently support multiple build targets and workflows, including: Verilator, FPGA, and DV (commercial RTL simulators, such as VCS and Xcelium). However, if you are new to the project, we recommend simulation with Verilator. This uses only free tools, and does not require any additional hardware such as an FPGA.

This guide will focus on the Verilator workflow, but indicate when those following FPGA or DV workflows should do something different. Just keep in mind, if you’re a new user and you don’t know you’re part of the FPGA or DV crowd, “Verilator” means you!

Step 0: Clone the OpenTitan Repository

Clone the OpenTitan repository:

git clone https://github.com/lowRISC/opentitan.git

If you wish to contribute to OpenTitan you will need to make a fork on GitHub and may wish to clone the fork instead. We have some notes for using GitHub which explain how to work with your own fork (and perform many other GitHub tasks) in the OpenTitan context.

Note: throughout the documentation $REPO_TOP refers to the path where the OpenTitan repository is checked out. Unless you’ve specified some other name in the clone, $REPO_TOP will be a directory called opentitan. You can create the environment variable by calling the following command from the same directory where you ran git clone:

export REPO_TOP=$PWD/opentitan

Step 1: Check System Requirements

OpenTitan installation requires Linux. If you do not have Linux, please stop right here and use the (experimental) Docker container. You can then skip to step 4 (building software).

If you do have Linux, you are still welcome to try the Docker container. However, as the container option is currently experimental, we recommend following the steps below to build manually if you plan on being a long-term user or contributor for the project.

Our continuous integration setup runs on Ubuntu 20.04 LTS, which gives us the most confidence that this distribution works out of the box. We do our best to support other distributions, but cannot guarantee they can be used “out of the box” and might require updates of packages. Please file a GitHub issue if you need help or would like to propose a change to increase compatibility with other distributions.

You will need at least 7GiB of available RAM in order to build the Verilator simulation. If you have an FPGA and download the bitstream from our cloud bucket rather than building it locally (the default setup) then this constraint does not apply.

If you are specifying a new machine to run top-level simulations of the whole of OpenTitan using Verilator, it is recommended that you have a minimum of 32GiB of physical RAM and at least 512GiB of SSD/HDD storage for the build tools, repository and Ubuntu installation.

Step 2: Install Package Manager Dependencies

Skip this step if using the Docker container.

A number of software packages from the distribution’s package manager are required. On Ubuntu 20.04, the required packages can be installed with the following command.

sed '/^#/d' ./apt-requirements.txt | xargs sudo apt install -y

Some tools in this repository are written in Python 3 and require Python dependencies to be installed through pip. We recommend installing the latest version of pip and setuptools (especially if on older systems such as Ubuntu 18.04) using:

python3 -m pip install --user -U pip "setuptools<66.0.0"

The pip installation instructions use the --user flag to install without root permissions. Binaries are installed to ~/.local/bin; check that this directory is listed in your PATH by running which pip3. It should show ~/.local/bin/pip3. If it doesn’t, add ~/.local/bin to your PATH, e.g. by adding the following line to your ~/.bashrc file:

export PATH=$PATH:~/.local/bin

Now install additional Python dependencies:

cd $REPO_TOP
pip3 install --user -r python-requirements.txt

Adjust GCC version (if needed)

On Ubuntu 18.04 the package build-essential includes the compilers gcc-7 and g++-7. But for the OpenTitan project gcc-9/g++-9 or higher is required, which has to be installed manually. Check that you have version 9 or higher of gcc installed by:

gcc --version

If your version is lower, you have to upgrade it manually. For this, first, add ubuntu-toolchain-r/test PPA to your system using the following commands:

sudo apt install software-properties-common
sudo add-apt-repository ppa:ubuntu-toolchain-r/test

Next, install the necessary GCC and G++ version by:

sudo apt update
sudo apt install gcc-9 g++-9

Finally, update the symbolic links for gcc and g++ using these commands:

sudo ln -sf /usr/bin/gcc-9 /usr/bin/gcc
sudo ln -sf /usr/bin/g++-9 /usr/bin/g++

Step 3: Install the LowRISC RISC-V Toolchain

Skip this step if using the Docker container.

To build device software you need a baremetal RISC-V toolchain (including, for example, a C compiler). Even if you already have one installed, we recommend using the prebuilt toolchain provided by lowRISC, because it is built with the specific patches and options that OpenTitan needs. You can install the toolchain using the util/get-toolchain.py script, which will download and install the toolchain to the default path, /tools/riscv.

cd $REPO_TOP
./util/get-toolchain.py

If you did not encounter errors running the script, you’re done and can go to step 4. If you did, read on.

Troubleshooting

If you need to install to a different path than /tools/riscv (for instance, if you do not have permission to write to the /tools directory), then you can specify a different location using the --install-dir option. Run ./util/get-toolchain.py --help for details. You can alternatively download the tarball starting with lowrisc-toolchain-rv32imcb- from GitHub releases and unpack it to the desired installation directory.

Assuming one of the above worked and you have installed to a non-standard location, you will need to set the TOOLCHAIN_PATH environment variable to match whatever path you used. For example, if I wanted to install to ~/ot_tools/riscv, then I would use:

./util/get-toolchain.py --install-dir ~/ot_tools/riscv
export TOOLCHAIN_PATH=~/ot_tools/riscv

Add the export command to your ~/.bashrc or equivalent to ensure that the TOOLCHAIN_PATH variable is set for future sessions. Check that it worked by opening a new terminal and running:

ls $TOOLCHAIN_PATH/bin/riscv32-unknown-elf-as

If that prints out the file path without errors, then you’ve successfully installed the toolchain. Otherwise, try to find the riscv32-unknown-elf-as file in your file system and make sure $TOOLCHAIN_PATH is correctly set.

Step 4: Set up your Simulation Tool or FPGA

Note: If you are using the pre-built Docker container, Verilator is already installed. Unless you know you need the FPGA or DV guides, you can skip this step.

In order to run the software, we need to have some way to emulate an OpenTitan chip. There are a few different options depending on your equipment and use-case. Follow the guide(s) that applies to you:

• Option 1 (Verilator setup, recommended for new users): Verilator guide, or
• Option 2 (FPGA setup): FPGA guide, or
• Option 3 (design verification setup): DV guide

Step 5: Build OpenTitan Software

Follow the dedicated guide to build OpenTitan’s software and run tests.

Step 6: Optional Additional Steps

If you have made it this far, congratulations! Hopefully you got a “Hello World!” demo running on OpenTitan using either the Verilator or FPGA targets.

Depending on the specific way you want to use or contribute to OpenTitan, there may be a few extra steps you want to do. In particular:

• If you want to contribute SystemVerilog code upstream to OpenTitan, follow step 6a to install Verible.
• If you want to debug on-chip OpenTitan software with GDB, follow step 6b to install OpenOCD.
• If you want to run supported formal verification flows for OpenTitan, using tools like JasperGold, follow step 6c to set up formal verification.
• If you want to simulate OpenTitan using Siemens Questa, follow step 6d to set it up.

It also may make sense to stick with the basic setup and come back to these steps if you find you need them later.

Step 6a: Install Verible (optional)

Verible is an open source SystemVerilog style linter and formatting tool. The style linter is relatively mature and we use it as part of our RTL design flow. The formatter is still under active development, and hence its usage is more experimental in OpenTitan.

You can download and build Verible from scratch as explained on the Verible GitHub page. But since this requires the Bazel build system the recommendation is to download and install a pre-built binary as described below.

Go to this page and download the correct binary archive for your machine.

The example below is for Ubuntu 20.04:

export VERIBLE_VERSION=v0.0-2135-gb534c1fe
wget https://github.com/google/verible/releases/download/${VERIBLE_VERSION}/verible-${VERIBLE_VERSION}-Ubuntu-20.04-focal-x86_64.tar.gz
tar -xf verible-${VERIBLE_VERSION}-Ubuntu-20.04-focal-x86_64.tar.gz

If you are using Ubuntu 18.04 then instead use:

export VERIBLE_VERSION=v0.0-2135-gb534c1fe
wget https://github.com/google/verible/releases/download/${VERIBLE_VERSION}/verible-${VERIBLE_VERSION}-Ubuntu-18.04-bionic-x86_64.tar.gz
tar -xf verible-${VERIBLE_VERSION}-Ubuntu-18.04-bionic-x86_64.tar.gz

Then install Verible within ‘tools’ using:

sudo mkdir -p /tools/verible/${VERIBLE_VERSION}/
sudo mv verible-${VERIBLE_VERSION}/* /tools/verible/${VERIBLE_VERSION}/

After installation you need to add /tools/verible/$VERIBLE_VERSION/bin to your PATH environment variable.

Note that we currently use version v0.0-2135-gb534c1fe, but it is expected that this version is going to be updated frequently, since the tool is under active development.

Step 6b: Install OpenOCD (optional)

See the OpenOCD install guide.

Step 6c: Set up formal verification (optional)

See the formal verification setup guide

Step 6d: Set up Siemens Questa (optional)

 Once a standard installation of Questa has been completed, add QUESTA_HOME as an environment variable which points to the Questa installation directory.

As of Questa version 21.4 there are some code incompatibilities with the OpenTitan code-base. See issue #9514 for the list of issues and temporary workarounds.

Step 7: Additional Resources

As you may have guessed, there are several other pieces of hardware and software, besides a “Hello World!” demo, that are being actively developed for the OpenTitan project. If you are interested in these, check out the additional resources below.

General

• Directory Structure
• GitHub Notes
• Building Documentation
• Design Methodology within OpenTitan

Hardware

• Designing Hardware
• OpenTitan Hardware

Software

• OpenTitan Software
• Writing and Building Software for OTBN
• Rust for Embedded C Programmers

Design Verification Setup

Before following this guide, make sure you’ve followed the dependency installation and software build instructions.

This document aims to enable a contributor to get started with a design verification (DV) effort within the OpenTitan project. While most of the focus is on development of a testbench from scratch, it should also be useful to understand how to contribute to an existing effort. Please refer to the DV methodology document for information on how design verification is done in OpenTitan.

Stages of DV

The life stages of a design / DV effort within the OpenTitan are described in the Hardware Development Stages document. It separates the life of DV into three broad stages: Initial Work, Under Test and Testing Complete. This document attempts to give guidance on how to get going with the first stage and have a smooth transition into the Under Test stage. They are not hard and fast rules but methods we have seen work well in the project. DV indeed cannot begin until the design has transitioned from Specification to the Development stage first. The design specification, once available, is used as a starting point.

Getting Started

The very first thing to do in any DV effort is to document the plan detailing the overall effort. This is done in conjunction with developing the initial testbench. It is recommended to use the uvmdvgen tool, which serves both needs.

The uvmdvgen tool provides the ability to generate the outputs in a specific directory. This should be set to the root of the DUT directory where the rtl directory exists. When the tool is run, it creates a dv directory, along with data and doc directories. The dv directory is where the complete testbench along with the collaterals to build and run tests can be found. It puts the documentation sources in doc and data directories respectively (which also exist alongside the rtl directory). It is recommended to grep for ‘TODO’ at this stage in all of these generated files to make some of the required fixes right way. One of these for example, is to create appropriate interfaces for the DUT-specific IOs and have them connected in the testbench (dv/tb/tb.sv).

Documentation and Initial Review

The skeleton DV document and the Hjson testplan should be addressed first. The DV documentation is not expected to be completed in full detail at this point. However, it is expected to list all the verification components needed and depict the planned testbench as a block diagram. Under the ‘design verification’ directory in the OpenTitan team drive, some sample testbench block diagrams are available in the .svg format, which can be used as a template. The Hjson testplan, on the other hand, is required to be completed. Please refer to the testplanner tool documentation for additional details on how to write the Hjson testplan. Once done, these documents are to be reviewed with the designer(s) and other project members for completeness and clarity.

UVM RAL Model

Before running any test, the UVM RAL model needs to exist (if the design contains CSRs). The DV simulation flow has been updated to generate the RAL model automatically at the start of the simulation. As such, nothing extra needs to be done. It can be created manually by invoking regtool:

$ util/regtool.py -s -t /path-to-dv /path-to-module/data/<dut>.hjson

The generated file is placed in the simulation build scratch area instead of being checked in.

Supported Simulators

The use of advanced verification constructs such as SystemVerilog classes (on which UVM is based on) requires commercial simulators. The DV simulation flow fully supports Synopsys VCS. There is support for Cadence Xcelium as well, which is being slowly ramped up.

Building and Running Tests

The uvmdvgen tool provides an empty shell sequence at dv/env/seq_lib/<dut>_sanity_vseq.sv for developing the sanity test. The sanity test can be run as-is by invoking dvsim.py, as a “hello world” step to bring the DUT out of reset.

$ util/dvsim/dvsim.py path/to/<dut>_sim_cfg.hjson -i <dut>_sanity [--waves <format>] [--tool xcelium]

The generated initial testbench is not expected to compile and elaborate successfully right away. There may be additional fixes required, which can hopefully be identified easily. Once the testbench compiles and elaborates without any errors or warnings, the sanity sequence can be developed further to access a major datapath and test the basic functionality of the DUT.

VCS is used as the default simulator. It can be switched to Xcelium by setting --tool xcelium on the command line.

To dump waves from the simulation, pass the --waves <format> argument to dvsim.py. If you are using Verdi for waveform viewing, then ‘–waves fsdb’ is probably the best option. For use with other viewers, ‘–waves shm’ is probably the best choice for Xcelium, and ‘–waves vpd’ with vcs.

Please refer to the DV simulation flow for additional details.

The uvmdvgen script also enables the user to run the full suite of CSR tests, if the DUT does have CSRs in it. The most basic CSR power-on-reset check test can be run by invoking:

$ util/dvsim/dvsim.py path/to/<dut>_sim_cfg.hjson -i <dut>_csr_hw_reset [--waves <format>] [--tool xcelium]

Please refer to CSR utilities for more information on how to add exclusions for the CSR tests.

Full DV

Running the sanity and CSR suite of tests while making progress toward reaching the V1 stage should provide a good reference in terms of how to develop tests as outlined in the testplan and running and debugging them. Please refer to the checklist to understand the key requirements for progressing through the subsequent verification stages and final signoff.

The UART DV area can be used as a canonical example for making progress. If it is not clear on how to proceed, feel free to file an issue requesting assistance.

Reproduce a DV failure that CI reported

Follow these steps to reproduce the failure

1. Make sure the version of VCS is the same as the one running in CI.

2. CI runs against an auto-generated merge commit, which effectively is generated by merging the pull request (PR) into the master branch. This “merge” branch is updated automatically by GitHub whenever the PR branch is pushed, or when the PR is closed and re-open. Retrieve this exact branch by running the following (assuming “upstream” is the name of your lowRISC/opentitan repository).

$ git fetch upstream pull/<PR_number>/merge
$ git checkout -b <temp_branch> FETCH_HEAD

3. This is the command that CI runs for the smoke regression.

$ util/dvsim/dvsim.py hw/top_earlgrey/dv/top_earlgrey_sim_cfgs.hjson -i smoke --fixed-seed=1

Since the CI runs tests with pseudo-random behaviour driven from ‘seed’ numbers, to be confident of reproducing the failure we must supply the exact seed that CI used.

Assume there is a failure in the uart_smoke test. To reproduce this with the DV simulation environment we use the following command, remembering to replace ‘’ with the seed number, and to choose an appropriate waveform ‘’:

$ util/dvsim/dvsim.py hw/ip/uart/dv/uart_sim_cfg.hjson -i uart_smoke --fixed-seed=<seed> [--waves <format>]

It is recommended to use ‘–waves fsdb’ if you are using Verdi, or ‘–waves vpd’ if you are using vcs but a different waveform viewer. With Xcelium (‘–tool xcelium’) but not Verdi, then ‘–waves shm’ is the preferred format.

For maximal portability, the standard ‘vcd’ format is supported by all simulators and waveform viewers, but please be forewarned that vcd files can become extremely large and are generally best avoided.

Formal Verification Setup

Before following this guide, make sure you’ve followed the dependency installation and software build instructions.

This document aims to enable a contributor to get started with a formal verification effort within the OpenTitan project. While most of the focus is on development of a testbench from scratch, it should also be useful to understand how to contribute to an existing effort.

Please refer to the OpenTitan Assertions for information on how formal verification is done in OpenTitan.

Formal property verification (FPV)

The formal property verification is used to prove assertions in the target. There are three sets of FPV jobs in OpenTitan. They are all under the directory hw/top_earlgrey/formal.

• top_earlgrey_fpv_ip_cfgs.hjson: List of IP targets.
• top_earlgrey_fpv_prim_cfgs.hjson: List of prim targets (such as counters, fifos, etc) that are usually imported by an IP.
• top_earlgrey_fpv_sec_cm_cfgs.hjson: List of IPs that contains standard security countermeasure assertions. This FPV environment only proves these security countermeasure assertions. Detailed description of this FPV use case is documented in Running FPV on security blocks for common countermeasure primitives.

To automatically create a FPV testbench, it is recommended to use the fpvgen tool to create a template. To run the FPV tests in dvsim, please add the target to the corresponding top_earlgrey_fpv_{category}_cfgs.hjson file , then run with command:

util/dvsim/dvsim.py hw/top_earlgrey/formal/top_earlgrey_fpv_{category}_cfgs.hjson --select-cfgs {target_name}

It is recommended to add the FPV target to lint script hw/top_earlgrey/lint/top_earlgrey_fpv_lint_cfgs.hjson to quickly find typos.

Formal connectivity verification

The connectivity verification is mainly used for exhaustively verifying system-level connections. User can specify the connection ports via a CSV format file in hw/top_earlgrey/formal/conn_csvs folder. User can trigger top_earlgrey’s connectivity test using dvsim:

util/dvsim/dvsim.py hw/top_earlgrey/formal/chip_conn_cfgs.hjson

The connectivity testplan is documented under hw/top_earlgrey/data/chip_conn_testplan.hjson.

Building (and Testing) Software

Before following this guide, make sure you have read the:

• main Getting Started instructions,
• install Verilator section of the Verilator guide, and
• OpenTitan Software documentation.

All OpenTitan software is built with Bazel. Additionally, most tests may be run with Bazel too.

TL;DR

To install the correct version of bazel, build, and run a single test with Verilator, run:

$REPO_TOP/bazelisk.sh test --test_output=streamed --disk_cache=~/bazel_cache //sw/device/tests:uart_smoketest_sim_verilator

This will take a while (an hour on a laptop is typical) if it’s your first build or the first after a git pull, because Bazel has to build the chip simulation. Future builds will be much faster; go get a coffee and come back later.

If the test worked, your OpenTitan setup is functional; you can build the software and run on-device tests using the Verilator simulation tool. See [Running Tests with Bazel](#running-tests with Bazel) for information on how to find and run other tests.

If the test didn’t work, read the full guide, especially the Troubleshooting section.

Installing Bazel

There are two ways to install the correct version of Bazel:

1. automatically, using the bazelisk.sh script provided in the repo, or
2. manually.

Automatic Installation

To simplify the installation of Bazel, and provide a means to seamlessly update the Bazel version we use in the future, we provide a shell script that acts as a wrapper for invocations of “bazel ...”. To use it, you have two options:

1. use “./bazelisk.sh ...” instead of “bazel ...” to invoke of Bazel subcommands, or
2. set the following alias (e.g., in your .bashrc file) to accomplish the same:

alias bazel="$REPO_TOP/bazelisk.sh"

Manual Installation

This section is optional and can be skipped if you completed the instructions above in Automatic Installation.

While the automatic installation is convenient, by installing Bazel directly, you can get some quality of life features like tab completion. If you haven’t yet installed Bazel, and would like to, you can add it to apt and install it on Ubuntu systems with the following commands as described in the Bazel documentation:

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" |
sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel-5.1.1
sudo ln -s /usr/bin/bazel-5.1.1 /usr/bin/bazel

or by following instructions for your system.

Building Software with Bazel

Running

bazel build //sw/...

will build all software in our repository. If you do not have Verilator installed yet, you can use the --define DISABLE_VERILATOR_BUILD=true flag to skip the jobs that depend on that.

In general, you can build any software target (and all of it’s dependencies) using the following syntax:

bazel build @<repository>//<package>:<target>

Since most targets are within the main Bazel repository (lowrisc_opentitan), you can often drop the “@<repository>” prefix. For example, to build the boot ROM we use for testing (also referred to as the test ROM), you can use

bazel build //sw/device/lib/testing/test_rom:test_rom

Additionally, some Bazel syntactic sugar enables dropping the target name when the target name matches the last subcomponent of the package name. For example, the following is equivalent to the above

bazel build //sw/device/lib/testing/test_rom

For more information on Bazel repositories, packages, and targets, please refer to the Bazel documentation.

Running Tests with Bazel

In addition to building software, Bazel is also used to build and run tests. There are two categories of OpenTitan tests Bazel can build and run:

1. on-host, and
2. on-device.

On-host tests are compiled and run on the host machine, while on-device tests are compiled and run on (simulated/emulated) OpenTitan hardware.

Examples of on-host tests are:

• unit tests for device software, such as DIF and ROM unit tests.
• any test for host software, such as opentitan{lib,tool}.

Examples of on-device tests are:

• chip-level tests.
• ROM functional tests

Test target names normally match file names (for instance, //sw/device/tests:uart_smoketest corresponds to sw/device/test/uart_smoketest.c). You can see all tests available under a given directory using bazel query, e.g.:

bazel query 'tests(//sw/device/tests/...)'

Tags and wildcards

TLDR: bazelisk.sh test --test_tag_filters=-cw310,-verilator,-vivado,-jtag,-eternal,-broken --build_tag_filters=-vivado,-verilator //... Should be able to run all the tests and build steps in OpenTitan that don’t require optional setup steps, and

You may find it useful to use wildcards to build/test all targets in the OpenTitan repository instead of individual targets. //sw/... is shorthand for all targets in sw that isn’t tagged with manual. If a a target (a test or build artifact) relies on optional parts of the “Getting Started” guide they should be tagged so they can be filtered out and users can bazelisk.sh test //... once they filter out the appropriate tags. We maintain or use the following tags to support this:

• broken is used to tag tests that are committed but should not be expected by CI or others to pass.
• cw310, cw310_test_rom, and cw310_rom are used to tag tests that depend on a correctly setup cw310 “Bergen Board” to emulate OpenTitan. The cw310 tag may be used in --test_tag_filters to enable concise filtering to select tests that run on this board and include or exclude them. Loading the bitstream is the slowest part of the test, so these tags can group tests with common bitstreams to accelerate the tests tagged cw310_test_rom.
• verilator is used to tag tests that depend on a verilated model of OpenTitan that can take a significant time to build. Verilated tests can still be built with --define DISABLE_VERILATOR_BUILD, but they will skip the invocation of Verilator and cannot be run.
• vivado is used to tag tests that critically depend on Vivado.
• jtag is used to tag tests that rely on a USB JTAG adapter connected like we have in CI.
• manual is a Bazel builtin that prevents targets from matching wildcards. Test suites are typically tagged manual so their contents match, but test suites don’t get built or run unless they’re intentionally invoked. Intermediate build artifacts may also be tagged with manual to prevent them from being unintentionally built if they cause other problems.
• skip_in_ci is used to tag ROM end-to-end tests that we currently skip in CI. ROM end-to-end tests are typically written for five life cycle states: TEST_UNLOCKED0, DEV, PROD, PROD_END, and RMA. This tag allows us to limit the tests run in CI to a single life cycle state, allowing CW310 tests to finish in a reasonable timeframe. We run tests for the remaining lifecycle states outside the CI in a less frequent manner.

ci/scripts/check-bazel-tags.sh performs some useful queries to ensure these tags are applied. These tags can then be used to filter tests using --build_tests_only --test_tag_filters=-cw310,-verilator,-vivado. These tags can also be used to filter builds using --build_tag_filters=-cw310,-verilator,-vivado.

--build_tests_only is important when matching wildcards if you aren’t using --build_tag_filters to prevent bazelisk.sh test //... from building targets that are filtered out by --test_tag_filters.

There is no way to filter out dependencies of a test_suite such as //sw/device/tests:uart_smoketest (Which is a suite that’s assembled by the opentitan_functest rule) from a build.

Running on-device Tests

On-device tests such as //sw/device/tests:uart_smoketest include multiple targets for different device simulation/emulation tools. Typically, you will only want to run one of these test targets at a time (for instance, only Verilator or only FPGA). Add _sim_verilator to the test name to run the test on Verilator only, and _fpga_cw310_rom or _fpga_cw310_test_rom to run the test on FPGA only.

You can check which Verilator tests are available under a given directory using:

bazel query 'attr(tags, verilator, tests(//sw/device/tests/...))'

For FPGA tests, just change the tag:

bazel query 'attr(tags, cw310, tests(//sw/device/tests/...))'

For more information, please refer to the Verilator and/or FPGA setup instructions.

Running on-host DIF Tests

The Device Interface Function or DIF libraries contain unit tests that run on the host and are built and run with Bazel. As shown below, you may use Bazel to query which tests are available, build and run all tests, or build and run only one test.

Querying which tests are available

bazel query 'tests(//sw/device/lib/dif:all)'

Building and running all tests

bazel test //sw/device/lib/dif:all

Building and running a single test

For example, building and testing the UART DIF library’s unit tests:

bazel test //sw/device/lib/dif:uart_unittest

Running on-host ROM Tests

Similar to the DIF libraries, you can query, build, and run all the ROM unit tests (which also run on the host) with Bazel.

Querying which (on-host) tests are available

Note, the ROM has both on-host and on-device tests. This query filters tests by their kind, i.e., only on-host tests.

bazel query 'kind(cc_.*, tests(//sw/device/silicon_creator/lib/...))'

Building and running all (on-host) tests

bazel test --test_tag_filters=-cw310,-dv,-verilator //sw/device/silicon_creator/lib/...

Building and running a single (on-host) test

For example, building and testing the ROM UART driver unit tests:

bazel test //sw/device/silicon_creator/lib/drivers:uart_unittest

Miscellaneous

Bazel-built Software Artifacts

As described in the OpenTitan Software documentation, there are three categories of OpenTitan software, all of which are built with Bazel. These include:

1. device software,
2. OTBN software,
3. host software,

Bazel produces various artifacts depending on the category of software that is built.

Device Artifacts

Device software is built and run on OpenTitan hardware. There are three OpenTitan “devices” for simulating/emulating OpenTitan hardware:

1. “sim_dv”: DV simulation (i.e., RTL simulation with commercial simulators),
2. “sim_verilator”: Verilator simulation (i.e., RTL simulation with the open source Verilator simulator),
3. “fpga_cw310”: FPGA.

Additionally, for each device, there are two types of software images that can be built, depending on the memory type the software is destined for, i.e.:

1. ROM,
2. flash,

To facilitate instantiating all build rules required to build the same artifacts across several devices and memories, we implement two OpenTitan-specific Bazel macros. These macros include:

• opentitan_rom_binary
• opentitan_flash_binary

Both macros instantiate build rules to produce software artifacts for each OpenTitan device above. Specifically, building either an opentitan_rom_binary or opentitan_flash_binary named <target>, destined to run on the OpenTitan device <device>, will output the following files under bazel-out/:

• <target>_<device>.elf: the linked program, in ELF format.
• <target>_<device>.bin: the linked program, as a plain binary with ELF debug information removed.
• <target>_<device>.dis: the disassembled program with inline source code.
• <target>_<device>.logs.txt: a textual database of addresses where LOG_* macros are invoked (for DV backdoor logging interface).
• <target>_<device>.rodata.txt: same as above, but contains the strings that are logged.
• <target>_<device>.*.vmem: a Verilog memory file which can be read by $readmemh() in Verilog code. Note, <device> will be in {sim_dv, sim_verilator, fpga_cw310}.

Additionally, if the opentitan_flash_binary is signed, then these files will also be under bazel-out/:

• <target>_<device>.<signing key name>.signed.bin: the same .bin file above, but with a valid signature field in the manifest.
• <target>_<device>.<signing key name>.signed.*.vmem: the same *.vmem file above, but with a valid signature field in the manifest.

OTBN Artifacts

OTBN programs use a specialized build flow (defined in rules/otbn.bzl). OTBN programs produce the following artifacts:

• <target>.o: unlinked object file usually representing a single assembly file
• <target>.elf: standalone executable binary representing one or more assembly/object files
• <target>.rv32embed.{a,o}: artifacts representing an OTBN binary, set up to be linked into a RISC-V program

In terms of Bazel rules:

• the otbn_library rule runs the assembler to create <target>.o artifacts, and
• the otbn_binary and otbn_sim_test rules run the linker on one or more .o files to create the .elf and .rv32embed.{a,o} artifacts.

Since OTBN has limited instruction memory, the best practice is to list each file individually as an otbn_library. This way, binary targets can easily include only the files they need.

OTBN programs run on the OTBN coprocessor, unlike standard “on-device” programs that run on the main processor (Ibex). There are two ways to run an OTBN program:

1. Run a standalone binary (.elf) on the specialized OTBN simulator.
2. Include a .rv32embed artifact in a C program that runs on Ibex, and create an on-device target as described in the previous section.

You can run .elf artifacts directly using the simulator as described in the OTBN README. The otbn_sim_test rule is a thin wrapper around otbn_binary. If you use it, bazel test will run the OTBN simulator for you and check the test result.

To include an OTBN program in a C program, you need to add the desired OTBN otbn_binary Bazel target to the deps list of the C program’s Bazel target. No #include is necessary, but you will likely need to initialize the symbols from the OTBN program as required by the OTBN driver you are using.

Host Artifacts

Host software is built and run on the host hardware (e.g., an x64 Linux machine). A final linked program in the ELF format is all that is produced for host software builds. Note, the file will not have an extension.

Disassembling Device Code

A disassembly of all executable sections is produced by the build system by default. It can be found by looking for files with the .dis extension next to the corresponding ELF file.

./bazelisk.sh build //sw/device/tests:uart_smoketest_prog_sim_verilator_dis

less "$(./bazelisk.sh outquery //sw/device/tests:uart_smoketest_prog_sim_verilator_dis)"

To get a different type of disassembly, e.g. one which includes data sections in addition to executable sections, objdump can be called manually. For example the following command shows how to disassemble all sections of the UART DIF smoke test interleaved with the actual source code:

./bazelisk.sh build --config=riscv32 //sw/device/tests:uart_smoketest_prog_sim_verilator.elf

riscv32-unknown-elf-objdump --disassemble-all --headers --line-numbers --source \
  "$(./bazelisk.sh outquery --config=riscv32 //sw/device/tests:uart_smoketest_prog_sim_verilator.elf)"

Refer to the output of riscv32-unknown-elf-objdump --help for a full list of options.

Troubleshooting {#troubleshooting}

Check CI {#troubleshooting-check-ci}

First, check the GitHub repository to make sure the CI check is succeeding for the commit you cloned. If there’s an issue with that commit (it would have a red “X” next to it), check out the most recent commit that passed CI (indicated by a green check mark). We try to always keep the main branch healthy, but the project is in active development and we’re not immune to temporary breaks.

Debugging a failed verilator test {#troubleshooting-verilator-test-failure}

If your bazelisk.sh build failed trying to run a test on Verilator, the first step is to see if you can build the chip simulation on its own:

./bazelisk.sh build //hw:verilator

This build can take a long time; it’s creating a simulation for the entire OpenTitan SoC. Expect up to an hour for a successful build, depending on your machine.

If the //hw:verilator build above ran for a while and then failed with a bunch of warnings about various .sv files, it may have run out of RAM. At the time of writing, our CI has 7GB of RAM, so that should be sufficient. If your system is close to that limit, you may want to exit web browsers or other RAM-intensive applications while the Verilator build runs.

If the //hw:verilator build failed pretty much immediately, try running util/check_tool_requirements.py to make sure you meet the tool requirements.

If the //hw:verilator build succeeeded, but running a particular test fails, try running a different test (you can find many options under sw/device/tests/). If that works, then it may be a problem with the specific test you’re running. See if you can build, but not run, the test with ./bazelisk.sh build instead of ./bazelisk.sh test. If the test fails to build, that indicates some issue with the source code or possibly the RISC-V toolchain installation.

Building documentation

The documentation for OpenTitan is available online. The creation of documentation is mainly based around the conversion from Markdown to HTML files with mdbook. Rules for how to write correct Markdown files can be found in the reference manual.

Building locally

There are a few project specific preprocessors. These preprocessors require the installation of the dependencies outlined in the previous section. util/site/build-docs.sh handles building all books in the repository and other auto-generated content, such as the API documentation generated by Doxygen.

Running the server

In order to run a local instance of the documentation server run the following command from the root of the project repository.

./util/site/build-docs.sh serve

This will execute the preprocessing, build the documentation and finally start a local server. The output will indicate at which address the local instance can be accessed. The default is http://0.0.0.0:9000.

Running e2e Tests

FPGA Setup

Before following this guide, make sure you’ve followed the dependency installation and software build instructions.

Do you want to try out OpenTitan, but don’t have a couple thousand or million dollars ready for an ASIC tapeout? Running OpenTitan on an FPGA board can be the answer!

Prerequisites

To use the OpenTitan on an FPGA you need two things:

• A supported FPGA board
• A tool from the FPGA vendor

Depending on the design/target combination that you want to synthesize you will need different tools and boards. Refer to the design documentation for information on what exactly is needed.

• Obtain an FPGA board

Obtain an FPGA bitstream

To run OpenTitan on an FPGA, you will need an FPGA bitstream. You can either download the latest bitstream for the ChipWhisperer CW310 board or build it yourself.

Download a Pre-built Bitstream

If you are using the ChipWhisperer CW310 board with the Xilinx Kintex 7 XC7K410T FPGA, you can download the latest passing pre-built bitstream.

For example, to download and unpack the bitstream, run the following:

mkdir -p /tmp/bitstream-latest
cd /tmp/bitstream-latest
curl https://storage.googleapis.com/opentitan-bitstreams/master/bitstream-latest.tar.gz -o bitstream-latest.tar.gz
tar -xvf bitstream-latest.tar.gz

By default, the bitstream is built with a version of the boot ROM used for testing (called the test ROM; pulled from sw/device/lib/testing/test_rom). There is also a version of the boot ROM used in production (called the ROM; pulled from sw/device/silicon_creator/rom). When the bitstream cache is used in bazel flows, the ROMs from the cache are not used. Instead, the bazel-built ROMs are spliced into the image to create new bitstreams, using the mechanism described in the FPGA Reference Manual. The metadata for the latest bitstream (the approximate creation time and the associated commit hash) is also available as a text file and can be downloaded separately.

Using the @bitstreams repository

OpenTitan’s build system automatically fetches pre-built bitstreams via the @bitstreams repository.

To keep the @bitstreams repository in sync with the current Git revision, install the Git post-checkout hook:

cp util/git/hooks/post-checkout .git/hooks/

Build an FPGA bitstream

Synthesizing a design for an FPGA board is simple with Bazel. While Bazel is the entry point for kicking off the FPGA synthesis, under the hood, it invokes FuseSoC, the hardware package manager / build system supported by OpenTitan. During the build process, the boot ROM is baked into the bitstream. As mentioned above, we maintain two boot ROM programs, one for testing (test ROM), and one for production (ROM).

To build an FPGA bitstream with the test ROM, use:

cd $REPO_TOP
bazel build //hw/bitstream/vivado:fpga_cw310_test_rom

To build an FPGA bitstream with the ROM, use:

cd $REPO_TOP
bazel build //hw/bitstream/vivado:fpga_cw310_rom

Note, building these bitstreams will require Vivado be installed on your system, with access to the proper licenses, described here. For general software development on the FPGA, Vivado must still be installed, but the Lab Edition is sufficient.

Dealing with FPGA Congestion Issues

The default Vivado tool placement may sometimes result in congested FPGA floorplans. When this happens, the implementation time and results become unpredictable. It may become necessary for the user to manually adjust certain placement. See this comment for a thorough analysis of one such situation and what changes were made to improve congestion.

Opening an existing project with the Vivado GUI

Sometimes, it may be desirable to open the generated project in the Vivado GUI for inspection. To this end, run:

. /tools/Xilinx/Vivado/2020.2/settings64.sh
cd $REPO_TOP
make -C $(dirname $(find bazel-out/* -wholename '*synth-vivado/Makefile')) build-gui

Now the Vivado GUI opens and loads the project.

Develop with the Vivado GUI

TODO(lowRISC/opentitan#13213): the below does not work with the Bazel FPGA bitstream build flow.

Sometimes it is helpful to use the Vivado GUI to debug a design. FuseSoC (the tool Bazel invokes) makes that easy, with one small caveat: by default FuseSoC copies all source files into a staging directory before the synthesis process starts. This behavior is helpful to create reproducible builds and avoids Vivado modifying checked-in source files. But during debugging this behavior is not helpful. The --no-export option of FuseSoC disables copying the source files into the staging area, and --setup instructs fusesoc to not start the synthesis process.

# Only create Vivado project directory by using FuseSoC directly (skipping Bazel invocation).
cd $REPO_TOP
fusesoc --cores-root . run --flag=fileset_top --target=synth --no-export --setup lowrisc:systems:chip_earlgrey_cw310

You can then navigate to the created project directory, and open Vivado

. /tools/Xilinx/Vivado/2020.2/settings64.sh
cd $REPO_TOP/build/lowrisc_systems_chip_earlgrey_cw310_0.1/synth-vivado/
vivado

Finally, using the Tcl console, you can kick off the project setup with

source lowrisc_systems_chip_earlgrey_cw310_0.1.tcl

Connecting the ChipWhisperer CW310 board

The ChipWhisperer CW310 board supports different power options. It is recommended to power the board via the included DC power adapter. To this end:

1. Set the SW2 switch (to the right of the barrel connector) up to the 5V Regulator option.
2. Set the switch below the barrel connector to the right towards the Barrel option.
3. Set the Control Power switch (bottom left corner, SW7) to the right.
4. Ensure the Tgt Power switch (above the fan, S1) is set to the right towards the Auto option.
5. Plug the DC power adapter into the barrel jack (J11) in the top left corner of the board.
6. Use a USB-C cable to connect your PC with the USB-C Data connector (J8) in the lower left corner on the board.

You can now use the following to monitor output from dmesg:

sudo dmesg -Hw

This should show which serial ports have been assigned, or if the board is having trouble connecting to USB. If dmesg reports a problem you can trigger a USB_RST with SW5. When properly connected, dmesg should identify the board, not show any errors, and the status light should flash. They should be named '/dev/ttyACM*', e.g. /dev/ttyACM1. To ensure that you have sufficient access permissions, set up the udev rules as explained in the Vivado installation instructions.

You will then need to run this command to configure the board. You only need to run it once.

bazel run //sw/host/opentitantool -- --interface=cw310 fpga set-pll

Check that it’s working by running the demo or a test, such as the uart_smoketest below.

cd $REPO_TOP
bazel test --test_output=streamed //sw/device/tests:uart_smoketest_fpga_cw310_test_rom

Troubleshooting

If the tests/demo aren’t working on the FPGA (especially if you get an error like SFDP header contains incorrect signature) then try adding --rcfile= to the set-pll command:

bazel run //sw/host/opentitantool -- --rcfile= --interface=cw310 fpga set-pll

It’s also worth pressing the USB_RST and USR_RESET buttons on the FPGA if you face continued errors.

Flash the bitstream onto the FPGA and bootstrap software into flash

There are two ways to load a bitstream on to the FPGA and bootstrap software into the OpenTitan embedded flash:

1. automatically, on single invocations of bazel test ....
2. manually, using multiple invocations of opentitantool, and Which one you use, will depend on how the build target is defined for the software you would like to test on the FPGA. Specifically, for software build targets defined in Bazel BUILD files using the opentitan_functest Bazel macro, you will use the latter (automatic) approach. Alternatively, for software build targets defined in Bazel BUILD files using the opentitan_flash_binary Bazel macro, you will use the former (manual) approach.

See below for details on both approaches.

Automatically loading FPGA bitstreams and bootstrapping software with Bazel

A majority of on-device software tests are defined using the custom opentitan_functest Bazel macro, which under the hood, instantiates several Bazel native.sh_test rules. In doing so, this macro provides a convenient interface for developers to run software tests on OpenTitan FPGA instances with a single invocation of bazel test .... For example, to run the UART smoke test (which is an opentitan_functest defined in sw/device/tests/BUILD) on FPGA hardware, and see the output in real time, use:

cd $REPO_TOP
bazel test --test_tag_filters=cw310 --test_output=streamed //sw/device/tests:uart_smoketest

or

cd $REPO_TOP
bazel test --test_output=streamed //sw/device/tests:uart_smoketest_fpga_cw310_test_rom

Under the hood, Bazel conveniently dispatches opentitantool to both:

• ensure the correct version of the FPGA bitstream has been loaded onto the FPGA, and
• bootstrap the desired software test image into the OpenTitan embedded flash.

To get a better understanding of the opentitantool functions Bazel invokes automatically, follow the instructions for manually loading FPGA bitstreams below.

Configuring Bazel to load the Vivado-built bitstream

By default, the above invocations of bazel test ... use the pre-built (Internet downloaded) FPGA bitstream. To instruct bazel to load the bitstream built earlier, or to have bazel build an FPGA bitstream on the fly, and load that bitstream onto the FPGA, add the --define bitstream=vivado flag to either of the above Bazel commands, for example, run:

bazel test --define bitstream=vivado --test_output=streamed //sw/device/tests:uart_smoketest_fpga_cw310_test_rom

Configuring Bazel to skip loading a bitstream

Alternatively, if you would like to instruct Bazel to skip loading any bitstream at all, and simply use the bitstream that is already loaded, add the --define bitstream=skip flag, for example, run:

bazel test --define bitstream=skip --test_output=streamed //sw/device/tests:uart_smoketest_fpga_cw310_test_rom

Manually loading FPGA bitstreams and bootstrapping OpenTitan software with opentitantool

Some on-device software targets are defined using the custom opentitan_flash_binary Bazel macro. Unlike the opentitan_functest macro, the opentitan_flash_binary macro does not instantiate any Bazel test rules under the hood. Therefore, to run such software on OpenTitan FPGA hardware, both a bitstream and the software target must be loaded manually onto the FPGA. Below, we describe how to accomplish this, and in doing so, we shed some light on the tasks that Bazel automates through the use of opentitan_functest Bazel rules.

Manually loading a bitstream onto the FPGA with opentitantool

Note: The following examples assume that you have a ~/.config/opentitantool/config with the proper --interface option. For the CW310, its contents would look like:

--interface=cw310

To flash the bitstream onto the FPGA using opentitantool, use the following command:

cd $REPO_TOP

### If you downloaded the bitstream from the Internet:
bazel run //sw/host/opentitantool fpga load-bitstream /tmp/bitstream-latest/lowrisc_systems_chip_earlgrey_cw310_0.1.bit.orig

### If you built the bitstream yourself:
bazel run //sw/host/opentitantool fpga load-bitstream $(ci/scripts/target-location.sh //hw/bitstream/vivado:fpga_cw310_test_rom)

Depending on the FPGA device, the flashing itself may take several seconds. After completion, a message like this should be visible from the UART:

I00000 test_rom.c:81] Version: earlgrey_silver_release_v5-5886-gde4cb1bb9, Build Date: 2022-06-13 09:17:56
I00001 test_rom.c:87] TestROM:6b2ca9a1
I00002 test_rom.c:118] Test ROM complete, jumping to flash!

Bootstrapping the demo software {#hello-world-demo}

The hello_world demo software shows off some capabilities of the OpenTitan hardware. To load hello_world into the FPGA on the ChipWhisperer CW310 board follow the steps shown below.

1. Generate the bitstream and flash it to the FPGA as described above.

2. Open a serial console (use the device file determined before) and connect. Settings: 115200 baud, 8 bits per byte, no software flow-control for sending and receiving data.

   screen /dev/ttyACM1 115200,cs8,-ixon,-ixoff
   

3. Run opentitantool.

   cd ${REPO_TOP}
   bazel run //sw/host/opentitantool -- --interface=cw310 fpga set-pll # This needs to be done only once.
   bazel build //sw/device/examples/hello_world:hello_world_fpga_cw310_bin
   bazel run //sw/host/opentitantool bootstrap $(ci/scripts/target-location.sh //sw/device/examples/hello_world:hello_world_fpga_cw310_bin)
   

   and then output like this should appear from the UART:

   I00000 test_rom.c:81] Version: earlgrey_silver_release_v5-5886-gde4cb1bb9, Build Date: 2022-06-13 09:17:56
   I00001 test_rom.c:87] TestROM:6b2ca9a1
   I00000 test_rom.c:81] Version: earlgrey_silver_release_v5-5886-gde4cb1bb9, Build Date: 2022-06-13 09:17:56
   I00001 test_rom.c:87] TestROM:6b2ca9a1
   I00002 test_rom.c:118] Test ROM complete, jumping to flash!
   I00000 hello_world.c:66] Hello World!
   I00001 hello_world.c:67] Built at: Jun 13 2022, 14:16:59
   I00002 demos.c:18] Watch the LEDs!
   I00003 hello_world.c:74] Try out the switches on the board
   I00004 hello_world.c:75] or type anything into the console window.
   I00005 hello_world.c:76] The LEDs show the ASCII code of the last character.
   

4. Observe the output both on the board and the serial console. Type any text into the console window.

5. Exit screen by pressing CTRL-a k, and confirm with y.

Troubleshooting

If the firmware load fails, try pressing the “USR-RST” button before loading the bitstream.

Connect with OpenOCD and debug

The CW310 supports JTAG-based debugging with OpenOCD and GDB via the standard ARM JTAG headers on the board (labeled USR Debug Headers). To use it, program the bitstream and bootstrap the desired firmware, then connect a JTAG adapter to one of the headers. For this guide, the Olimex ARM-USB-TINY-H JTAG adapter was used.

After bootstrapping the firmware, the TAP straps may need to be set. As of this writing, the FPGA images are typically programmed to be in the RMA lifecycle state, and the TAP straps are sampled continuously in that state. To connect the JTAG chain to the CPU’s TAP, adjust the strap values with opentitantool. Assuming opentitantool has been built and that the current directory is the root of the workspace, run these commands:

./bazel-bin/sw/host/opentitantool/opentitantool \
        --interface cw310 \
        gpio write TAP_STRAP0 false
./bazel-bin/sw/host/opentitantool/opentitantool \
        --interface cw310 \
        gpio write TAP_STRAP1 true

Connect a JTAG adapter to one of the headers. For the Olimex ARM-USB-TINY-H, use the classic ARM JTAG header (J13) and make sure switch S2 is set to 3.3 V. Depending on the adapter’s default state, OpenTitan may be held in reset when the adapter is initially connected. This reset will come under software control once OpenOCD initializes the driver.

Device permissions: udev rules

The JTAG adapter’s device node in /dev must have read-write permissions. Otherwise, OpenOCD will fail because it’s unable to open the USB device. The udev rule below matches the ARM-USB-TINY-H adapter, sets the octal mode mask to 0666, and creates a symlink at /dev/jtag_adapter_arm_usb_tiny_h.

# [/etc/udev/rules.d/90-jtag-adapter.rules]

SUBSYSTEM=="usb", ATTRS{idVendor}=="15ba", ATTRS{idProduct}=="002a", MODE="0666", SYMLINK+="jtag_adapter_arm_usb_tiny_h"

Now, reload the udev rules and reconnect the JTAG adapter.

# Reload the udev rules.
sudo udevadm control --reload-rules
sudo udevadm trigger

# Physically disconnect and reconnect the JTAG adapter, or fake it:
sudo udevadm trigger --verbose --type=subsystems --action=remove --subsystem-match=usb --attr-match="idVendor=15ba"
sudo udevadm trigger --verbose --type=subsystems --action=add --subsystem-match=usb --attr-match="idVendor=15ba"

# Print the permissions of the USB device. This command should print "666".
stat --dereference -c '%a' /dev/jtag_adapter_arm_usb_tiny_h

To connect the ChipWhisperer CW310 FPGA board with OpenOCD, run the following command:

cd $REPO_TOP
openocd -f <adapter-config.cfg> \
        -c "adapter speed 500; transport select jtag; reset_config trst_and_srst" \
        -f util/openocd/target/lowrisc-earlgrey.cfg

For the Olimex ARM-USB-TINY-H with a Debian-based distro, the adapter configuration would be at /usr/share/openocd/scripts/interface/ftdi/olimex-arm-usb-tiny-h.cfg. So for that particular case, the command would be the following:

cd $REPO_TOP
openocd -f /usr/share/openocd/scripts/interface/ftdi/olimex-arm-usb-tiny-h.cfg \
        -c "adapter speed 500; transport select jtag; reset_config trst_and_srst" \
        -f util/openocd/target/lowrisc-earlgrey.cfg

Example OpenOCD output:

Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
trst_and_srst separate srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst

Info : Hardware thread awareness created
force hard breakpoints
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : JTAG tap: riscv.tap tap/device found: 0x04f5484d (mfg: 0x426 (Google Inc), part: 0x4f54, ver: 0x0)
Info : datacount=2 progbufsize=8
Info : Examined RISC-V core; found 1 harts
Info :  hart 0: XLEN=32, misa=0x40101106
Info : starting gdb server for riscv.tap.0 on 3333
Info : Listening on port 3333 for gdb connections

Note that the reset_config command may need to be adjusted for the particular JTAG adapter in use. TRSTn is available on the 20-pin ARM JTAG header only. Use srst_only if the adapter only supports SRSTn.

See the install instructions for guidance on installing OpenOCD.

To actually debug through OpenOCD, it must either be connected through telnet or GDB.

Debug with OpenOCD

The following is an example for using telnet

telnet localhost 4444 // or whatever port that is specificed by the openocd command above
mdw 0x8000 0x10 // read 16 bytes at address 0x8000

Debug with GDB

First, make sure the device software has been built with debug symbols (by default Bazel does not build software with debug symbols). For example, to build and test the UART smoke test with debug symbols, you can add --copt=-g flag to the bazel test ... command:

cd $REPO_TOP
bazel test --copt=-g --test_output=streamed //sw/device/tests:uart_smoketest_fpga_cw310_test_rom

Then a connection between OpenOCD and GDB may be established with:

cd $REPO_TOP
./bazelisk.sh build --config=riscv32 //sw/device/tests:uart_smoketest_prog_fpga_cw310.elf
riscv32-unknown-elf-gdb -ex "target extended-remote :3333" -ex "info reg" \
  "$(./bazelisk.sh outquery --config=riscv32 //sw/device/tests:uart_smoketest_prog_fpga_cw310.elf)"

The above will print out the contents of the registers upon successs. Note that you should have the RISC-V toolchain installed and on your PATH. For example, if you followed the Getting Started instructions, then make sure /tools/riscv/bin is on your PATH.

Common operations with GDB

Examine 16 memory words in the hex format starting at 0x200005c0

(gdb) x/16xw 0x200005c0

Press enter again to print the next 16 words. Use help x to get a description of the command.

If the memory content contains program text it can be disassembled

(gdb) disassemble 0x200005c0,0x200005c0+16*4

Displaying the memory content can also be delegated to OpenOCD

(gdb) monitor mdw 0x200005c0 16

Use monitor help to get a list of supported commands.

To single-step use stepi or step

(gdb) stepi

stepi single-steps an instruction, step single-steps a line of source code. When testing debugging against the hello_world binary it is likely you will break into a delay loop. Here the step command will seem to hang as it will attempt to step over the whole delay loop with a sequence of single-step instructions which may take quite some time!

To change the program which is debugged the file command can be used. This will update the symbols which are used to get information about the program. It is especially useful in the context of our rom.elf, which resides in the ROM region, which will eventually jump to a different executable as part of the flash region.

(gdb) file sw/device/examples/hello_world/sw.elf
(gdb) disassemble 0x200005c0,0x200005c0+16*4

The output of the disassemble should now contain additional information.

Reproducing FPGA CI Failures Locally

When an FPGA test fails in CI, it can be helpful to run the tests locally with the version of the bitstream generated by the failing CI run. To avoid rebuilding the bitstream, you can download the bitstream artifact from the Azure Pipeline CI run and use opentitantool to load the bitstream manually.

To download the bitstream:

1. Open your PR on Github and navigate to the “Checks” tab.
2. On the left sidebar, expand the “Azure Pipelines” menu.
3. Open the “CI (CW310’s Earl Grey Bitstream)” job and click on “View more details on Azure Pipelines”.
4. Click on “1 artifact produced”.
5. Click on the three dots for “partial-build-bin-chip_earlgrey_cw310”.
6. You can either download the artifact directly or download with the URL.

Note that Azure does not allow you to download the artifact with wget or curl by default, so to use the download URL, you need to specify a user-agent header. For example, to download with curl, you can use the following command

curl --output /tmp/artifact.tar.gz -H 'user-agent: Mozilla/5.0' <download_URL>

After extracting the artifact, the bitstream is located at build-bin/hw/top_earlgrey/lowrisc_systems_chip_earlgrey_cw310_0.1.bit.{splice,orig}. The .splice bitstream has the ROM spliced in, and the .orig bitstream has the test ROM.

Next, load the bitstream with opentitantool, and run the test. The FPGA tests attempt to load the latest bitstream by default, but because we wish to use the bitstream that we just loaded, we need to tell Bazel to skip the automatic bitstream loading.

# Load the bitstream with opentitantool
bazel run //sw/host/opentitantool --interface=cw310 fpga load-bitstream <path_to_your_bitstream>

# Run the broken test locally, showing all test output and skipping the bitstream loading
bazel test <broken_test_rule> --define bitstream=skip --test_output=streamed

Verilator Setup

Before following this guide, make sure you’ve followed the dependency installation instructions.

About Verilator

Verilator is a cycle-accurate simulation tool. It translates synthesizable Verilog code into a simulation program in C++, which is then compiled and executed.

Install Verilator

Even though Verilator is packaged for most Linux distributions these versions tend to be too old to be usable. We recommend compiling Verilator from source, as outlined here.

Fetch, build and install Verilator itself (this should be done outside the $REPO_TOP directory). Note that Verilator 4.210 will not build with GCC 12.0 or later, so it will need to be built with an older toolchain. The example below assumes gcc-11 and g++-11 are installed on the system.

export VERILATOR_VERSION=4.210

git clone https://github.com/verilator/verilator.git
cd verilator
git checkout v$VERILATOR_VERSION

autoconf
CC=gcc-11 CXX=g++-11 ./configure --prefix=/tools/verilator/$VERILATOR_VERSION
CC=gcc-11 CXX=g++-11 make
sudo CC=gcc-11 CXX=g++-11 make install

The make step can take several minutes.

After installation you need to add /tools/verilator/$VERILATOR_VERSION/bin to your PATH environment variable. Also add it to your ~/.bashrc or equivalent so that it’s on the PATH in the future, like this:

export PATH=/tools/verilator/$VERILATOR_VERSION/bin:$PATH

Check your installation by running:

$ verilator --version
Verilator 4.210 2021-07-07 rev v4.210 (mod)

Troubleshooting

If you need to install to a different location than /tools/verilator/..., you can pass a different directory to ./configure --prefix above and add your/install/location/bin to PATH instead.

Running Software on a Verilator Simulation with Bazel

First the RTL must be built into a simulator binary. This is done by running fusesoc, which collects up RTL code and passes it to Verilator to generate and then compile a C++ model. Next software must be built to run on the simulated hardware. There are 4 memory types on OpenTitan hardware: ROM, Flash, OTP, and SRAM. Software images need to be provided for ROM, Flash, and OTP (SRAM is populated at runtime). By default, the system will first execute out of ROM and then jump to Flash. The OTP image does not contain executable code, rather it contains root secrets, runtime configuration data, and life cycle state. (By default, the life cycle state is set to RMA, which enables debugging features such as the JTAG interface for the main processor.) Lastly, the Verilator simulation binary must be run with the correct arguments.

Thankfully, Bazel (and opentitantool) simplify this process by providing a single interface for performing all of the above steps. Moreover, Bazel automatically connects to the simulated UART (via opentitantool) to print the test output in real time.

For example, to run the UART smoke test on Verilator simulated hardware, and see the output in real time, use

cd $REPO_TOP
bazel test --test_output=streamed //sw/device/tests:uart_smoketest_sim_verilator

or

cd $REPO_TOP
bazel test --test_tag_filters=verilator --test_output=streamed //sw/device/tests:uart_smoketest

You should expect to see something like:

Invoking: sw/host/opentitantool/opentitantool --rcfile= --logging=info --interface=verilator --verilator-bin=hw/build.verilator_real/sim-verilator/Vchip_sim_tb --verilator-rom=sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem --verilator-flash=sw/device/tests/uart_smoketest_prog_sim_verilator.64.scr.vmem --verilator-otp=hw/ip/otp_ctrl/data/img_rma.vmem console --exit-failure=(FAIL|FAULT).*\n --exit-success=PASS.*\n --timeout=3600s
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::subprocess] Spawning verilator: "hw/build.verilator_real/sim-verilator/Vchip_sim_tb" ["--meminit=rom,sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem", "--meminit=flash,sw/device/tests/uart_smoketest_prog_sim_verilator.64.scr.vmem", "--meminit=otp,hw/ip/otp_ctrl/data/img_rma.vmem"]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] Simulation of OpenTitan Earl Grey
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] =================================
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] Tracing can be toggled by sending SIGUSR1 to this process:
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ kill -USR1 3422749
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] GPIO: FIFO pipes created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read (read) and $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write (write) for 32-bit wide GPIO.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] GPIO: To measure the values of the pins as driven by the device, run
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ cat $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read  # '0' low, '1' high, 'X' floating
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] GPIO: To drive the pins, run a command like
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ echo 'h09 l31' > $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write  # Pull the pin 9 high, and pin 31 low.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] SPI: Created /dev/pts/9 for spi0. Connect to it with any terminal program, e.g.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ screen /dev/pts/9
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] NOTE: a SPI transaction is run for every 4 characters entered.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] SPI: Monitor output file created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/spi0.log. Works well with tail:
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ tail -f $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/spi0.log
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] UART: Created /dev/pts/10 for uart0. Connect to it with any terminal program, e.g.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ screen /dev/pts/10
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] UART: Additionally writing all UART output to 'uart0.log'.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] USB: Monitor output file created at $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/usb0.log. Works well with tail:
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] $ tail -f $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/usb0.log
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] JTAG: Virtual JTAG interface dmi0 is listening on port 44853. Use
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] OpenOCD and the following configuration to connect:
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]   interface remote_bitbang
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]   remote_bitbang_host localhost
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]   remote_bitbang_port 44853
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout] Simulation running, end by pressing CTRL-c.
[2022-06-09T08:08:16Z INFO  opentitanlib::transport::verilator::stdout]
[2022-06-09T08:08:17Z INFO  opentitanlib::transport::verilator::transport] Verilator started with the following interaces:
[2022-06-09T08:08:17Z INFO  opentitanlib::transport::verilator::transport] gpio_read = $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-read
[2022-06-09T08:08:17Z INFO  opentitanlib::transport::verilator::transport] gpio_write = $HOME/.cache/bazel/_bazel_ttrippel/3d92022c091a734228e22679f3ac7c7f/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/uart_smoketest_sim_verilator.runfiles/lowrisc_opentitan/gpio0-write
[2022-06-09T08:08:17Z INFO  opentitanlib::transport::verilator::transport] uart = /dev/pts/10
[2022-06-09T08:08:17Z INFO  opentitanlib::transport::verilator::transport] spi = /dev/pts/9
Starting interactive console
[CTRL+C] to exit.

I00000 test_rom.c:81] Version:    earlgrey_silver_release_v5-5775-gefa09d3b8
Build Date: 2022-06-09, 00:12:35

I00001 test_rom.c:118] Test ROM complete, jumping to flash!
I00000 status.c:28] PASS!


Exiting interactive console.
[2022-06-09T08:09:38Z INFO  opentitantool::command::console] ExitSuccess("PASS!\r\n")

For most use cases, interacting with the UART is all you will need and you can stop here. However, if you want to interact with the simulation in additional ways, there are more options listed below.

Execution Log

All executed instructions in the loaded software into Verilator simulations are logged to the file trace_core_00000000.log. By default this file is stored somewhere in ~/.cache/bazel, you can find it using the following command:

find ~/.cache/bazel -name "trace_core_00000000.log"

The columns in this file are tab separated; change the tab width in your editor if the columns don’t appear clearly, or open the file in a spreadsheet application.

Interact with GPIO (optional)

The simulation includes a DPI module to map general-purpose I/O (GPIO) pins to two POSIX FIFO files: one for input, and one for output. Observe the gpio0-read file for outputs (in the same directory as the trace):

cat gpio0-read

To drive input pins write to the gpio0-write file. A command consists of the desired state: h for high, and l for low, and the decimal pin number. Multiple commands can be issued by separating them with a single space.

echo 'h09 l31' > gpio0-write  # Pull the pin 9 high, and pin 31 low.

Connect with OpenOCD to the JTAG port and use GDB (optional)

The simulation includes a “virtual JTAG” port to which OpenOCD can connect using its remote_bitbang driver. All necessary configuration files are included in this repository.

See the OpenOCD install instructions for guidance on installing OpenOCD.

Run the simulation with Bazel, making sure to build the device software with debug symbols using

cd $REPO_TOP
bazel test --copt=-g --test_output=streamed //sw/device/tests:uart_smoketest_sim_verilator

Then, connect with OpenOCD using the following command.

cd $REPO_TOP
/tools/openocd/bin/openocd -s util/openocd -f board/lowrisc-earlgrey-verilator.cfg

Lastly, connect GDB using the following command (noting it needs to be altered to point to the sw binary in use).

cd $REPO_TOP
riscv32-unknown-elf-gdb -ex "target extended-remote :3333" -ex "info reg" \
  "$(./bazelisk.sh outquery --config=riscv32 //sw/device/tests:uart_smoketest_prog_sim_verilator.elf)"

SPI device test interface (optional)

The simulation contains code to monitor the SPI bus and provide a host interface to allow interaction with the spi_device. When starting the simulation you should see a message like

SPI: Created /dev/pts/4 for spi0. Connect to it with any terminal program, e.g.
$ screen /dev/pts/4
NOTE: a SPI transaction is run for every 4 characters entered.
SPI: Monitor output file created at /auto/homes/mdh10/github/opentitan/spi0.log. Works well with tail:
$ tail -f /auto/homes/mdh10/github/opentitan/spi0.log

Use any terminal program, e.g. screen or microcom to connect to the simulation.

screen /dev/pts/4

Microcom seems less likely to send unexpected control codes when starting:

microcom -p /dev/pts/4

The terminal will accept (but not echo) characters. After 4 characters are received a 4-byte SPI packet is sent containing the characters. The four characters received from the SPI transaction are echoed to the terminal. The hello_world code will print out the bytes received from the SPI port (substituting _ for non-printable characters). The hello_world code initially sets the SPI transmitter to return SPI! (so that should echo after the four characters are typed) and when bytes are received it will invert their bottom bit and set them for transmission in the next transfer (thus the Nth set of four characters typed should have an echo of the N-1th set with bottom bit inverted).

The SPI monitor output is written to a file. It may be monitored with tail -f which conveniently notices when the file is truncated on a new run, so does not need restarting between simulations. The output consists of a textual “waveform” representing the SPI signals.

Generating waveforms (optional)

TODO(lowRISC/opentitan#13042): the below does not work with the Bazel test running flow.

With the --trace argument the simulation generates a FST signal trace which can be viewed with Gtkwave (only). Tracing slows down the simulation by roughly factor of 1000.

cd $REPO_TOP
build/lowrisc_dv_chip_verilator_sim_0.1/sim-verilator/Vchip_sim_tb \
  --meminit=rom,build-bin/sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem \
  --meminit=flash,build-bin/sw/device/examples/hello_world/hello_world_sim_verilator.64.scr.vmem \
  --meminit=otp,build-bin/sw/device/otp_img/otp_img_sim_verilator.vmem \
  --trace
gtkwave sim.fst

Install OpenOCD

OpenOCD is a tool to connect with the target chip over JTAG and similar transports. It also provides a GDB server which is an “intermediate” when debugging software on the chip with GDB.

It is recommended to use the regular upstream version of OpenOCD instead of the RISC-V downstream fork.

It is trivial to install OpenOCD because we manage the dependency with Bazel. The Bazel-built OpenOCD binary lives at //third_party/openocd:openocd_bin. It’s not runnable, but we also provide a runnable wrapper: //third_party/openocd.

OpenOCD also ships with a library of config files. Instead of using use whichever config files happen to be installed on the system, prefer the Bazel-exposed config files that match OpenOCD source. Currently, we only expose OpenTitan’s default JTAG adapter config as //third_party/openocd:jtag_adapter_cfg.

# Manually run OpenOCD:
./bazelisk.sh run //third_party/openocd -- arg1 arg2

# Get the path of the OpenOCD binary:
./bazelisk.sh outquery //third_party/openocd:openocd_bin

Install Vivado

Generating a bitstream for Xilinx devices requires a Vivado installation. Please note that the “WebPACK” edition does not support the Xilinx Kintex 7 XC7K410T used on the CW310 board.

For software development, Vivado is still necessary for most workflows. However, the (free) Lab Edition is sufficient, and it has a significantly smaller installation footprint. For example, Vivado’s updatemem tool is used to splice ROM images into the bitstream, and this is included in the Lab Edition.

Install Xilinx Vivado

Vivado Version: The recommendation is to use Vivado 2020.2.

Following the arrival of Vivado ML Edition, you will need to follow the links for that, eg. Products -> Hardware Development -> Vivado ML. Then click on ‘Vivado Archive’ in the Version list and locate version 2020.2 of Vivado Design Suite.

See Download and Installation for installation instructions.

When asked what edition to install, choose “Vivado HL Design Edition”. Note: If you are only developing software, you may select the “Lab Edition” instead. On the feature selection screen, select at least the following features:

After installing Vivado, you will need to add Vivado’s paths to your shell environment. See Launching the Vivado IDE from the Command Line on Windows or Linux for instructions.

Device permissions: udev rules

To program any FPGAs the user using Vivado typically needs to have permissions to access USB devices connected to the PC. Depending on your security policy you can take different steps to enable this access. One way of doing so is given in the udev rule outlined below.

To do so, create a file named /etc/udev/rules.d/90-lowrisc.rules and add the following content to it:

# Grant access to board peripherals connected over USB:
# - The USB devices itself (used e.g. by Vivado to program the FPGA)
# - Virtual UART at /dev/tty/XXX

# NewAE Technology Inc. ChipWhisperer boards e.g. CW310, CW305, CW-Lite, CW-Husky
ACTION=="add|change", SUBSYSTEM=="usb|tty", ATTRS{idVendor}=="2b3e", ATTRS{idProduct}=="ace[0-9]|c[3-6][0-9][0-9]", MODE="0666"

# Future Technology Devices International, Ltd FT2232C/D/H Dual UART/FIFO IC
# used on Digilent boards
ACTION=="add|change", SUBSYSTEM=="usb|tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6010", ATTRS{manufacturer}=="Digilent", MODE="0666"

# Future Technology Devices International, Ltd FT232 Serial (UART) IC
ACTION=="add|change", SUBSYSTEM=="usb|tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", MODE="0666"

You then need to reload the udev rules:

sudo udevadm control --reload