SDRAM Memory controllers for the Research community Current Version: 0.3.0
To develop HDL based SDRAM memory controllers that are, to the greatest extent possible, i) open source, ii) FPGA vendor agnostic, iii) simple to modify, and iv) able to easily close timing after controller modifications.
Note: Mem-Re is part of Project Morpheus, which is an effort to build an open source reconfigurable hardware operating system (analogous to Linux in the software world).
The design is likely to have some primitives, which are typically proprietary ASIC chips embedded within the FPGA fabric. Examples of these primitives include PLLs, SERDES units, DDR signalling, global clock buffers, phasers and tristate logic. These primitives are chip architecture specific, and can even differ across FPGAs from the same vendor. Use of the primitives is important for a number of reasons such as:
-
High speed interfaces typically require the FPGA to operate at frequencies significantly higher than what the fabric can easily close timing for. While FPGA technologies have continued to improve and support faster internal clocks, so have the speeds of external interfaces since we consistently target higher performance to keep pace with growing requirements/constraints of workloads. As a result, this disparity between frequencies was, is and will likely continue to be a problem for FPGAs. By using ASICs embedded within the FPGA fabric, we can offload the high frequency data path to the dedicated circuit and close timing.
-
While HDL is a low level assembly-like language, it is still an abstraction of a physical circuit. And as with almost all abstractions, they trade off expressibility for programmability. As a result, there are certain circuits which cannot be effectively expressed within HDL; a shortcoming addressed through use of primitives.
-
Even if a circuit is expresable and can be built in a stable manner within the FPGA fabric, if the primitive is common enough across FPGAs, it is usually a good idea to provide the option of using either the primitive or its HDL equivalent. This is because the latter would i) consume the already limited reconfigurable fabric (making it difficult to place and route designs), ii) unlikely achieve the same levels of performance, and iii) have a relatively higher energy cost.
Our goal is to:
-
Use as much basic HDL as possible. Anything that doesn’t deal with clock generation or a high frequency DDR data path will likely be HDL code.
-
Provide loose circuit coupling, appropriate interfaces and compilation flags for facilitating users in replacing primitives in the default design with equivalent circuits.
-
Minimize, but not completely eliminate, code reuse. Striking the right balance between redundancy and lines-of-code is important for ensuring a design is both readable and easy to modify - High code reuse can sometimes make modifications difficult since a change affects all instantiations of the circuit, and not just the target part of the controller.
-
Provide any constraints that help reduce the effort of closing timing, provided that the constraints are applicable, with little or no modifications, to most FPGA chips.
-
Proprietary IP blocks only allow a limited number of parameters to be tuned. As a result, the controller is typically not application specific. To get something that is tuned to a particular application, we either build a custom controller from scratch, or build wrappers for the vendor IP block - neither of which is an efficient approach. Moreover, even when dealing with general explorations, such as system level research into memory controllers, we hit the same roadblocks since low level access to the data and control paths is typically not possible. Finally, there is the cost of these IP blocks.
-
Open source memory controllers typically suffer from multiple drawbacks. For example, typically these controllers: i) can be difficult to read and understand, ii) are built with a large number of source files and complex hierarchies that substantially increase any effort to modify and/or debug the design, iii) have only been tested in simulation, iv) lack one or more core features, v) have poor out of box performance and/or resource usage, vi) are not vendor agnostic, and vii) are accompanied with little or no documentation.
-
Through community centric growth, we can bridge the gap between proprietary and open source hardware, while simultaneously addressing drawbacks shared by both - similar to how it has been done in the software world. Thus, this project does not aim to provide the best possible solution, but rather aims to provide an efficient and effective platform which enables developers to collaborate towards innovation that outperforms the state of the art.
The initial memory we are targeting is the DDR3 controller due its simplicity versus the newer DDR4 and upcoming (at the time of writing) DDR5.
The initial board we are targeting is the Digilent Arty A35, which has a Xilinx 7-series chip. While the Nextpnr flow does support this chip, we will be using Vivado (free for this chip) for initial development in order to reduce the debugging effort timeframes. Once we have a working design, we will use it to test the open source tools to determine if they can meet the required performance/timing.
We will be using Verilog HDL in the design.
-Frequency -Precision (Phases) -Feedback
power_up;
// INITIALIZE SECTION
zq_calibration (1); // perform Long ZQ Calibration
load_mode (3, 14'b00000000000000); // Extended Mode Register (3)
nop (tmrd-1);
load_mode (2, {14'b00001000_000_000} | mr_cwl<<3); // Extended Mode Register 2 with DCC Disable
nop (tmrd-1);
load_mode (1, 14'b0000010110); // Extended Mode Register with DLL Enable, AL=CL-1
nop (tmrd-1);
load_mode (0, {14'b0_0_000_1_0_000_1_0_00} | mr_wr<<9 | mr_cl<<2); // Mode Register with DLL Reset
nop (max(TDLLK,512));
odt_out <= 1; // turn on odt
nop (10);
// Random Act -> Write -> Read -> Precharge
for (r_i = 0; r_i < 2048; r_i = r_i + 1) begin
r_bank = $urandom_range (8);
r_row = $urandom_range (1<<ROW_BITS);
r_col = $urandom_range (1<<COL_BITS);
r_data = {$urandom,$urandom,$urandom,$urandom,$urandom,$urandom,$urandom,$urandom};
activate (r_bank, r_row);
nop (trcd);
write (r_bank, r_col, 0, 0, 0, r_data);
nop (wl + bl/2 + twtr);
read (r_bank, r_col, 0, 0);
nop (rl + bl/2);
precharge (r_bank, 0);
nop (trp);
end- Implemented design but state machine unstable at 400MHz (stable at 100MHz).
- V-1 was working in simulation.
- V-1 has a single clock (400MHz), single state machine design.
- Tested PLL, DDR and tristate primitives.
- Attempt to find manually optimize to reduce critical paths
- state machine is still unstable at 400MHz.
- Revised goal is to get it working at a dual frequency (100/400 MHz).
- Implemented V-2 where the state machine is driven at 100MHz
- Data output signalling is done at a higher frequency.
- State machine is now stable and does not glitch.
- Still having difficulties with routing design and generating 400MHz clock.
- New approach: Try running at 100M/200M
- Verify state machine sequence
- Data will likely be corrupted
- Tested with manual read/write inputs and a 3 level sanity check
- read, write, read
- read, write, read, reset design, reprogram board, read, write, read
- read, write, read, power cycle board, reprogram board, read, write, read
- Results showed that state machine sequence is good.
Updated git repo
- Implemented V-3 with 200M/400M clock and separation of state machine seq/comb logic
- Compromise
- more code in multiple places now needs to be updated
- But design is more stable, easier to understand, and timing is easier to visualize
- Real time verification of 100M, 200M and 400M clocks.
- Switched from PLL to Multi mode clock manager (recommended for user clocks).
- Wrote UART based software runtime for better testing.
- V-3 working in Xilinx’s own behavioral simulation.
- However, still some bugs with reads and writes
- Bug fixing a data alignment problem.
- Improved ack/busy signalling.
- Bugfixes for the host interface - ended up running the UART controller at 200MHz for simplicity and reliability. Added in extra states for more stability in interfacing the controller logic.
- Bugfixes for the software runtime - reduced complexity of script for more reliable testing.
- Bugfixes to the host interface and software runtime helped reliably test the design
- Some manual optimizations done to reduce slack and failing endpoints.
- Design working! Am able to do 26 writes to different rows on each bank (8x26 today) and then read them afterwards.
- Validated with the three step sanity check approach outlined in the 3/25 update
- Limitations:
- requires two write operations before it can be read
- Otherwise only every alternating byte in the word gets updated
- Likely due to lack of timing closure
- Write alignment turned off currently to reduce complexity and eliminate source of error
- Lowest 2 bytes of write data copied throughout the word
- Only works for certain rows
- The simulation model used to build the design is generic (12 vs 13 pins), and so this limitation will likely require updating the addressing logic.
- two reads required
- First shows data from the previous transaction. Second shows the correct one.
- Likely a bug in the host interface logic since LEDs show the correct value on the first read.
- Timing not closed - unpredictable results when modified
- requires two write operations before it can be read
Updated git repo
- Mapped design to yosys+nextpnr
- Nextpnr Xilinx only supports PLL and ISERDES units
- Requires updating the constraints file
- Initially just tested PLL
- Unsuccessful
- Unpredictable circuit behavior if generated freq > 200 MHz or if phase shift requested.
- Able to generate 200M and 400M clocks by cascading two PLLs
- Inefficient approach
- PLL is a very limited resource.
- Loading the clocks (e.g. inverting the clock) causes the circuit to behave unpredictably.
- Attempted to port design to Genesys 2 board
- Timing closure successful
- However board does not respond
- Likely due to the addressing logic
- Put this on the stack for when we focus on the portability aspect of the design.
- Explored use of the SERDES units
- Located on the same bank as the IO pads
- Can substantially improve timing
- However, are not well documented
- Goal is to create a SERDES+wrapper module so that the current control logic/signalling does not need to be modified.
- Currently strobe pins from the DRAM (dqs) are used to clock the read shift registers directly at positive and negative edges.
- Another goal is to document the SERDES units ourselves.
- Attempted to close timing without SERDES unit
- More manual optimizations
- Ran Xilinx timing wizard
- Manually floorplanned design
- Added multi-cycle exceptions
- Added double buffering for the read data
- Some improvements if the right clock is used for the second buffer (currently 200MHz)
- Conclusion: The 7-series architecture makes it difficult to do this without using the SERDES primitives.
- For example, the strobe IO pad is located half a chip away from the global clock buffers.
- Poor placement prevents strobe from being treated as a clock.
- Likely unable to use the dedicated clock network.
PLL and Differential Signalling demo using a fully open source toolchain for the Icestorm FPGA
DDR3 Memory Interface on Xilinx Zynq SOC – Free Software Compatible