OFMAP_DRAM_TRACE cycles jump backwards

[chihyu-box]

While inspecting OFMAP_DRAM_TRACE.csv, I noticed that the cycles numbers in the first column do not keep increasing; they go up to a certain value and then jump back to a smaller value, and some cycle values appear twice with different OFMAP addresses. I am not sure if this is intended behavior or a bug.

Command: python3 scalesim/scale.py -c configs/scale.cfg -t topologies/conv_nets/gemm.csv -i gemm

Using configs/scale.cfg with the following relevant settings:

- ArrayHeight = 64, ArrayWidth = 64
- IfmapSramSzkB = 32, FilterSramSzkB = 32, OfmapSramSzkB = 32
- Bandwidth = 64, MemoryBanks:   1
- IfmapSRAMBankBandwidth = 64, IfmapSRAMBankNum = 1
- FilterSRAMBankBandwidth = 64, FilterSRAMBankNum = 1
- Dataflow = ws
- InterfaceBandwidth = USER
- UseRamulatorTrace = False

Topology file: topologies/conv_nets/gemm.csv:

Layer,M,N,K
1,320,64,64

After the run, I inspect:
results/scale_example_run_32x32_ws/layer0/OFMAP_DRAM_TRACE.csv

In OFMAP_DRAM_TRACE.csv for layer0 there is a region like:

...
668.0,20016224.0,20016161.0,20016098.0,...
669.0,20016288.0,20016225.0,20016162.0,...
509.0,20016352.0,-1.0,-1.0,...,-1.0
510.0,20016289.0,20016226.0,20016163.0,...
511.0,20016353.0,20016290.0,20016227.0,...
...
573.0,20020280.0,20020217.0,20020154.0,...,20020479.0,-1.0

So:

• The cycle numbers increase up to 668 / 669, then the next line has cycle 509, and then the sequence continues 510, 511, …, 573.
• An awk script over the first column shows that many cycles in [509,573] appear twice (two different rows have the same cycle value, but different address sets).

If this trace is interpreted literally as “these DRAM writes happen in cycle X”, then some cycles effectively carry more than 64 writes (because the same cycle appears on two rows with different addresses), even though Bandwidth : 64 is specified in the config.
At the same time, DETAILED_ACCESS_REPORT.csv for this layer reports DRAM OFMAP start/stop cycles roughly as 414–669, so those “extra” lines after 669 do not change the min/max, but they are still present in the raw DRAM trace.

The OFMAP DRAM trace seems to be generated as follows:

# scalesim/memory/double_buffered_scratchpad_mem.py

def service_memory_requests(self, ifmap_demand_mat, filter_demand_mat, ofmap_demand_mat):
    ...
    ofmap_serviced_cycles = []

    for i in range(ofmap_lines):
        cycle_arr = np.zeros((1,1)) + i + self.stall_cycles
        ...
        ofmap_cycle_out = self.ofmap_buf.service_writes(ofmap_demand_line, cycle_arr)
        ofmap_serviced_cycles += [ofmap_cycle_out[0]]
        ...
    ...
    self.ofmap_buf.empty_all_buffers(ofmap_serviced_cycles[-1])

And in the write buffer:

# scalesim/memory/write_buffer.py

def empty_all_buffers(self, cycle):
    """
    Method to drain all of the active buffer.
    """
    self.append_to_trace_mat(force=True)

    if self.trace_matrix_empty:
        return

    while self.drain_buf_start_line_id < self.trace_matrix.shape[0]:
        self.drain_end_cycle = self.empty_drain_buf(empty_start_cycle=cycle)
        cycle = self.drain_end_cycle + 1

My understanding is:

• Earlier in the run, whenever the OFMAP buffer fills, empty_drain_buf is called and drains some rows of trace_matrix to DRAM; this can push self.drain_end_cycle up to a cycle such as 669.
• ofmap_serviced_cycles[-1] records the cycle of the last OFMAP write to SRAM, which can be smaller (e.g., around 509) when the last OFMAP demand rows are all -1.
• When empty_all_buffers(ofmap_serviced_cycles[-1]) runs, it starts emitting DRAM writes from this earlier cycle, so new writes appear at cycles that are smaller than some already emitted DRAM cycles. This seems to explain why the cycle sequence in OFMAP_DRAM_TRACE.csv goes up to 668/669 and then jumps back to 509, and why cycles 509–573 appear twice.

Question:

• Is it expected that OFMAP_DRAM_TRACE.csv can contain cycle numbers that first increase, then jump back to a smaller value, and that the same cycle can appear on multiple rows with different addresses?
• Or is this behavior unintended, coming from how the starting cycle is chosen for write_buffer.empty_all_buffers?