There are two distinct data paths through the iTOP system: the trigger path and the event path. Each system consists of both a sending and receiving channel. Data flow is primarily from the front-end to the back-end. Although much rarer, commands sometimes need to be sent from the back-end to the front-end. These do not use much link bandwidth, but should be treated with higher priority than sending data from the front-end to the back-end. Specifics on each of the data paths and formats for specific data are given below.

The trigger path connects the front-end readout (via SCROD) to the TRG_FIN. The data flow is primarily from the SCROD to the TRG_FIN, and consists of channel and channel-level trigger timing information for all hits that the front-end observes. For the trigger path, the primary data entity is a 32-bit word. This proposal is a minor adaptation of Xin Gao's data format proposal, which can be found here.

Bit 31 high indicates this is a data word.

PMT[2:0] - The PMT on the SCROD corresponding to the hit. A SCROD services 8 SL10 MCP-PMTs.

CHANNEL[3:0] - The channel is the channel number within the PMT.

TIME[23:0] - Timing information (24 bits).

To-Do:

• Fix the numbering scheme for PMTs and channels. The most natural numbering on the front-end electronics is actually by ASIC and channel number. That is actually more flexible since it doesn't depend on how many PMTs or how many channels per PMT. On the other hand, the simulation results used to generate PDFs that interpret these numbers may prefer a different scheme.
• The timing formatting here has not been fixed in any way. It should include some information on revolution/bunch/etc, but exactly how to do this needs work. This word may end up being divided into multiple sub-words.

The general control word structure is:

Bit 31 low indicates this is a control word.

WORD_TYPE[2:0] - Identification of which control word this is. Current assigned words:

• 000 - Link Initialization (SCROD –> TRG_FIN)
• 001 - Wait-to-transmit (TRG_FIN –> SCROD)

This word is sent from the SCROD to the TRG_FIN to inform the downstream TRG_FIN which front-end it's talking to. It is anticipated that this will be sent once upon the link coming up.

Bit 31 low indicates this is a control word.

Bits 30:28 are the WORD_TYPE, which is “000”

MODULE[3:0] - The iTOP module within the Belle II detector. There are 16 total modules, so 4-bits are reserved.

FIBER[1:0] - The fiberoptic link for this module (also a proxy for the electronics module number within the iTOP module). There are 4 front-end board stacks per iTOP module, and each has a separate trigger link, so 2-bits are reserved.

Bits 21:0 are unused, or can be reserved for other use if needed later.

This word is sent from the TRG_FIN to the SCROD to request a reprieve from receiving data. It is anticipated that this will be sent once upon the link coming up.

Bit 31 low indicates this is a control word.

Bits 30:28 are the WORD_TYPE, which is “001”

WAIT_CODE[3:0] - This is a requested delay, in units of 8-clock cycles, that the SCROD should wait until sending further trigger data. For example, “0001” would be a request of 8 clock cycles of delay, and “0010” would be a request of 16 clock cycles of delay.

Bits 23:0 are unused, or can be reserved for other use if needed later.

No other words are defined at the moment. However, bits 30:28 can be used to support up to six other control words. These can be added as needed.

The event path connects the front-end readout (via SCROD) to the back-end electronics. In standalone (non-Belle II infrastructure) situations, this is probably the DSP_cPCI. For the final Belle II data path, this is the DSP_FIN. The data is primarily from the SCROD to one of these back-ends, and consists primarily of waveforms from a triggered event, though other types of data may also flow along this link, such as auxiliary monitoring information (temperatures, feedback settings, etc.). The opposite direction (back-end to SCROD) is the primary mechanism to send slow-control commands to the front-end, including DAC settings, software triggers, and ASIC control bits. The primary data entity is a variable length packet composed of a number of 32-bit words.

As part of initial development, a fixed-length packet protocol was developed. Information on this protocol (which is now being phased out!) can be found here.

The general format for variable-size packets is the following, where N is the number of words in a given packet:

Specifics on words 2 through N-2 each packet type are given below.

Command packets are passed from the data acquisition system to the front-end modules. The data path for such packets should be somewhat flexible, as some commands may need to be sent to FPGAs or DSPs on the DSP_FIN or the DSP_cPCI, in addition to commands that are going to the front-end modules.

Multiple commands may be packed into a single packet, in which case each command will be executed as soon as the command checksum is validated. Each command should be acknowledged by a response packet, and the response packet should include an ID number associated with the command, so that the originator of the command can see which commands have been acknowledged.

A ping command simply generates a response packet from the SCROD. It can be used to check whether a link is alive and functioning or probe the SCROD ID on a given channel.

To-Do:

• The SCROD commands assume we can handle everything with the memory mapped address space. We may need to add further commands as needs arise.

To-Do:

• No specific commands are currently specified for the DSPs or FPGAs on the DSP_cPCI or DSP_FIN boards, but I anticipate some will be necessary. For example, we may need to send pedestal or timing constants to the memory on one of these boards for use by the DSPs. We could use the upper bits of the destination word to designate where the command is going (i.e., SCROD, back-end FPGA, DSP), but this should be elaborated upon.

There are two types of packets that can come back from the SCROD, those that are responses to commands, and those from triggered events.

Each command received by SCROD should be followed by an acknowledgment packet of the following form. Although the SCROD may receive many commands in a single packet, it will respond with an individual packet for each command, immediately after the command is executed. If a command is successfully executed, the response will be:

Further words depend on the command sent. The following are currently planned:

The ping acknowledge does not include anything beyond the command ID. So word 6 would be the checksum, and the final word of the packet.

Both reads and writes to registers should respond with a response from the same register. In this way you can immediately see if the register was able to be set.

Open questions/options:

• We could also echo back the value of the read after a write command, but I think it can be the responsibility of the command issuer to check that with a follow-on read command.

The bad command flags are:

• Bit 0: packet size over expected maximum
• Bit 1: packet packet type not understood
• Bit 2: mismatched command destination (revision and ID did not match broadcast or the SCROD ID read from the general purpose EEPROM)
• Bit 3: command type was not understood
• Bit 4: bad command checksum
• Bit 5: bad packet checksum
• All other bits unused, for now

Event data is output whenever a trigger is received, either hardware or software. It consists of a single header packet, waveform packets if any waveforms are available to be read, and auxiliary packets. A SCROD will always send a header packet upon receipt of a trigger, whether there were any waveforms to be read or not. Auxiliary packets are not yet defined, but are reserved in case we find need of them later.

*Please note that the trigger_type bits 0 and 1 can BOTH be high. This indicates that a hardware or software trigger was received, and the other type of trigger was also received before the first could be fully processed. The second trigger is ignored, but this may indicate that the data may not be suitable for use, depending on the conditions.

• The definition for “reference window”: This window is ALWAYS the last window of analog memory sampled. It effectively is a reference marker on the ring buffer of analog memory that tells you where the system trigger came in, regardless of whether that trigger was from the hardware system trigger or from a software command. In some cases, the reference window may be delayed from this trigger. This occurs when registers are set that allow sampling to continue after the receipt of the system trigger (for example, when you know your signal arrives shortly after the trigger instead of before). Even in these cases, it should still be used along with the “AsicWindow” to determine a coarse time for the first sample of a waveform window.

To-Do:

• These still need to be defined, if it turns out that we need them.