diff options
author | José Pekkarinen <jose.pekkarinen@nokia.com> | 2016-04-11 10:41:07 +0300 |
---|---|---|
committer | José Pekkarinen <jose.pekkarinen@nokia.com> | 2016-04-13 08:17:18 +0300 |
commit | e09b41010ba33a20a87472ee821fa407a5b8da36 (patch) | |
tree | d10dc367189862e7ca5c592f033dc3726e1df4e3 /kernel/Documentation/mic/scif_overview.txt | |
parent | f93b97fd65072de626c074dbe099a1fff05ce060 (diff) |
These changes are the raw update to linux-4.4.6-rt14. Kernel sources
are taken from kernel.org, and rt patch from the rt wiki download page.
During the rebasing, the following patch collided:
Force tick interrupt and get rid of softirq magic(I70131fb85).
Collisions have been removed because its logic was found on the
source already.
Change-Id: I7f57a4081d9deaa0d9ccfc41a6c8daccdee3b769
Signed-off-by: José Pekkarinen <jose.pekkarinen@nokia.com>
Diffstat (limited to 'kernel/Documentation/mic/scif_overview.txt')
-rw-r--r-- | kernel/Documentation/mic/scif_overview.txt | 98 |
1 files changed, 98 insertions, 0 deletions
diff --git a/kernel/Documentation/mic/scif_overview.txt b/kernel/Documentation/mic/scif_overview.txt new file mode 100644 index 000000000..0a280d986 --- /dev/null +++ b/kernel/Documentation/mic/scif_overview.txt @@ -0,0 +1,98 @@ +The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low +level communications API across PCIe currently implemented for MIC. Currently +SCIF provides inter-node communication within a single host platform, where a +node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of +communicating over the PCIe bus while providing an API that is symmetric +across all the nodes in the PCIe network. An important design objective for SCIF +is to deliver the maximum possible performance given the communication +abilities of the hardware. SCIF has been used to implement an offload compiler +runtime and OFED support for MPI implementations for MIC coprocessors. + +==== SCIF API Components ==== +The SCIF API has the following parts: +1. Connection establishment using a client server model +2. Byte stream messaging intended for short messages +3. Node enumeration to determine online nodes +4. Poll semantics for detection of incoming connections and messages +5. Memory registration to pin down pages +6. Remote memory mapping for low latency CPU accesses via mmap +7. Remote DMA (RDMA) for high bandwidth DMA transfers +8. Fence APIs for RDMA synchronization + +SCIF exposes the notion of a connection which can be used by peer processes on +nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A +process in a SCIF node initiates a SCIF connection to a peer process on a +different node via a SCIF "endpoint". SCIF endpoints support messaging APIs +which are similar to connection oriented socket APIs. Connected SCIF endpoints +can also register local memory which is followed by data transfer using either +DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and +kernel mode clients which are functionally equivalent. + +==== SCIF Performance for MIC ==== +DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus +SCIF shows the performance advantages of SCIF for HPC applications and runtimes. + + Comparison of TCP and SCIF based BW + + Throughput (GB/sec) + 8 + PCIe Bandwidth ****** + + TCP ###### + 7 + ************************************** SCIF %%%%%% + | %%%%%%%%%%%%%%%%%%% + 6 + %%%% + | %% + | %%% + 5 + %% + | %% + 4 + %% + | %% + 3 + %% + | % + 2 + %% + | %% + | % + 1 + + + ###################################### + 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- + 1 10 100 1000 10000 100000 + Transfer Size (KBytes) + +SCIF allows memory sharing via mmap(..) between processes on different PCIe +nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap +latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. + +SCIF has a user space library which is a thin IOCTL wrapper providing a user +space API similar to the kernel API in scif.h. The SCIF user space library +is distributed @ https://software.intel.com/en-us/mic-developer + +Here is some pseudo code for an example of how two applications on two PCIe +nodes would typically use the SCIF API: + +Process A (on node A) Process B (on node B) + +/* get online node information */ +scif_get_node_ids(..) scif_get_node_ids(..) +scif_open(..) scif_open(..) +scif_bind(..) scif_bind(..) +scif_listen(..) +scif_accept(..) scif_connect(..) +/* SCIF connection established */ + +/* Send and receive short messages */ +scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..) + +/* Register memory */ +scif_register(..) scif_register(..) + +/* RDMA */ +scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..) + +/* Fence DMAs */ +scif_fence_signal(..) scif_fence_signal(..) + +mmap(..) mmap(..) + +/* Access remote registered memory */ + +/* Close the endpoints */ +scif_close(..) scif_close(..) |