Symmetric Communication Interface (SCIF)

The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors.

SCIF API Components

The SCIF API has the following parts:

  1. Connection establishment using a client server model

  2. Byte stream messaging intended for short messages

  3. Node enumeration to determine online nodes

  4. Poll semantics for detection of incoming connections and messages

  5. Memory registration to pin down pages

  6. Remote memory mapping for low latency CPU accesses via mmap

  7. Remote DMA (RDMA) for high bandwidth DMA transfers

  8. Fence APIs for RDMA synchronization

SCIF exposes the notion of a connection which can be used by peer processes on nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A process in a SCIF node initiates a SCIF connection to a peer process on a different node via a SCIF "endpoint". SCIF endpoints support messaging APIs which are similar to connection oriented socket APIs. Connected SCIF endpoints can also register local memory which is followed by data transfer using either DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and kernel mode clients which are functionally equivalent.

SCIF Performance for MIC

DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus SCIF shows the performance advantages of SCIF for HPC applications and runtimes:

           Comparison of TCP and SCIF based BW

Throughput (GB/sec)
  8 +                                             PCIe Bandwidth ******
    +                                                        TCP ######
  7 +    **************************************             SCIF %%%%%%
    |                       %%%%%%%%%%%%%%%%%%%
  6 +                   %%%%
    |                 %%
    |               %%%
  5 +              %%
    |            %%
  4 +           %%
    |          %%
  3 +         %%
    |        %
  2 +      %%
    |     %%
    |    %
  1 +
    +    ######################################
  0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-
    1       10     100      1000   10000   100000
                 Transfer Size (KBytes)

SCIF allows memory sharing via mmap(..) between processes on different PCIe nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.

SCIF has a user space library which is a thin IOCTL wrapper providing a user space API similar to the kernel API in scif.h. The SCIF user space library is distributed @ https://software.intel.com/en-us/mic-developer

Here is some pseudo code for an example of how two applications on two PCIe nodes would typically use the SCIF API:

Process A (on node A)                 Process B (on node B)

/* get online node information */
scif_get_node_ids(..)                 scif_get_node_ids(..)
scif_open(..)                         scif_open(..)
scif_bind(..)                         scif_bind(..)
scif_listen(..)
scif_accept(..)                               scif_connect(..)
/* SCIF connection established */

/* Send and receive short messages */
scif_send(..)/scif_recv(..)           scif_send(..)/scif_recv(..)

/* Register memory */
scif_register(..)                     scif_register(..)

/* RDMA */
scif_readfrom(..)/scif_writeto(..)    scif_readfrom(..)/scif_writeto(..)

/* Fence DMAs */
scif_fence_signal(..)                 scif_fence_signal(..)

mmap(..)                              mmap(..)

/* Access remote registered memory */

/* Close the endpoints */
scif_close(..)                                scif_close(..)