VSI OpenVMS x86-64 Driver Developer Guide for the I/O Buffer Descriptor

The user buffer that is passed to a device driver for I/O on platforms prior to x86-64 was typically described by a three-element triplet within the I/O Request Packet (IRP) consisting of the following:

IRP$L_SVAPTE is a 32-bit System Virtual Address (SVA) that points to the first element in a virtually contiguous list of Page Table Entries (PTEs).

If a buffer is in system space, IRP$L_SVAPTE will normally point to the first element of a list of real PTEs in a page table. PTEs (and the PT pages that contain them) are as virtually contiguous as the buffers that they map, so incrementing a SVAPTE by the size of a PTE simply crosses PT boundaries as needed.

If a buffer is in process space, which cannot be accessed from kernel mode, IRP$L_SVAPTE will normally point to the first element of a list of PTE copies in a DIOBM structure that is also located in the IRP. However, a driver may not modify these PTEs to point to a different page of memory or modify the protection bits contained within these PTEs.

Drivers on these platforms also have to handle the byte offset into the first page of the buffer.

Prior to x86-64, every driver needing to perform DMA had to include this user buffer mapping code within the driver itself. This meant having copies of this mapping code in every driver. If one driver needed changing, then other drivers would likely also need changing.

OpenVMS on x86-64 does not implement the 32-bit Page Table space. As PTEs now all have 64-bit SVAs, and since x86-64 supports multiple page sizes and PTE formats, direct access to PTEs outside of the memory management code is not supported. This prohibits device drivers from accessing SVAPTEs. The method of mapping user buffers using the SVAPTE/BOFF/BCNT triplet has been replaced with a different buffer mapping mechanism on x86-64.

OpenVMS on x86-64 provides a new data structure called an IOBD. This new structure, created in the I/O Exec, contains all of the required user buffer mapping information necessary for performing DMA. Device drivers need only copy the mapping information into their respective scatter/gather structures and hand it off to the device.

This document describes the device driver interface for using an IOBD to map a user buffer for DMA on the x86-64 platform.

A base IOBD has space for up to four Extents. If the user buffer may be mapped in four Extents or less, only a base IODB is required.

In a base IOBD, the Extent vectors for the first Extent are:

If the user buffer is fragmented such that it requires more than four Extents to fully map, an auxiliary IOBD is required. An auxiliary IOBD is sized dynamically to hold the exact number of Extents required to map this buffer and is allocated from non-paged pool.

In an auxiliary IOBD, the Extent vectors for the first Extent are:

All IOBDs contain the IOBD$IL_FLAGS longword. The IOBD$IL_FLAGS longword contains bits that are used for IOBD management. The following table describes the bit definitions in the IOBD$IL_FLAGS longword:

An auxiliary IOBD can be recognized by the IOBD$V_AUX_IOBD bit being set in IOBD$IL_FLAGS. Because auxiliary IOBDs always come from non-paged pool, each will have IOBD$V_REL_DEALLOC set, indicating that when the IOBD is released it must be deallocated. An auxiliary IOBD is never reused.

The following table shows four ways of looking at a single buffer that spans six pages, assuming an 8KB page size. All values are hexadecimal values.

An Extent is used to describe all of (or a segment of) a buffer by a 64-bit address and a 32-bit length, each of which is quadword aligned.

The following is the structure of an Extent from extdef.h. The fields with a reference number are the fields that a driver uses when scanning the Extent list to build the scatter/gather list for the device.

$ LIB/TEXT/EXTRACT=EXTDEF/OUT=TT: SYS$SYSDEVICE:[VMS$COMMON.SYSLIB]SYS$LIB_C.TLB
typedef struct _ext {
    __union {
             unsigned __int64 ext$q_address; 
             __struct {
                       unsigned int ext$l_address_low; 
                       unsigned int ext$l_address_high; 
             } ext$r_addr_longwords;
   } ext$r_address_union;
    __union {
             __struct {
                       int ext$l_length; 
                       int ext$l_mbz; 
             } ext$r_length_struct;
             unsigned __int64 ext$q_length; 
   } ext$r_length_union
   PTE ext$r_common_pte;      /* PTE bits common to all pages in EXT */
} EXT;

The following header files contain the symbols and offsets necessary for IOBD use for drivers written in C:

The following header files contain the symbols and offsets necessary for IOBD use for drivers written in MACRO:

For most I/O requests (e.g. $QIO), IOBDs are used by drivers as follows:

#include <extdef.h> /* Extent (address + length) structure */
EXT_PQ extent = 0L;
int byte_count = <total_byte_count>;
extent = irp->irp$pq_extent;
index = 0;
/* Break inside the loop */
for (;;)
{
        (*sge)->Addr[index] = extent->ext$q_address;
        (*sge)->Len[index] = extent->ext$l_length;

        /* Exit loop if no more bytes to map */
        if ((bcnt -= ext$l_length) == 0)
            break;        /* Done. Mapped all data */

        /* More to go */
        extent++;         /* Step to next Extent */
        (*sge)++;         /* Advance controller’s SGE pointer */
        index++;          /* Index to next Extent */
}
/* All done */
return SS$_NORMAL;

In certain scenarios, a driver needs to allocate and map an internal buffer for driver use. In this case, the driver will have to manage (allocate, fill, release) an IOBD as follows:

#include <iobddef.h>             /* I/O Buffer Descriptor structure */
#include <extdef.h>              /* Extent (address + length) structure */
IOBD_PQ iobd = NULL;
EXT_PQ extent = 0L;
int byte_count = <total_byte_count>;
int index;

/* Get a buffer out of non-paged pool */
exe_std$alononpaged(BUFFER_SIZE, &size, (void**)&va_buffer);

/* Allocate and fill an IOBD with Extents that map allocated buffer */
status = ioc$create_iobd(va_buffer, size, IOBD$M_NORESWAIT, iobd);
if ($VMS_STATUS_SUCCESS(status)) {
      status = ioc$first_extent(*iobd, ext, &extent_boff);
      if (!$VMS_STATUS_SUCCESS(status)) {
          ioc$release_iobd(*iobd);  /* Error status */
          return (status);
      }
}
/* Success status */

extent = irp->irp$pq_extent;
index = 0;
/* Break inside the loop */
for (;;)
{
        (*sge)->Addr[index] = extent->ext$q_address;
        (*sge)->Len[index] = extent->ext$l_length;

         /* Exit loop if no more bytes to map */
         if ((bcnt -= ext$l_length) == 0)
                  break;     /* Done. Mapped all data */

         /* More to go */
         extent++;           /* Step to next Extent */
         (*sge)++;           /* Advance controller’s SGE pointer */
         index++;            /* Index to next Extent */
}

ioc$release_iobd(*iobd);            /* Done with IOBD */

/* All done */
return SS$_NORMAL;

These counters are used to determine the level of IOBD use and to help detect or troubleshoot IOBD related pool leakage.

The counters with a reference number are useful for driver developers, whereas the other counters are for VSI use.

$ ANALYZE/SYSTEM
SDA> READ/EXEC
SDA> SHOW SYMBOL IOBD_*

Symbols sorted by name 
---------------------- 
IOBD_CTR$Q_CREATE_CALLS        = FFFFFFFF.804BF0C0 : 00000000.000527A0
IOBD_CTR$Q_EXT_ALLOCS           = FFFFFFFF.804BF0F8 : 00000000.0008BA9A
IOBD_CTR$Q_EXT_COUNT_HISTOGRAM  = FFFFFFFF.804BF148 : 00000000.05E08F5F
IOBD_CTR$Q_EXT_DEALLOCS         = FFFFFFFF.804BF100 : 00000000.0008BA9A
IOBD_CTR$Q_EXT_POOL             = FFFFFFFF.804BF108 : 00000000.00000000
IOBD_CTR$Q_FILL_CALLS          = FFFFFFFF.804BF0B8 : 00000000.05DB7147
IOBD_CTR$Q_FIRST_EXTENT_CALLS  = FFFFFFFF.804BF0D0 : 00000000.00000000
IOBD_CTR$Q_INV_ATT_IOBDS        = FFFFFFFF.804BF140 : 00000000.00000000
IOBD_CTR$Q_INV_IRP_SIZES        = FFFFFFFF.804BF120 : 00000000.00000000
IOBD_CTR$Q_INV_IRP_TYPES        = FFFFFFFF.804BF118 : 00000000.00000000
IOBD_CTR$Q_INV_PTE_TRANS        = FFFFFFFF.804BF110 : 00000000.00000000
IOBD_CTR$Q_INV_UCB_SIZES        = FFFFFFFF.804BF138 : 00000000.00000000
IOBD_CTR$Q_INV_UCB_SVAS         = FFFFFFFF.804BF128 : 00000000.00000000
IOBD_CTR$Q_INV_UCB_TYPES        = FFFFFFFF.804BF130 : 00000000.00000000
IOBD_CTR$Q_IOBD_ALLOCS         = FFFFFFFF.804BF0E0 : 00000000.000527A0
IOBD_CTR$Q_IOBD_DEALLOCS       = FFFFFFFF.804BF0E8 : 00000000.000527A0
IOBD_CTR$Q_IOBD_POOL           = FFFFFFFF.804BF0F0 : 00000000.00000000
IOBD_CTR$Q_NEXT_SEGMENT_CALLS   = FFFFFFFF.804BF0D8 : 00000000.00000000
IOBD_CTR$Q_PTE_4KB_CHECKS       = FFFFFFFF.804BF0A8 : 00000000.00000000
IOBD_CTR$Q_PTE_4KB_MISMATCHES   = FFFFFFFF.804BF0B0 : 00000000.00000000
IOBD_CTR$Q_RELEASE_CALLS       = FFFFFFFF.804BF0C8 : 00000000.000DE23A