#LyX 2.0 created this file. For more info see http://www.lyx.org/ \lyxformat 413 \begin_document \begin_header \textclass book \begin_preamble \usepackage[Lenny]{myfncychap} \usepackage{listings} \usepackage{color} \usepackage{geometry} \usepackage{tikz} \usepackage{array} \usetikzlibrary{positioning,shadows,arrows,shapes,patterns} \usepackage{verbatim} \tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white, bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \renewcommand{\chaptermark}[1]{\markboth{\thechapter.\ #1}{}} \renewcommand{\sectionmark}[1]{\markright{\thesection.\ #1}} \fancyhead{} \fancyhead[LE]{\bfseries\leftmark} \fancyhead[RO]{\bfseries\rightmark} \fancyfoot{} \fancyfoot[LE,RO]{\bfseries\thepage} \fancypagestyle{plain}{ \renewcommand{\headrulewidth}{0pt} \fancyhead{} \fancyhead[LE]{} \fancyhead[RO]{} \fancyfoot{} \fancyfoot[LE,RO]{\bfseries\thepage} } \fancypagestyle{Contents}{ \fancyfoot{} \fancyfoot[LE,RO]{\bfseries\thepage} } \def\contentsname{Table of Contents} \definecolor{listinggray}{gray}{0.95} \lstset{basicstyle=\small,keywordstyle=,tabsize=3,escapechar=`,extendedchars=true} \lstset{backgroundcolor=\color{listinggray},rulecolor=\color{black}} \lstset{commentstyle=\textit, stringstyle=\upshape,showspaces=false} \lstset{showstringspaces=false} \lstset{frame=single} \lstset{breaklines=true} \lstset{language=C} \lstset{basicstyle=\footnotesize} \lstset{columns=flexible} \end_preamble \use_default_options true \begin_modules theorems-ams \end_modules \maintain_unincluded_children false \language american \language_package default \inputencoding auto \fontencoding global \font_roman palatino \font_sans default \font_typewriter default \font_default_family default \use_non_tex_fonts false \font_sc false \font_osf false \font_sf_scale 100 \font_tt_scale 100 \graphics default \default_output_format default \output_sync 0 \bibtex_command default \index_command default \paperfontsize 10 \spacing single \use_hyperref true \pdf_title "Linux Graphics Drivers: an Introduction" \pdf_author "Stéphane Marchesin" \pdf_bookmarks true \pdf_bookmarksnumbered false \pdf_bookmarksopen false \pdf_bookmarksopenlevel 1 \pdf_breaklinks false \pdf_pdfborder true \pdf_colorlinks true \pdf_backref false \pdf_pdfusetitle true \pdf_quoted_options "linkcolor=cyan" \papersize b5paper \use_geometry true \use_amsmath 1 \use_esint 1 \use_mhchem 1 \use_mathdots 1 \cite_engine basic \use_bibtopic false \use_indices false \paperorientation portrait \suppress_date false \use_refstyle 0 \index Index \shortcut idx \color #008000 \end_index \leftmargin 2.5cm \topmargin 2.5cm \rightmargin 1.7cm \bottommargin 2.5cm \secnumdepth 3 \tocdepth 3 \paragraph_separation skip \defskip medskip \quotes_language english \papercolumns 1 \papersides 2 \paperpagestyle fancy \bullet 0 0 17 -1 \tracking_changes false \output_changes false \html_math_output 0 \html_css_as_file 0 \html_be_strict false \end_header \begin_body \begin_layout Title \lang english Linux Graphics Drivers: an Introduction \begin_inset Newline newline \end_inset \size small Version 3 \end_layout \begin_layout Author \lang english Stéphane Marchesin \begin_inset Newline newline \end_inset \end_layout \begin_layout Standard \lang english \begin_inset CommandInset toc LatexCommand tableofcontents \end_inset \end_layout \begin_layout Chapter \lang english Introduction \begin_inset CommandInset label LatexCommand label name "cha:Introduction" \end_inset \end_layout \begin_layout Standard \lang french \begin_inset ERT status open \begin_layout Plain Layout \backslash markboth{ }{ Introduction } \end_layout \end_inset \lang english Accelerating graphics is a complex art which suffers a mostly unjustified reputation of being voodoo magic. This book is intended as an introduction to the inner workings and development of graphics drivers under Linux. Throughout this whole book, knowledge of C programming is expected, along with some familiarity with graphics processors. Although its primary audience is the graphics driver developer, this book details the internals of the full Linux graphics stack and therefore can also be useful to application developers seeking to enhance their vision of the Linux graphics world: one can hope to improve the performance of one's applications through better understanding the Linux graphics stack. In this day and age of pervasive 3D graphics and GPU computing, a better comprehension of graphics is a must have! \end_layout \begin_layout Section \lang english Book overview \end_layout \begin_layout Standard \lang english The book starts with an introduction of relevant hardware concepts (Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:A-Look-at" \end_inset ). Only concepts directly relevant to the graphics driver business are presented there. Then we paint a high-level view of the Linux graphics stack in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:The-Big-Picture" \end_inset and its evolution over the years. Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Framebuffer-Drivers" \end_inset introduces framebuffer drivers, a basic form of graphics drivers under Linux that, although primitive, sees wide usage in the embedded space. Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:The-DRM-Kernel" \end_inset introduces the Direct Rendering Manager (or DRM), a kernel module which is in charge of arbitrating all graphics activity going on in a Linux system. The next chapter (Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:X.Org-Drivers" \end_inset ) focuses on X.Org drivers and the existing acceleration APIs available to the developer. Video decoding sees its own dedicated part in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Video-Decoding" \end_inset . We then move on to 3D acceleration with Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:OpenGL" \end_inset where we introduce the basic concepts of OpenGL. Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Mesa" \end_inset and \begin_inset CommandInset ref LatexCommand ref reference "cha:Gallium-3D" \end_inset are dedicated to Mesa and Gallium 3D, the two foundations of 3D graphics acceleration under Linux used as the framework for 3D drivers. Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:GPU-Computing" \end_inset tackles an emerging field, GPU computing. Next, we discuss suspend and resume in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Suspend-and-Resume" \end_inset . We then discuss two side issues with Linux graphics drivers: technical specifications in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Technical-Specifications" \end_inset and what you should do aside pure development in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Beyond-Development" \end_inset . Finally, we conclude in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Conclusions" \end_inset . \end_layout \begin_layout Standard \lang english Each chapter finishes with the \begin_inset Quotes eld \end_inset takeaways \begin_inset Quotes erd \end_inset , a number of relevant points that we made during said chapter. \end_layout \begin_layout Section \lang english What this book does not cover \end_layout \begin_layout Standard \lang english Computer graphics move at a fast pace, and this book is not about the past. Obsolete hardware (isa, vlb, ...), old standards (the vga standard and its dreadful int10, vesa), outdated techniques (user space modesetting) and old X11 servers (Xsun, XFree86, KDrive...) will not be detailed. \end_layout \begin_layout Chapter \lang english A Look at the Hardware \begin_inset CommandInset label LatexCommand label name "cha:A-Look-at" \end_inset \end_layout \begin_layout Standard \lang english Before diving any further into the subject of graphics drivers, we need to understand the graphics hardware. This chapter is by no means intended to be a complete description of all the inner workings of your average computer and its graphics hardware, but only as an introduction thereof. The goal of this section is just to \begin_inset Quotes eld \end_inset cover the bases \begin_inset Quotes erd \end_inset on the knowledge we will require later on. Notably, most hardware concepts that will subsequently be needed are introduced here. Although we sometimes have to go through architecture-specific hoops, we try to stay as generic as possible and the concepts detailed thereafter should generalize well. \end_layout \begin_layout Section \lang english Hardware Overview \end_layout \begin_layout Standard \lang english Today all computers are architectured the same way: a central processor and a number of peripherals. In order to exchange data, these peripherals are interconnected by a bus over which all communications go. Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:Peripheral-interconnection-in" \end_inset outlines the layout of peripherals in a standard computer. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, myarrowtwoside/.style={<->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=1.5cm] (CPU) {CPU \backslash \backslash }; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=1.5cm, below=1.0cm of CPU] (chipset) {chipset \backslash \backslash }; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=1.5cm, left=0.8cm of chipset] (memory) {System \backslash \backslash Memory}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=1.5cm, right=0.8cm of CPU] (GPU) {Graphics \backslash \backslash Card}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=1.5cm, right=0.8cm of GPU] (network) {Network \backslash \backslash Card}; \end_layout \begin_layout Plain Layout \backslash node[right = 0.8cm of network] {$ \backslash cdots$}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 12cm, below=3cm of GPU] (bus) {Bus}; \end_layout \begin_layout Plain Layout \backslash draw[myarrowtwoside] (CPU.south) -> ++(0,-1.0) (chipset); \end_layout \begin_layout Plain Layout \backslash draw[myarrowtwoside] (chipset.south) -> ++(0,-1.0) (bus); \end_layout \begin_layout Plain Layout \backslash draw[myarrowtwoside] (GPU.south) -> ++(0,-3) (bus); \end_layout \begin_layout Plain Layout \backslash draw[myarrowtwoside] (memory.east) -> ++(0.8,0) (chipset); \end_layout \begin_layout Plain Layout \backslash draw[myarrowtwoside] (network.south) -> ++(0,-3) (bus); \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:Peripheral-interconnection-in" \end_inset Peripheral interconnection in a typical computer. \end_layout \end_inset \end_layout \end_inset The first user of the bus is the CPU. The CPU uses the bus to access system memory and other peripherals. However, the CPU is not the only one able to write and read data to the peripherals, the peripherals themselves also have the capability to exchange information directly. In particular, a peripheral which has the ability to read and write to memory without the CPU intervention is said to be DMA (Direct Memory Access) capable, and the memory transaction is usually called a DMA. This type of transaction is interesting, because it allows the driver to use the GPU instead of the CPU to do memory transfers. Since the CPU doesn't need to actively work any more to achieve those transfers , and since it allows better asynchronicity between the CPU and the GPU, better performance can be attained. Common uses of DMA include improving the performance of texture uploads or streaming video. Today, all graphics processors feature this ability (named DMA bus mastering) which consists in the card requesting and subsequently taking control of the bus for a number of microseconds. \end_layout \begin_layout Standard \lang english If a peripheral has the ability to achieve DMA to or from an uncontiguous list of memory pages (which is very convenient when the data is not contiguous in memory), it is said to have DMA scatter-gather capability (as it can scatter data to different memory pages, or gather data from different pages). \end_layout \begin_layout Standard \lang english Notice that the DMA capability can be a downside in some cases. For example on real time systems, this means the CPU is unable to access the bus while a DMA transaction is in progress, and since DMA transactions happen asynchronously this can lead to missing a real time scheduling deadline. Another example is small DMA memory transfers, where the CPU overhead of setting up the DMA is greater than the gain in asynchronicity and therefore transfers slow down. So while DMA has a lot of advantages from a performance viewpoint, there are situations where it should be avoided. \end_layout \begin_layout Section \lang english Bus types \end_layout \begin_layout Standard \lang english Buses connect the machine peripherals together; each and every communication between different peripherals goes over (at least) one bus. In particular, a bus is the way most graphics card are connected to the rest of the computer (one notable exception being the case of some embedded systems, where the GPU is directly connected to the CPU). As shown in Table \begin_inset CommandInset ref LatexCommand ref reference "fig:Common-bus-types" \end_inset , there are many bus types suitable for graphics: PCI, AGP, PCI-X, PCI-express to name a (relevant) few. All the bus types we will detail are variants of the PCI bus type, however some of them feature singular improvements over the original PCI design. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset Tabular \begin_inset Text \begin_layout Plain Layout \lang english Bus type \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Bus width \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Frequency \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Bandwidth \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Capabilities \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english PCI \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 32 bits \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 33 Mhz \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 133 MB/s (33 Mhz) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english - \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english AGP \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 32 bits \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 66 Mhz \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 2100 MB/s (8x) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english SBA, FW, \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english GART \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english PCI-X \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 64 bits \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 33, 66, \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 533 MB/s (66 Mhz) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english - \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 133 Mhz \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english PCI-Express (1.0) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Serial \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 1.25 Ghz \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 4 GB/s (16 lanes) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english - \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english PCI-Express (3.0) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Serial \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 4 Ghz \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 16 GB/s (16 lanes) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english - \end_layout \end_inset \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:Common-bus-types" \end_inset Common bus types. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english PCI (Peripheral Component Interconnect) \end_layout \begin_layout Standard \lang english PCI is the most basic bus allowing connecting graphics peripherals today. One of its key feature is called bus mastering. This feature allows a given peripheral to take hold of the bus for a given number of cycles and do a complete transaction (called a DMA, Direct Memory Access). The PCI bus is coherent, which means that no explicit flushes are required for the memory to be coherent across devices. \end_layout \begin_layout Subsubsection* \lang english AGP (Accelerated Graphics Port) \end_layout \begin_layout Standard \lang english AGP is essentially a modified PCI bus with a number of extra features compared to its ancestor. Most importantly, it is faster thanks to a higher clock speed and the ability to send 2, 4 or 8 bits per lane on each clock tick (for AGP 2x, 4x and 8x respectively). AGP also three distinctive features: \end_layout \begin_layout Itemize \lang english The first feature is AGP GART (Graphics Aperture Remapping Table), a simple form of IOMMU (as will be seen in section \begin_inset CommandInset ref LatexCommand ref reference "sec:Virtual-and-Physical" \end_inset ). It allows taking a (non contiguous) set of physical memory pages out of system memory and exposing it to the GPU for its use as a contiguous area. This increases the amount of memory usable by the GPU at little cost, and creates a convenient area for sharing data between the CPU and the GPU (AGP graphics cards can do fast DMA to/from this area, and since the GART area is a chunk of system RAM, CPU access is a lot faster than VRAM). One notable drawback is that the GART area is not coherent, and therefore writes to GART (be it from the GPU or CPU) need to be flushed before transactio ns from the other party can begin. Another drawback is that only a single GART area is handled by the hardware, and it has to be sub-allocated by the driver. \end_layout \begin_layout Itemize \lang english The second feature is AGP side band addressing (SBA). Side band addressing consists in 8 extra bus bits used as an address bus. Instead of multiplexing the bus bandwidth between addresses and data, the nominal AGP bandwidth can be dedicated to data only. This feature is transparent to the driver developer. \end_layout \begin_layout Itemize \lang english The third feature is AGP Fast Writes (FW). Fast writes allow sending data to the graphics card directly, without having the card initiate a DMA. This feature is also transparent for the driver developer. \end_layout \begin_layout Standard \lang english Keep in mind that these last two features are known to be unstable on a wide range of hardware, and oftentimes require chipset-specific hacks to work properly. Therefore it is advisable not to enable them. In fact, they are an extremely frequent cause for strange hardware errors on AGP cards. \end_layout \begin_layout Subsubsection* \lang english PCI-X \end_layout \begin_layout Standard \lang english PCI-X was developed as a faster PCI for server boards, and very few graphics peripherals exist in this format (some Matrox G550 cards). It is not to be confused with PCI-Express, which sees real widespread usage. \end_layout \begin_layout Subsubsection* \lang english PCI-Express (PCI-E) \end_layout \begin_layout Standard \lang english PCI-Express is the new generation of PCI devices. It has more advantages than a simple improved PCI. \end_layout \begin_layout Standard \lang english Finally, it is important to note that, depending on the architecture, the CPU-GPU communication does not always relies on a bus. This is especially common on embedded systems where the GPU and the CPU are on a single die. In that case the CPU can access the GPU registers directly. \end_layout \begin_layout Section \lang english Virtual and Physical Memory \begin_inset CommandInset label LatexCommand label name "sec:Virtual-and-Physical" \end_inset \end_layout \begin_layout Standard \lang english The term \begin_inset Quotes eld \end_inset memory \begin_inset Quotes erd \end_inset has two main different meanings: \end_layout \begin_layout Itemize \lang english Physical memory. Physical memory is real, hardware memory, as stored in the memory chips. \end_layout \begin_layout Itemize \lang english Virtual memory. Virtual memory is a translation of physical memory addresses allowing user space applications to see their allocated chunks as if they were contiguous while they are fragmented and scattered on the chips. \end_layout \begin_layout Standard \lang english In order to simplify programming, it is easier to handle contiguous memory areas. It is easy to allocate a small contiguous area, but allocating a bigger memory chunk would require as much contiguous physical memory which is difficult if not impossible to achieve shortly after bootup because of memory fragmentation. Therefore, a mechanism is required to keep the appearance of a contiguous piece of memory to the application while using scattered pieces. \end_layout \begin_layout Standard \lang english To achieve this, memory is split into pages. For the scope of this book, it is sufficient to say that a memory page is a collection contiguous bytes in physical memory \begin_inset Foot status open \begin_layout Plain Layout \lang english On x86 and x86-64, a page is usually 4096 bytes long, although different sizes are possible on other architectures or with huge pages. \end_layout \end_inset In order to make a scattered list of physical pages seem contiguous in virtual space, a piece of hardware called MMU (memory mapping unit) converts virtual addresses (used in applications) into physical addresses (used for actually accessing memory) using a page table as shown on Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:MMU-and-IOMMU" \end_inset . In case a page does not exist in virtual space (and therefore not in the MMU table), the MMU is able to signal it, which provides the basic mechanism for reporting access to non-existent memory areas. This in turn is used by the system to implement advanced memory programming like swapping or on-the-fly page instantiations. As the MMU is only effective for CPU access to memory, virtual addresses are not relevant to the hardware since it is not able to match them to physical addresses. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, \end_layout \begin_layout Plain Layout mynode2/.style={rectangle,rounded corners,draw=black, top color=white, bottom color=red!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (CPU) {CPU}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=of CPU] (GPU) {GPU}; \end_layout \begin_layout Plain Layout \backslash node[mynode, below=1.0cm of GPU] (gpummu) {GPU VM engine}; \end_layout \begin_layout Plain Layout \backslash node[mynode, below=4cm of CPU] (mmu) {MMU}; \end_layout \begin_layout Plain Layout \backslash node[mynode, below=4cm of GPU] (iommu) {IOMMU}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, left=1cm of mmu] (mmupt) {MMU page table}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, right=1cm of iommu] (iommupt) {IOMMU page table}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, right=1cm of gpummu] (gpummupt) {GPU VM page table}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width=5cm, below=2cm of mmu, xshift=1.5cm] (memory) {Memory}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (CPU.south) -| (mmu.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (GPU.south) -| (gpummu.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (gpummu.south) -| (iommu.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (mmu.south) -> ++(0,-2) (memory); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (iommu.south) -> ++(0,-2) (memory); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (mmu) -> (mmupt); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (iommu) -> (iommupt); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (gpummu) -> (gpummupt); \end_layout \begin_layout Plain Layout \backslash node[anchor=west] at (2.5,-1.0) {GPU Virtual Address}; \end_layout \begin_layout Plain Layout \backslash node[anchor=west] at (2.5,-3.5) {Bus Address}; \end_layout \begin_layout Plain Layout \backslash node at (-1.5,-1.5) {Virtual Address}; \end_layout \begin_layout Plain Layout \backslash node at (-1.5,-6.5) {Physical Address}; \end_layout \begin_layout Plain Layout \backslash node[anchor=west] at (2.5,-6.5) {Physical Address}; \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:MMU-and-IOMMU" \end_inset MMU and IOMMU. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english While the MMU only works for CPU accesses, it has an equivalent for peripherals: the IOMMU. As shown on figure \begin_inset CommandInset ref LatexCommand ref reference "fig:MMU-and-IOMMU" \end_inset , an IOMMU is the same as an MMU except that it virtualizes the address space of peripherals. The IOMMU can see various incarnations, either on the motherboard chipset (in which case it is shared between all peripherals) or on the graphics card itself (where it will be called AGP GART, PCI GART). The job of the IOMMU is to translate memory addresses from the peripherals into physical addresses. In particular, this allows \begin_inset Quotes eld \end_inset fooling \begin_inset Quotes erd \end_inset a device into restricting its DMAs to a given range of memory and it is required for better security and hardware virtualization. \end_layout \begin_layout Standard \lang english A special case of IOMMU is the Linux swiotlb which allocates a contiguous piece of physical memory at boot (which makes it feasible to have a large contiguous physical allocation since there is no fragmentation yet) and uses it for DMA. As the memory is physically contiguous, no page translation is required and therefore a DMA can occur to and from this memory range. However, this means that this memory (64MB by default) is preallocated and will not be used for anything else. \end_layout \begin_layout Standard \lang english AGP GART is another special case of IOMMU present with AGP graphics cards which exposes a single linear area to the card. In that case the IOMMU is embedded in the AGP chipset, on the motherboard. The AGP GART area is exposed as a linear area of virtual memory to the system. \begin_inset Note Note status open \begin_layout Plain Layout \lang english Say it's linear in physical and virtual memory \end_layout \end_inset \end_layout \begin_layout Standard \lang english Yet another special case of IOMMU is the PCI GART present on some GPUs, which allows exposing a chunk of system memory to the card. In that case the IOMMU table is embedded in the graphics card, and often the physical memory used does not need to be contiguous. \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english http://images.google.fr/images?hl=fr&source=hp&q=page+table&btnG=Recherche+d'image s&gbv=2&aq=f&oq= \end_layout \begin_layout Plain Layout \lang english http://pages.cs.wisc.edu/~bart/537/lecturenotes/s16.html \end_layout \begin_layout Plain Layout \lang english http://a.michelizza.free.fr/pmwiki.php?n=TutoOS.Mm3 \end_layout \begin_layout Plain Layout \lang english http://lwn.net/Articles/106177/ \end_layout \begin_layout Plain Layout \lang english http://www.vocw.edu.vn/content/m10106/latest/ \end_layout \begin_layout Plain Layout \lang english http://cs.nyu.edu/courses/spring05/G22.2250-001/lectures/lecture-08.html \end_layout \end_inset \end_layout \begin_layout Standard \lang english Obviously, with so many different memory types, performance is not homogeneous; not all combination of accesses are fast, depending on whether they involve the CPU, the GPU, or bus transfers. Another issue which arises is memory coherence: how can one ensure that memory is coherent across devices, in particular that data written by the CPU is available to the GPU (or the opposite). These two issues are correlated, as higher performance usually means a lower level of memory coherence, and vice-versa. \end_layout \begin_layout Standard \lang english As far as setting the memory caching parameters goes, there are two ways to set caching attributes on memory ranges: \end_layout \begin_layout Itemize \lang english MTRRs. An MTRR (Memory Type Range Register) is a register describing attributes for a range of given physical memory. Each MTRR consists of a starting physical address, a size, and a caching type. The number of MTRRs depends on the system, but it is usually very small (less than a dozen). Although these apply to a physical memory range, the effect works on the corresponding virtual memory pages. This for example makes it possible to map pages with a specific caching type. \begin_inset Note Note status open \begin_layout Plain Layout \lang english XXX add examples \end_layout \end_inset \end_layout \begin_layout Itemize \lang english PAT (Page Attribute Table) allows setting per-page memory attributes. Instead of relying on a limited number of memory ranges like with MTRRs, it is possible to specify caching attributes on a per-page basis. However it is an extension only available on recent x86 processors. \end_layout \begin_layout Standard \lang english On top of these, one can use explicit caching instructions on some architectures , for example on x86 \emph on movntq \emph default is an uncached mov instruction and \emph on clflush \emph default can selectively flush cache lines. \end_layout \begin_layout Standard \lang english There are three caching modes, usable both through MTRRs and PAT on system memory: \end_layout \begin_layout Itemize \lang english UC (UnCached) memory is uncached. CPU read/writes to this area are uncached, and each memory write instruction triggers an actual immediate memory write. This is helpful to ensure that information has been actually written so as to avoid CPU/GPU race conditions. \end_layout \begin_layout Itemize \lang english WC (Write Combine) memory is uncached, but CPU writes are combined together in order to improve the performance. This is useful to improve performance in situations where uncached memory is required, but where combining the writes together has no adverse effects. \end_layout \begin_layout Itemize \lang english WB (Write Back) memory is cached. This is the default mode and leads to the best performance for CPU accesses. However this does not ensure that memory writes are propagated to central memory after a finite time. \end_layout \begin_layout Standard \lang english Notice that these caching modes apply to the CPU only, the GPU accesses are not directly affected by the current caching mode. However, when the GPU has to access an area of memory which was previously filled by the CPU, uncached modes ensure that the memory writes are actually done, and are not pending sitting in a CPU cache. Another way to achieve the same effect is the use of cache flushing instruction s present on some x86 processors (like cflush). However this is less portable than using the caching modes. Yet another (portable) way is the use of memory barriers, which ensures that pending memory writes have been committed to main memory before moving on. \end_layout \begin_layout Standard \lang english Obviously with so many different caching modes, not all accesses have the same performance: \end_layout \begin_layout Itemize \lang english When it comes to CPU access to system memory, uncached mode provides the worst performance, write back provides the best performance, and write combine is in between. \end_layout \begin_layout Itemize \lang english When the CPU accesses the video memory from a discrete card, all accesses are extremely slow, be they reads or writes, as each access needs a cycle on the bus. Therefore it is not recommended to access large areas of VRAM with the CPU. Furthermore on some GPUs synchronizing is required or this could cause a GPU hang. \end_layout \begin_layout Itemize \lang english Obviously the GPU accessing VRAM is extremely fast. \end_layout \begin_layout Itemize \lang english GPU access to system ram is unaffected by the caching mode, but still has to go over the bus. This is the case of DMA transactions. As those happen asynchronously, they can be considered \begin_inset Quotes eld \end_inset free \begin_inset Quotes erd \end_inset from the viewpoint of the CPU, however there is a non-negligible setup cost involved for each DMA transaction. This is why, when transferring small amounts of memory, a DMA transaction is not always better than a direct CPU access. \end_layout \begin_layout Standard \lang english Finally, one last important point to make about memory is the notion of memory barriers and write posting. In the case of a cached (Write Combine or Write Back) memory area, a memory barrier ensures that pending writes have actually been committed to memory. This is used, for example, before asking the GPU to read a given memory area. For I/O areas, a similar technique called write posting exists: it consists in doing a dummy read inside the I/O area which will, as a side effect, wait until pending writes have taken effect before completing. \end_layout \begin_layout Section \lang english Anatomy of a Graphics Card \end_layout \begin_layout Standard \lang english Today, a graphics card is basically a computer-in-the-computer. It is a complex beast with a dedicated processor on a separate card, and features its own computation units, its own bus, and its own memory. This section gives an overview of a graphics card, including those elements. \end_layout \begin_layout Subsubsection* \lang english Graphics Memory \end_layout \begin_layout Standard \lang english The GPU's memory, which we will from now on refer to as video memory, can be either real, dedicated, on-card memory (in the case of a discrete card), or memory shared with the CPU (also known as \begin_inset Quotes eld \end_inset stolen memory \begin_inset Quotes erd \end_inset or \begin_inset Quotes eld \end_inset carveout \begin_inset Quotes erd \end_inset in the case of an integrated card). Notice that the case of shared memory has interesting implications, as it means that system to video memory copies can be virtually free if implemente d properly. In the case of dedicated memory it means that transfers back and forth will need to happen, and they will be limited by the speed of the bus. \end_layout \begin_layout Standard \lang english It is not uncommon for modern GPUs to feature a form of virtual memory as well, allowing to map different resources (real video memory of system memory) into the GPU address space. This is very similar to the CPU's virtual memory, but uses a completely separate hardware implementation. For example, older Radeon cards (actually since Rage 128) feature a number of surfaces which you can map into the GPU address space, each of which is a contiguous memory resource (video ram, AGP, PCI). Old Nvidia cards (everything up to NV40) have a similar concept based on objects which describe an area of memory which can then be bound to a given use. Recent cards (starting with NV50 and R800) let you build the address space page by page, with the ability of picking system and dedicated video memory pages at will. The similarity of these with a CPU virtual address space is very striking, in fact unmapped page access can be signaled to the system through an interrupt and acted upon in a video memory page fault handler. However, be careful playing with those as the implication here is that driver developers have to juggle with multiple address spaces from the CPU and GPU which are going to be fundamentally different. \end_layout \begin_layout Subsubsection* \lang english Surfaces \end_layout \begin_layout Standard \lang english Surfaces are the basic sources and targets for all rendering. Although they can be called differently (textures, render targets, buffers...) the basic idea is always the same. Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:The-layout-of" \end_inset depicts the layout of a graphics surface. The surface width is rounded up to what we call the pitch because of hardware limitations (usually to the next multiple of some power of 2) and therefore there exists a dead zone of pixels which goes unused. The graphics surface has a number of characteristics: \end_layout \begin_layout Itemize \lang english The pixel format of the surface. A pixel color is represented memory by its red, green and blue components, plus an alpha component used as the opacity for blending. The number of bits for a whole pixel usually matches hardware sizes (8,16 or 32 bits) but the repartition of the bits between the four components does not have to match those. The number of bits used for each pixels is referred to as bits per pixel, or \emph on bpp \emph default . Common pixel formats include 888 RGBX, 8888 RGBA, 565 RGB, 5551, RGBA, 4444 RGBA \begin_inset Note Note status open \begin_layout Plain Layout \lang english , YUV12, YUY16 \end_layout \end_inset . Notice that most cards today work natively in ABGR 8888. \end_layout \begin_layout Itemize \lang english Width and height are the most obvious characteristics, and are given in pixels. \end_layout \begin_layout Itemize \lang english The pitch is the width in bytes (not in pixels!) of the surface, including the dead zone pixels. The pitch is convenient for computing memory usages, for example the size of the surface should be computed by \begin_inset Formula $height\times pitch$ \end_inset and not \begin_inset Formula $height\times width\times bpp$ \end_inset in order to include the dead zone. \end_layout \begin_layout Standard \lang english Notice that surfaces are not always stored linearly in video memory, in fact for performance reasons it is extremely common that they are not, as this improves the locality of the memory accesses when rendering. Such surfaces are called \emph on tiled \emph default . The exact layout of a tiled surface is highly dependent on the hardware, but is usually a form of space-filling curve like the Z curve or Hilbert's curve. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash hspace{-4cm} \end_layout \end_inset \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (2,2) rectangle (10,7); \end_layout \begin_layout Plain Layout \backslash draw[pattern = north east lines] (8.5,2) rectangle (10,7); \end_layout \begin_layout Plain Layout \backslash draw[<->] (2,7.5) -- +(6.5,0); \end_layout \begin_layout Plain Layout \backslash draw[<->] (1.5,2) -- +(0,5); \end_layout \begin_layout Plain Layout \backslash draw[<->] (2,1.5) -- +(8,0); \end_layout \begin_layout Plain Layout \backslash node at (6,8) {Surface width}; \end_layout \begin_layout Plain Layout \backslash node at (6,1) {Surface pitch}; \end_layout \begin_layout Plain Layout \backslash node at (0,4.5) {Surface height}; \end_layout \begin_layout Plain Layout \backslash node at (5.2,4.5) {Used pixels}; \end_layout \begin_layout Plain Layout \backslash node at (9.2,4.8) {Dead}; \end_layout \begin_layout Plain Layout \backslash node at (9.2,4.3) {zone}; \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:The-layout-of" \end_inset The layout of a surface. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english 2D Engine \end_layout \begin_layout Standard \lang english The 2D engine, or blitter, is the hardware used for 2D acceleration. Blitters have been one of the earliest form of graphics acceleration and are still extremely widespread today. Generally, a 2D engine is capable of the following operations: \end_layout \begin_layout Itemize \lang english Blits. Blits are a copy of a memory rectangle from one place to another by the GPU. The source and destination can be either video or system memory. \end_layout \begin_layout Itemize \lang english Solid fills. Solid fills consist in filling a rectangle memory area with a color. Note that this can also include the alpha channel. \end_layout \begin_layout Itemize \lang english Alpha blits. Alpha blits use the alpha component of pixels from of a surface to achieve transparency [porter & duff]. \end_layout \begin_layout Itemize \lang english Stretched blits. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash hspace{-2cm} \end_layout \end_inset \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout % Source \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (2,2) rectangle (8,6); \end_layout \begin_layout Plain Layout \backslash draw[pattern = north east lines] (7,2) rectangle (8,6); \end_layout \begin_layout Plain Layout \backslash node at (4,7) {Blit width}; \end_layout \begin_layout Plain Layout \backslash draw[<->] (3,6.5) -- +(2,0); \end_layout \begin_layout Plain Layout \backslash node at (0,4.5) {Blit height}; \end_layout \begin_layout Plain Layout \backslash draw[<->] (1.5,3.5) -- +(0,2); \end_layout \begin_layout Plain Layout \backslash draw[<->] (2,1.5) -- +(6,0); \end_layout \begin_layout Plain Layout \backslash node at (5,1) {Src pitch}; \end_layout \begin_layout Plain Layout % source pixels \end_layout \begin_layout Plain Layout \backslash draw (3,3.5) rectangle (5,5.5); \end_layout \begin_layout Plain Layout \backslash node at (4,4.5) {Src pixels}; \end_layout \begin_layout Plain Layout % Destination \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (9,2) rectangle (12,6); \end_layout \begin_layout Plain Layout \backslash draw[pattern = north east lines] (11.5,2) rectangle (12,6); \end_layout \begin_layout Plain Layout \backslash draw[<->] (9,1.5) -- +(3,0); \end_layout \begin_layout Plain Layout \backslash node at (10.5,1) {Dst pitch}; \end_layout \begin_layout Plain Layout % destination pixels \end_layout \begin_layout Plain Layout \backslash draw (9.2,2.5) rectangle (11.2,4.5); \end_layout \begin_layout Plain Layout \backslash node at (10.2,3.5) {Dst pixels}; \end_layout \begin_layout Plain Layout % relier les zones src/dst de copie \end_layout \begin_layout Plain Layout \backslash draw[-,style=dashed] (9.2,2.5) -- (3,3.5); \end_layout \begin_layout Plain Layout \backslash draw[-,style=dashed] (11.2,2.5) -- (5,3.5); \end_layout \begin_layout Plain Layout \backslash draw[-,style=dashed] (11.2,4.5) -- (5,5.5); \end_layout \begin_layout Plain Layout \backslash draw[-,style=dashed] (9.2,4.5) -- (3,5.5); \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:Blitting-between-two" \end_inset Blitting between two different surfaces. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:Blitting-between-two" \end_inset shows an example of blitting a rectangle between two different surfaces. This operation is defined by the following parameters: the source and destinati on coordinates, the source and destination pitches, and the blit width and height. However, this is limited to 2D coordinates, usually no perspective or transform ation is possible with a blitting engine. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash hspace{-4cm} \end_layout \end_inset \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (2,2) rectangle (10,7); \end_layout \begin_layout Plain Layout \backslash draw[pattern = north east lines] (8.5,2) rectangle (10,7); \end_layout \begin_layout Plain Layout \backslash draw[<->] (2,7.5) -- +(6.5,0); \end_layout \begin_layout Plain Layout \backslash draw[<->] (1.5,2) -- +(0,5); \end_layout \begin_layout Plain Layout \backslash draw[<->] (2,1.5) -- +(8,0); \end_layout \begin_layout Plain Layout \backslash node at (6,8) {Surface width}; \end_layout \begin_layout Plain Layout \backslash node at (6,1) {Surface pitch}; \end_layout \begin_layout Plain Layout \backslash node at (0,4.5) {Surface height}; \end_layout \begin_layout Plain Layout % source pixels \end_layout \begin_layout Plain Layout \backslash draw (4,3.5) rectangle (8,6.5); \end_layout \begin_layout Plain Layout \backslash node at (6,6) {Src pixels}; \end_layout \begin_layout Plain Layout % destination pixels \end_layout \begin_layout Plain Layout \backslash draw (2.5,2.5) rectangle (6.5,5.5); \end_layout \begin_layout Plain Layout \backslash node at (4,3) {Dst pixels}; \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:Overlapping-blit-inside" \end_inset Overlapping blit inside a surface. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english When a blit happens between two overlapping source and destination surfaces, the semantics of the copy is not trivially defined, especially if one considers that what happens for a blit is not a simple move of a rectangle, but is done pixel-by-pixel at the core. As seen on Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:Overlapping-blit-inside" \end_inset , if one does a line-by-line copy top to bottom, some source pixels will be modified as a side effect. Therefore, the notion of blitting direction was introduced into the blitters. In this case, for a proper copy a bottom to top copy is required. Some cards will determine the blitting direction automatically according to surface overlap (for example nvidia GPUs), and others will not, in which case this has to be handled by the driver. This is why some GPUs actually support negative pitches in order to tell the 2D engine to go backwards. \end_layout \begin_layout Standard \lang english Finally, keep in mind that not all current graphics accelerators feature a 2D engine. Since 3D acceleration is technically a super-set of 2D acceleration, it is possible to implement 2D acceleration using the 3D engine (and this idea is one of the core ideas behind the Gallium 3D design, which will be detailed in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Gallium-3D" \end_inset ). And indeed some drivers use the 3D engine to implement 2D which allows GPU makers to completely part with the transistors otherwise dedicated to it. Yet some other cards do not dedicate the transistors, but microprogram 2D operations on top of 3D operations inside the GPU (this is the case for nVidia cards since nv10 and up to nv50, and for the Radeon R600 series which have an optional firmware that implements 2D on top of 3D). This sometimes has an impact on mixing 2D and 3D operations since those now share hardware units. \end_layout \begin_layout Subsubsection* \lang english 3D Engine \end_layout \begin_layout Standard \lang english A 3D engine is also called rasterization engine. It contains a series of stages which exchange data in a pipeline (1-directional ) fashion. \end_layout \begin_layout Standard \lang english vertex -> geom -> fragment \end_layout \begin_layout Standard \lang english graphics fifo \end_layout \begin_layout Standard \lang english DMA \end_layout \begin_layout Standard \lang english http://www.x.org/wiki/Development/Documentation/HowVideoCardsWork \end_layout \begin_layout Standard \lang english To attain better cache locality, the textures and surface are often tiled. Tiling means that the texture isn't stored linearly in GPU memory, but instead is stored so as to make pixels which are close in texture space also close in memory space. Examples are the Z-order curve and the Hilbert curve. \end_layout \begin_layout Subsubsection* \lang english Overlays and hardware sprites \end_layout \begin_layout Standard \lang english Overlays and hardware sprites happen at the scanout stage. \end_layout \begin_layout Standard \lang english Common use is for mouse pointer and play movies. \end_layout \begin_layout Standard \lang english On embedded platforms, the overlays can be used to avoid repeatedly blitting the same surface \end_layout \begin_layout Standard Overlay pixel formats, RGB/RGBA/YUV/YUVA... \end_layout \begin_layout Standard Which overlay to scanout from is usually picked by either a priority, or a colorkey \end_layout \begin_layout Subsubsection* \lang english Scanout \end_layout \begin_layout Standard \lang english The last stage of a graphics display is presenting the information onto a display device, or screen. \end_layout \begin_layout Standard \lang english Display devices are the last link of the graphics chain. They are charged with presenting the pictures to the user. \end_layout \begin_layout Standard \lang english digital vs analog signal \end_layout \begin_layout Standard \lang english hsync, vsync \end_layout \begin_layout Standard \lang english sync on green \end_layout \begin_layout Standard \lang english Connectors and encoders: CRTC,TMDS, LVDS, DVI-I, DVI-A, DVI-D, VGA (D-SUB 15 is the proper name) \end_layout \begin_layout Section \lang english Programming the card \end_layout \begin_layout Standard \lang english Each PCI card exposes a number of PCI resources; \emph on lspci -v \emph default lists these resources. These can be, but are not limited to, BIOSes, MMIO ranges, video memory (or only some part of it). As the total PCI resource size is limited, oftentimes a card will only expose part of its video memory as a resource, and the only way to access the remaining memory is through DMA from other, reachable areas (in a way similar to bounce pages). This is increasingly common as the video memory sizes keep growing while the PCI resource space stays limited. \end_layout \begin_layout Subsubsection* \lang english MMIO \end_layout \begin_layout Standard \lang english MMIO is the most direct access to the card. A range of addresses is exposed to the CPU, where each write goes directly to the GPU. This allows the simplest form of communication of commands from the CPU to the GPU. This type of programming is synchronous; writes are done by the CPU and executed on the GPU in a lockstep fashion. This leads to sub-par performance because each access turns into a packet on the bus and because the CPU has to wait for previous GPU commands to complete before submitting subsequent commands. For this reason MMIO is only used in the non-performance critical paths of today's drivers. \end_layout \begin_layout Subsubsection* \lang english DMA \end_layout \begin_layout Standard \lang english A direct memory access (DMA) is the use by a peripheral of the bus mastering feature of the bus. This allows one peripheral to talk directly to another, without intervention from the CPU. In the graphics card case, the two most common uses of DMAs are: \end_layout \begin_layout Itemize \lang english Transfers by the GPU to and from system memory (for reading textures and writing buffers). This allows implementing things like texturing over AGP or PCI, and hardware-ac celerated texture transfers. \end_layout \begin_layout Itemize \lang english The implementation of command FIFO. As MMIO between the CPU and GPU is synchronous and graphics drivers inherently use a lot of I/O, a faster means of communicating with the card is required. The command FIFO is a piece of memory (either system memory or more rarely video memory) shared between the graphics card and the CPU, where the CPU places command for later execution by the GPU. Then the GPU reads the FIFO asynchronously using DMA and executes the commands. This model allows asynchronous execution of the CPU and GPU command flows and thus leads to higher performance. \end_layout \begin_layout Subsubsection* \lang english Interrupts \end_layout \begin_layout Standard \lang english Interrupts are a way for hardware peripherals in general, and GPUs in particular , to signal events to the CPU. Usage examples for interrupts include signaling completion of a graphics command, signaling a vertical blanking event, reporting a GPU error, ... When an interrupt is raised by the peripheral, the CPU executes a small routine called an interrupt handler, which preempts other current executions. There is a maximum execution time for an interrupt handler, so the drivers have to keep it short (not more than a few microseconds). In order to execute more code, the common solution is to schedule a tasklet from the interrupt handler. \end_layout \begin_layout Section \lang english Graphics Hardware Examples \end_layout \begin_layout Subsection \lang english Forward renderers \end_layout \begin_layout Standard \lang english Forward renderers (i.e. classical renderers) are GPU which render the primitives as they are submitted to the rendering API, and for each of those primitives it is drawn entirely before moving on to the next one. This is the most straightforward way of rendering 3D primitives. This is the approach used in most desktop-range GPUs. \end_layout \begin_layout Subsubsection* \lang english ATI \end_layout \begin_layout Standard \lang english Shader engine 4+1 \end_layout \begin_layout Subsubsection* \lang english Nvidia \end_layout \begin_layout Standard \lang english NVidia hardware has multiple specificities compared to other architectures. The first one is the availability of multiple contexts, which is implemented using multiple command fifos (similar to what some high-end infiniband networking cards do) and a context switching mechanism to commute between those fifos. A small firmware is used for context switches between contexts, which is responsible for saving the graphics card state to a portion of memory and restoring another context. A scheduling system using the round robin algorithm handles the selection of the contexts, and the timeslice is programmable. \end_layout \begin_layout Standard \lang english The second specificity is the notion of graphics objects. Nvidia hardware features two levels of GPU access: the first one is at the raw level and is used for context switches, an the second one is the graphics objects which microprogram the raw level to achieve high level functionality (for example 2D or 3D acceleration). \end_layout \begin_layout Standard \lang english Shader engine nv40/nv50 \end_layout \begin_layout Standard \lang english http://nouveau.freedesktop.org/wiki/HonzaHavlicek \end_layout \begin_layout Subsection Deferred Renderers \end_layout \begin_layout Standard Deferred renderers are a different design for GPUs. Instead of rendering each 3D primitive as it is submitted by the rendering API, the driver stores it in memory and when it notices the end of a frame, it issues a single hardware call to render the whole scene. This has a number of advantages over classic architectures: \end_layout \begin_layout Itemize Much better rendering locality can be achieved by splitting the screen into tiles (usually in the \begin_inset Formula $16\times16$ \end_inset to \begin_inset Formula $32\times32$ \end_inset pixel range). The GPU can then iterate over these tiles, and for each of those can resolve per-pixel depth in an internal (mini) zbuffer. Once the whole tile is rendered it can be written back to video memory, saving precious bandwidth. Similarly, since visibility is determined before fetching texture data, only the useful texture data is read (again saving bandwidth) and the fragment shaders are only executed for visible fragments (which saves computation power). \end_layout \begin_layout Itemize If the depth buffer values are not required, they don't need to be written to memory. The depth buffer resolution can happen per-tile inside the GPU and never be written back to video memory, therefore saving video memory bandwith and space. \end_layout \begin_layout Standard Of course tiled renders require the ability to store the whole scene in memory before starting, and will also add latency since you need to wait for an end of frame even before you start drawing. The latency problem can be partially hidden by drawing a given frame on the GPU while the driver already allows the application to submit data for the next frame. However in some situations (readbacks, cross-process synchronization) it's not always possible to avoid it. \end_layout \begin_layout Standard All in all, the deferred renderers are particularly useful for embedded platforms where the bandwidth is generally very scarce and the applications are simple enough that the additional latency and the limitations of the approach don't matter. \end_layout \begin_layout Subsubsection* \lang english SGX \end_layout \begin_layout Standard \lang english The SGX is an example of a deferred rendering GPU. It uses a tiling architecture. \end_layout \begin_layout Standard \lang english The SGX shaders combine blending and depth test \end_layout \begin_layout Standard \lang english Another example of a deferred renderer is the Mali family of GPUs. \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english There are multiple memory domains in a computer, and they are not coherent. \end_layout \begin_layout Itemize \lang english A GPU is a completely separate computer with its own bus, address space and computational units. \end_layout \begin_layout Itemize \lang english Communication between the CPU and GPU is achieved over a bus, which has non-trivial performance implications. \end_layout \begin_layout Itemize \lang english GPUs can be programmed using two modes: MMIO and command FIFOs. \end_layout \begin_layout Itemize \lang english There is no standard output method for display devices. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english The Big Picture \begin_inset CommandInset label LatexCommand label name "cha:The-Big-Picture" \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english X, how it works (encapsulating) with indirect (glx) 3D with kernel FB + picture. This is how utah-glx used to work. \end_layout \begin_layout Plain Layout \lang english DRI : bypassing encapsulation for performance-critical operations with kernel FB + picture \end_layout \end_inset \end_layout \begin_layout Standard \lang english The Linux graphics stack has seen numerous evolutions over the years. The purpose of this section is to detail that history, as well as give the justification behind the changes which have been made over the years. Today, the design is still strongly rooted in that history, and this section will explain that historyto better motivate the current design of the Linux graphics stack. \end_layout \begin_layout Section \lang english The X11 infrastructure \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement tbh wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (application) {Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, below=of application] (xlib) {Xlib}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (application.south) -> (xlib.north); \end_layout \begin_layout Plain Layout \backslash draw (1,-1) rectangle (6,-5.2); \end_layout \begin_layout Plain Layout \backslash node at (3.5,-1.2) {X server}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=2cm of xlib] (xserver) {DIX}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xlib.east) -> (xserver.west); \end_layout \begin_layout Plain Layout \backslash node[mynode, below=1cm of xserver] (driver) {DDX (Driver)}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xserver.south) -> (driver.north); \end_layout \begin_layout Plain Layout \backslash node[mynode, below=1cm of driver] (hardware) {Hardware}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (driver.south) -> (hardware.north); \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english The X11 architecture. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english DIX (Device-Independent X), DDX (Device-Dependent X), \end_layout \begin_layout Standard \lang english modules \end_layout \begin_layout Standard \lang english Xlib \end_layout \begin_layout Standard \lang english socket \end_layout \begin_layout Standard \lang english X protocol \end_layout \begin_layout Standard \lang english X extensions \end_layout \begin_layout Standard \lang english shm -> shared memory for transport \end_layout \begin_layout Standard \lang english XCB -> asynchronous \end_layout \begin_layout Standard \lang english Another notable X extension is Xv, which will be discussed in further detail in the video decoding chapter. \end_layout \begin_layout Section \lang english The DRI/DRM infrastructure \end_layout \begin_layout Standard \lang english Initially (when Linux first supported graphics hardware acceleration), only a single piece of code would access the graphics card directly: the XFree86 server. The design was as follows: by running with super-user privileges, the XFree86 server could access the card from user space and did not require kernel support to implement 2D acceleration. The advantage of such a design was its simplicity, and the fact that the XFree86 server could be easily ported from one operating system to another since it required no kernel component. For years this was the most widespread X server design (although there were notable exceptions, like XSun which implemented modesetting in the kernel for some drivers). \end_layout \begin_layout Standard \lang english Later on, Utah-GLX, the first hardware-independent 3D accelerated design, came to Linux. Utah-GLX basically consists in an additional user space 3D driver implementing GLX, and directly accesses the graphics hardware from user space, in a way similar to the 2D driver. In a time where the 3D hardware was clearly separated from 2D (because the functionality used for 2D and 3D was completely different, or because the 3D card was a completely separate card, à la 3Dfx), it made sense to have a completely separate driver. Furthermore, direct access to the hardware from user space was the simplest approach and the shortest road to getting 3D acceleration going under Linux. \end_layout \begin_layout Standard \lang english At the same time, framebuffer drivers (which will be detailed in Chapter \begin_inset CommandInset ref LatexCommand ref reference "cha:Framebuffer-Drivers" \end_inset ) were getting increasingly widespread, and represented another component that could simultaneously access the graphics hardware directly. To avoid potential conflicts between the framebuffer and XFree86 drivers, it was decided that on VT switches the kernel would emit a signal to the X server telling it to save the graphics hardware state. Asking each driver to save its complete GPU state on VT switches made the drivers more fragile, and life became more difficult for developers who suddenly faced bug-prone interaction between different drivers. Keep in mind that there were at least two possibilities for XFree86 drivers (xf86-video-vesa and a native XFree86 driver) and two possibilities for kernel framebuffer drivers (vesafb and a native framebuffer driver), so each GPU had at least four combinations of cohabitating drivers. \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english Tikz documentation : http://www.texample.net/tikz/examples/ \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement H wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (x11application) {X11 Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of x11application] (glapplication) {OpenGL Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of glapplication] (fbapplication) {Framebuffer Applicati on}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below=1cm of x11application, xshift = 1.7cm] (xorg) {XFree86}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (x11application.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift=-2cm] (2ddriver) {2D Driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) ++ (-2,0) -> ++(0,-1) (2ddriver); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift= 2cm] (glxdriver) {Utah GLX driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) ++(2,0) -> ++(0,-1) (glxdriver); \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 12cm , below=3cm of 2ddriver, xshift=5cm] (hardware) {Graphics Hardware}; \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-5.2) -- (11,-5.2); \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-7.2) -- (11,-7.2); \end_layout \begin_layout Plain Layout \backslash node at (4.6,-1) {GLX}; \end_layout \begin_layout Plain Layout \backslash node at (10,-5) {User Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7) {Kernel Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7.5) {Hardware}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glxdriver.south) -> ++(0,-3.0) (hardware); \end_layout \begin_layout Plain Layout \backslash node[mynode, below=5.1cm of fbapplication] (fbdriver) {FB driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbapplication) -> (fbdriver); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbdriver.south) -> ++(0,-1) (hardware); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (2ddriver.south) -> ++(0,-3) (hardware); \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:Early-implementation-of" \end_inset Early implementation of the Linux graphics stack using Utah-GLX. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english Obviously, this model had drawbacks. First, it required that unprivileged user space applications be allowed to access the graphics hardware for 3D. Second, as can be seen on Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:Early-implementation-of" \end_inset all GL acceleration had to be indirect through the X protocol, which would slow it down significantly, especially for data-intensive functions like texture uploads. Because of growing concerns about the security in Linux and performance shortcomings, another model was required. \end_layout \begin_layout Standard \lang english To address the reliability and security concerns with the Utah-GLX model, the DRI model was put together; it was used in both XFree86 and its successor, X.Org. This model relies on an additional kernel component whose duty is to check the correctness of the 3D command stream, security-wise. The main change is now that instead of accessing the card directly, the unprivileged OpenGL application would submit command buffers to the kernel, which would check them for security and then pass them to the hardware for execution. The advantage of this model is that trusting user space is no longer required. Notice that although this would have been possible, the 2D command stream from XFree86 still did not go through the DRM, and therefore the X server still required super-user privileges to be able to map the GPU registers directly. \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement H wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (x11application) {X11 Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of x11application] (glapplication) {OpenGL Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of glapplication] (fbapplication) {Framebuffer Applicati on}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below=1cm of x11application, xshift = 1.7cm] (xorg) {X.Org}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (x11application.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift=-2cm] (2ddriver) {2D Driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) ++ (-2,0) -> ++(0,-1) (2ddriver); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift= 2cm] (glxdriver) {OpenGL DRI driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) ++(1.3,0) -> ++(0,-3.1) (glxdriver); \end_layout \begin_layout Plain Layout \backslash node at (6,-2.1) {DRI}; \end_layout \begin_layout Plain Layout \backslash node[mynode, below = 0.9cm of glxdriver] (drm) {DRM}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glxdriver.south) -> ++(0,-0.9) (drm); \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 12cm , below=3cm of 2ddriver, xshift=5cm] (hardware) {Graphics Hardware}; \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-5.2) -- (11,-5.2); \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-7.2) -- (11,-7.2); \end_layout \begin_layout Plain Layout \backslash node at (4.6,-1) {GLX}; \end_layout \begin_layout Plain Layout \backslash node at (10,-5) {User Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7) {Kernel Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7.5) {Hardware}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbapplication) -> ++(0,-5.65) (drm); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (2ddriver.south) -> ++(0,-3) (drm); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (drm.south) -> ++(0,-1.0) (hardware); \end_layout \begin_layout Plain Layout \backslash node[mynode, below=5.1cm of fbapplication] (fbdriver) {FB driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbapplication) -> (fbdriver); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbdriver.south) -> ++(0,-1) (hardware); \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english The old picture of the Linux graphics stack. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english The current stack evolved from a new set of needs. First, requiring the X server to have super-user privileges always had serious security implications. Second, with the previous design different drivers were touching a single piece of hardware, which would often cause issues. In order to resolve this the key is two-fold: first, merge the kernel framebuff er functionality into the DRM module and second, have X.Org access the graphics card through the DRM module and run unprivileged. This is called Kernel Modesetting (KMS); in this model the DRM module is now responsible for providing modesetting services both as a framebuffer driver and to X.Org. \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement H wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (x11application) {X11 Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of x11application] (glapplication) {OpenGL Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of glapplication] (fbapplication) {Framebuffer Applicati on}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below=1cm of x11application, xshift = 1.7cm] (xorg) {X.Org}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (x11application.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) -> ++(0,-1) (xorg); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift=-2cm] (2ddriver) {2D Driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) ++ (-2,0) -> ++(0,-1) (2ddriver); \end_layout \begin_layout Plain Layout \backslash node[mynode, below = of xorg, xshift= 2cm] (glxdriver) {OpenGL DRI driver}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) ++(1,0) -> ++(0,-1) (glxdriver); \end_layout \begin_layout Plain Layout \backslash node at (3.5,-3.1) {AIGLX}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) ++(1.3,0) -> ++(0,-3.1) (glxdriver); \end_layout \begin_layout Plain Layout \backslash node at (6,-2.1) {DRI}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 12cm, below = 0.9cm of glxdriver, xshift = 1cm] (drm) {DRM}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glxdriver.south) -> ++(0,-0.9) (drm); \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 12cm , below=3cm of 2ddriver, xshift=5cm] (hardware) {Graphics Hardware}; \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-5.2) -- (11,-5.2); \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-7.2) -- (11,-7.2); \end_layout \begin_layout Plain Layout \backslash node at (4.6,-1) {GLX}; \end_layout \begin_layout Plain Layout \backslash node at (10,-5) {User Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7) {Kernel Space}; \end_layout \begin_layout Plain Layout \backslash node at (10,-7.5) {Hardware}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbapplication) -> ++(0,-5.65) (drm); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (2ddriver.south) -> ++(0,-0.9) (drm); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (drm.south) -> ++(0,-1.0) (hardware); \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english The new picture of the Linux graphics stack. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english VT switches \end_layout \begin_layout Standard \lang english http://dri.sourceforge.net/doc/dri_data_flow.html \end_layout \begin_layout Standard \lang english http://dri.sourceforge.net/doc/dri_control_flow.html \end_layout \begin_layout Standard \lang english http://nouveau.freedesktop.org/wiki/GraphicStackOverview \end_layout \begin_layout Standard \lang english http://people.freedesktop.org/~ajax/dri-explanation.txt \end_layout \begin_layout Standard \lang english http://dri.sourceforge.net/doc/DRIintro.html \end_layout \begin_layout Standard \lang english http://jonsmirl.googlepages.com/graphics.html \end_layout \begin_layout Standard \lang english http://wiki.x.org/wiki/Development/Documentation/Glossary \end_layout \begin_layout Standard \lang english http://mjules.littleboboy.net/carnet/index.php?post/2006/11/15/89-comment-marche-x1 1-xorg-et-toute-la-clique-5-partie \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Applications communicate with X.Org through a specific library which encapsulates drawing calls. \end_layout \begin_layout Itemize \lang english The current DRI design has evolved over time in a number of significant steps. \end_layout \begin_layout Itemize \lang english In a modern stack, all graphics hardware activity is moderated by a kernel module, the DRM. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Framebuffer Drivers \begin_inset CommandInset label LatexCommand label name "cha:Framebuffer-Drivers" \end_inset \end_layout \begin_layout Standard \lang english Framebuffer drivers are the simplest form of graphics drivers under Linux. A framebuffer driver is a kernel graphics driver exposing its interface through /dev/fb* (generally /dev/fb0). This interface implements limited functionality (basically it allows setting a video mode and drawing to a linear framebuffer), and the framebuffer drivers are therefore extremely simple and easy to create. Despite their simplicity, framebuffer drivers are still a relevant option if the only thing you are after is a basic two-dimensional display with no bells and whistles. It is also useful to know how framebuffer drivers work when implementing framebuffer acceleration for a kernel modesetting DRM driver, as the accelerati on callbacks are the same. In short, framebuffer drivers are especially interesting for embedded systems, where memory footprint is essential, or when the intended applications do not require advanced graphics acceleration. \end_layout \begin_layout Standard \lang english At the core, a framebuffer driver implements the following functionality: \end_layout \begin_layout Itemize \lang english Modesetting. Modesetting consists in configuring video mode to get a picture on the screen. This includes choosing the video resolution and refresh rates. \end_layout \begin_layout Itemize \lang english Optional 2d acceleration. Framebuffer drivers can provide basic 2D acceleration used to accelerate the linux console. These operations include copies in video memory and solid fills. Acceleration is sometimes made available to user space through a hook (the user space must then program card specific MMIO registers, and this requires root privileges). \end_layout \begin_layout Standard \lang english By implementing only these two pieces, framebuffer drivers remain the simplest and most amenable form of linux graphics drivers. Framebuffer drivers do not always rely on a specific card model (like nvidia or ATI). Drivers on top of vesa, EFI or Openfirmware exist; instead of accessing the graphics hardware directly, these drivers call firmware functions to achieve modesetting and 2D acceleration. \end_layout \begin_layout Standard \lang english http://www.linux-fbdev.org/HOWTO/index.html \end_layout \begin_layout Section \lang english Creating a framebuffer driver \end_layout \begin_layout Standard \lang english struct platform_driver with a probe function \end_layout \begin_layout Standard \lang english probe function in charge of creating the fb_info struct and register_framebuffer () on it. \end_layout \begin_layout Section \lang english Framebuffer operations \end_layout \begin_layout Standard \lang english The framebuffer operations structure is how non-modesetting framebuffer callbacks are set. Different callbacks can be set depending on what functionality you wish to implement, like fills, copies, or cursor handling. By filling struct fb_ops callbacks, one can implement the following functions: \end_layout \begin_layout Subsubsection* \lang english Set color register \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_setcolreg(unsigned regno, unsigned red, unsigned green, unsigned blue, unsigned transp, struct fb_info *info); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Set color registers in batch \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_setcmap(struct fb_cmap *cmap, struct fb_info *info); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Blank display \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_blank(int blank, struct fb_info *info); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Pan display \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_pan_display(struct fb_var_screeninfo *var, struct fb_info *info); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Draws a solid rectangle \end_layout \begin_layout Standard \lang english Draws a solid rectangle on the framebuffer. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void fb_fillrect(struct fb_info *info, const struct fb_fillrect *rect); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Copy data from area to another \end_layout \begin_layout Standard \lang english This is mostly useful for accelerating console scrolling. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void fb_copyarea(struct fb_info *info, const struct fb_copyarea *region); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Draws an image to the display \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void fb_imageblit(struct fb_info *info, const struct fb_image *image); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Draws cursor \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_cursor(struct fb_info *info, struct fb_cursor *cursor); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Rotates the display \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void fb_rotate(struct fb_info *info, int angle); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Wait for blit idle, optional \end_layout \begin_layout Standard \lang english It will be called by the cfb_* fallback functions if it's available. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout int fb_sync(struct fb_info *info); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english Note that common framebuffer functions (cfb) are available if you do not want to implement everything for your device specifically. These functions are cfb_fillrect, cfb_copyarea and cfb_imageblit and will perform the corresponding function in a generic, unoptimized fashion using the CPU. \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Framebuffer drivers are the simplest form of linux graphics driver, requiring little implementation work. \end_layout \begin_layout Itemize \lang english Framebuffer drivers deliver a low memory footprint and thus are useful for embedded devices. \end_layout \begin_layout Itemize \lang english Implementing acceleration is optional as software fallback functions exist. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english The Direct Rendering Manager \begin_inset CommandInset label LatexCommand label name "cha:The-DRM-Kernel" \end_inset \end_layout \begin_layout Standard \lang english The use of a kernel module is a requirement in a complex world. This kernel module is called the Direct Rendering Manager (DRM, not to be confused with Digital Rights Management) and serves multiple purposes: \end_layout \begin_layout Itemize \lang english Put critical initialization of the card in the kernel, for example uploading firmwares or setting up DMA areas. \end_layout \begin_layout Itemize \lang english Share the rendering hardware between multiple user space components, and arbitrate access. \end_layout \begin_layout Itemize \lang english Enforce security by preventing applications from performing DMA to arbitrary memory regions, and more generally from programming the card in any way that could result in a security hole. \end_layout \begin_layout Itemize \lang english Manage the memory of the card, by providing video memory allocation functionalit y to user space. \end_layout \begin_layout Itemize \lang english More recently, the DRM was improved to achieve modesetting. This simplifies the previous situation where both the DRM and the framebuffer driver were fighting to access the same GPU. Instead, the framebuffer driver has been removed and framebuffer support is implemented from within the DRM. \end_layout \begin_layout Standard \lang english Kernel module (DRM) \end_layout \begin_layout Standard \lang english Global DRI/DRM user space/kernel scheme (figure with libdrm - drm - entry points - multiple user space apps) \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement H wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \backslash node[mynode] (xorg) {X.Org}; \end_layout \begin_layout Plain Layout \backslash node[mynode, right=0.5cm of xorg] (glapplication) {OpenGL Application}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below= of xorg, xshift = 2.2cm] (libdrm) {libdrm}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (xorg.south) -> ++(0,-1) (libdrm); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (glapplication.south) -> ++(0,-1) (libdrm); \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below= of libdrm] (drm) {drm}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (libdrm.south) -> ++(0,-1) (drm); \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 6cm, below= of drm] (hardware) {Graphics Hardware}; \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (drm.south) -> ++(0,-1.0) (hardware); \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-3.2) -- (9,-3.2); \end_layout \begin_layout Plain Layout \backslash draw [thick, dotted] (-1.8,-5.2) -- (9,-5.2); \end_layout \begin_layout Plain Layout \backslash node at (8,-3) {User Space}; \end_layout \begin_layout Plain Layout \backslash node at (8,-5) {Kernel Space}; \end_layout \begin_layout Plain Layout \backslash node at (8,-5.5) {Hardware}; \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english Accessing the DRM through libdrm. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english When designing a Linux graphics driver aiming for more than simple framebuffer support, a DRM component is the first thing to do. One should derive a design that is both efficient and enforces security. The DRI/DRM scheme can be implemented in different ways and the interface is indeed entirely card-specific. Do not always follow the existing models that other drivers use, innovate! \end_layout \begin_layout Section \lang english DRM batch buffer submission model \end_layout \begin_layout Standard \lang english At the core of the DRM design is the DRM_GEM_EXECBUFFER ioctl; which lets a user space application submit a batch buffer to the kernel, which in turns puts it on the GPU. This ioctl allows many things like sharing the hardware, managing memory and enforcing memory protection. \end_layout \begin_layout Subsection \lang english Hardware sharing \end_layout \begin_layout Standard \lang english One of the duties of the DRM is to multiplex the GPU itself between multiple user space processes. \lang american Given that the GPU holds the graphics state, a problem arises when multiple applications use the same GPU: if nothing is done, the applications can stomp over each other's state. Depending on the hardware, there are two main cases: \end_layout \begin_layout Itemize When the GPU features hardware state tracking, the hardware sharing is simpler since each application can send to a separate context, and the GPU itself tracks each application's state. This is the way the nouveau driver works. \end_layout \begin_layout Itemize \lang english When the GPU doesn't have multiple hardware contexts, the common way of multiplexing the hardware is to reemit the state at the begining of each batch buffer; it's the way the intel and radeon drivers multiplex the GPU. Note that this duty of reemitting the state relies on user space entirely. If the user space doesn't reemit the state at the begining of each batch buffer, the state from other DRM processes will leak into it. \end_layout \begin_layout Standard \lang english The DRM also prevents simultaneous access to the same hardware. \end_layout \begin_layout Subsection \lang english Memory management and security \end_layout \begin_layout Standard \lang english The kernel has the ability to move memory areas around to handle high memory pressure situations. Depending on the hardware, there are two ways to do this: \end_layout \begin_layout Itemize If the hardware has complete memory protection and virtualization, then it is possible to page memory resources into the GPU as they get allocated and isolate them per-process. In this case not much is required to support memory protection for GPU memory. \end_layout \begin_layout Itemize \lang english When the hardware doesn't have memory protection, this can still be achieved entirely in the kernel, transparently to user space. In order to allow relocations to work for a user space process which is otherwise unaware of them, the command submission ioctl will rewrite the command buffers in the kernel by replacing all the hardware offsets to their current locations. This is possible since the kernel knows about the current position of all memory buffers. \begin_inset Newline newline \end_inset To prevent access to arbitrary GPU memory, the command submission ioctl can also check that each of these offsets is owned by the calling process, and reject the batch buffer if it isn't. That way it is possible to implement memory protection when the hardware doesn't provide the functionality. Although there is some overhead for doing command buffer parsing, it is bearable, and this is how the open source radeon drivers work. \end_layout \begin_layout Subsubsection GEM Buffer management \end_layout \begin_layout Standard GPU buffer allocation and management is done in the kernel and exposed to the userspace through the GEM interface. The resulting buffer is then called an object or a buffer object (BO). \end_layout \begin_layout Standard GEM was introduced by Intel in 2008 to simplify Tungsten Graphics' TTM (Translat ion Table Maps) and became the standard way to allocate, read, write and sharing BOs between applications. It is however impossible to share buffers between drivers through GEM. \end_layout \begin_layout Standard As the GEM interface is driver-dependent, it is best to use the driver-dependent libdrm calls to manage your buffers. However, the general outline of the interface remains fairly constant. \end_layout \begin_layout Subsubsection* \lang english Object Allocation & Destruction \end_layout \begin_layout Standard Object allocation is done by calling the GEM_CREATE function. It should take the required size of the buffer as an argument and return a handle that is non-zero. This handle will be used later on as a buffer identifier. \end_layout \begin_layout Subsubsection* \lang english Reading and writing from/to a buffer object \end_layout \begin_layout Standard There are two ways to read/write from/to a buffer object: \end_layout \begin_layout Itemize pread/pwrite : syscalls to read/write to/from a buffer object. They both take the BO offset, the size and where the data should be read from/written to; \end_layout \begin_layout Itemize mapping the BO : it is possible for the userspace to map the BO in its memory space. Mapping takes the BO offset and size that should be mapped as an argument. It then returns the address of the BO in the user's virtual address space. \end_layout \begin_layout Subsubsection* Sharing buffer objects between applications \end_layout \begin_layout Standard The GEM interface specifies a way for applications/clients to share buffers. In order to share a buffer, a client should: \end_layout \begin_layout Enumerate Call GEM_FLINK with the handle of the buffer object to be shared. A uint32 name will be returned. \end_layout \begin_layout Enumerate Transfer this name to the clients that should access the buffer object. \end_layout \begin_layout Enumerate The remote clients should open the BO by name, using the GEM_OPEN IOCTL. \end_layout \begin_layout Standard By default, no buffers are shared between clients. However, when one buffer is shared, any authenticated client can read/write the BO. No access control is done between authenticated clients. \end_layout \begin_layout Standard A client is said to be authenticated if its euid is 0 (root) or if an already-au thenticated client added this client to the authenticated-clients list. \end_layout \begin_layout Standard The procedure to authenticate a non-root client is: \end_layout \begin_layout Enumerate Get a magic cookie from the dri interface (drm_getmagic). \end_layout \begin_layout Enumerate Send the magic cookie to an already-authenticated client. \end_layout \begin_layout Enumerate The authenticated client calls drm_authmagic to authenticate the cookie and thus the original client. \end_layout \begin_layout Subsubsection* Learning more about GEM \end_layout \begin_layout Standard The best documentation for GEM is the source code of the drivers exposing it (libdrm for userspace, the drm driver for kernelspace). However, it is worth reading the initial proposal called \begin_inset CommandInset href LatexCommand href name "GEM - the Graphics Execution Manager" target "https://lwn.net/Articles/283798/" \end_inset from Intel. \end_layout \begin_layout Subsubsection \lang english DMA-Buf : Sharing buffer objects between drivers \end_layout \begin_layout Standard \lang english DMA-Buf is a driver-independent buffer-sharing mechanism for clients. \end_layout \begin_layout Standard \lang english Unlike GEM buffer sharing, with DMA-Buf a fd is returned to the client willing to export a buffer. It is then the client's responsibility to export this fd to the other clients using a UNIX socket as explained by this \begin_inset CommandInset href LatexCommand href name "email from Greg A. Woods" target "http://archives.neohapsis.com/archives/postfix/2000-09/1476.html" \end_inset . This sharing solution is more secure since clients cannot access buffers unless explicitly allowed by the exporter, whereas with GEM names are guessable. \end_layout \begin_layout Standard \lang english For more information, see /Documentation/dma-buf-sharing.txt in your kernel tree. \end_layout \begin_layout Section \lang english Modesetting \end_layout \begin_layout Standard \lang english Modesetting is the act of setting a mode on the card to display. This can range from extremely simple procedures (calling a VGA interrupt or VESA call is a basic form of modesetting) to directly programming the card registers (which brings along the advantage of not needing to rely on a VGA or VESA layer). Historically, this was achieved in user space by the DDX. \end_layout \begin_layout Standard \lang english However, these days it makes more sense to put it in the kernel once and for all, and share it between different GPU users (framebuffer drivers, DDXes, EGL stacks...). This extension to modesetting is called kernel modesetting (also known as KMS). A number of concepts are used by the modesetting interface (those concepts are mainly inherited from the Randr 1.2 specification). \end_layout \begin_layout Subsubsection* \lang english Crtc \end_layout \begin_layout Standard \lang english Crtc is in charge of reading the framebuffer memory and routes the data to an encoder. A CRTC can usually be connected to any encoder. The number of connectable screens is usually limited by the number of CRTCs. Most cards used to be limited to 2 CRTCs but AMD introduced eyefinity in 2010 which increased their CRTC count to 6. NVidia bumped the number of CRTC from 2 to 4 on Kepler. Matrox is also known for being able to drive up to 12 screens. \end_layout \begin_layout Subsubsection* \lang english Encoder \end_layout \begin_layout Standard \lang english Encoder encodes the pixel data for a connector. \end_layout \begin_layout Subsubsection* \lang english Connector \end_layout \begin_layout Standard \lang english The connector is name given to the physical output on the card (DVI, Dsub, Svideo...). Notice that connectors can get their data from multiple encoders (for example DVI-I which can feed both analog and digital signals) \end_layout \begin_layout Standard \lang english Also, on embedded or old hardware, it is common to have encoders and connectors merged for simplicity/power efficiency reasons. \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement H wide false sideways false status open \begin_layout Plain Layout \align center \begin_inset Graphics filename crtc-encoder-connector.dot \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english CRTC - Encoder - Connector representation \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Section \lang english libdrm \end_layout \begin_layout Standard \lang english libdrm is a small (but growing) component that interfaces between user space and the DRM module, and allows calling into the entry points. \end_layout \begin_layout Standard \lang english Obviously security should not rely on components from libdrm because it is an unprivileged user space component \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english The DRM manages all graphics activity in a modern linux graphics stack. \end_layout \begin_layout Itemize \lang english It is the only trusted piece of the stack and is responsible for security. Therefore it shall not trust the other components. \end_layout \begin_layout Itemize \lang english It provides basic graphics functionality: modesetting, framebuffer driver, memory management. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english X.Org Drivers \begin_inset CommandInset label LatexCommand label name "cha:X.Org-Drivers" \end_inset \end_layout \begin_layout Standard \lang english This chapter covers the implementation of a 2D driver for the Xorg server. Xorg loads drivers from shared object files depending on the hardware detected in the system and any xorg.conf configuration files found at startup. There are multiple ways to implement a 2D X.Org driver: ShadowFB, XAA, EXA. \end_layout \begin_layout Standard \lang english One simple way of implementing X.Org support is through the xf86-video-fbdev module. This module implements X.Org on top of an existing, in-kernel framebuffer driver. It can be a \begin_inset Quotes eld \end_inset good enough \begin_inset Quotes erd \end_inset option if all you need is basic X compatibility. Similiarly the xf86-video-modesetting driver implements X.Org on top of an existing in-kernel KMS driver, again without hardware specific acceleration. \end_layout \begin_layout Standard \lang english Gallium Xorg state tracker \end_layout \begin_layout Standard \lang english http://www.x.org/wiki/DriverDevelopment \end_layout \begin_layout Section \lang english Creating a basic driver \end_layout \begin_layout Standard \lang english Mandatory entry points \end_layout \begin_layout Subsubsection* \lang english PreInit \end_layout \begin_layout Standard \lang english This function is in charge of the initialization. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool PreInit (ScrnInfoPtr pScreen, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english ScreenInit \end_layout \begin_layout Standard \lang english This function gets called on startup. It is responsible for setting up all per-screen state. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool ScreenInit(int scrnIndex, ScreenPtr screen, int argc, char **argv); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english EnterVT \end_layout \begin_layout Standard \lang english This is called when VT switching back to the X server. In a KMS-enabled X driver, this will only need to acquire the DRM master bit and set the video mode. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool EnterVT(int scrnIndex, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english LeaveVT \end_layout \begin_layout Standard \lang english This is called when switching away from the X server to another VT. In a KMS-enabled X driver, this only needs to drop the DRM master bit. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool LeaveVT(int scrnIndex, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english Optional functions (but very useful) \end_layout \begin_layout Subsubsection* \lang english SwitchMode \end_layout \begin_layout Standard \lang english Sets a video mode. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool SwitchMode(int scrnIndex, DisplayModePtr mode, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english AdjustFrame \end_layout \begin_layout Standard \lang english This function is used to initialize the Start Address - the first displayed location in the video memory (randr 1.3 panning) \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void AdjustFrame(int scrnIndex, int x, int y, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english FreeScreen \end_layout \begin_layout Standard \lang english Cleanup the ScreenInit \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void FreeScreen(int scrnIndex, int flags); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Section \lang english ShadowFB acceleration \end_layout \begin_layout Standard \lang english ShadowFB provides no acceleration proper, a copy of the framebuffer is kept in system memory. The driver implements a single hook that copies graphics from system to video memory. This can be implemented using either a DMA copy, or a CPU copy (depending on the hardware and copy size, either can be better). \end_layout \begin_layout Standard \lang english Despite the name, shadowFB is not to be confused with the kernel framebuffer drivers. \end_layout \begin_layout Standard \lang english Although ShadowFB is a very basic design, it can result in a more efficient and responsive desktop than an incomplete implementation of EXA. \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english Insérer une image avec la propagation shadowfb \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (0,2) rectangle (5,6); \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (6,2) rectangle (11,6); \end_layout \begin_layout Plain Layout % \backslash draw[<->] (2,7.5) -- +(6.5,0); \end_layout \begin_layout Plain Layout % \backslash draw[<->] (1.5,2) -- +(0,5); \end_layout \begin_layout Plain Layout % \backslash draw[<->] (2,1.5) -- +(8,0); \end_layout \begin_layout Plain Layout \backslash node at (2.5,1.5) {Shadow surface}; \end_layout \begin_layout Plain Layout \backslash node at (8.5,1.5) {Video ram surface}; \end_layout \begin_layout Plain Layout % source pixels \end_layout \begin_layout Plain Layout \backslash draw (2,2.5) rectangle (4,4); \end_layout \begin_layout Plain Layout \backslash node at (3,3) {Dirty pixels}; \end_layout \begin_layout Plain Layout % destination pixels \end_layout \begin_layout Plain Layout \backslash draw (8,2.5) rectangle (10,4); \end_layout \begin_layout Plain Layout \backslash node at (9,3) {Dst pixels}; \end_layout \begin_layout Plain Layout % fleches de copie \end_layout \begin_layout Plain Layout \backslash draw[->] (3,3.25) -- +(6,0); \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english Shadowfb acceleration. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard To implement shadowFB acceleration, a driver simply calls Bool ShadowFBInit(Scre enPtr pScreen, RefreshAreaFuncPtr refreshArea ). refreshArea is a function pointer with the following profile: \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void RefreshAreaFuncPtr(ScrnInfoPtr pScreen, int numBoxes, BoxPtr pBox); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard The callback should refresh numBoxes contained in the pBox[] array. It can be achieved either with a CPU copy to video memory or with a DMA on the GPU. \end_layout \begin_layout Section \lang english XAA acceleration \end_layout \begin_layout Standard \lang english XAA was the XFree86 Acceleration Architecture, an interface inside Xorg implemented by drivers for 2D acceleration. It was created for XFree86 3.3, and was rewritten for XFree86 4.0. EXA was developed as a replacement, and after years of offering both interface s, XAA was removed from the Xorg server after the Xorg 1.12 release. XAA is thus useful only for maintaining existing drivers and new drivers should write to another interface instead. \end_layout \begin_layout Standard \lang english Scanline based acceleration \end_layout \begin_layout Standard \lang english Offscreen area, same pitch as the screen \end_layout \begin_layout Section \lang english EXA acceleration \end_layout \begin_layout Standard \lang english EXA is an interface inside X.Org implemented by drivers for 2D acceleration. It was originally designed as KAA in the Kdriver X server, and then was adapted into X.Org. The interface used is pretty simple; for each acceleration function three hooks are available: PrepareAction, Action and FinishAction. PrepareAction is called once before the operation is used. Action can be called many times in a row after a single PrepareAction call for different surfaces. FinishAction is called after all the Action calls have been made. The number of Action calls can be just one or many, but the PrepareAction and FinishAction function will always be called once, first and last. The PrepareAction functions return a boolean, and can return false if they fail at accelerating the specific type of operation, in which case a software fallback is used instead. Otherwise the function returns true and subsequent Action calls are expected to succeed. \end_layout \begin_layout Standard \lang english EXA is implemented in the driver as a series of callbacks; the following gives a detail of the EXA acceleration functions that a driver should implement ; some of them like Composite() are optional. \end_layout \begin_layout Subsubsection* \lang english Solid \end_layout \begin_layout Standard \lang english Solid just fills an area with a solid color (RGBA). Just like with all EXA callbacks, there are three main functions: the first one is the Prepare() function which sets the graphics state required to do solid fills. The second one is the Solid() function which actually does the solid fill. The last one is Done() which signals to the driver that the current series of Solid() calls is over, so that it can restore graphics state and/or flush required GPU states. \end_layout \begin_layout Standard Note that Solid() can be called many times in a row between a Prepare() and Done() (the same applies to other EXA callbacks). \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool PrepareSolid(PixmapPtr pPixmap, int alu, Pixel planemask, Pixel fg); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void Solid(PixmapPtr pPixmap, int x1, int y1, int x2, int y2); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void DoneSolid(PixmapPtr pPixmap); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Copy \end_layout \begin_layout Standard \lang english Copy is a simple blit function; it copies a rectangle area in video memory from one pixmap to another pixmap (possibly the same one). Just like with Solid() it has a Prepare/Copy/Done triplet of callbacks, and Copy() can be called many times in a row. \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool PrepareCopy(PixmapPtr pSrcPixmap, PixmapPtr pDstPixmap, int dx, int dy, int alu, Pixel planemask); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void Copy(PixmapPtr pDstPixmap, int srcX, int srcY, int dstX, int dstY, int width, int height); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout void DoneCopy(PixmapPtr pDstPixmap); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english Composite \end_layout \begin_layout Standard \lang english Composite is an optional interface used to achieve composite operations like blending. This allows accelerating 2D desktop effects like blending, scaling, operations with masks... All in all, the composite() hook is sufficient to implement a basic 2D composite window manager (KDE and XFCE are examples of window compositors indirectly using EXA trough the Render API to implement compositing). \end_layout \begin_layout Standard \lang english If the driver doesn't support the required operation, it is free to return false, in which case the EXA layer will call into the pixman library as a software fallback. Of course this will be done on the CPU as a fallback. \end_layout \begin_layout Subsubsection* \lang english UploadToScreen \end_layout \begin_layout Standard \lang english UploadToScreen copies an area from system memory to video memory \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool UploadToScreen(PixmapPtr pDst, int x, int y, int w, int h, char *src, int src_pitch); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english DownloadFromScreen \end_layout \begin_layout Standard \lang english DownloadFromScreen copies an area from video memory to system memory \end_layout \begin_layout Standard \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{lstlisting}{} \end_layout \begin_layout Plain Layout Bool DownloadFromScreen(PixmapPtr pSrc, int x, int y, int w, int h, char *dst, int dst_pitch); \end_layout \begin_layout Plain Layout \backslash end{lstlisting}{} \end_layout \end_inset \end_layout \begin_layout Subsubsection* \lang english PrepareAccess \end_layout \begin_layout Standard \lang english PrepareAccess makes the pixmap accessible from the CPU. This includes mapping it into memory, copying it from unmappable video memory, untiling the pixmap... What this exactly does is very dependent from the GPU, but the core of the matter is that you must provide a piece of CPU-accessible memory which is stored in a linear form. This can be achieved by either mapping GPU memory into the CPU domain with a linear view, or by doing a copy from GPU to CPU memory. \end_layout \begin_layout Subsubsection* \lang english FinishAccess \end_layout \begin_layout Standard \lang english FinishAccess is called once the pixmap is done being accessed, and must undo what PrepareAccess did to make the pixmap usable by the GPU again. \end_layout \begin_layout Subsubsection* \lang english A note about EXA performance \end_layout \begin_layout Standard \lang english EXA Pixmap migration. EXA tries to be smart about pixmap migration, and will only migrate the parts of a pixmap that are required for an operation. Migration heuristics Greedy/Mixed/Driver. Since fallbacks might require pixmap migration, it is not always better to implement some of the composite interface. For example if the usage pattern often calls operations A->B->C and only B is GPU accelerated, A will be done on the CPU, B will trigger a migration to GPU memory and will do the operation on the GPU, and C will trigger a migration back to system memory and do the third operation on the CPU. Since the overhead and cost of pixmap migration is so high, the end result is probably slower than doing all three operations on the CPU, and therefore the newly added composite() functionality actually results in a slowdown! \end_layout \begin_layout Standard \lang english As a side effect, it is often better to profile before implementing specific EXA composite() functions, and look at the common calling patterns; a very common example is antialiased fonts (which will also show different calling patterns if subpixel rendering is enabled or not). \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Multiple choices exist for accelerating 2D in X.Org. \end_layout \begin_layout Itemize \lang english The most efficient one is EXA, which puts all the smart optimizations in a common piece of code, and leaves the driver implementation very simple. \end_layout \begin_layout Itemize \lang english Today, most 2D acceleration is implemented using the 3D engine of the graphics card. \end_layout \begin_layout Itemize \lang english If your card cannot accelerate 2D operations, shadowfb is the path to take. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Video Decoding \begin_inset CommandInset label LatexCommand label name "cha:Video-Decoding" \end_inset \end_layout \begin_layout Section \lang english Video Standards \end_layout \begin_layout Standard \lang english H262 (mpeg 2, DVD) \end_layout \begin_layout Standard \lang english H263 (divx/mpeg4) \end_layout \begin_layout Standard \lang english H264 (used on blu-ray) \end_layout \begin_layout Section \lang english Video decoding pipeline \end_layout \begin_layout Standard \lang english Two typical video pipelines : mpeg2 and h264 \end_layout \begin_layout Subsubsection* \lang english The H262 decoding pipeline \end_layout \begin_layout Standard \lang english iDCT -> MC -> CSC -> Final display \end_layout \begin_layout Subsubsection* \lang english The H.264 decoding pipeline \end_layout \begin_layout Standard \lang english entropy decoding -> iDCT -> MC -> CSC -> Final display \end_layout \begin_layout Subsection \lang english Entropy \end_layout \begin_layout Standard \lang english Entropy encoding is a lossless compression phase. It is the last stage of encoding and therefore also the first stage of decoding. \end_layout \begin_layout Standard \lang english CABAC/CAVLC \end_layout \begin_layout Subsection \lang english Inverse DCT \end_layout \begin_layout Subsection \lang english Motion Compensation \end_layout \begin_layout Subsection \lang english Color Space Conversion \end_layout \begin_layout Standard \lang english A color space is the way color is represented. \end_layout \begin_layout Standard \lang english In order to save space, perception principles are applied to video encoding. \lang american The color space conversion stage exploits the fact that the human eye is more perceptive in the luminance than the chrominance domain, and separates the color data into those two components. Intuitively, the luminance is the amount of light in the color, and the chrominance is the name of the color (red, green, yellow...) and its amount of saturation. \end_layout \begin_layout Standard The quality of the chrominance data can be lowered to conserve space, for example by downsampling it. Then the missing chrominance data is interpolated at play time while minimally impacting the perceived video quality. \end_layout \begin_layout Standard \lang english This is how the YUV color space works: it separates the color information into one luminance component (Y) and two chrominance components (U and V). The chrominance information is less relevant to the human eye than the luminance, so usually chrominance is subsampled and luminance is at the original resolution. Therefore, the Y plane usually has a higher resolution than the U and V planes. For example, in the YUV420 video format, the U and V data is subsampled by a factor of two in both the X and Y directions, so U and V planes have a quarter of the number of pixels compared to the Y plane. \end_layout \begin_layout Standard \lang english Converting to YUV and downsampling the U and V planes can result in huge bandwidth gains. For example in the case of an RGB888 video encoded in YV12, the YUV version uses 12 bits per pixel compared to the original 24 bit per pixel. \end_layout \begin_layout Standard \lang english The color conversion from the YUV to the RGB color space is a linear relation, and can therefore be represented by a matrix/vector multiply, the matrix being the conversion formula. \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \lang english \begin_inset Formula $\left[\begin{array}{c} R\\ G\\ B \end{array}\right]=\left[\begin{array}{ccc} 1 & 0 & 1.13983\\ 1 & -0.39465 & -0.58060\\ 1 & 2.03211 & 0 \end{array}\right]\left[\begin{array}{c} Y\\ U\\ V \end{array}\right]$ \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:YUV-to-RGB" \end_inset YUV to RGB Conversion formula as per ITU-R RB recommendation 601. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \lang english \begin_inset Formula $\left[\begin{array}{c} R\\ G\\ B \end{array}\right]=\left[\begin{array}{ccc} 1 & 0 & 1.28033\\ 1 & -0.21482 & -0.38059\\ 1 & 2.12798 & 0 \end{array}\right]\left[\begin{array}{c} Y\\ U\\ V \end{array}\right]$ \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:YUV-to-RGB-1" \end_inset YUV to RGB Conversion formula as per ITU-R RB recommendation 709. \end_layout \end_inset \end_layout \end_inset Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:YUV-to-RGB" \end_inset shows the conversion matrices from ITU-R BT Recommendation 601 (standard definition content) and recommendation 709 (intended for high definition content). Notice that although these matrices are very similar, there are numerical differences which will result in slight off-colored rendering if one is used in place of the other. This is indeed often the case that video decoders with YUV to RGB hardware are used to playback high definition content but still use the ITU-R RB recommendation 601 color space conversion matrix. Since the colors are only slightly wrong, this problem is commonly overlooked, whereas most hardware features at least a BT601/BT709 switch, or a fully programmable conversion matrix. \end_layout \begin_layout Standard \lang english YUV Planar and packed (interlaced) formats on Figure \begin_inset CommandInset ref LatexCommand ref reference "fig:YUV-layouts-in" \end_inset . \end_layout \begin_layout Standard \lang english \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \noindent \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (0,2) rectangle (5,6); \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (6,2) rectangle (8.5,4); \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (10,2) rectangle (12.5,4); \end_layout \begin_layout Plain Layout \backslash node at (2.5,1.5) {Y plane}; \end_layout \begin_layout Plain Layout \backslash node at (7.25,1.5) {U plane}; \end_layout \begin_layout Plain Layout \backslash node at (11.25,1.5) {V plane}; \end_layout \begin_layout Plain Layout \backslash node at (1.25,5.5) {$Y_{0}$ $Y_{1}$ $Y_{2}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout \backslash node at (6.75,3.5) {$U_{0}$ $U_{1}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout \backslash node at (10.75,3.5) {$V_{0}$ $V_{1}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \noindent \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (0,2) rectangle (5,6); \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (6,2) rectangle (11,4); \end_layout \begin_layout Plain Layout \backslash node at (2.5,1.5) {Y plane}; \end_layout \begin_layout Plain Layout \backslash node at (7.25,1.5) {UV plane}; \end_layout \begin_layout Plain Layout \backslash node at (1.25,5.5) {$Y_{0}$ $Y_{1}$ $Y_{2}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout \backslash node at (7.25,3.5) {$U_{0}$ $V_{0}$ $U_{1}$ $V_{1}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \noindent \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=1em, minimum size=3em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} } \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash tikz{ \end_layout \begin_layout Plain Layout \backslash draw[top color=white, bottom color=yellow!50, drop shadow,very thick, inner sep=1em] (0,2) rectangle (10,6); \end_layout \begin_layout Plain Layout \backslash node at (2.5,1.5) {YUV plane}; \end_layout \begin_layout Plain Layout \backslash node at (1.25,5.5) {$Y_{0}$ $U_{0}$ $Y_{1}$ $V_{0}$ $Y_{2}$ $U_{1}$ $Y_{3}$ $V_{1}$ $ \backslash cdots$}; \end_layout \begin_layout Plain Layout % faux noeud pour pas que la légende soit collée \end_layout \begin_layout Plain Layout \backslash node at (6,0.5) { }; \end_layout \begin_layout Plain Layout \backslash node at (14,4) { }; \end_layout \begin_layout Plain Layout } \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \noindent \lang english \begin_inset Caption \begin_layout Plain Layout \lang english \begin_inset CommandInset label LatexCommand label name "fig:YUV-layouts-in" \end_inset YUV layouts in memory: planar format example (YV12, top), partially interleaved format example (NV12, middle), fully interleaved format example (YUY2, bottom). \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english figure schema planar vs interlaced \end_layout \end_inset \end_layout \begin_layout Standard \lang english Plane order (YV12 vs NV12) \end_layout \begin_layout Standard \lang english Order of the planes (YV12, I420) \end_layout \begin_layout Standard \lang english http://en.wikipedia.org/wiki/YUV \end_layout \begin_layout Standard \lang english http://www.fourcc.org/yuv.php \end_layout \begin_layout Standard \lang english http://www.glennchan.info/articles/articles.html \end_layout \begin_layout Standard \lang english http://www.poynton.com/papers/SMPTE_98_YYZ_Luma/index.html \end_layout \begin_layout Standard \lang english \begin_inset Float table wide false sideways false status open \begin_layout Plain Layout \align center \lang english \begin_inset Tabular \begin_inset Text \begin_layout Plain Layout \lang english Format name \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Y:U:V bits per pixel \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Comments \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english YV12 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 8:2:2 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 1 Y plane, 1 V 2*2 sub-sampled plane, 1 U 2*2 sampled plane \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Same as I420 except U and V are reversed. \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english I420 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 8:2:2 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 1 Y plane, 1 U 2*2 sub-sampled plane, 1 V 2*2 sub-sampled plane \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Same as YV12 except U and V are reversed. \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english NV12 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 8:2:2 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 1 Y plane, 1 packed U+V 2*2 sub-sampled plane \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Convenient for hardware implementation on 3D-capable GPUs \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english YUY2 (YUYV) \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 8:4:4 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english 1 Packed YUV plane \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \lang english Packed as Y0U0Y1V0 \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \begin_inset Text \begin_layout Plain Layout \end_layout \end_inset \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english Common YUV color space formats \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard The final stage of video decoding is video upscaling. Video upscaling consists in upscaling a video frame from its native resolution to the screen resolution. It can be done by specialized hardware, the 3D engine, or the 2D blitter if it has scaling capabilities. \lang english Since the conversion from YUV space to RGB space is linear, filtered upscaling can be done either in the YUV or RGB space, which conveniently allows using bilinear texture filtering which is available on 3D hardware to sample the YUV data. This allows a single pass color space conversion and scaling. For example, bi-linear filtering will work just fine with three textures for the three Y, U and V planes. Notice that higher quality can be obtained at the expense of performance by using better filtering modes, such as bi-cubic [citer papier hadwiger], even though this can prove to be costly. A trade-off can be achieved by implementing bi-cubic filtering for the (most eye-visible) Y plane, and keeping bi-linear filtering for U and V planes. \end_layout \begin_layout Standard \lang english If the hardware cannot achieve color space conversion and scaling at the same time (for example if you have a YUV->RGB blitter and a shader less 3D engine), it is equivalent to first do color space conversion from YUV to RGB and then scale in RGB space. Again this is only possible when the color conversion operation is linear, in particular that means gamma correction has to be ignored. \end_layout \begin_layout Standard gamma conversion \end_layout \begin_layout Standard using a conversion shader or a conversion texture lookup \end_layout \begin_layout Section \lang english Video decoding APIs \end_layout \begin_layout Subsubsection* \lang english Xv \end_layout \begin_layout Standard \lang english Xv is simply about CSC ans scaling. In order to implement Xv, a typical X.Org driver will have to implement this space conversion. Although the Xv API is a little complex for what it implements, the gits of it consists in the PutImage function, which puts an YUV image on screen. Multiple YUV formats can be handled, planar or interlaced mainly. Note that Xv has RGB support as well. Thanks to the bandwidth gains and DMA transfers, even an Xv implementation already provides a relevant level of video decoding acceleration, and can prove sufficient depending on the target hardware (for example, it can prove to be fine when coupled with a powerful CPU to decode H264 content). \end_layout \begin_layout Subsubsection* \lang english XvMC \end_layout \begin_layout Standard \lang english idct + mc +csc \end_layout \begin_layout Subsubsection* \lang english VAAPI \end_layout \begin_layout Standard \lang english VAAPI was initially created for intel's poulsbo video decoding. The API is very tailored to embedded platforms and has many entry points, at different pipeline stages, which makes it more complex to implement. \end_layout \begin_layout Subsubsection* \lang english VDPAU \end_layout \begin_layout Standard \lang english The VDPAU was initiated by nvidia for H264 & VC1 decoding support \end_layout \begin_layout Subsubsection* \lang english XvBA \end_layout \begin_layout Standard \lang english All 3 APIs are intended for full \end_layout \begin_layout Subsubsection* \lang english OpenMax \end_layout \begin_layout Standard \lang english http://x264dev.multimedia.cx \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english A video decoding pipeline consists in multiple stages chained together. \end_layout \begin_layout Itemize \lang english Color space conversion and scaling is the most important stage, and if your driver implements only one operation for simplicity, this is it. \end_layout \begin_layout Itemize \lang english Implementing a full pipeline can provide a high performance boost, and save battery life on mobile systems. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english OpenGL \begin_inset CommandInset label LatexCommand label name "cha:OpenGL" \end_inset \end_layout \begin_layout Standard \lang english OpenGL is a specification. There are many OpenGL implementations, both hardware accelerated and in software. As a driver author, our job is sometimes to provide a hardware-accelerated OpenGL implementation. In this section we describe the OpenGL pipeline from the point of view of the driver. \end_layout \begin_layout Standard \lang english OpenGL ARB, khronos, bla bla... \end_layout \begin_layout Section \lang english The OpenGL Rendering Pipeline \end_layout \begin_layout Standard \lang english \begin_inset Float figure placement tbh wide false sideways false status open \begin_layout Plain Layout \noindent \align center \lang english \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{tikzpicture}[node distance=1cm, auto] \end_layout \begin_layout Plain Layout \backslash tikzset{ mynode/.style={rectangle,rounded corners,draw=black, top color=white , bottom color=yellow!50,very thick, inner sep=0.5em, minimum size=2em, text centered, drop shadow}, myarrow/.style={->, >=latex', shorten >=1pt, thick}, mylabel/.style={text width=7em, text centered} , mynode2/.style={rec tangle,rounded corners,draw=black, top color=white, bottom color=red!50,very thick, inner sep=0.5em, minimum size=2em, text centered, drop shadow}, mynode3/.s tyle={rectangle,rounded corners,draw=black, top color=white, bottom color=green! 50,very thick, inner sep=0.5em, minimum size=2em, text centered, drop shadow},} \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm] (vertex) {Vertex Shader}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of vertex] (vertexcalls) {glDrawElemen ts, glArrayElement, glDrawArrays}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, text width = 3cm, left=1cm of vertex] (vertexprog) {Vertex Shader Program}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of vertex] (geom) {Geometry Shader}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, text width = 3cm, left=1cm of geom] (geomprog) {Geometry Shader Program}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of geom] (clip) {Clipping}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of clip] (viewport) {Viewport}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of viewport] (viewportcalls) {glViewpo rt}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of viewport] (cull) {Face Culling}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of cull] (cullcalls) {glCullFace, glFrontFace, glPolygonMode, glEnable(GL \backslash _CULL \backslash _FACE)}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, text width = 3cm, left=1cm of viewport] (uniforms) {Uniforms and Samplers}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of cull] (rast) {Rasterization}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of rast] (rastcalls) {glPolygonOffset, glPointSize, glLineStipple, glLineWidth}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of rast] (frag) {Fragment Shader}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, text width = 3cm, left=1cm of frag] (fragprog) {Fragment Shader Program}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of frag] (scissor) {Scissor}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of scissor] (scissorcalls) {glScissor, glEnable(GL \backslash _SCISSOR)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of scissor] (multisample) {Multisample }; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of multisample] (multisamplecalls) {glSampleCoverage, glEnable(GL \backslash _MULTISAMPLE)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of multisample] (stencil) {Stencil}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of stencil] (stencilcalls) {glStencilF unc, glStentilMask, glStencilOp, glEnable(GL \backslash _STENCIL \backslash _TEST)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of stencil] (depth) {Depth}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of depth] (depthcalls) {glDepthFunc, glDepthMask, glEnable(GL \backslash _DEPTH \backslash _TEST)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of depth] (query) {Occlusion Query}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of query] (querycall) {glGenQueries, glIsQuery, glBeginQuery, glEndQuery, glGetQuery*}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of query] (blend) {Blending}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of blend] (blendcalls) {glBlendColor, glBlendFunc, glBlendEquation, glEnable(GL \backslash _BLEND)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of blend] (dither) {Dithering}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of dither] (dithercalls) {glEnable(GL \backslash _DITHER)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of dither] (logicop) {Logic Op}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of logicop] (logicopcalls) {glLogicOp, glEnable(GL \backslash _COLOR \backslash _LOGIC \backslash _OP)}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of logicop] (mask) {Masking}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of mask] (maskcalls) {glColorMask, glIndexMask}; \end_layout \begin_layout Plain Layout \backslash node[mynode, text width = 3cm, below=0.3cm of mask] (fbcon) {Framebuffer Control}; \end_layout \begin_layout Plain Layout \backslash node[mynode3, text width = 4cm, right=1cm of fbcon] (fbconcalls) {glDrawBuffer, glDrawBuffers}; \end_layout \begin_layout Plain Layout \backslash node[mynode2, text width = 3cm, below=0.3cm of fbcon] (fb) {Framebuffer}; \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (vertex.south) -> (geom.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (geom.south) -> (clip.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (clip.south) -> (viewport.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (viewport.south) -> (cull.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (cull.south) -> (rast.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (rast.south) -> (frag.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (frag.south) -> (scissor.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (scissor.south) -> (multisample.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (multisample.south) -> (stencil.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (stencil.south) -> (depth.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (depth.south) -> (query.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (query.south) -> (blend.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (blend.south) -> (dither.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (dither.south) -> (logicop.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (logicop.south) -> (mask.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (mask.south) -> (fbcon.north); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fbcon.south) -> (fb.north); \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (uniforms.east) -> (vertex.west); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (uniforms.east) -> (geom.west); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (uniforms.east) -> (frag.west); \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (vertexprog.east) -> (vertex.west); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (geomprog.east) -> (geom.west); \end_layout \begin_layout Plain Layout \backslash draw[myarrow] (fragprog.east) -> (frag.west); \end_layout \begin_layout Plain Layout \end_layout \begin_layout Plain Layout \backslash end{tikzpicture} \end_layout \begin_layout Plain Layout \end_layout \end_inset \end_layout \begin_layout Plain Layout \lang english \begin_inset Caption \begin_layout Plain Layout \lang english The OpenGL 3.2 pipeline. \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Subsection \lang english Vertex processing \end_layout \begin_layout Standard \lang english vertex stage \end_layout \begin_layout Standard \lang english vertex buffers \end_layout \begin_layout Subsection \lang english Geometry processing \end_layout \begin_layout Subsection \lang english Fragment processing \end_layout \begin_layout Standard \lang english Rasterization \end_layout \begin_layout Standard \lang english Render buffers \end_layout \begin_layout Standard \lang english Textures \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english OpenGL is a suite of stages arranged in a pipeline. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Mesa \begin_inset CommandInset label LatexCommand label name "cha:Mesa" \end_inset \end_layout \begin_layout Standard \lang english Mesa is both a software OpenGL implementation, and the common rendering architecture for all open source hardware accelerated graphics drivers. We now describe the internals of Mesa and the available interfaces and infrastructure required for graphics drivers. \end_layout \begin_layout Section \lang english Mesa \end_layout \begin_layout Standard \lang english Mesa serves two major purposes: \end_layout \begin_layout Itemize \lang english Mesa is a software implementation of OpenGL. It is considered to be the reference implementation and is useful in checking conformance, seeing that the official OpenGL conformance tests are not publicly available. \end_layout \begin_layout Itemize \lang english Mesa provides the OpenGL entry points for Open Source graphics drivers under linux. \end_layout \begin_layout Standard \lang english In this section, we will focus on the second point. \end_layout \begin_layout Section \lang english Mesa internals \end_layout \begin_layout Subsection \lang english Textures in mesa \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Mesa is the reference OpenGL implementation under Linux. \end_layout \begin_layout Itemize \lang english All Open Source graphics drivers use Mesa for 3D \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Gallium 3D \begin_inset CommandInset label LatexCommand label name "cha:Gallium-3D" \end_inset \end_layout \begin_layout Standard \lang english Gallium 3D is the Future of 3D Acceleration. \end_layout \begin_layout Standard \lang english http://jrfonseca.blogspot.com/2008/04/gallium3d-introduction.html \end_layout \begin_layout Standard \lang english http://people.freedesktop.org/~csimpson/gallium-docs/ \end_layout \begin_layout Section \lang english Gallium3D: a plan for a new generation of hardware \end_layout \begin_layout Standard With the advent of shaders and thus programmable GPU hardware, the combinatorial space of hardware functions exposed by GPUs got a lot smaller. Gallium3D builds on this observation and tries to abstract away how GPUs work internally into an universally useable interface, where shaders are sitting at the core and the rest of the interface is there to control how they are executed. Keep in mind that Gallium3D is just a thin layer interface inside MESA and so is a moving target. There is no effort going into keeping the interface stable at any point; if something inside it does not fit newly introduced GPUs the interface is changed to cope with those changed capabilities. \end_layout \begin_layout Standard \lang english global diagram \end_layout \begin_layout Section \lang english Winsys \end_layout \begin_layout Standard The name Winsys is actually a bit of a misnomer, but it hasn't changed since Gallium was introduced. The Winsys provides the hardware drivers with an internal fixed interface to talk to the operating system. You can easily imagine a driver for the same hardware working on different OSes, where you have to use differing interfaces to get hardware access through the kernel. The Winsys is meant to abstract those differences away. Consequently drivers that are not meant to run on different OSes don't need to implement a Winsys, although it might prove to be beneficial to stick to this strict layering. \end_layout \begin_layout Section \lang english Pipe driver \end_layout \begin_layout Standard \lang english Pipe drivers are the components that actually implement the Gallium interface. They talk directly to the hardware, through the Winsys or direct usage of the respective operating system routines. The pipe driver has to fully hide how your hardware works internally. \end_layout \begin_layout Section \lang english State trackers \end_layout \begin_layout Standard \lang english State trackers sit on top of the Gallium interface and implement a specific API like OpenGL, OpenCL or other custom APIs that use the GPU to accelerate something. They do so by turning the API calls into hardware-independent acceleration operations covered by Gallium. \end_layout \begin_layout Standard \lang english Gallium abstracts away most of the internals of the GPU, however it cannot hide the different featuresets exposed by the large numbers of GPUs used today. Some things are hidden internally, like some drivers using a JIT compiler to do fallbacks when the GPU can't execute a specific shader and it seems worthwhile to do a software fallback, but mostly the differences in capabilitie s are also reflected into the Gallium interface. So before a state tracker can use a specific part of the Gallium interface it has to ask the pipe driver it is running on currently if it supports that one feature. This is done using the respective pipe_cap, pipe_caf and pipe_shader_cap function calls, depending on what the state tracker wants to know. \end_layout \begin_layout Section \lang english Writing Gallium3D drivers \end_layout \begin_layout Standard \lang english screen \end_layout \begin_layout Standard \lang english context \end_layout \begin_layout Standard \lang english pipe_transfer \end_layout \begin_layout Section \lang english Shaders in Gallium \begin_inset Note Note status open \begin_layout Plain Layout \lang english Do we want to go this deeply into gallium stuff here? \end_layout \end_inset \end_layout \begin_layout Standard \lang english In order to operate shaders, Gallium features an internal shader description language which uses 4-component vectors. We will later refer to the 4 components of a vector as x,y,z,w. In particular, v.x is the first component of vector v, v.xyzw are all 4 component s of v in that order, and swizzling is allowed, for example v.wzyx reverses the component order. It is also legal to replicate a component, for example v.xxxx means the x component of v is cloned to all four components and v.yyzz means two identical y and two identical z. \end_layout \begin_layout Standard \lang english These components usually carry no semantics, and despite their coordinate names they can just as easily represent a color or an opacity value. \end_layout \begin_layout Standard \lang english TGSI instruction set \end_layout \begin_layout Standard \lang english mesa/src/gallium/auxiliary/tgsi/tgsi-instruction-set.txt \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Gallium 3D is the new graphics API. \end_layout \begin_layout Itemize \lang english Everything is converted to a shader internally, fixed functionality is gone. \end_layout \begin_layout Itemize \lang english Drivers are simpler than classic Mesa drivers, as one only has to implement shaders to get all fixed functionality to work. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english GPU Computing \begin_inset CommandInset label LatexCommand label name "cha:GPU-Computing" \end_inset \end_layout \begin_layout Chapter \lang english Suspend and Resume \begin_inset CommandInset label LatexCommand label name "cha:Suspend-and-Resume" \end_inset \end_layout \begin_layout Standard \lang english VT switches \end_layout \begin_layout Standard \lang english Card state \end_layout \begin_layout Standard \lang english Suspend/resume hooks in the DRM \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Suspend and resume has long been very clumsy, but this is solved now thanks to the DRM implementing more functionality. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Technical Specifications \begin_inset CommandInset label LatexCommand label name "cha:Technical-Specifications" \end_inset \end_layout \begin_layout Standard \lang english Technical specifications are the nuts and bolts of graphics driver work. Without hardware specifications, no work can be started. However, manufacturing companies are usually wary of sharing said specification s, as they think this will hinder their business. While this claim is false (because you can't copy a GPU from just its specifica tions), it is still very widespread and prevents a lot of hardware from being properly documented. Therefore, getting hold of hardware specifications will be the first major step in any graphics driver development project. \end_layout \begin_layout Section \lang english Obtaining official specifications \end_layout \begin_layout Paragraph* \lang english Public specifications \end_layout \begin_layout Standard \lang english Some vendors distribute the technical documentation for their hardware publicly without restrictions. \end_layout \begin_layout Standard \lang english Sometimes, things can be as simple as asking the vendor, who might share the documentation (possibly under NDA, see below). \end_layout \begin_layout Paragraph* \lang english NDA (Non-Disclosure Agreement) \end_layout \begin_layout Standard \lang english Put simply, an NDA is a contract signed between the developer and the hardware company, by which the developer agrees not to spread the docs he received. However, there can be more restrictions in an NDA. \end_layout \begin_layout Standard \lang english Terms of the NDA \end_layout \begin_layout Standard \lang english Before signing an NDA, think. Whatever lawyers say, there is no such thing as a \begin_inset Quotes eld \end_inset standard \begin_inset Quotes erd \end_inset NDA, you can always negotiate. \end_layout \begin_layout Standard \lang english Can Open Source drivers be written from that documentation under that NDA? \end_layout \begin_layout Standard \lang english What happens when the NDA expires? Can code still be free, are you bound by any clause? \end_layout \begin_layout Standard \lang english What about yourself? Are you prevented from doing further work on this hardware? \end_layout \begin_layout Section \lang english Reverse engineering \end_layout \begin_layout Standard \lang english When specifications are not easily available or just incomplete, an alternate route is reverse engineering. Reverse engineering consists in figuring out the specifications for a given piece of hardware by yourself, for example by looking at what a black-box binary driver does to the hardware under certain circumstances. \end_layout \begin_layout Standard \lang english Reverse engineering is not just a tool to obtain missing hardware specifications , it is also a strong means of Open Source advocacy. Once a reverse engineered driver exists and ships in linux distributions, pressure shifts on the hardware vendor for support. This, in turn, can force the vendor to support Open Source drivers. \end_layout \begin_layout Standard \lang english not as difficult as it seems, requires organization, being rigorous. Write down all bits of information (even incomplete bits), share it among developers, try to work out bits one by one. Do not hesitate writing ad-hoc tools, as they will save precious time down the road (if you hesitate, you have crossed the line already!). \end_layout \begin_layout Subsubsection* \lang english Mmiotrace \end_layout \begin_layout Standard \lang english The basic idea behind mmio-trace is simple: it first hooks the ioremap call, and therefore prevents mapping of a designated I/O area. Subsequently, accesses to this area will generate page faults, which are caught by the kernel. For each page fault, the faulting instruction is decoded to figure out the write or read address, along with the value written/read. The page is put back, the faulting instruction is then single-stepped, and the page is then removed again. Execution then continues as usual. \end_layout \begin_layout Standard \lang english mmio trace is now part of the official Linux kernels. Therefore, any pre-existing driver can be traced. \end_layout \begin_layout Subsubsection* \lang english Libsegfault \end_layout \begin_layout Standard \lang english libsegfault is similar to mmio-trace in the way it works: after removing some pages which one want to track accesses to, it will generate a segmentation fault on each access and therefore be able to report each access. The difference is that libsegfault is a user space tool while mmio-trace is a kernel tool. \end_layout \begin_layout Subsubsection* \lang english Valgrind-mmt \end_layout \begin_layout Standard \lang english Valgrind is a dynamic recompiling and instrumentation framework. Valgrint-mmt is a plugin for valgrind which implements tracing of read and writes to a certain range of memory addresses, usually an mmio range accessed from user space. Memory accesses are dynamically instrumented thanks to valgrind and each access to the zones we want to see traced is logged. \end_layout \begin_layout Subsubsection* \lang english vbetool/vbtracetool \end_layout \begin_layout Subsubsection* \lang english Virtualization \end_layout \begin_layout Standard \lang english Finally, one last pre-existing tool to help reverse engineering is virtualizatio n. By running a proprietary driver in a controlled environment, one can figure out the inner workings of a GPU. The plan is then to write an emulated GPU while doing the reverse engineering (which imposes the use of an open source virtualization solution like Qemu). \end_layout \begin_layout Subsubsection* \lang english Ad-hoc tools \end_layout \begin_layout Standard \lang english In addition to these generic tools, you will often find it useful to implement your own additional tools, tailored for specific needs. Renouveau is an example of such a tool that integrates the reverse engineering mechanisms, the command decoding and printing. In order to achieve decoding of the commands, it carries a database of the graphics commands of nvidia GPUs. This allows quick testing of new database entries. Headers generated from this database are later used in the driver development process. \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Technical specifications of course very important for authoring graphics drivers. \end_layout \begin_layout Itemize \lang english NDAs can have unforeseen implications on yourself and your work. \end_layout \begin_layout Itemize \lang english When they are unavailable, incomplete or just plain wrong, reverse engineering can help you figure out how the hardware actually works. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Beyond Development \begin_inset CommandInset label LatexCommand label name "cha:Beyond-Development" \end_inset \end_layout \begin_layout Section \lang english Testing for conformance \end_layout \begin_layout Subsubsection* \lang english Rendercheck \end_layout \begin_layout Subsubsection* \lang english OpenGL conformance test suite \end_layout \begin_layout Standard \lang english The official OpenGL testing suite is not publicly available, and (paying) Khronos Membership is required. Instead, most developers use alternate sources for test programs. \end_layout \begin_layout Subsubsection* \lang english Piglit \end_layout \begin_layout Subsubsection* \lang english glean \end_layout \begin_layout Standard \lang english glean.sourceforge.net \end_layout \begin_layout Subsubsection* \lang english Mesa demos \end_layout \begin_layout Standard \lang english mesa/progs/* \end_layout \begin_layout Section \lang english Debugging \end_layout \begin_layout Subsubsection* \lang english gdb and X.Org \end_layout \begin_layout Standard \lang english gdb needs to run on a terminal emulator while the application debug might be with a lock held. That might result in a deadlock between the application stuck with a lock and gdb waiting to be able to output text. \end_layout \begin_layout Subsubsection* \lang english printk debug \end_layout \begin_layout Subsubsection* \lang english crash \end_layout \begin_layout Standard \lang english TODO: (gdb overlay to analyze vmcores) \end_layout \begin_layout Subsubsection* \lang english kgdb \end_layout \begin_layout Subsubsection* \lang english serial console \end_layout \begin_layout Subsubsection* \lang english diskdump \end_layout \begin_layout Subsubsection* \lang english linux-uml \end_layout \begin_layout Subsubsection* \lang english systemtap \end_layout \begin_layout Section \lang english Upstreaming \end_layout \begin_layout Standard \lang english Submitting your code for inclusion in the official trees is an important part of the graphics driver development process under linux. There are multiple motivations for doing this. \end_layout \begin_layout Standard \lang english First, this allows end users to get hold of your driver more easily. \end_layout \begin_layout Standard \lang english Second, this makes it easier for your driver maintenance in the future: in the event of interface changes, \end_layout \begin_layout Standard \lang english Why upstream? \end_layout \begin_layout Standard \lang english How? \end_layout \begin_layout Standard \lang english When? \end_layout \begin_layout Standard \lang english \begin_inset Box Shadowbox position "t" hor_pos "c" has_inner_box 1 inner_pos "t" use_parbox 0 use_makebox 0 width "100col%" special "none" height "1in" height_special "totalheight" status open \begin_layout Plain Layout \lang english Takeaways: \end_layout \begin_layout Itemize \lang english Thoroughly testing all your changes can save you the cost of bisection later on. \end_layout \begin_layout Itemize \lang english Debugging is not easy for graphics drivers. \end_layout \begin_layout Itemize \lang english By upstreaming your code in official repositories, you save yourself the burden of adapting it to ever-moving programming interfaces in X.Org, Mesa and the kernel. \end_layout \end_inset \end_layout \begin_layout Chapter \lang english Conclusions \begin_inset CommandInset label LatexCommand label name "cha:Conclusions" \end_inset \end_layout \begin_layout Standard \lang english \begin_inset Note Note status open \begin_layout Plain Layout \lang english Bordel à caser quelque part : \end_layout \begin_layout Plain Layout \lang english - la composition, avec XRender ou avec GLX + GL_EXT_texture_from_pixmap, expliquer les différences \end_layout \begin_layout Plain Layout \lang english - XGL, AIGLX \end_layout \end_inset \end_layout \end_body \end_document