summaryrefslogtreecommitdiff
path: root/drivers/dma
AgeCommit message (Collapse)AuthorFilesLines
2008-05-20iop-adma: fixup some kzalloc/memset confusionsChristophe Jaillet1-4/+2
1) Remove an explicit memset(.., 0, ...) to a variable allocated with kzalloc (i.e. 'dest'). 2) Allocate 'src' with kmalloc instead of kzalloc as all elements of the 'src' buffer are initialized in a 'for(...)' loop just after. 3) remove useless 'sizeof(u8)', which always returns 1, when computing the size of the memory to be allocated. Signed-off-by: Christophe Jaillet <christophe.jaillet@wanadoo.fr> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-21DMA engine: typo fixesSebastian Siewior1-3/+3
Spelling fixes for dmaengine.[ch] Signed-off-by: Sebastian Siewior <bigeasy@linutronix.de> Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com> Signed-off-by: Jesper Juhl <jesper.juhl@gmail.com>
2008-04-17dmaengine: ack to flags: make use of the unused bits in the 'ack' fieldDan Williams4-41/+44
'ack' is currently a simple integer that flags whether or not a client is done touching fields in the given descriptor. It is effectively just a single bit of information. Converting this to a flags parameter allows the other bits to be put to use to control completion actions, like dma-unmap, and capture results, like xor-zero-sum == 0. Changes are one of: 1/ convert all open-coded ->ack manipulations to use async_tx_ack and async_tx_test_ack. 2/ set the ack bit at prep time where possible 3/ make drivers store the flags at prep time 4/ add flags to the device_prep_dma_interrupt prototype Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-17iop-adma: remove the workaround for missed interrupts on iop3xxDan Williams1-5/+0
This workaround was covering the dependency submission bug in async_tx. Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-17async_tx: kill ->device_dependency_addedDan Williams4-28/+0
DMA drivers no longer need to be notified of dependency submission events as async_tx_run_dependencies and async_tx_channel_switch will handle the scheduling and execution of dependent operations. [sfr@canb.auug.org.au: extend this for fsldma] Acked-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-17async_tx: fix multiple dependency submissionDan Williams2-6/+5
Shrink struct dma_async_tx_descriptor and introduce async_tx_channel_switch to properly inject a channel switch interrupt in the descriptor stream. This simplifies the locking model as drivers no longer need to handle dma_async_tx_descriptor.lock. Acked-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-17fsldma: Split the MPC83xx event from MPC85xx and refine irq codes.Zhang Wei1-5/+21
Split MPC83xx EOCDI event from MPC85xx EOLNI event, which is also need to update cookie and start the next transfer. The DMA channel irq handler function code is refined. The patch is tested on MPC8377MDS board. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by; Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-04-17fsldma: Remove CONFIG_FSL_DMA_SELFTEST, keep fsl_dma_self_test() running always.Zhang Wei2-18/+3
Always enabling the fsl_dma_self_test() to ensure the DMA controller should works well after the driver probed. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-31[POWERPC] fsldma: Use compatiable binding as specKumar Gala1-4/+4
Documentation/powerpc/booting-without-of.txt specifies the compatiables we should bind to for this driver (elo, eloplus). Use these instead of the extremely specific 'mpc8540' and 'mpc8349' compatiables. Acked-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-03-30fix the broken annotations in fsldmaAl Viro1-20/+27
a) every bitwise declaration will give a unique type; use typedefs. b) no need to bother with the stuff pointed to by iomem pointers, unless it's accessed directly. noderef will force us to use helpers anyway. Signed-off-by: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-03-30ioat_dca __iomem annotationsAl Viro1-2/+2
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-03-18fsldma: Fix the DMA halt when using DMA_INTERRUPT async_tx transfer.Zhang Wei2-0/+31
The DMA_INTERRUPT async_tx is a NULL transfer, thus the BCR(count register) is 0. When the transfer started with a byte count of zero, the DMA controller will triger a PE(programming error) event and halt, not a normal interrupt. I add special codes for PE event and DMA_INTERRUPT async_tx testing. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-13iop-adma.c: replace remaining __FUNCTION__ occurrencesHarvey Harrison1-16/+16
__FUNCTION__ is gcc-specific, use __func__ Signed-off-by: Harvey Harrison <harvey.harrison@gmail.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-13fsldma: Add a completed cookie updated action in DMA finish interrupt.Zhang Wei1-1/+1
The patch 'fsldma: do not cleanup descriptors in hardirq context' (commit 222ccf9ab838a1ca7163969fabd2cddc10403fb5) removed descriptors cleanup function to tasklet but the completed cookie do not updated. Thus, the DMA controller will get lots of duplicated transfer interrupts. Just make a completed cookie update in interrupt handler. And keep other cleanup jobs in tasklet function. Tested-by: Sebastian Siewior <bigeasy@linutronix.de> Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-13fsldma: Add device_prep_dma_interrupt support to fsldma.cZhang Wei1-0/+27
This is a bug that I assigned DMA_INTERRUPT capability to fsldma but missing device_prep_dma_interrupt function. For a bug in dmaengine.c the driver passed BUG_ON() checking. The patch fixes it. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-13dmaengine: Fix a bug about BUG_ON() on DMA engine capability DMA_INTERRUPT.Zhang Wei1-1/+1
The device->device_prep_dma_interrupt function is used by DMA_INTERRUPT capability, not DMA_ZERO_SUM. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-13fsldma: Fix fsldma.c warning messages when it's compiled under PPC64.Zhang Wei1-13/+16
There are warning messages reported by Stephen Rothwell with ARCH=powerpc allmodconfig build: drivers/dma/fsldma.c: In function 'fsl_dma_prep_memcpy': drivers/dma/fsldma.c:439: warning: comparison of distinct pointer types lacks a cast drivers/dma/fsldma.c: In function 'fsl_chan_xfer_ld_queue': drivers/dma/fsldma.c:584: warning: format '%016llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' drivers/dma/fsldma.c: In function 'fsl_dma_chan_do_interrupt': drivers/dma/fsldma.c:668: warning: format '%x' expects type 'unsigned int', but argument 5 has type 'dma_addr_t' drivers/dma/fsldma.c:684: warning: format '%016llx' expects type 'long long unsigned int', but argument 4 has type 'dma_addr_t' drivers/dma/fsldma.c:684: warning: format '%016llx' expects type 'long long unsigned int', but argument 5 has type 'dma_addr_t' drivers/dma/fsldma.c:701: warning: format '%02x' expects type 'unsigned int', but argument 4 has type 'dma_addr_t' drivers/dma/fsldma.c: In function 'fsl_dma_self_test': drivers/dma/fsldma.c:840: warning: format '%d' expects type 'int', but argument 5 has type 'size_t' drivers/dma/fsldma.c: In function 'of_fsl_dma_probe': drivers/dma/fsldma.c:1010: warning: format '%08x' expects type 'unsigned int', but argument 5 has type 'resource_size_t' This patch fixed the above warning messages. Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-03-04ioat: fix 'ack' handling, driver must ensure that 'ack' is zeroDan Williams1-0/+2
Initialize 'ack' to zero in case the descriptor has been recycled. Prevents "kernel BUG at crypto/async_tx/async_xor.c:185!" Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-by: Shannon Nelson <shannon.nelson@intel.com> Cc: stable@kernel.org
2008-03-04fsldma: do not cleanup descriptors in hardirq contextDan Williams1-1/+0
"Cleaning" descriptors involves calling pending callbacks and clients assume that their callback will only ever happen in softirq context. Delay cleanup to the tasklet. Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-by: Zhang Wei <wei.zhang@freescale.com>
2008-03-04dmaengine: add driver for Freescale MPC85xx DMA controllerZhang Wei4-1/+1276
The driver implements DMA engine API for Freescale MPC85xx DMA controller, which could be used by devices in the silicon. The driver supports the Basic mode of Freescale MPC85xx DMA controller. The MPC85xx processors supported include MPC8540/60, MPC8555, MPC8548, MPC8641 and so on. The MPC83xx(MPC8349, MPC8360) are also supported. [kamalesh@linux.vnet.ibm.com: build fix] [dan.j.williams@intel.com: merge mm fixes, rebase on async_tx-2.6.25] Signed-off-by: Zhang Wei <wei.zhang@freescale.com> Signed-off-by: Ebony Zhu <ebony.zhu@freescale.com> Acked-by: Kumar Gala <galak@gate.crashing.org> Cc: Shannon Nelson <shannon.nelson@intel.com> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-02-06async_tx: replace 'int_en' with operation preparation flagsDan Williams2-12/+12
Pass a full set of flags to drivers' per-operation 'prep' routines. Currently the only flag passed is DMA_PREP_INTERRUPT. The expectation is that arch-specific async_tx_find_channel() implementations can exploit this capability to find the best channel for an operation. Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-by: Shannon Nelson <shannon.nelson@intel.com> Reviewed-by: Haavard Skinnemoen <hskinnemoen@atmel.com>
2008-02-06async_tx: kill tx_set_src and tx_set_dest methodsDan Williams4-126/+87
The tx_set_src and tx_set_dest methods were originally implemented to allow an array of addresses to be passed down from async_xor to the dmaengine driver while minimizing stack overhead. Removing these methods allows drivers to have all transaction parameters available at 'prep' time, saves two function pointers in struct dma_async_tx_descriptor, and reduces the number of indirect branches.. A consequence of moving this data to the 'prep' routine is that multi-source routines like async_xor need temporary storage to convert an array of linear addresses into an array of dma addresses. In order to keep the same stack footprint of the previous implementation the input array is reused as storage for the dma addresses. This requires that sizeof(dma_addr_t) be less than or equal to sizeof(void *). As a consequence CONFIG_DMADEVICES now depends on !CONFIG_HIGHMEM64G. It also requires that drivers be able to make descriptor resources available when the 'prep' routine is polled. Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-by: Shannon Nelson <shannon.nelson@intel.com>
2008-02-06iop-adma: use LIST_HEAD instead of LIST_HEAD_INITDenis Cheng1-1/+1
these three list_head are all local variables, but can also use LIST_HEAD. Signed-off-by: Denis Cheng <crquan@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2008-01-24DMA: Convert from class_device to device for DMA engineTony Jones1-21/+22
Signed-off-by: Tony Jones <tonyj@suse.de> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Cc: Shannon Nelson <shannon.nelson@intel.com> Cc: Kay Sievers <kay.sievers@vrfy.org> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
2007-12-17I/OAT: fix null device in call to dev_err()Shannon Nelson1-1/+1
We can't use the device in a dev_err() after a kzalloc failure or after the kfree, so simplify it to the pdev that was originally passed in. Cc: Eric Sesterhenn <snakebyte@gmx.de> Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-17I/OAT: fixups from code commentsShannon Nelson2-66/+78
A few fixups from Andrew's code comments. - removed "static inline" forward-declares - changed use of min() to min_t() - removed some unnecessary NULL initializations - removed a couple of BUG() calls Fixes this: drivers/dma/ioat_dma.c: In function `ioat1_tx_submit': drivers/dma/ioat_dma.c:177: sorry, unimplemented: inlining failed in call to '__ioat1_dma_memcpy_issue_pending': function body not available drivers/dma/ioat_dma.c:268: sorry, unimplemented: called from here Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: "Williams, Dan J" <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-11-29dmaengine: correct invalid assumptions in the Kconfig textHaavard Skinnemoen1-3/+5
This patch corrects recently changed (and now invalid) Kconfig descriptions for the DMA engine framework: - Non-Intel(R) hardware also has DMA engines; - DMA is used for more than memcpy and RAID offloading. In fact, on most platforms memcpy and RAID aren't factors, and DMA exists so that peripherals can transfer data to/from memory while the CPU does other work. Signed-off-by: Haavard Skinnemoen <hskinnemoen@atmel.com> Signed-off-by: David Brownell <dbrownell@users.sourceforge.net> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-11-14I/OAT: Add support for version 2 of ioatdma deviceShannon Nelson6-145/+779
Add support for version 2 of the ioatdma device. This device handles the descriptor chain and DCA services slightly differently: - Instead of moving the dma descriptors between a busy and an idle chain, this new version uses a single circular chain so that we don't have rewrite the next_descriptor pointers as we add new requests, and the device doesn't need to re-read the last descriptor. - The new device has the DCA tags defined internally instead of needing them defined statically. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: "Williams, Dan J" <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-11-14dmaengine: fix broken device refcountingHaavard Skinnemoen1-11/+6
When a DMA device is unregistered, its reference count is decremented twice for each channel: Once dma_class_dev_release() and once in dma_chan_cleanup(). This may result in the DMA device driver's remove() function completing before all channels have been cleaned up, causing lots of use-after-free fun. Fix it by incrementing the device's reference count twice for each channel during registration. [dan.j.williams@intel.com: kill unnecessary client refcounting] Signed-off-by: Haavard Skinnemoen <hskinnemoen@atmel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: <stable@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-30Remove bogus default y for DMAR and NET_DMAAndi Kleen1-1/+0
No reason I can think of of making them default y Most people don't have the hardware and with default y they just pollute lots of configs during make oldconfig. Signed-off-by: Andi Kleen <ak@suse.de> Acked-by: Jeff Garzik <jeff@garzik.org> Acked-by: "Nelson, Shannon" <shannon.nelson@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-18I/OAT: Add completion callback for async_tx interface useShannon Nelson1-0/+25
The async_tx interface includes a completion callback. This adds support for using that callback, including using interrupts on completion. [akpm@linux-foundation.org: various fixes] Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-18I/OAT: Tighten descriptor setup performanceShannon Nelson2-84/+95
The change to the async_tx interface cost this driver some performance by spreading the descriptor setup across several functions, including multiple passes over the new descriptor chain. Here we bring the work back into one primary function and only do one pass. [akpm@linux-foundation.org: cleanups, uninline] Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-18I/OAT: clean up error handling and some print messagesShannon Nelson4-18/+31
Make better use of dev_err(), and catch an error where the transaction creation might fail. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-18I/OAT: clean up of dca provider start and stopShannon Nelson2-8/+7
Don't start ioat_dca if ioat_dma didn't start, and then stop ioat_dca before stopping ioat_dma. Since the ioat_dma side does the pci device work, This takes care of ioat_dca trying to use a bad device reference. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-18I/OAT: cleanup pci issuesShannon Nelson2-18/+10
Reorder the pci release actions Letting go of the resources in the right order helps get rid of occasional kernel complaints. Fix the pci_driver object name [Randy Dunlap] Rename the struct pci_driver data so that false section mismatch warnings won't be produced. Cc: Randy Dunlap <randy.dunlap@oracle.com> Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-17Remove "unsafe" from module structRusty Russell1-5/+4
Adrian Bunk points out that "unsafe" was used to mark modules touched by the deprecated MOD_INC_USE_COUNT interface, which has long gone. It's time to remove the member from the module structure, as well. If you want a module which can't unload, don't register an exit function. (Vlad Yasevich says SCTP is now safe to unload, so just remove the __unsafe there). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Acked-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: Dan Williams <dan.j.williams@intel.com> Acked-by: Vlad Yasevich <vladislav.yasevich@hp.com> Cc: Sridhar Samudrala <sri@us.ibm.com> Cc: Adrian Bunk <bunk@stusta.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: Add DCA servicesShannon Nelson5-27/+318
Add code to connect to the DCA driver and provide cpu tags for use by drivers that would like to use Direct Cache Access hints. [Adrian Bunk] Several Kconfig cleanup items [Andrew Morten, Chris Leech] Fix for using cpu_physical_id() even when built for uni-processor Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: Add support for MSI and MSI-XShannon Nelson3-66/+298
Add support for MSI and MSI-X interrupt handling, including the ability to choose the desired interrupt method. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> [bunk@kernel.org: drivers/dma/ioat_dma.c: make 3 functions static] Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: Split PCI startup from DMA handling codeShannon Nelson5-146/+254
Split the general PCI startup from the DMA handling code in order to prepare for adding support for DCA services and future versions of the ioatdma device. [Rusty Russell] Removal of __unsafe() usage. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: code cleanup from checkpatch outputShannon Nelson1-88/+110
Take care of a bunch of little code nits in ioatdma files Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: Rename the source fileShannon Nelson2-1/+2
Rename the ioatdma.c file in preparation for splitting into multiple files, which will allow for easier adding new functionality. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16I/OAT: New device idsShannon Nelson1-2/+3
Add device ids for new revs of the Intel I/OAT DMA engine Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Acked-by: David S. Miller <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-08-26[IOAT]: ioatdma needs to to play nice in a multi-dma-client worldShannon Nelson1-14/+4
Now that the DMA engine has a multi-client interface, fix the ioatdma driver to play along. At the same time, remove a couple of unnecessary reads and writes. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2007-08-14[IOAT]: Remove redundant struct member to avoid descriptor cache missShannon Nelson2-6/+4
The layout for struct ioat_desc_sw is non-optimal and causes an extra cache hit for every descriptor processed. By tightening up the struct layout and removing one item, we pull in the fields that get used in the speedpath and get a little better performance. Before: ------- struct ioat_desc_sw { struct ioat_dma_descriptor * hw; /* 0 8 */ struct list_head node; /* 8 16 */ int tx_cnt; /* 24 4 */ /* XXX 4 bytes hole, try to pack */ dma_addr_t src; /* 32 8 */ __u32 src_len; /* 40 4 */ /* XXX 4 bytes hole, try to pack */ dma_addr_t dst; /* 48 8 */ __u32 dst_len; /* 56 4 */ /* XXX 4 bytes hole, try to pack */ /* --- cacheline 1 boundary (64 bytes) --- */ struct dma_async_tx_descriptor async_tx; /* 64 144 */ /* --- cacheline 3 boundary (192 bytes) was 16 bytes ago --- */ /* size: 208, cachelines: 4 */ /* sum members: 196, holes: 3, sum holes: 12 */ /* last cacheline: 16 bytes */ }; /* definitions: 1 */ After: ------ struct ioat_desc_sw { struct ioat_dma_descriptor * hw; /* 0 8 */ struct list_head node; /* 8 16 */ int tx_cnt; /* 24 4 */ __u32 len; /* 28 4 */ dma_addr_t src; /* 32 8 */ dma_addr_t dst; /* 40 8 */ struct dma_async_tx_descriptor async_tx; /* 48 144 */ /* --- cacheline 3 boundary (192 bytes) --- */ /* size: 192, cachelines: 3 */ }; /* definitions: 1 */ Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2007-07-31[NET_DMA]: remove unused dma_memcpy_to_kernel_iovecShannon Nelson1-27/+0
Al Viro pointed out that dma_memcpy_to_kernel_iovec() really was unreachable and thus unused. The code originally was there to support in-kernel dma needs, but since it remains unused, we'll pull it out. Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2007-07-16dma-mapping: prevent dma dependent code from linking on !HAS_DMA archsDan Williams1-1/+1
Continuing the work started in 411f0f3edc141a582190d3605cadd1d993abb6df ... This enables code with a dma path, that compiles away, to build without requiring additional code factoring. It also prevents code that calls dma_alloc_coherent and dma_free_coherent from linking whereas previously the code would hit a BUG() at run time. Finally, it allows archs that set !HAS_DMA to delete their asm/dma-mapping.h file. Cc: Cornelia Huck <cornelia.huck@de.ibm.com> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: John W. Linville <linville@tuxdriver.com> Cc: Kyle McMartin <kyle@parisc-linux.org> Cc: James Bottomley <James.Bottomley@SteelEye.com> Cc: Tejun Heo <htejun@gmail.com> Cc: Jeff Garzik <jeff@garzik.org> Cc: <geert@linux-m68k.org> Cc: <zippel@linux-m68k.org> Cc: <spyro@f2s.com> Cc: <ysato@users.sourceforge.jp> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-13ioatdma: add the unisys "i/oat" pci vendor/device idDan Williams1-0/+2
Cc: John Magolan <john.magolan@unisys.com> Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2007-07-13dmaengine: driver for the iop32x, iop33x, and iop13xx raid enginesDan Williams3-0/+1477
The Intel(R) IOP series of i/o processors integrate an Xscale core with raid acceleration engines. The capabilities per platform are: iop219: (2) copy engines iop321: (2) copy engines (1) xor and block fill engine iop33x: (2) copy and crc32c engines (1) xor, xor zero sum, pq, pq zero sum, and block fill engine iop34x (iop13xx): (2) copy, crc32c, xor, xor zero sum, and block fill engines (1) copy, crc32c, xor, xor zero sum, pq, pq zero sum, and block fill engine The driver supports the features of the async_tx api: * asynchronous notification of operation completion * implicit (interupt triggered) handling of inter-channel transaction dependencies The driver adapts to the platform it is running by two methods. 1/ #include <asm/arch/adma.h> which defines the hardware specific iop_chan_* and iop_desc_* routines as a series of static inline functions 2/ The private platform data attached to the platform_device defines the capabilities of the channels 20070626: Callbacks are run in a tasklet. Given the recent discussion on LKML about killing tasklets in favor of workqueues I did a quick conversion of the driver. Raid5 resync performance dropped from 50MB/s to 30MB/s, so the tasklet implementation remains until a generic softirq interface is available. Changelog: * fixed a slot allocation bug in do_iop13xx_adma_xor that caused too few slots to be requested eventually leading to data corruption * enabled the slot allocation routine to attempt to free slots before returning -ENOMEM * switched the cleanup routine to solely use the software chain and the status register to determine if a descriptor is complete. This is necessary to support other IOP engines that do not have status writeback capability * make the driver iop generic * modified the allocation routines to understand allocating a group of slots for a single operation * added a null xor initialization operation for the xor only channel on iop3xx * support xor operations on buffers larger than the hardware maximum * split the do_* routines into separate prep, src/dest set, submit stages * added async_tx support (dependent operations initiation at cleanup time) * simplified group handling * added interrupt support (callbacks via tasklets) * brought the pending depth inline with ioat (i.e. 4 descriptors) * drop dma mapping methods, suggested by Chris Leech * don't use inline in C files, Adrian Bunk * remove static tasklet declarations * make iop_adma_alloc_slots easier to read and remove chances for a corrupted descriptor chain * fix locking bug in iop_adma_alloc_chan_resources, Benjamin Herrenschmidt * convert capabilities over to dma_cap_mask_t * fixup sparse warnings * add descriptor flush before iop_chan_enable * checkpatch.pl fixes * gpl v2 only correction * move set_src, set_dest, submit to async_tx methods * move group_list and phys to async_tx Cc: Russell King <rmk@arm.linux.org.uk> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2007-07-13async_tx: add the async_tx apiDan Williams1-3/+2
The async_tx api provides methods for describing a chain of asynchronous bulk memory transfers/transforms with support for inter-transactional dependencies. It is implemented as a dmaengine client that smooths over the details of different hardware offload engine implementations. Code that is written to the api can optimize for asynchronous operation and the api will fit the chain of operations to the available offload resources. I imagine that any piece of ADMA hardware would register with the 'async_*' subsystem, and a call to async_X would be routed as appropriate, or be run in-line. - Neil Brown async_tx exploits the capabilities of struct dma_async_tx_descriptor to provide an api of the following general format: struct dma_async_tx_descriptor * async_<operation>(..., struct dma_async_tx_descriptor *depend_tx, dma_async_tx_callback cb_fn, void *cb_param) { struct dma_chan *chan = async_tx_find_channel(depend_tx, <operation>); struct dma_device *device = chan ? chan->device : NULL; int int_en = cb_fn ? 1 : 0; struct dma_async_tx_descriptor *tx = device ? device->device_prep_dma_<operation>(chan, len, int_en) : NULL; if (tx) { /* run <operation> asynchronously */ ... tx->tx_set_dest(addr, tx, index); ... tx->tx_set_src(addr, tx, index); ... async_tx_submit(chan, tx, flags, depend_tx, cb_fn, cb_param); } else { /* run <operation> synchronously */ ... <operation> ... async_tx_sync_epilog(flags, depend_tx, cb_fn, cb_param); } return tx; } async_tx_find_channel() returns a capable channel from its pool. The channel pool is organized as a per-cpu array of channel pointers. The async_tx_rebalance() routine is tasked with managing these arrays. In the uniprocessor case async_tx_rebalance() tries to spread responsibility evenly over channels of similar capabilities. For example if there are two copy+xor channels, one will handle copy operations and the other will handle xor. In the SMP case async_tx_rebalance() attempts to spread the operations evenly over the cpus, e.g. cpu0 gets copy channel0 and xor channel0 while cpu1 gets copy channel 1 and xor channel 1. When a dependency is specified async_tx_find_channel defaults to keeping the operation on the same channel. A xor->copy->xor chain will stay on one channel if it supports both operation types, otherwise the transaction will transition between a copy and a xor resource. Currently the raid5 implementation in the MD raid456 driver has been converted to the async_tx api. A driver for the offload engines on the Intel Xscale series of I/O processors, iop-adma, is provided in a later commit. With the iop-adma driver and async_tx, raid456 is able to offload copy, xor, and xor-zero-sum operations to hardware engines. On iop342 tiobench showed higher throughput for sequential writes (20 - 30% improvement) and sequential reads to a degraded array (40 - 55% improvement). For the other cases performance was roughly equal, +/- a few percentage points. On a x86-smp platform the performance of the async_tx implementation (in synchronous mode) was also +/- a few percentage points of the original implementation. According to 'top' on iop342 CPU utilization drops from ~50% to ~15% during a 'resync' while the speed according to /proc/mdstat doubles from ~25 MB/s to ~50 MB/s. The tiobench command line used for testing was: tiobench --size 2048 --block 4096 --block 131072 --dir /mnt/raid --numruns 5 * iop342 had 1GB of memory available Details: * if CONFIG_DMA_ENGINE=n the asynchronous path is compiled away by making async_tx_find_channel a static inline routine that always returns NULL * when a callback is specified for a given transaction an interrupt will fire at operation completion time and the callback will occur in a tasklet. if the the channel does not support interrupts then a live polling wait will be performed * the api is written as a dmaengine client that requests all available channels * In support of dependencies the api implicitly schedules channel-switch interrupts. The interrupt triggers the cleanup tasklet which causes pending operations to be scheduled on the next channel * Xor engines treat an xor destination address differently than a software xor routine. To the software routine the destination address is an implied source, whereas engines treat it as a write-only destination. This patch modifies the xor_blocks routine to take a an explicit destination address to mirror the hardware. Changelog: * fixed a leftover debug print * don't allow callbacks in async_interrupt_cond * fixed xor_block changes * fixed usage of ASYNC_TX_XOR_DROP_DEST * drop dma mapping methods, suggested by Chris Leech * printk warning fixups from Andrew Morton * don't use inline in C files, Adrian Bunk * select the API when MD is enabled * BUG_ON xor source counts <= 1 * implicitly handle hardware concerns like channel switching and interrupts, Neil Brown * remove the per operation type list, and distribute operation capabilities evenly amongst the available channels * simplify async_tx_find_channel to optimize the fast path * introduce the channel_table_initialized flag to prevent early calls to the api * reorganize the code to mimic crypto * include mm.h as not all archs include it in dma-mapping.h * make the Kconfig options non-user visible, Adrian Bunk * move async_tx under crypto since it is meant as 'core' functionality, and the two may share algorithms in the future * move large inline functions into c files * checkpatch.pl fixes * gpl v2 only correction Cc: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-By: NeilBrown <neilb@suse.de>
2007-07-13dmaengine: make clients responsible for managing channelsDan Williams3-110/+111
The current implementation assumes that a channel will only be used by one client at a time. In order to enable channel sharing the dmaengine core is changed to a model where clients subscribe to channel-available-events. Instead of tracking how many channels a client wants and how many it has received the core just broadcasts the available channels and lets the clients optionally take a reference. The core learns about the clients' needs at dma_event_callback time. In support of multiple operation types, clients can specify a capability mask to only be notified of channels that satisfy a certain set of capabilities. Changelog: * removed DMA_TX_ARRAY_INIT, no longer needed * dma_client_chan_free -> dma_chan_release: switch to global reference counting only at device unregistration time, before it was also happening at client unregistration time * clients now return dma_state_client to dmaengine (ack, dup, nak) * checkpatch.pl fixes * fixup merge with git-ioat Cc: Chris Leech <christopher.leech@intel.com> Signed-off-by: Shannon Nelson <shannon.nelson@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Acked-by: David S. Miller <davem@davemloft.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-12 02:13:21 +0000