summaryrefslogtreecommitdiff
path: root/arch/arm/mm/cache-v6.S
AgeCommit message (Collapse)AuthorFilesLines
2024-05-07ARM: 9393/1: mm: Use conditionals for CFI branchesLinus Walleij1-0/+2
Commit 9385/2 introduced a few branches inside function prototypes when using CFI in order to deal with the situation where CFI inserts a few bytes of function information in front of the symbol. This is not good for older CPUs where every cycle counts. Commit 9386/2 alleviated the situation a bit by using aliases for the cache functions with identical signatures. This leaves the coherent cache flush functions *_coherent_kern_range() with these branches to the corresponing *_coherent_user_range() around, since their return type differ and they therefore cannot be aliased. Solve this by a simple ifdef so at least we can use fallthroughs when compiling without CFI enabled. Link: https://lore.kernel.org/linux-arm-kernel/Zi+e9M%2Ff5b%2FSto9H@shell.armlinux.org.uk/ Suggested-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
2024-04-29ARM: 9387/2: mm: Rewrite cacheflush vtables in CFI safe CLinus Walleij1-8/+0
Instead of defining all cache flush operations with an assembly macro in proc-macros.S, provide an explicit struct cpu_cache_fns for each CPU cache type in mm/cache.c. As a side effect from rewriting the vtables in C, we can avoid the aliasing for the "louis" cache callback, instead we can just assign the NN_flush_kern_cache_all() function to the louis callback in the C vtable. As the louis cache callback is called explicitly (not through the vtable) if we only have one type of cache support compiled in, we need an ifdef quirk for this in the !MULTI_CACHE case. Feroceon and XScale have some dma mapping quirk, in this case we can just define two structs and assign all but one callback to the main implementation; since each of them invoked define_cache_functions twice they require MULTI_CACHE by definition so the compiled-in shortcut is not used on these variants. Tested-by: Kees Cook <keescook@chromium.org> Reviewed-by: Sami Tolvanen <samitolvanen@google.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
2024-04-29ARM: 9385/2: mm: Type-annotate all cache assembly routinesLinus Walleij1-18/+23
Tag all references to assembly functions with SYM_TYPED_FUNC_START() and SYM_FUNC_END() so they also become CFI-safe. When we add SYM_TYPED_FUNC_START() to assembly calls, a function prototype signature will be emitted into the object file at (pc-4) at the call site, so that the KCFI runtime check can compare this to the expected call. Example: 8011ae38: a540670c .word 0xa540670c 8011ae3c <v7_flush_icache_all>: 8011ae3c: e3a00000 mov r0, #0 8011ae40: ee070f11 mcr 15, 0, r0, cr7, cr1, {0} 8011ae44: e12fff1e bx lr This means no "fallthrough" code can enter a SYM_TYPED_FUNC_START() call from above it: there will be a function prototype signature there, so those are consistently converted to a branch or ret lr depending on context. Tested-by: Kees Cook <keescook@chromium.org> Reviewed-by: Sami Tolvanen <samitolvanen@google.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
2023-12-22ARM: Delete ARM11MPCore (ARM11 ARMv6K SMP) supportLinus Walleij1-31/+0
This ARM11 SMP configuration was one of the first SMP configurations the ARM kernel supported, but it has the downside of odd DMA handling, odd cache tagging, and often (as of recent) completely broken cache handling on the ARM RealView PB11MPCore test chips. To boot the platform it was necessary to completely disable the cache. When it comes to the EB 11MPCore it is unclear if this ever worked. These reference designs are now the only ARMv6K SMP platforms. As only reference designs of purely academic interest remain, and since the special-cased DMA and PMU code is hard to maintain and doesn't really work, it is not really worth our time. Delete the ARM11MPCore support along with: - The special DMA quirk CONFIG_DMA_CACHE_RWFO that is only used on ARMv6K SMP, and we are the last ARMV6K system leaving the building and the cache handling is awkward, so good-bye. - The special PMU handling that was only used by ARM11MPCore. The following is left behind: - TIMER_OF_DECLARE(arm_twd_11mp, "arm,arm11mp-twd-timer", ...) in arch/arm/kernel/smp_twd.c, this is still in use by Marvell MMP3 arch/arm/boot/dts/marvell/mmp3.dtsi - IRQCHIP_DECLARE(arm11mp_gic, "arm,arm11mp-gic", ...) in drivers/irqchip/irq-gic.c, this is still in use by Marvell MMP3 arch/arm/boot/dts/marvell/mmp3.dtsi - A compatible for the arm11mpcore SCU, since this was mistakedly used for the Cortex-A9 version of RealView EB. These are unfortunate but will need to be kept around for compatibility. New Marvell-specific compatibles should however probably be added. Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Liviu Dudau <liviu.dudau@arm.com> Link: https://lore.kernel.org/r/20231207-drop-11mpcore-v2-1-560b396f3bf5@linaro.org Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2022-11-08ARM: 9263/1: use .arch directives instead of assembler command line flagsNick Desaulniers1-0/+2
Similar to commit a6c30873ee4a ("ARM: 8989/1: use .fpu assembler directives instead of assembler arguments"). GCC and GNU binutils support setting the "sub arch" via -march=, -Wa,-march, target function attribute, and .arch assembler directive. Clang was missing support for -Wa,-march=, but this was implemented in clang-13. The behavior of both GCC and Clang is to prefer -Wa,-march= over -march= for assembler and assembler-with-cpp sources, but Clang will warn about the -march= being unused. clang: warning: argument unused during compilation: '-march=armv6k' [-Wunused-command-line-argument] Since most assembler is non-conditionally assembled with one sub arch (modulo arch/arm/delay-loop.S which conditionally is assembled as armv4 based on CONFIG_ARCH_RPC, and arch/arm/mach-at91/pm-suspend.S which is conditionally assembled as armv7-a based on CONFIG_CPU_V7), prefer the .arch assembler directive. Add a few more instances found in compile testing as found by Arnd and Nathan. Link: https://github.com/llvm/llvm-project/commit/1d51c699b9e2ebc5bcfdbe85c74cc871426333d4 Link: https://bugs.llvm.org/show_bug.cgi?id=48894 Link: https://github.com/ClangBuiltLinux/linux/issues/1195 Link: https://github.com/ClangBuiltLinux/linux/issues/1315 Suggested-by: Arnd Bergmann <arnd@arndb.de> Suggested-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Arnd Bergmann <arnd@arndb.de> Tested-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nick Desaulniers <ndesaulniers@google.com> Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500Thomas Gleixner1-4/+1
Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation # extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 4122 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Enrico Weigelt <info@metux.net> Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-02-26ARM: 8844/1: use unified assembler in assembly filesStefan Agner1-4/+4
Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2014-07-18ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+Russell King1-10/+10
ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-09-25ARM: mm: implement LoUIS API for cache maintenance opsLorenzo Pieralisi1-0/+3
ARM v7 architecture introduced the concept of cache levels and related control registers. New processors like A7 and A15 embed an L2 unified cache controller that becomes part of the cache level hierarchy. Some operations in the kernel like cpu_suspend and __cpu_disable do not require a flush of the entire cache hierarchy to DRAM but just the cache levels belonging to the Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems correspond to L1. The current cache flushing API used in cpu_suspend and __cpu_disable, flush_cache_all(), ends up flushing the whole cache hierarchy since for v7 it cleans and invalidates all cache levels up to Level of Coherency (LoC) which cripples system performance when used in hot paths like hotplug and cpuidle. Therefore a new kernel cache maintenance API must be added to cope with latest ARM system requirements. This patch adds flush_cache_louis() to the ARM kernel cache maintenance API. This function cleans and invalidates all data cache levels up to the Level of Unification Inner Shareable (LoUIS) and invalidates the instruction cache for processors that support it (> v7). This patch also creates an alias of the cache LoUIS function to flush_kern_all for all processor versions prior to v7, so that the current cache flushing behaviour is unchanged for those processors. v7 cache maintenance code implements a cache LoUIS function that cleans and invalidates the D-cache up to LoUIS and invalidates the I-cache, according to the new API. Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Tested-by: Shawn Guo <shawn.guo@linaro.org>
2012-05-02ARM: 7408/1: cacheflush: return error to userspace when flushing syscall failsWill Deacon1-6/+4
The cacheflush syscall can fail for two reasons: (1) The arguments are invalid (nonsensical address range or no VMA) (2) The region generates a translation fault on a VIPT or PIPT cache This patch allows do_cache_op to return an error code to userspace in the case of the above. The various coherent_user_range implementations are modified to return 0 in the case of VIVT caches or -EFAULT in the case of an abort on v6/v7 cores. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-07-07ARM: mm: cache-v6: Use the new processor struct macrosDave Martin1-13/+2
Signed-off-by: Dave Martin <dave.martin@linaro.org>
2011-05-26ARM: 6941/1: cache: ensure MVA is cacheline aligned in flush_kern_dcache_areaWill Deacon1-0/+1
The v6 and v7 implementations of flush_kern_dcache_area do not align the passed MVA to the size of a cacheline in the data cache. If a misaligned address is used, only a subset of the requested area will be flushed. This has been observed to cause failures in SMP boot where the secondary_data initialised by the primary CPU is not cacheline aligned, causing the secondary CPUs to read incorrect values for their pgd and stack pointers. This patch ensures that the base address is cacheline aligned before flushing the d-cache. Cc: <stable@kernel.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-12-14ARM: 6535/1: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fixValentine Barshak1-8/+20
Cache ownership must be acquired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never acquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary. Fix this by reading/writing data if needed, before performing cache operations. While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer. Cc: stable@kernel.org Signed-off-by: Valentine Barshak <vbarshak@mvista.com> Signed-off-by: George G. Davis <gdavis@mvista.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-10-04ARM: 6405/1: Handle __flush_icache_all for CONFIG_SMP_ON_UPTony Lindgren1-11/+19
Do this by adding flush_icache_all to cache_fns for ARMv6 and 7. As flush_icache_all may neeed to be called from flush_kern_cache_all, add it as the first entry in the cache_fns. Note that now we can remove the ARM_ERRATA_411920 dependency to !SMP so it can be selected on UP ARMv6 processors, such as omap2. Signed-off-by: Tony Lindgren <tony@atomide.com> Signed-off-by: Anand Gadiyar <gadiyar@ti.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-01ARM: 6188/1: Add a config option for the ARM11MPCore DMA cache maintenance ↵Catalin Marinas1-3/+12
workaround Commit f4d6477f introduced a workaround for the lack of hardware broadcasting of the cache maintenance operations on ARM11MPCore. However, the workaround is only valid on CPUs that do not do speculative loads into the D-cache. This patch adds a Kconfig option with the corresponding help to make the above clear. When the DMA_CACHE_RWFO option is disabled, the kernel behaviour is that prior to the f4d6477f commit. This also allows ARMv6 UP processors with speculative loads to work correctly. For other processors, a different workaround may be needed. Cc: Ronen Shitrit <rshitrit@marvell.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-01ARM: 6187/1: The v6_dma_inv_range() function must preserve data on SMPCatalin Marinas1-1/+2
A recent patch for DMA cache maintenance on ARM11MPCore added a write for ownership trick to the v6_dma_inv_range() function. Such operation destroys data already present in the buffer. However, this function is used with with dma_sync_single_for_device() which is supposed to preserve the existing data transfered into the buffer. This patch adds a combination of read/write for ownership to preserve the original data. Reported-by: Ronen Shitrit <rshitrit@marvell.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-05-08ARM: 6111/1: Implement read/write for ownership in the ARMv6 DMA cache opsCatalin Marinas1-4/+13
The Snoop Control Unit on the ARM11MPCore hardware does not detect the cache operations and the dma_cache_maint*() functions may leave stale cache entries on other CPUs. The solution implemented in this patch performs a Read or Write For Ownership in the ARMv6 DMA cache maintenance functions. These LDR/STR instructions change the cache line state to shared or exclusive so that the cache maintenance operation has the desired effect. Tested-by: George G. Davis <gdavis@mvista.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-02-15ARM: dma-mapping: fix for speculative prefetchingRussell King1-4/+6
ARMv6 and ARMv7 CPUs can perform speculative prefetching, which makes DMA cache coherency handling slightly more interesting. Rather than being able to rely upon the CPU not accessing the DMA buffer until DMA has completed, we now must expect that the cache could be loaded with possibly stale data from the DMA buffer. Where DMA involves data being transferred to the device, we clean the cache before handing it over for DMA, otherwise we invalidate the buffer to get rid of potential writebacks. On DMA Completion, if data was transferred from the device, we invalidate the buffer to get rid of any stale speculative prefetches. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk> Tested-By: Santosh Shilimkar <santosh.shilimkar@ti.com>
2010-02-15ARM: dma-mapping: remove dmac_clean_range and dmac_inv_rangeRussell King1-4/+2
These are now unused, and so can be removed. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk> Tested-By: Santosh Shilimkar <santosh.shilimkar@ti.com>
2010-02-15ARM: dma-mapping: provide per-cpu type map/unmap functionsRussell King1-0/+26
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk> Tested-By: Santosh Shilimkar <santosh.shilimkar@ti.com>
2009-12-14ARM: add size argument to __cpuc_flush_dcache_pageRussell King1-5/+6
... and rename the function since it no longer operates on just pages. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-10-07ARM: 5746/1: Handle possible translation errors in ARMv6/v7 coherent_user_rangeCatalin Marinas1-2/+18
This is needed because applications using the sys_cacheflush system call can pass a memory range which isn't mapped yet even though the corresponding vma is valid. The patch also adds unwinding annotations for correct backtraces from the coherent_user_range() functions. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-04-30[ARM] 5488/1: ARM errata: Invalidation of the Instruction Cache operation ↵Catalin Marinas1-0/+33
can fail This patch implements the recommended workaround for erratum 411920 (ARM1136, ARM1156, ARM1176). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-03-10[ARM] 3356/1: Workaround for the ARM1136 I-cache invalidation problemCatalin Marinas1-3/+4
Patch from Catalin Marinas ARM1136 erratum 371025 (category 2) specifies that, under rare conditions, an invalidate I-cache by MVA (line or range) operation can fail to invalidate a cache line. The recommended workaround is to either invalidate the entire I-cache or invalidate the range by set/way rather than MVA. Note that for a 16K cache size, invalidating a 4K page by set/way is equivalent to invalidating the entire I-cache. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-02-01[ARM] 3294/1: don't invalidate individual BTB entries on ARMv6Nicolas Pitre1-12/+6
Patch from Nicolas Pitre Doing so adds a much larger cost to the loop than the cost implied by simply invalidating the whole BTB at once. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-09-30[ARM] 2940/1: Fix BTB entry flush in arch/arm/mm/cache-v6.SGen FUKATSU1-1/+8
Patch from Gen FUKATSU Invalidate BTB entry instruction flushes two instruction at a time. Therefore this instruction should be done four times after invalidate instruction cache line. Signed-off-by: Gen Fukatsu Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-16Linux-2.6.12-rc2Linus Torvalds1-0/+227
Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!