l10n: use translations of .uno: commands from LibreOffice

This is a combination of 22 commits.

l10n: New script for collecting the used .uno: commands.

Change-Id: I9c6f3f03a4042f8e5ba1f258ce468fc0f4abd169
Reviewed-on: https://gerrit.libreoffice.org/45981
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 99161841dcee3b066153a7f7af0e98d6ef7f8a2c)

l10n: Don't repeat the strings in the menu definition, use the uno commands.

Change-Id: If8ccd753cd88056c64c10e3f9f7b951208fafdb8
Reviewed-on: https://gerrit.libreoffice.org/45983
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 5b87f2fca5d6f84d0ac637021e87e2dc41dd13c7)

l10n: "type: 'unocommand'" is implicit when there is "uno:" in the item.

Change-Id: I2fe50e4860848920f551cc6a3fda5ebe5f49b3f2
Reviewed-on: https://gerrit.libreoffice.org/46011
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit d38664c023f374dc66361d00be89c08a0e5bbc20)

l10n: Fix mis-typed .uno: commands.

Change-Id: Ia2d695d584d48b92def364c3936655909aabd076
Reviewed-on: https://gerrit.libreoffice.org/46012
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit e68789ac2c9c3bfa661ae815ff27cd4bb038bc2c)

l10n: Don't extract uno: when name: is provided for that entry.

Change-Id: I85619915cf66a71c7ab982c2949390bd82df9abb
Reviewed-on: https://gerrit.libreoffice.org/46013
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 293c5d45083823ae1c1f4e0519928c0b5d23dced)

l10n: Implement --check to notice not covered .uno: commands early.

Change-Id: I928f2cef8d9a869c10fa6c6370a7ce1c811631c8
Reviewed-on: https://gerrit.libreoffice.org/46014
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 457f929357adf7f52e0fc9281d41df3f29859467)

l10n: Further decrease the use of own Online strings...

...if those can be deducted from .uno: commands.

Change-Id: I9c2459a12ee95e936a7ec37f6270012d811cdf86
Reviewed-on: https://gerrit.libreoffice.org/46018
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 70ecee2bb6398aafdf6c71aeecf6e9eb6330fb70)

l10n: Let's avoid redirect, and write the unocommands.js directly.

Change-Id: Ifda0de27881ba415ff062368483524080dce5189
Reviewed-on: https://gerrit.libreoffice.org/46020
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit b6fa6f376c41d6787ab1cbb52218e7ff1064b83a)

l10n: Generate translations of the .uno: commands + update the current ones.

Change-Id: I77377412e4776a830f15973a35b83e8eca299f05
Reviewed-on: https://gerrit.libreoffice.org/46047
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit b2bfb2349ec2934a611bb4c1393e9f17aedb0ba2)

l10n: Translate also the right-click menu .uno: commands.

Change-Id: Ie5cedaaa10febe78ba319f9e9b03fdd44c94f468
Reviewed-on: https://gerrit.libreoffice.org/46048
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 861c252d5204dfac7b8f5fb4505efcc29eafeb3d)

l10n: len(sys.argv) is 1, when the command is invoked without options

Change-Id: Ie9a47955e9a031ee0484c0703f67bad7077bbd76
Reviewed-on: https://gerrit.libreoffice.org/46078
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit f096cf33977620f368ecc7c5b64164ab2303060b)

l10n: remove obsolete unocommandsl10n.sh

Change-Id: I4e9945407aff4a4fa1444079f8ecec1782a71c9c
Reviewed-on: https://gerrit.libreoffice.org/46079
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit 4f0d5d7a2431bf5e132bdcd85285c10c0fd21322)

l10n: Make sure to distinct between apps + between main menu and context menu.

The strings can differ considerably, making some menu entries not
understandable at all.

Change-Id: I36ffa8b01df119a7655a711ec859391af16836c9
Reviewed-on: https://gerrit.libreoffice.org/46097
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit 53d9593edb32bbc5db3fe430ac5f20a1bdd4e127)

l10n: Fix the Table menu in Impress.

Change-Id: I2cb8b98ecc1a18174b26dfaa9c96ad3829ff5989
Reviewed-on: https://gerrit.libreoffice.org/46099
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit be100fe06abbeabed05b0c8a02c4f6067fcfc774)

l10n: first check for 'lang', then require unocommands

Change-Id: Ib8f94bbdd163a89bef4886d68816628e124e9ef8
(cherry picked from commit d94a53a0e363f6edf3a139d26c4b9603ef14389f)

l10n: get Language menu translations for free

Change-Id: I8b5628457cf713d8fdb0d4de6935b7e1d54ad093
Reviewed-on: https://gerrit.libreoffice.org/46158
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit 53b8a60d5d85fc1a3909809ea51a23cb20453bea)

l10n: Calc and Impress should use _UNO('.uno:LanguageMenu') too.

Change-Id: I987821d9d9b0f40baea1879d28177e8e0c58a3a3
Reviewed-on: https://gerrit.libreoffice.org/46418
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>
(cherry picked from commit ba61ec9da2c073d2d728c5bb8df8912f5117fd5e)

l10n: Calc row, column and sheet tab context menu translations from LibreOffice

Change-Id: I1f566a71e5fdc66304e82bc2bcfb0f9ced2f5d2a
Reviewed-on: https://gerrit.libreoffice.org/46430
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit e8847b9ef174288b73650a4e63e8a6b0cfb43b4c)

l10n: Some toolbar button tooltip translations from LibreOffice

Change-Id: I4a8d050c7ae8efc7d1932f16dc35455f1a182d8a
Reviewed-on: https://gerrit.libreoffice.org/46444
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit d51a760957480ea412519f677dae7a2294c324cd)

l10n: localize 'Undo' and 'Redo' labels on Document Repair dialog

Change-Id: Ibd4236f85a6db92b7d6456f6f22115b35079da83
Reviewed-on: https://gerrit.libreoffice.org/47252
Reviewed-by: Jan Holesovsky <kendy@collabora.com>
Tested-by: Jan Holesovsky <kendy@collabora.com>
(cherry picked from commit f8bf2c28608c139279aaa484603da8cc539ff948)

l10n: 5 more toolbar strings from LO core

Change-Id: I398a0d7c28a8295c3b8e5077903b5fa7d3d5c45c
Reviewed-on: https://gerrit.libreoffice.org/47310
Reviewed-by: Andras Timar <andras.timar@collabora.com>
Tested-by: Andras Timar <andras.timar@collabora.com>

2 files changed, 359 insertions, 12 deletions

diff --git a/scripts/unocommands.py b/scripts/unocommands.py
new file mode 100755
index 000000000..d3b7a887d
--- /dev/null
+++ b/scripts/unocommands.py
@@ -0,0 +1,359 @@
+#!/usr/bin/env python
+# -*- tab-width: 4; indent-tabs-mode: nil; py-indent-offset: 4 -*-
+#
+# This file is part of the LibreOffice project.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+#
+
+import os
+import polib
+import re
+import sys
+from lxml import etree
+
+def usageAndExit():
+    message = """usage: {program} [--check|--update|--translate] online_dir [...]
+
+Checks, extracts, or translates .uno: command descriptions from the
+LibreOffice XCU files.
+
+Check whether all the commands in the menus have their descriptions in
+unocommands.js:
+
+    {program} --check /path/to/online
+
+Update the unocommands.js by fetching the .uno: commands descriptions from the
+core.git.  This is what you want to do after you add new .uno: commands or
+dialogs to the menus:
+
+    {program} --update /path/to/online /path/to/loffice
+
+Update the translations of unocommands.js before releasing:
+
+    {program} --translate /path/to/online /path/to/translations
+
+"""
+    print(message.format(program = os.path.basename(sys.argv[0])))
+    exit(1)
+
+# Extract uno commands name from lines like "  'Command1', 'Command2',"
+def commandsFromLine(line):
+    commands = []
+
+    inCommand = False
+    command = ''
+    for c in line:
+        if c == "'":
+            inCommand = not inCommand
+            # command ended, collect it
+            if not inCommand and command != '':
+                commands += [ command ]
+                command = ''
+        elif inCommand:
+            command += c
+
+    return commands
+
+# Extract uno commands name from lines like "  {uno: '.uno:Command3',"
+def commandFromMenuLine(line):
+    m = re.search(r"\buno: *'\.uno:([^']*)'", line)
+    if m:
+        return [ m.group(1) ]
+
+    m = re.search(r"\b_UNO\('.uno:([^']*)'", line)
+    if m:
+        return [ m.group(1) ]
+
+    return []
+
+# Extract all the uno commands we are using in the Online menu
+def extractMenuCommands(path):
+    commands = []
+
+    # extract from the menu specifications
+    f = open(path + '/loleaflet/src/control/Control.Menubar.js', 'r')
+    for line in f:
+        if line.find("uno:") >= 0 and line.find("name:") < 0:
+            commands += commandFromMenuLine(line)
+        elif line.find("_UNO(") >= 0:
+            commands += commandFromMenuLine(line)
+
+    # may the list unique
+    return set(commands)
+
+# Extract all the uno commands we are using in the Online context menu
+def extractContextCommands(path):
+    commands = []
+
+    # extract from the comments whitelist
+    f = open(path + '/loleaflet/src/control/Control.ContextMenu.js', 'r')
+    readingCommands = False
+    for line in f:
+        if line.find('UNOCOMMANDS_EXTRACT_START') >= 0:
+            readingCommands = True
+        elif line.find('UNOCOMMANDS_EXTRACT_END') >= 0:
+            readingCommands = False
+        elif readingCommands:
+            commands += commandsFromLine(line)
+
+    f = open(path + '/loleaflet/src/control/Control.ColumnHeader.js', 'r')
+    for line in f:
+        if line.find("_UNO(") >= 0:
+            commands += commandFromMenuLine(line)
+
+    f = open(path + '/loleaflet/src/control/Control.RowHeader.js', 'r')
+    for line in f:
+        if line.find("_UNO(") >= 0:
+            commands += commandFromMenuLine(line)
+
+    f = open(path + '/loleaflet/src/control/Control.Tabs.js', 'r')
+    for line in f:
+        if line.find("_UNO(") >= 0:
+            commands += commandFromMenuLine(line)
+
+    # may the list unique
+    return set(commands)
+
+# Extract all the uno commands we are using in the Online toolbar
+def extractToolbarCommands(path):
+    commands = []
+
+    # extract from the menu specifications
+    f = open(path + '/loleaflet/dist/toolbar/toolbar.js', 'r')
+    for line in f:
+        if line.find("_UNO(") >= 0:
+            commands += commandFromMenuLine(line)
+
+    # may the list unique
+    return set(commands)
+
+# Create mapping between the commands and appropriate strings
+def collectCommandsFromXCU(xcu, descriptions, commands, label, type):
+    root = etree.parse(xcu)
+    nodes = root.xpath("/oor:component-data/node/node/node", namespaces = {
+        'oor': 'http://openoffice.org/2001/registry',
+        })
+    for node in nodes:
+        # extract the uno command name
+        unoCommand = node.get('{http://openoffice.org/2001/registry}name')
+        unoCommand = unoCommand[5:]
+
+        if unoCommand in commands:
+            # normal labels
+            textElement = node.xpath('prop[@oor:name="' + label + '"]/value', namespaces = {
+                'oor': 'http://openoffice.org/2001/registry',
+                })
+
+            if len(textElement) == 1:
+                # extract the uno command's English text
+                text = ''.join(textElement[0].itertext())
+                command = {}
+                if unoCommand in descriptions.keys():
+                    command = descriptions[unoCommand]
+
+                if not type in command:
+                    command[type] = {}
+
+                menuType = 'menu'
+                if label == 'PopupLabel' or label == 'TooltipLabel':
+                    menuType = 'context'
+
+                if menuType in command[type]:
+                    continue
+
+                command[type][menuType] = text
+
+                descriptions[unoCommand] = command
+
+    return descriptions
+
+# Print commands from all the XCU files, and collect them too
+def writeUnocommandsJS(onlineDir, lofficeDir, menuCommands, contextCommands, toolbarCommands):
+    descriptions = {}
+    dir = lofficeDir + '/officecfg/registry/data/org/openoffice/Office/UI'
+    for file in os.listdir(dir):
+        if file.endswith('.xcu'):
+            type = 'global';
+            if file.startswith('Writer'):
+                type = 'text'
+            elif file.startswith('Calc'):
+                type = 'spreadsheet'
+            elif file.startswith('DrawImpress'):
+                type = 'presentation'
+
+            # main menu
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, menuCommands, 'ContextLabel', type)
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, contextCommands, 'ContextLabel', type)
+
+            # right-click menu
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, contextCommands, 'PopupLabel', type)
+
+            # toolbar
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, toolbarCommands, 'PopupLabel', type)
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, toolbarCommands, 'TooltipLabel', type)
+
+            # fallbacks
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, menuCommands, 'Label', type)
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, contextCommands, 'Label', type)
+            descriptions = collectCommandsFromXCU(os.path.join(dir, file), descriptions, toolbarCommands, 'Label', type)
+
+    # output the unocommands.js
+    f = open(onlineDir + '/loleaflet/unocommands.js', 'w')
+    f.write('''// Don't modify, generated using unocommands.py
+
+var unoCommandsArray = {\n''')
+
+    for key in sorted(descriptions.keys()):
+        #f.write(('    ' + key + ": _('" + descriptions[key] + "'),\n").encode('utf-8'))
+        f.write(('\t' + key + ':{').encode('utf-8'))
+        for type in sorted(descriptions[key].keys()):
+            f.write((type + ':{').encode('utf-8'))
+            for menuType in sorted(descriptions[key][type].keys()):
+                f.write((menuType + ":_('" + descriptions[key][type][menuType] + "'),").encode('utf-8'))
+            f.write(('},').encode('utf-8'))
+        f.write(('},\n').encode('utf-8'))
+
+    f.write('''};
+
+global._UNO = function(string, component, isContext) {
+\tvar command = string.substr(5);
+\tvar context = 'menu';
+\tif (isContext === true) {
+\t\tcontext = 'context';
+\t}
+\tvar entry = unoCommandsArray[command];
+\tif (entry === undefined) {
+\t\treturn command;
+\t}
+\tvar componentEntry = entry[component];
+\tif (componentEntry === undefined) {
+\t\tcomponentEntry = entry['global'];
+\t\tif (componentEntry === undefined) {
+\t\t\treturn command;
+\t\t}
+\t}
+\tvar text = componentEntry[context];
+\tif (text === undefined) {
+\t\ttext = componentEntry['menu'];
+\t\tif (text === undefined) {
+\t\t\treturn command;
+\t\t}
+\t}
+
+\treturn text.replace('~', '');
+}\n''')
+
+    return descriptions
+
+# Read the uno commands present in the unocommands.js for checking
+def parseUnocommandsJS(onlineDir):
+    strings = {}
+
+    f = open(onlineDir + '/loleaflet/unocommands.js', 'r')
+    readingCommands = False
+    for line in f:
+        line = line.decode('utf-8')
+        m = re.match(r"\t([^:]*):.*", line)
+        if m:
+            command = m.group(1)
+
+            n = re.findall(r"_\('([^']*)'\)", line)
+            if n:
+                strings[command] = n
+
+    return strings
+
+# Generate translation JSONs for the .uno: commands
+def writeTranslations(onlineDir, translationsDir, strings):
+    keys = set(strings.keys())
+
+    dir = translationsDir + '/source/'
+    for lang in os.listdir(dir):
+        poFile = dir + lang + '/officecfg/registry/data/org/openoffice/Office/UI.po'
+        if not os.path.isfile(poFile):
+            continue
+
+        sys.stderr.write('Generating ' + lang + '...\n')
+
+        po = polib.pofile(poFile, autodetect_encoding=False, encoding="utf-8", wrapwidth=-1)
+
+        translations = {}
+        for entry in po.translated_entries():
+            m = re.search(r"\.uno:([^\n]*)\n", entry.msgctxt)
+            if m:
+                command = m.group(1)
+                if command in keys:
+                    for text in strings[command]:
+                        if text == entry.msgid:
+                            translations[entry.msgid] = entry.msgstr
+
+        f = open(onlineDir + '/loleaflet/dist/l10n/uno/' + lang + '.json', 'w')
+        f.write('{\n')
+
+        writeComma = False
+        for key in sorted(translations.keys()):
+            if writeComma:
+                f.write(',\n')
+            else:
+                writeComma = True
+            f.write(('"' + key + '":"' + translations[key] + '"').encode('utf-8'))
+
+        f.write('\n}\n')
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        usageAndExit()
+
+    check = False
+    translate = False
+    onlineDir = ''
+    lofficeDir = ''
+    translationsDir = ''
+    if (sys.argv[1] == '--check'):
+        if len(sys.argv) != 3:
+            usageAndExit()
+
+        check = True
+        onlineDir = sys.argv[2]
+    elif (sys.argv[1] == '--translate'):
+        translate = True
+        if len(sys.argv) != 4:
+            usageAndExit()
+
+        onlineDir = sys.argv[2]
+        translationsDir = sys.argv[3]
+    elif (sys.argv[1] == "--update"):
+        if len(sys.argv) != 4:
+            usageAndExit()
+
+        onlineDir = sys.argv[2]
+        lofficeDir = sys.argv[3]
+    else:
+        usageAndExit()
+
+    menuCommands = extractMenuCommands(onlineDir)
+    contextCommands = extractContextCommands(onlineDir)
+    toolbarCommands = extractToolbarCommands(onlineDir)
+
+    processedCommands = set([])
+    parsed = {}
+    if (check or translate):
+        parsed = parseUnocommandsJS(onlineDir)
+        processedCommands = set(parsed.keys())
+    else:
+        written = writeUnocommandsJS(onlineDir, lofficeDir, menuCommands, contextCommands, toolbarCommands)
+        processedCommands = set(written.keys())
+
+    # check that we have translations for everything
+    dif = (menuCommands | contextCommands | toolbarCommands) - processedCommands
+    if len(dif) > 0:
+        sys.stderr.write("ERROR: The following commands are not covered in unocommands.js, run scripts/unocommands.py --update:\n\n.uno:" + '\n.uno:'.join(dif) + "\n\n")
+        exit(1)
+
+    if (translate):
+        writeTranslations(onlineDir, translationsDir, parsed)
+
+# vim: set shiftwidth=4 softtabstop=4 expandtab:
diff --git a/scripts/unocommandsl10n.sh b/scripts/unocommandsl10n.sh
deleted file mode 100755
index 397d5be5f..000000000
--- a/scripts/unocommandsl10n.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-LODIR=/home/timar/cp-5.1
-LEAFLETDIR=/home/timar/online
-
-for lang in `ls -1 $LODIR/translations/source`; do
-echo -ne "{" > $lang.json
-for unocommand in `grep -Pzo "(?s)^(\s*)\N*whitelist: .*?{.*?^\1}" $LEAFLETDIR/loleaflet/src/control/Control.ContextMenu.js | grep -Po "'.*?'" | tr -d "'" | sed -e "s/^/.uno:/"`;do grep -A4 $unocommand $LODIR/translations/source/$lang/officecfg/registry/data/org/openoffice/Office/UI.po | grep -P "msgid|msgstr" | tr -d "~" | sed -e "s/msgid //" -e "s/msgstr //";done | paste -d" " - - | sort | uniq | sed -e 's/" "/": "/' -e 's/$/, /' | grep -v '""' | tr -d "\n" | sed -e "s/, $//" >> $lang.json
-echo -ne "}" >> $lang.json
-done
-
-