Locale Support in Qt 5

From Qt Wiki
Jump to navigation Jump to search


WARNING: This page is a WORK-IN-PROGRESS and so may not be complete or accurate.

Early Qt localization support was weak with a design based on Windows, but with its own code and data to allow to consistent cross-platform support in an era when localization was consistently poor on all platforms. Over time new features have been introduced, but the current QLocale implementation now lags a long way behind modern localization libraries. It is incomplete, inconsistent, not fully integrated with the host system locale, and unnecessarily bloats the core library. It is clear that Qt needs a new approach.

This page documents the current state of Qt localization and researches possible solutions to the issues.

Current Support

Qt provides two core options for localization:

  • System Locale: Use the default system locale settings, the implementation being platform specific.
  • Custom Locale: An application can create any custom locale supported by the CLDR data embedded in QtCore which is then used by Qt's internal code routines. These may or may not match the host's available locales.

In theory, the System Locale should use the facilities of the host system, falling back to Qt's CLDR-based facilities where the host system doesn't provide the required data or API. In practice, Windows and Mac use an inconsistent mixture of the host facilities and Qt's CLDR-based facilities. On all other platforms, such as Linux and Android, Qt's built-in CLDR-based facilities are always used.

A number of key requirements for future improvements have been identified:

  • Minimise locale data shipped with Qt
  • Minimise localization code needing to be maintained by Qt
  • Integrate fully with the user/system locale on all platforms
  • Integrate fully with any user level overrides
  • Add support for time zones
  • Add support for calendar systems
  • Add support for collation
  • Add support for advanced formatting/parsing options such as spell-out, ordinal, duration, etc.

These can be summarised as reducing the Qt code base, integrating fully with the host platform, and supporting advanced features.

In the past, Qt has emphasised consistent behaviour across all platforms as being the key driver, at the expense of Qt apps looking and behaving the same as all other apps on the users platform, i.e. making life easier for the developer rather than the end user. A better balance needs to be found between the two positions, but with the emphasis on fitting in with the users expectations for the fundamentals, i.e. date and number formats must be consistent with the environment teh app is running in unless the developer has good reason. This means better using the host facilities for the core localization features, and providing optional advanced features for those apps that need them.

Solution Options

A number of options have been investigated for implementing these improvements.

The initial plan was to extend the existing QLocale code and data with new code, code donated from KDE, and more data from CLDR. As a test case calendar system support was added using KDE code and CLDR data, but this proved to be unacceptable in terms of the library size required by the new data. It would also result in a lot of new code that Qt would struggle to code and support, especially in difficult areas like collation. It also didn't improve host integration.

The second plan was to utilise ICU as the localization back-end on all platforms to minimise code and data requirements within Qt. This had the advantage of a single code base and consistent behaviour and feature set across platforms. The main disadvantage was that ICU does not respect a users personal preference overrides or even the host settings, so could be inconsistent. This option proved unworkable however due to resistance from Windows devs to the extra dependency and download, Apple App Store policies preventing linking to ICU, and Android only shipping the Java version.

The third plan adopted was to implement individual system back-ends for each host platform to fully utilise the host system locale resources. While more code than option 2, and limited to a lowest common denominator feature set, it would at least provide a fully integrated appearance with the host system, and is the only design pattern that will work on all platforms without additional large amounts of code or data or external library dependencies. Unfortunately the lowest common feature set was too low, limited to what the Win32 API in Windows Vista or Windows Embedded 2013 supports, a very limited feature set little better than the current QLocale support. Given that Windows 7 will need to be supported for at least 5 more years (official end-of-support is 2020) then we are unlikely to be be able to add any advanced formatting support into QtCore during the 5.x series, and possibly even a future 6.x series. This will fail to meet the needs of many developers who require advanced formatting.

A fourth plan is now proposed. this will be to improve the existing QLocale code base where possible, adding a small number of missing features, with a new separate Qt Add-on library provided for advanced features based on ICU. This is the option now documented below. Detailed planning for the ICU option 2 can be found at Qt-5-ICU. Many of these details will still apply to the new separate library, host-based plan.

Localization Features

This section describes the different localization features, their current implementation in Qt, and possible changes.

User and System Locales

The System Locale is the default locale configured for the entire system, i.e. the default any new user or a root user would have. The User Locale is the locale chosen by the current user which may differ from the System Locale, and/or may have individual settings customised by the user, e.g. usually date format.

  • Windows and Mac have System and User locales, but Qt only uses/exposes the User Locale including user customizations.
  • Unix has System and User locale IDs, but Qt only uses/exposes the User IDs. The user can customize setting groups but not individual settings.

Ideally, QLocale would provide separate static methods for systemLocale() and userLocale(), and a way on UNIX to customize individual overrides.

Custom Locales

A Custom Locale is one where the app developer creates a locale of their own choice, for example if they want to show output in different currencies, languages or locations. Qt currently supports this by shipping a sub-set of the CLDR data for the features it supports. This is only useful for a very small set of developers on Windows and Mac at the cost of a lot of data in qtcore. The data is currently required for all other platforms though.

  • Windows provides a limited set of locales depending on the regional version installed or language packs downloaded, and allows passing of the required locale into it's localization api
  • Mac and most Unix systems ship the full set of locales from CLDR/ICU so do not need Qt's CLDR data
  • WinRT and Android may also ship the full set of CLDR data (TODO: confirm)

In short, on all main platforms host data can be utilised to create custom locales, reducing the need to ship Qt's copy of CLDR on those platforms. It can also be argued this is an advanced feature that should be enabled if required by the app developer on platforms that don't provide enough host data, either by enabling Qt's inbuilt data or choosing to build against ICU.

Locale Changes

In theory, all Qt apps should fix the locale used at start-up and not change that locale until the entire UI can be refreshed at the same time. The app should respond to a change locale system event by refreshing the QLocale and then refreshing the GUI. Unfortunately this facility appears to be incomplete on most if not all platforms, with no obvious api available. (More research required).

  • On Windows and Unix, the locale ID is fixed on creation and an internal refresh call is available.
  • On Mac, CFLocaleCopyCurrent is used for every call then released, so the cached object may be reused but this is not guaranteed. No internal refresh call is available

This support needs to be properly completed for all platforms.

Number Localization

All number localization is done using the internal Qt code:

  • System Locales use cached host data for number symbols
  • Custom Locales use Qt's CLDR data for number symbols
  • All locales use Qt internal code routines, and libdouble where available
  • The Qt internal code only supports single-character number symbols whereas CLDR and host data can be multi-character (TODO: check how many actually are)
  • The Qt internal code does not support alternative number-grouping such as Chinese or Indian.
  • The QLocale API requires code routines for int64, uint64, and double, all other types can be derived from these
  • The Qt internal code is reused by other Qt classes QByteArray, QString, QTextStream, QIpAddress, QVersionNumber, QtGui::QValidator, and in external modules QtXmlPatterns and QtDeclarative (TODO: check if really need to use Qt internal routines, i.e. if only doing non-localized C conversions could use host, or if localized could use public QLocale api)

Ideally Qt would not have to implement it's own number localization code as it is complex and hard to get exactly right for all cases and so imposes a maintenance burden and possibly a performance burden. Ideally we would also support advanced number localization options such as Spell-out, Ordinal, etc as done by ICU-based platforms.

There exists 3 options here:

  • Continue to use our own code, but cleaned-up and enhanced with multi-char and grouping support
  • Use the host facilities
  • Use another external library or the standard C/C++ library

A number of factors affect this decision:

  • Using our own code is a maintenance burden
  • Using the host localization code may not support the functionality used by QLocale or other Qt code
  • The standard C/C++ code options only support basic number formatting and not advanced options as in ICU/WinRT/OSX
  • The Win32 API only supports standard C/C++ library formatting
  • The ICU C API only supports int32, int64 and double, it does not support unsigned int64 until ICU 52
  • The standard C/C++ libraries do not integrate with ICU or Qt's own CLDR data and do not provide easy ways to override the settings used

As some platforms cannot support the minimum Qt localization requirements, we must continue to offer an internal version or use an external library that takes the required symbols. It may not make sense to only use this code for selected platforms and host code on others, so it could be used on all platforms and improved to include the required features. Alternatively we could clean up our existing code to match the new C++11 std:: api calls and provide a thin wrapper around whichever is appropriate to use on a given platform.

Currency Localization

Current QLocale support is as follows:

  • Host system data is used for Mac and Windows System Locales
  • Qt's copy of CLDR data is used for system locales on all other platforms, and for all custom locales on all platforms
  • No currency parsing is supported
  • On Mac and Windows sytem locales currency formatting uses the host facilities
  • On all other platforms and for custom locales Qt's own code is used
  • The Qt api requires support for int64, unit64 and double

Parsing support could be added.

Date/Time Localization

Current QLocale support is as follows:

  • Host system data is used for Mac and Windows System Locales
  • Qt's copy of CLDR data is used for system locales on all other platforms, and for all custom locales on all platforms
  • All date parsing is done by Qt's own code, no host or standard library code is used
  • All custom date formatting (dd-MMM-yyyy etc) is done by Qt's own code, no host or standard library code is used
  • Fixed format date formatting (LongDate, ShortDate, etc) is done by the host on Mac and Wondows, and Qt's own code on all other platforms and custom locales

Ideally Qt would not have to implement it's own date/time localization code as it is complex and hard to get exactly right for all cases and so imposes a maintenance burden and possibly a performance burden. Unlike number localization, it may be possible to use the host facilities for all parsing and formatting, except for CLDR which will still require internal Qt routines.

A number of problems exist with the Qt code:

  • No support for non-Gregorian Calendar Systems, but the host data and formatting may return names and symbols for these
  • Time zone names are not properly supported
  • Not all standard format codes or formats supported
  • Advanced formatting options not supported
  • The format codes are a weird hybrid of Windows and Unicode
  • No easy way to format time with default format but no seconds, or with 24-hour time instead of 12-hour time

Given that most platforms use the Unicode standard format codes, it may make sense to convert all internal code to directly work with this standard and to store the CLDR data in this format instead of converting it first to Qt format. This will improve performance and reduce discrepancies. The only format conversions then required will be on Win32, and when the api is called with Qt format codes.

User Customization on Linux

A major issue that needs solving is user customizations on UNIX desktop platforms. This is of particular interest to KDE developers.

Currently if a UNIX user wishes to customize their locale settings, they are restricted to changing the locale code used for a group of functionality. For example, if a en_GB user wants to always use the ISO date format instead of the default, they need to set LC_TIME to a locale code that has an ISO date format, but otherwise has all the other settings the same as en_GB. This can be hard to find, and may not even exist. Such changes also do not take effect globally until the user has logged out and back in again.

ICU also offers no way for a user to define any customizations, although the app programmer is free to override any settings they want to. The ideal solution would be for ICU to implement a standard way to read in user customizations, but it currently doesn't offer this, or indeed respect user customizations on any other platform like Windows or Mac.

One solution would be for QLocale to read in a file with user-level customizations from a standard location for that desktop or system. This file could be in the CLDR JSON formats so would be in a universal standard and readable by QtCore. When the user uses the desktop locale config program to change a setting, this file would be written out, along with a custom POSIX file for other toolkits to use.

Solution Design

The solution proposed is split into two sections, changes to the current QLocale code to implement what new features and structural changes are possible in Qt5 series, and a new QtGlobalization library for advanced localization.


Long term, once Windows 7 support is dropped in Qt6 (?), QLocale would be dropped and QtGlobalization would be merged into QtCore.

A major issue with this design is the heavy overlap of shared code between QtCore and QtGlobalization i.e. the Linux backend in QtCore will implement ICU support, which will be replicated and enhanced in QtGlobalization This would be a necessary evil however to allow for QtGlobalization to be built on Windows 7 using ICU. This would not be possible if QtGlobalization was to use a shared private api exported by QtCore. It may still be possible for Mac and WinRT to share their implementations in this way. In this event, the resulting QtGlobalization library could be very small as a result. An alternative is to make the new API part of QtCore but with Windows 7 devs advised that if they don't use ICU then the new API will not actually provide the advanced features. This would be controversial but may be simpler.

QLocale Changes

A possible plan would be to implement a major structural change on the backend:

  • Create a new QPlatformLocale private base class that caches the locale code, number and date symbols and returns them via a Qlocale-style api. The base class will implement a C locale by default. Derived classes will be created for each platform to load/reload the settings for the system or custom locale requested (Win32, WinRT, Mac, ICU, and CLDR). Date formatting codes returned will use the Unicode standard.
  • The local matching and listing code will now be platform based rather than CLDR based
  • Create a new Win32 backend using Locale Names instead of LCIDs and extend to support querying host-based custom locales. This backend will define the features all other backends will be capable of providing
  • Create a new Mac backend using NSLocale rather than CFLocale and extend to support querying host-based custom locales
  • Create a new Qt CLDR backend
  • Make the Qt CLDR data import confgurable so embedded platforms can configure what locales to include
  • Create a new ICU backend as the default system and custom locale backend for all Unix systems
  • Create a new WinRT backend to support system and custom locales
  • Create a new Android backend to support system and custom locales using JNI
  • Change QLocalePrivate to embed a QPlatformLocale, and modify all code to refer to this when querying for data
  • Have a compile switch to choose -locale-backend as 'system' (i.e. mac, win32, winrt, icu), 'icu' (to force ICU), or 'cldr' (to force CLDR)
  • Have a compile switch for Windows Vista/7 for -custom-locale of 'system', 'icu' or 'cldr'
  • Both compile switches default to 'system'
  • Switch to always using Qt date formatting code for consistency, and use Unicode date format codes internally

The Mac, Win32 and CLDR backend changes must be implemented at the same time, with the ICU and Android backends then able to be added later. The date formatting code changes should also be implemented in the initial set of changes to save later changes to the backends.

Once these structural changes are made we can then improve the existing QLocale features

  • Add full number grouping and multi-byte symbol support to our number parse/format code
  • Add a few extra date formatting options
  • Add ISO codes support
  • Add calendar system support using the host facilities with an optional CLDR based fallback (this would bloat the CLDR data, so optional and configurable)
  • Update QTimeZone support to the new minimum platform features

QtGlobalization

For the advanced formatting features we will need a new Qt Add-on library (QtGlobalization?) that is essentially a very thin wrapper around the CLDR/ICU style api:

  • A Qt Add-on that ships as part of the main Qt release so it may be relied upon
  • The api implemented will be the ICU/CLDR style api as implemented on all modern platforms
  • The module will implement api support for as advanced a feature set as possible, with devs checking at runtime which features may or may not be available in a local host. Where a dev does call an unavailable feature thsi will degrade gracefully by default
  • On Mac wrap the Mac api
  • On WinRT wrap the WinRT api
  • On Android wrap the Android api via JNI
  • On Windows Vista/7 wrap ICU but with user customizations included in the default user locale
  • On all other platforms wrap ICU
  • Allow choice of always using ICU if the dev wants to
  • The API design will be completely modern and independent of QLocale, with the aim of replacing it in Qt6

The exact split of core functions and advanced functions is yet to be determined, as is how to share code required in both.

Platform Support

While we cannot use ICU directly on all platforms, all platforms except Win32 extended (Windows 7 and Windows Embedded 2013) use ICU or CLDR data as a base and so have a consistent feature set, object model and api that we can design to. Careful API design using new classes for the new features, an API that can be queried for supported features, and sensible fallback options where a feature is unsupported on minor platforms will allow for different levels of support on different platform versions while allowing the latest versions to make most new features available.

Choosing the supported lowest common set of advanced features is slightly tricky depending on whether we choose the lowest deployment version of every platform supported (currently RHEL supporting ICU 4.2), or choose the most widely installed version (say OSX 10.8 supporting ICU 49), or choose the ideal version we want (say ICU 52) and implement bridging code in the interim.

As at Qt5.7 the minimum deployment platforms are:

Platform Reference Official Community ICU
Windows 7 or Vista? 7 or Vista? 7 or Vista? Win32 Extended only
Windows Embedded Compact 2013 Compact 2013 Compact 2013 Win32 Extended only
Windows Phone 8.1 8.1? Own API
Windows RT 8.1 8.1? Own API
Mac OSX 10.8 10.8 10.8? 49 via Own API
Mac iOS 8.0 5.0? ? via Own API
Linux Ubuntu 14.04 14.04 11.10? 4.4 or 52
Linux RHEL 6.6 6.6 ? 4.2
Linux OpenSuse 13.1 13.1 ? 51
Android 4.1 (API Level 16) 4.1 (API Level 16) 4.8 via Own API
QNX SDP 6.6 ?

ICU

ICU ships standard on all modern Linux and BSD distros, and is shipped standard with QNX. With POSIX locale functions being clearly deficient (i.e. no calendar system support, poor collation, etc) then this is the preferred back-end for use on all Unix-like platforms, including embedded and QNX. One limiting factor is that the C++ ABI is not stable between versions so the C API must be used instead, and this lacks many of the features of the C++ API.

The defining platform here is RHEL 6.6 which only uses ICU 4.2, but it could be argued that RHEL is mostly a server distro and so localization is perhaps not as high a profile requirement there, so a later target of ICU 51 (OpenSUSE 13.1) could be used with a degraded feature-set on RHEL.

Handling of user-level overrides could occur at app level, but a platform wide solution should be sought (e.g. so KDE can define desktop-wide overrides).

Linux ICU
RHEL 6.6 4.2
Ubuntu 11.10 LTS (Oneiric) 4.4
Ubuntu 12.04 LTS (Precise) 4.8
Ubuntu 14.04 LTS (Trusty) 52
OpenSuse 13.1 51
OpenSuse 13.2 53
QNX 6.6.0 ?

Mac OS X / iOS

Mac uses ICU for localization, but does not ship the ICU headers and prohibits apps that link to ICU from the App Store. Instead we must use the native Mac API which in reality is a thin wrapper around ICU with a simplified API and some Mac convenience methods added. The Mac API also uses any user level overrides which using ICU directly would not. One problem is that the wrapper API's may not have been updated to provide access to new features in each ICU release used.

From Qt 5.7 onwards OSX 10.8 and ICU 49 are the lowest supported version.

OSX ICU
10.6 4.0
10.7 4.6
10.8 49
10.9 51
10.10 53
10.11 55

Reference: http://opensource.apple.com/

Android

Android uses ICU for localization, but only ships the Java version of the library and data files. We will use the Java Android api via JNI.

From Qt 5.7 onwards Android 4.1 and ICU 4.8 are the lowest supported version.

Version Codename API Level ICU
2.3.3 Gingerbread 10 4.4
3.0 Honeycomb 11 4.4
4.0 Ice Cream Sandwich 14 4.6
4.1 Jelly Bean 16 4.8
4.3 Jelly Bean 18 50
4.4 Kit Kat 19 51
5.0 Lollipop 21 53

Reference: http://developer.android.com/reference/java/util/Locale.html

Win32

Windows Vista, Windows 7 and Windows Embedded 2013 continue to use the Win32 API for localization functions, supplemented with some new functions using Locale Names in place of LCID's and improved calendar and custom locale data access. Unfortunately, the core number and date formatter calls have not changed and so remain the lowest common denominator.

Windows Runtime

WinRT and Windows Phone provide advanced localization functions clearly based on CLDR data and broadly comparable to ICU. An initial review of the WinRT api indicates all required features are exposed, but this needs to be fully documented.

Embedded / Fallback

Some embedded platforms may prefer not to ship ICU, or may only have very simple localization requirements. It will be required to provide a fall-back implementation for these platforms. This could be the existing QLocale code and database, but that could be a substantial maintenance burden. The alternative is to ship a simple C-locale back-end or a pass-through to POSIX. One option is to provide good documentation on building the minimal ICU required to support the embedded platform's locales.

API Design

The new API will be implemented as a set of new classes completely separate to the existing QLocale class:

  • QLocaleCode
  • QNumberFormatter
  • QCurrencyFormatter
  • QDateTimeFormatter
  • QCalendar
  • QTimeZone

There are a number of very strong reasons for this:

  • This is the design pattern used by ICU, OSX, Windows, and Java that all devs are already familiar with
  • It is more efficient as ICU splits the locale resource files that are loaded by the Number and DateTime formatters, so a monolithic class would take longer to load
  • It represents a clear break with the old API and format codes, making it clearer to devs that behaviour and format codes have changed
  • It prevents API bloat by having a single formatter api for different formatter types rather than multiple calls with prefixes or extra enums
  • It allows different levels of feature support for different formatter types on different platforms or versions while keeping the same simple api across them all

Components

QTimeZone

QTimeZone has been successfully implemented in Qt 5.2 using separate back-ends for each system but a common API. The design of this class will be copied for much of the new QLocale implementation, especially QCalendar.

In 5.3 a number of new features are still required, including a QEvent for TimeZoneChanged and a QTimeZoneDatabase class to load TZ databases on any platform.

Old design details can be found at http://wiki.qt.io/Qt-5-QTimeZone

QCalendar

QCalendar will follow the design of QTimeZone to wrap the system provided calendar calculators.

Not all platforms equally support the same set of calendar systems, although this is slowly converging thanks to the increasing use of CLDR and ICU. While QCalendar will have to define an enum for all possible calendar systems, it will also have to provide an availableCalendarSystems() api to describe what systems are available to be set in the formatter.

QCalendar will implement baseline support for as many calendars as possible so that even where a calendar system is not available on a host system it may still be used as an optional calculation class, but not in the formatter.

ICU-based systems integrate the Time Zone into the Calendar to allow for fully accurate conversions of date/times, although this is strictly not necessary. It should allow calendars like Hebrew and Islamic to start their days at sundown or any other time, but doesn't. The wrapper will need to follow this design.

ICU

Supported range in ICU 56: Julian day numbers of -0x7F000000 to +0x7F000000. This corresponds to years from ~5,800,000 BCE to ~5,800,000 CE. Was previously wider range.

Date/Time Components:

  • UCAL_ERA
  • UCAL_YEAR
  • UCAL_MONTH
  • UCAL_WEEK_OF_YEAR
  • UCAL_WEEK_OF_MONTH
  • UCAL_DAY_OF_YEAR
  • UCAL_DAY_OF_WEEK
  • UCAL_DAY_OF_WEEK_IN_MONTH
  • UCAL_AM_PM
  • UCAL_HOUR
  • UCAL_HOUR_OF_DAY
  • UCAL_MINUTE
  • UCAL_SECOND
  • UCAL_MILLISECOND
  • UCAL_ZONE_OFFSET
  • UCAL_DST_OFFSET
  • UCAL_YEAR_WOY
  • UCAL_DOW_LOCAL
  • UCAL_EXTENDED_YEAR
  • UCAL_JULIAN_DAY
  • UCAL_MILLISECONDS_IN_DAY
  • UCAL_IS_LEAP_MONTH
  • UCAL_FIELD_COUNT
  • UCAL_DAY_OF_MONTH

Other:

  • Min/Max values
  • Lenient
  • Gregorian change date
  • First day of week
  • Min days in first week
  • ICU 4.4 Weekday/Weekend
  • ICU 49 Ambiguous Wall Time
  • ICU 50 TZ Transitions
  • ICU 51 Get TZ ID
  • ICU 52 Windows TZ ID

Mac.

API Docs say enum deprecated since 10.10? OSX 10.11 System Preferences lists all below as available, plus Amete Alem, Umm al-Qura, Islamic Tabular. Perhaps uses locale keyword instead?

  • NSGregorianCalendar
  • NSBuddhistCalendar
  • NSChineseCalendar
  • NSHebrewCalendar
  • NSIslamicCalendar
  • NSIslamicCivilCalendar
  • NSJapaneseCalendar
  • NSRepublicOfChinaCalendar
  • NSPersianCalendar
  • NSIndianCalendar
  • NSISO8601Calendar - Doc says not implemented?

ICU C++: http://icu-project.org/apiref/icu4c/classicu_1_1Calendar.html

Calendar ICU Mac Android WinRT Win32 KDE
Gregorian gregorian
Chinese Lunar chinese
Coptic coptic
Dangi (Korean Lunar) dangi (ICU 51)
Ethiopic ethiopic
Ethiopic Al Amate ???
Indian National indian
Jalali persian
Hebrew hebrew
Islamic Civil islamic
Islamic Um-al-Qura islamic-umalqura (ICU 52)
Islamic Tabular islamic-tbla (ICU 52)
Japanese (Gregorian) japanese
Taiwan (Gregorian) taiwan
Thai (Gregorian) buddist
Retrieved from "https://wiki.qt.io/index.php?title=Locale_Support_in_Qt_5&oldid=20180"