Locale Support in Qt 5

Early Qt localization support was weak with a design based on Windows APIs, but with its own code and data. Over time new features have been introduced based on CLDR data, but the current QLocale implementation is still incomplete, inconsistent, not fully integrated with the host system locale, and unnecessarily bloats the core library. It is clear that Qt needs a new approach.

Current Support

Qt provides two core options for localization:

• System Locale: Use the default system locale settings, the implementation being platform specific.
• Custom Locale: An application can replace the default system locale with any custom locale supported by the CLDR data embedded in QtCore which is then used by Qt's internal code routines.

On Windows, Mac and Blackberry the System Locale uses the localization data of the host system, falling back to embedded CLDR data where the host system doesn't provide the required data, and using an inconsistent mixture of of the host parse/format code and Qt's own code.

On all other platforms such as Linux and Android the embedded CLDR data is always used instead.

A number of key requirements for future improvements were defined:

• Minimise locale data shipped with Qt
• Minimise localization code needing to be maintained by Qt
• Integrate fully with the system locale on all platforms
• Integrate fully with any user level overrides
• Add support for time zones
• Add support for calendar systems
• Add support for localized collation
• Add support for advanced formatting/parsing options

A number of options have been investigated for implementing these improvements.

The initial plan was to extend the existing QLocale code and data with new code, code from KDE, and more data from CLDR As a test case calendar system support was added using KDE code and CLDR data, but this proved to be unacceptable in terms of the library size required by the new data. It would also result in a lot of new code that Qt would struggle to code and support, especially in difficult areas like collation.

The second plan was to utilise ICU as the localization back-end on all platforms to minimise code and data requirements within Qt. This had the advantage of a single code base and consistent behaviour and feature set across platforms. The main disadvantage was that ICU does not respect a users personal preference overrides or the host settings, so would be inconsistent. This option proved unworkable however due to resistance from Windows devs to the extra dependency and download, Apple App Store policies preventing linking to ICU, and Android only shipping the Java version.

The third plan adopted is to implement individual system back-ends for each host platform to fully utilise the system locale resources. While more code than option 2, and limited to a lowest common denominator feature set, it will at least provide a fully integrated appearance with the host system, and is the only design pattern that will work on all platforms. This is the option now documented below.

Detailed planning for the ICU option can be found at Qt-5-ICU. Many of these details will still apply to the new host-based plan.

Platform Support

While we cannot use ICU directly on all platforms, all platforms except Win32 either use ICU or CLDR as a base and so have a consistent feature set, object model and api that we can design to. With Win32 due to be dropped soon we no longer need to be concerned about not offering new features on that platform and can include in QtCore new api to expose the new features. The older platforms that do not support the new features will always have a fallback to the old feature set supported by QLocale. Careful API design using new classes for the new features, an API that can be queried for supported features, and sensible fallback options where a feature is unsupported will allow for different levels of support on different platform versions while allowing the latest versions to make the most new features available.

Choosing the supported lowest common set of advanced features is slightly tricky depending on whether we choose the lowest working version of each platform (say OSX 10.6 forcing ICU 4.0) or the lowest Reference or Officially supported platform (say OSX 10.8 and RHEL 6.6 and Android 4.4 yielding ICU xxx).

As at Qt5.6 the minimum supported platforms are:

Platform	Reference	Official	Community
Windows	7	7	7
Windows Embedded			7.0?
Windows Phone	8.1	8.1?
Windows RT	8.1	8.1?
Mac OSX	10.8	10.8	10.7?
Mac iOS	8.0	5.0?
Linux Ubuntu	14.04	14.04	11.10?
Linux RHEL	6.6	6.6	?
Linux OpenSuse	13.1	13.1	?
Android	4.4 (NDK 10c)	2.3.3 (API level 10)	?
QNX			SDP 6.6

ICU

ICU ships standard on all modern Linux and BSD distros, and is shipped standard with QNX. With POSIX locale functions being clearly deficient (i.e. no calendar system support, poor collation, etc) then this is the preferred back-end for use on all Unix-like platforms, including embedded and QNX.

The limiting factor here is RHEL 6 which only uses ICU 4.2, but it could be argued that RHEL is mostly a server distro and so localization is perhaps not as high a profile requirement there.

Linux	ICU
RHEL 6.5	4.2
RHEL 6.6	4.2
Ubuntu 10.04 LTS (Lucid)	4.2
Ubuntu 11.10 LTS (Oneiric)	4.4
Ubuntu 12.04 LTS (Precise)	4.8
Ubuntu 14.04 LTS (Trusty)	52
OpenSuse 13.1	51
OpenSuse 13.2	53
QNX 6.6.0

Mac OS X / iOS

Mac uses ICU for localization, but does not ship the ICU headers and prohibits apps that link to ICU from the App Store. Instead we must use the native Mac API which in reality is a thin wrapper around ICU with a simplified API and some Mac convenience methods added. The Mac API also uses any user level overrides which using ICU directly would not. One problem is that the wrapper API's may not have been updated to provide access to new features in each ICU release used.

Qt 5.4 supports 10.6 at community level, and 10.7 at Official and Reference level. Qt 5.5. drops 10.7 as an Official and Reference platform, making OSX 10.8 and ICU 49 the lowest supported version.

OSX	ICU
10.6	4.0
10.7	4.6
10.8	49
10.9	51
10.10	53

Reference: http://opensource.apple.com/

Android

Android uses ICU for localization, but only ships the Java version of the library and data files. We will use the Java Android api via JNI.

The ICU version changes in Android are as follows:

Version	Codename	API Level	ICU
2.3.3	Gingerbread	10	4.4
3.0	Honeycomb	11	4.4
4.0	Ice Cream Sandwich	14	4.6
4.1	Jelly Bean	16	4.8
4.3	Jelly Bean	18	50
4.4	Kit Kat	19	51
5.0	Lollipop	21	53

Reference: http://developer.android.com/reference/java/util/Locale.html

WinRT

WinRT does not document what version of ICU or CLDR it uses, but it is clear that it does so. An initial review of the Windows api indicates are required features are exposed.

Windows

WIndows 7 and later provides advanced localization functions clearly based on CLDR and broadly comparable to ICU. Windows Vista lacks many of these new API relying mainly on Win32 API's, but with a < 2% market share can be safely ignored provided sensible fallbacks occur. Windows XP only has Win32 API which lack most of the required advanced functions, but basic functionality can still be supported using some fallbacks. The issue comes with what platform Qt is compiled for, runtime checks may not be sufficient to work around missing API calls.

Embedded / Fallback

Some embedded platforms may prefer not to ship ICU, or may only have very simple localization requirements. It may be possible to provide a fall-back implementation for these platforms. This could be the existing QLocale code and database, but that would be a substantial maintenance burden. The alternative is to ship a simple C-locale back-end.

API Design

The new API will be implemented as a set of new classes completely separate to the existing QLocale class:

• QLocaleCode
• QNumberFormatter
• QDateTimeFormatter
• QCalendar
• QTimeZone

There are a number of very strong reasons for this:

• This is the design pattern used by ICU, OSX, Windows, and Java that all devs are already familiar with
• It is more efficient as ICU splits the locale resource files that are loaded by the Number and DateTime formatters, so a monolithic class would take longer to load
• It represents a clear break with the old API and format codes, making it clearer to devs that behaviour and codes have changed
• It prevents API bloat by having a single formatter api for different formatter types rather than multiple calls with prefixes or extra enums
• It allows different levels of feature support for different formatter types on different platforms or versions while keeping the same simple api across them all

The old QLocale backend code will be removed and replaced with calls to the new code. It is expected this compatibility layer will take considerable effort and testing to ensure backwards compatible behaviour.

Components

QTimeZone

QTimeZone has been successfully implemented in Qt 5.2 using separate back-ends for each system but a common API. The design of this class will be copied for much of the new QLocale implementation, especially QCalendar.

In 5.3 a number of new features are still required, including a QEvent for TimeZoneChanged and a QTimeZoneDatabase class to load TZ databases on any platform.

Old design details can be found at http://wiki.qt.io/Qt-5-QTimeZone

QCalendar

QCalendar will follow the design of QTimeZone to wrap the system provided calendar calculators.

Not all platforms equally support the same set of calendar systems, although this is slowly converging thanks to the increasing use of CLDR and ICU. While QCalendar will have to define an enum for all possible calendar systems, it will also have to provide an availableCalendarSystems() api to describe what systems are available to be set in the formatter.

QCalendar will implement baseline support for as many calendars as possible so that even where a calendar system is not available on a host system it may still be used as an optional calculation class, but not in the formatter.

ICU C++: http://icu-project.org/apiref/icu4c/classicu_1_1Calendar.html

• ICU C: http://icu-project.org/apiref/icu4c/ucal_8h.html
• Android / ICU4J: http://developer.android.com/reference/java/util/Calendar.html
• Mac Cocoa: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSCalendar_Class/Reference/NSCalendar.html
• WinRT: http://msdn.microsoft.com/en-US/library/windows/apps/windows.globalization.calendar.aspx
• Win32: http://msdn.microsoft.com/en-us/library/windows/desktop/dd319114(v=vs.85).aspx