Use UnicodeData.txt to create UTF-8 ctype map.
authoryuripv <yuripv@FreeBSD.org>
Sat, 17 Nov 2018 10:36:00 +0000 (10:36 +0000)
committeryuripv <yuripv@FreeBSD.org>
Sat, 17 Nov 2018 10:36:00 +0000 (10:36 +0000)
commitc6e4d2410627586bd7eb2730fa89e21234536509
tree86662b0245f6b0951302b38c2ce514547f6fbd98
parent82a106e39f2b2593f9f5d7bf939529e6aa4b6d20
Use UnicodeData.txt to create UTF-8 ctype map.

This should provide more complete coverage of currently defined Unicode
characters as compared to manually assembled one we use currently.

Comparison of original and new UTF-8 ctype maps by character class:

TYPE    ORIG    NEW
alnum   94229   126029
alpha   93557   125419
blank   4       2
cntrl   73      137685
digit   469     622
graph   109615  137203
lower   1478    2145
print   109641  137222
punct   3428    797
rune    110481  274907
space   33      24
upper   983     1781
xdigit  469     622

Large number of added cntrl definitions is due to the fact that private-use
planes are currently defined as such, this can change in the future.

Discussed with: bapt
Approved by: kib (mentor, implicit)
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D17842
share/ctypedef/C.UTF-8.src
tools/tools/locale/Makefile
tools/tools/locale/etc/common.UTF-8.src [deleted file]
tools/tools/locale/etc/manual-input.UTF-8 [deleted file]
tools/tools/locale/tools/utf8-rollup.pl