1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
|
* Copyright (C) 2004-2012, International Business Machines
* Corporation and others. All Rights Reserved.
*
* file name: changes.txt
* encoding: US-ASCII
* tab size: 8 (not used)
* indentation:4
*
* created on: 2004may06
* created by: Markus W. Scherer
*
* change log for Unicode updates
---------------------------------------------------------------------------- ***
Future Unicode update
Tools simplified since the Unicode 6.1 update. See
- http://site.icu-project.org/design/props/ppucd
- http://bugs.icu-project.org/trac/wiki/Markus/ReviewTicket8972
* Unicode version numbers
- icutools/unicode/makedefs.sh was deleted, so one fewer place for version & path updates
* file preparation
- ucdcopy.py, idna2nrm.py and genpname/preparse.pl replaced by preparseucd.py:
- ~/svn.icu/tools/trunk/src/unicode$ py/preparseucd.py ~/uni61/20120118 ~/svn.icu/trunk/src ~/svn.icu/tools/trunk/src
- This writes files (especially ppucd.txt) to the ICU4C unidata and testdata subfolders.
- Check test file diffs for previously commented-out, known-failing data lines;
probably need to keep those commented out.
* PropertyValueAliases.txt changes
- Script codes that are in ISO 15924 but not in Unicode are now listed in
preparseucd.py, in the _scripts_only_in_iso15924 variable.
If there are new ISO codes, then add them.
If Unicode adds some of them, then remove them from the .py variable.
* UnicodeData.txt changes
- No more manual changes for CJK ranges for algorithmic names;
those are now written to ppucd.txt and genprops reads them from there.
* generate core properties data files (makeprops.sh was deleted)
- ~/svn.icu/tools/trunk/dbg/unicode$ c/genprops/genprops ~/svn.icu/trunk/src
* no more manual updates of source/data/unidata/norm2/nfkc_cf.txt
- it is now generated by preparseucd.py
* no more separate idna2nrm.py run and manual copying to generate source/data/unidata/norm2/uts46.txt
- it is now generated by preparseucd.py
- make sure that the Unicode data folder passed into preparseucd.py
includes a copy of http://www.unicode.org/Public/idna/6.1.0/IdnaMappingTable.txt
(can be in some subfolder)
* generate normalization data files
- ~/svn.icu/trunk/dbg$ export LD_LIBRARY_PATH=~/svn.icu/trunk/dbg/lib
- ~/svn.icu/trunk/dbg$ SRC_DATA_IN=~/svn.icu/trunk/src/source/data/in
- ~/svn.icu/trunk/dbg$ UNIDATA=~/svn.icu/trunk/src/source/data/unidata
- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfc.nrm -s $UNIDATA/norm2 nfc.txt
- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt
- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/nfkc_cf.nrm -s $UNIDATA/norm2 nfc.txt nfkc.txt nfkc_cf.txt
- ~/svn.icu/trunk/dbg$ bin/gennorm2 -o $SRC_DATA_IN/uts46.nrm -s $UNIDATA/norm2 nfc.txt uts46.txt
* build ICU (make install)
* build Unicode tools using CMake+make
* new way to call genuca (makeuca.sh was deleted)
- ~/svn.icu/tools/trunk/dbg/unicode$ c/genuca/genuca -i ~/svn.icu/trunk/dbg/data/out/build/icudt49l ~/svn.icu/trunk/src
---------------------------------------------------------------------------- ***
Unicode 6.1 update
*** ICU Trac
- ticket 8995 final update to Unicode 6.1
- ticket 8994 regenerate source/layout/CanonData.cpp
- ticket 8961 support Unicode "Age" value *names*
- ticket 8963 support multiple character name aliases & types
- ticket 8827 "update ICU to Unicode 6.1"
- C++ branches/markus/uni61 at r30864 from trunk at r30843
- Java branches/markus/uni61 at r30865 from trunk at r30863
*** Unicode version numbers
- makedata.mak
- uchar.h
(configure.in & configure: have been modified to extract the version from uchar.h)
- com.ibm.icu.util.VersionInfo
- icutools/unicode/makedefs.sh
+ also review & update other definitions in that file,
e.g. the ICU version in this path: BLD_DATA_FILES=$ICU_BLD/data/out/build/icudt49l
*** data files & enums & parser code
* file preparation
~/svn.icu/tools/trunk/src/unicode/c/genprops/misc$ ./ucdcopy.py ~/uni61/20111205/ucd ~/uni61/processed
- This prepares both unidata and testdata files in respective output subfolders.
- Check test file diffs for previously commented-out, known-failing data lines;
probably need to keep those commented out.
* PropertyValueAliases.txt changes
- 11 new block names:
Arabic_Extended_A
Arabic_Mathematical_Alphabetic_Symbols
Chakma
Meetei_Mayek_Extensions
Meroitic_Cursive
Meroitic_Hieroglyphs
Miao
Sharada
Sora_Sompeng
Sundanese_Supplement
Takri
-> add to uchar.h
-> add to UCharacter.UnicodeBlock IDs
Eclipse find UBLOCK_([^ ]+) = ([0-9]+), (/.+)
replace public static final int \1_ID = \2; \3
-> add to UCharacter.UnicodeBlock objects
Eclipse find UBLOCK_([^ ]+) = [0-9]+, (/.+)
replace public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2
- 1 new Joining_Group (jg) value:
Rohingya_Yeh
-> uchar.h & UCharacter.JoiningGroup
- 2 new Line_Break (lb) values:
CJ=Conditional_Japanese_Starter
HL=Hebrew_Letter
-> uchar.h & UCharacter.LineBreak
- 7 new scripts:
sc ; Cakm ; Chakma
sc ; Merc ; Meroitic_Cursive
sc ; Mero ; Meroitic_Hieroglyphs
sc ; Plrd ; Miao
sc ; Shrd ; Sharada
sc ; Sora ; Sora_Sompeng
sc ; Takr ; Takri
-> remove these from SyntheticPropertyValueAliases.txt
-> fix expectedLong names in cucdapi.c/TestUScriptCodeAPI()
and in com.ibm.icu.dev.test.lang.TestUScript.java
- 2 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html
(added 2011-06-21)
Khoj 322 Khojki
Tirh 326 Tirhuta
and another one added 2011-12-09
Hluw 080 Anatolian Hieroglyphs (Luwian Hieroglyphs, Hittite Hieroglyphs)
-> uscript.h
-> com.ibm.icu.lang.UScript
find USCRIPT_([^ ]+) *= ([0-9]+),(.+)
replace public static final int \1 = \2;\3
-> SyntheticPropertyValueAliases.txt
-> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI()
and in com.ibm.icu.dev.test.lang.TestUScript.java
* UnicodeData.txt changes
- the last Unihan code point changes from U+9FCB to U+9FCC
search for both 9FCB (end) and 9FCC (limit) (regex 9FC[BC], case-insensitive)
+ do change gennames.c
+ do change swapCJK() in ucol.cpp & ImplicitCEGenerator.java
* DerivedBidiClass.txt changes
- 2 new default-AL blocks:
# Arabic Extended-A: U+08A0 - U+08FF (was default-R)
# Arabic Mathematical Alphabetic Symbols:
# U+1EE00 - U+1EEFF (was default-R)
- 2 new default-R blocks:
# Meroitic Hieroglyphs:
# U+10980 - U+1099F
# Meroitic Cursive: U+109A0 - U+109FF
-> should be picked up by the explicit data in the file
* NameAliases.txt changes
- from
# Each line has two fields
# First field: Code point
# Second field: Alias
- to
# Each line has three fields, as described here:
#
# First field: Code point
# Second field: Alias
# Third field: Type
- Also, the file previously allowed multiple aliases but only now does it
actually provide multiple, even multiple of the same type. For example,
FEFF;BYTE ORDER MARK;alternate
FEFF;BOM;abbreviation
FEFF;ZWNBSP;abbreviation
- This breaks our gennames parser, unames.icu data structure, and API.
Fix gennames to only pick up "correction" aliases.
New ticket #8963 for further changes.
* run genpname/preparse.pl (on Linux)
+ cd ~/svn.icu/tools/trunk/src/unicode/c/genpname
+ make sure that data.h is writable
+ perl preparse.pl ~/svn.icu/trunk/src > out.txt
+ preparse.pl shows no errors, out.txt Info and Warning lines look ok
* build ICU (make install)
so that the tools build can pick up the new definitions from the installed header files.
* build Unicode tools (at least genpname) using CMake+make
* run genpname
(builds both pnames.icu and propname_data.h)
- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in
- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/common --csource
* build ICU (make install)
* build Unicode tools using CMake+make
* update source/data/unidata/norm2/nfkc_cf.txt
- follow the instructions in nfkc_cf.txt for updating it from DerivedNormalizationProps.txt
* update source/data/unidata/norm2/uts46.txt
- download http://www.unicode.org/Public/idna/6.1.0/IdnaMappingTable.txt
to ~/svn.icu/tools/trunk/src/unicode/py
- adjust idna2nrm.py to remove "; NV8": For UTS #46, we do not care about "not valid in IDNA2008".
- ~/svn.icu/tools/trunk/src/unicode/py$ ./idna2nrm.py
- ~/svn.icu/tools/trunk/src/unicode/py$ cp uts46.txt ~/svn.icu/trunk/src/source/data/unidata/norm2
* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to
sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar)
- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters
- Unicode 6.0..6.1: U+2260, U+226E, U+226F
- nothing new in 6.1, no test file to update
* generate core properties data files
- in initial bootstrapping, change the UCA version
in source/data/unidata/FractionalUCA.txt to match the new Unicode version
- ~/svn.icu/tools/trunk/src/unicode$ ./makeprops.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU & tools
+ if genrb fails to build coll/root.res with an U_INVALID_FORMAT_ERROR,
check if the UCA version in FractionalUCA.txt matches the new Unicode version
(see step above)
- run makeuca.sh so that genuca picks up the new case mappings and nfc.nrm:
~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU & tools
* update Java data files
- refresh just the UCD-related files, just to be safe
- see (ICU4C)/source/data/icu4j-readme.txt
- mkdir /tmp/icu4j
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
output:
...
Unicode .icu files built to ./out/build/icudt49l
mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt49b
mkdir -p ./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt49b
echo pnames.icu ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt
LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt49l.dat ./out/icu4j/icudt49b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt49l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt49b
mv ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/zoneinfo64.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/metaZones.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/timezoneTypes.res" ./out/icu4j/"com/ibm/icu/impl/data/icudt49b/windowsZones.res" "./out/icu4j/tzdata/com/ibm/icu/impl/data/icudt49b"
jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt49b/
mkdir -p /tmp/icu4j/main/shared/data
cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data
jar cf ./out/icu4j/icutzdata.jar -C ./out/icu4j/tzdata com/ibm/icu/impl/data/icudt49b/
mkdir -p /tmp/icu4j/main/shared/data
cp ./out/icu4j/icutzdata.jar /tmp/icu4j/main/shared/data
make[1]: Leaving directory `/home/mscherer/svn.icu/trunk/bld/data'
- copy the big-endian Unicode data files to another location,
separate from the other data files
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/brkitr
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt49b
~/svn.icu/trunk/bld/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/cnvalias.icu
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt49b
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/brkitr
- refresh ICU4J
~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt49b
* refresh Java test .txt files
- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode
* test ICU so far, fix test code where necessary
- temporarily ignore collation issues that look like UCA/UCD mismatches,
until UCA data is updated
* UCA
- get output from Mark's tools; look in
http://www.unicode.org/Public/UCA/6.1.0/CollationAuxiliary-<dev. version>.txt
- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt
- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt
(note removing the underscore before "Rules")
- update (ICU)/source/test/testdata/CollationTest_*.txt
and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt
with output from Mark's Unicode tools (..._CLDR_..._SHORT.txt)
- check test file diffs for previously commented-out, known-failing data lines;
probably need to keep those commented out
- check FractionalUCA.txt for manual changes of lead bytes from IMPLICIT to Hani
- run makeuca.sh:
~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU4C
- refresh ICU4J collation data:
(subset of instructions above for properties data refresh, except copies all coll/*)
~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
~/svn.icu/trunk/bld$ mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt49b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt49b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt49b
- run all tests with the *_SHORT.txt or the full files (the full ones have comments, useful for debugging)
- note on intltest: if collate/UCAConformanceTest fails, then
utility/MultithreadTest/TestCollators will fail as well;
fix the conformance test before looking into the multi-thread test
* When refreshing all of ICU4J data from ICU4C
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data
or
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install
*** LayoutEngine script information
(For details see the Unicode 5.2 change log below.)
* Run icu4j-tools: com.ibm.icu.dev.tool.layout.ScriptNameBuilder.
This generates LEScripts.h, LELanguages.h, ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp
in the working directory.
(It also generates ScriptRunData.cpp, which is no longer needed.)
The generated files have a current copyright date and "@draft" statement.
- diff current <icu>/source/layout files vs. generated ones
~/svn.icu4j/trunk/src$ kdiff3 ~/svn.icu/trunk/src/source/layout tools/misc/src/com/ibm/icu/dev/tool/layout
review and manually merge desired changes;
fix gratuitous changes, incorrect @draft and missing aliases;
Unicode-derived script codes should be "born stable" like constants in uchar.h, uscript.h etc.
- if you just copy the above files, then
fix mixed line endings, review the diffs as above and restore changes to API tags etc.;
manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h
*** merge the Unicode update branches back onto the trunk
- do not merge the icudata.jar and testdata.jar,
instead rebuild them from merged & tested ICU4C
---------------------------------------------------------------------------- ***
ICU 4.8 (no Unicode update, just new script codes)
* 9 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html
(added 2010-12-21)
Afak 439 Afaka
Jurc 510 Jurchen
Mroo 199 Mro, Mru
Nshu 499 Nüshu
Shrd 319 Sharada, Śāradā
Sora 398 Sora Sompeng
Takr 321 Takri, Ṭākrī, Ṭāṅkrī
Tang 520 Tangut
Wole 480 Woleai
-> uscript.h
-> com.ibm.icu.lang.UScript
find USCRIPT_([^ ]+) *= ([0-9]+),(.+)
replace public static final int \1 = \2;\3
-> genpname/SyntheticPropertyValueAliases.txt
-> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI()
and in com.ibm.icu.dev.test.lang.TestUScript.java
* run genpname/preparse.pl (on Linux)
+ cd ~/svn.icu/tools/trunk/src/unicode/c/genpname
+ make sure that data.h is writable
+ perl preparse.pl ~/svn.icu/trunk/src > out.txt
+ preparse.pl shows no errors, out.txt Info and Warning lines look ok
* rebuild Unicode tools (at least genpname) using make
- You might first need to "make install" ICU so that the tools build can pick
up the new definitions from the installed header files.
* run genpname
(builds both pnames.icu and propname_data.h)
- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in
- ~/svn.icu/tools/trunk/bld/unicode/c$ genpname/genpname -v -d ~/svn.icu/trunk/src/source/common --csource
- rebuild ICU & tools
* run genprops
- ~/svn.icu/tools/trunk/bld/unicode/c$ genprops/genprops -d ~/svn.icu/trunk/src/source/data/in -s ~/svn.icu/trunk/src/source/data/unidata -i ~/svn.icu/trunk/dbg/data/out/build/icudt48l -u 6.0
- ~/svn.icu/tools/trunk/bld/unicode/c$ genprops/genprops -d ~/svn.icu/trunk/src/source/common --csource -s ~/svn.icu/trunk/src/source/data/unidata -i ~/svn.icu/trunk/dbg/data/out/build/icudt48l -u 6.0
- rebuild ICU & tools
* update Java data files
- refresh just the UCD-related files, just to be safe
- see (ICU4C)/source/data/icu4j-readme.txt
- mkdir /tmp/icu4j
- ~/svn.icu/trunk/dbg$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
- copy the big-endian Unicode data files to another location,
separate from the other data files
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt48b
~/svn.icu/trunk/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt48b/pnames.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt48b
~/svn.icu/trunk/dbg/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt48b/uprops.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt48b
- refresh ICU4J
~/svn.icu/trunk/dbg/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt48b
* should have updated the layout engine script codes but forgot
---------------------------------------------------------------------------- ***
Unicode 6.0 update
*** related ICU Trac tickets
7264 Unicode 6.0 Update
*** Unicode version numbers
- makedata.mak
- uchar.h
(configure.in & configure: have been modified to extract the version from uchar.h)
- com.ibm.icu.util.VersionInfo
*** data files & enums & parser code
* file preparation
~/svn.icu/tools/trunk/src/unicode/c/genprops/misc$ ./ucdcopy.py ~/uni60/20100720/ucd ~/uni60/processed
- This now prepares both unidata and testdata files in respective output subfolders.
* PropertyAliases.txt changes
- new Script_Extensions property defined in the new ScriptExtensions.txt file
but not listed in PropertyAliases.txt; reported to unicode.org;
-> added to tools/trunk/src/unicode/c/genpname/SyntheticPropertyAliases.txt
scx; Script_Extensions
-> uchar.h with new UProperty section
-> com.ibm.icu.lang.UProperty, parallel with uchar.h
* PropertyValueAliases.txt changes
- 12 new block names:
Alchemical_Symbols
Bamum_Supplement
Batak
Brahmi
CJK_Unified_Ideographs_Extension_D
Emoticons
Ethiopic_Extended_A
Kana_Supplement
Mandaic
Miscellaneous_Symbols_And_Pictographs
Playing_Cards
Transport_And_Map_Symbols
-> add to uchar.h
-> add to UCharacter.UnicodeBlock
Eclipse find UBLOCK_([^ ]+) = [0-9]+, (/.+)
replace public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2
- Joining_Group (jg) values:
Teh_Marbuta_Goal becomes the new canonical value for the old Hamza_On_Heh_Goal which becomes an alias
-> uchar.h & UCharacter.JoiningGroup
- 3 new scripts:
sc ; Batk ; Batak
sc ; Brah ; Brahmi
sc ; Mand ; Mandaic
-> remove these from SyntheticPropertyValueAliases.txt
-> add alias USCRIPT_MANDAIC to USCRIPT_MANDAEAN
-> fix expectedLong names in cucdapi.c/TestUScriptCodeAPI()
and in com.ibm.icu.dev.test.lang.TestUScript.java
- 13 new script codes from ISO 15924 http://www.unicode.org/iso15924/codechanges.html
(added 2009-11-11..2010-07-18)
Bass 259 Bassa Vah
Dupl 755 Duployan shortand
Elba 226 Elbasan
Gran 343 Grantha
Kpel 436 Kpelle
Loma 437 Loma
Mend 438 Mende
Merc 101 Meroitic Cursive
Narb 106 Old North Arabian
Nbat 159 Nabataean
Palm 126 Palmyrene
Sind 318 Sindhi
Wara 262 Warang Citi
-> uscript.h
-> com.ibm.icu.lang.UScript
find USCRIPT_([^ ]+) *= ([0-9]+),(.+)
replace public static final int \1 = \2;\3
-> SyntheticPropertyValueAliases.txt
-> add to expectedLong and expectedShort names in cintltst/cucdapi.c/TestUScriptCodeAPI()
and in com.ibm.icu.dev.test.lang.TestUScript.java
- ISO 15924 name change
Mero 100 Meroitic Hieroglyphs (was Meroitic)
-> add new alias USCRIPT_MEROITIC_HIEROGLYPHS to USCRIPT_MEROITIC
- property value alias added for Cham, was already moved out of SyntheticPropertyValueAliases.txt
* UnicodeData.txt changes
- new CJK block:
2B740;<CJK Ideograph Extension D, First>;Lo;0;L;;;;;N;;;;;
2B81D;<CJK Ideograph Extension D, Last>;Lo;0;L;;;;;N;;;;;
-> add to tools/trunk/src/unicode/c/gennames/gennames.c, with new ucdVersion
* build Unicode tools using CMake+make
* run genpname/preparse.pl (on Linux)
+ cd ~/svn.icu/tools/trunk/src/unicode/c/genpname
+ make sure that data.h is writable
+ perl preparse.pl ~/svn.icu/trunk/src > out.txt
+ preparse.pl shows no errors, out.txt Info and Warning lines look ok
* rebuild Unicode tools (at least genpname) using make
- You might first need to "make install" ICU so that the tools build can pick
up the new definitions from the installed header files.
* run genpname
- ~/svn.icu/tools/trunk/bld/unicode$ c/genpname/genpname -v -d ~/svn.icu/trunk/src/source/data/in
- rebuild ICU & tools
* update source/data/unidata/norm2/nfkc_cf.txt
- follow the instructions in nfkc_cf.txt for updating it from DerivedNormalizationProps.txt
* update source/data/unidata/norm2/uts46.txt
- download http://www.unicode.org/Public/idna/6.0.0/IdnaMappingTable.txt
to ~/svn.icu/tools/trunk/src/unicode/py
- adjust idna2nrm.py to handle new disallowed_STD3_valid and disallowed_STD3_mapped values
- ~/svn.icu/tools/trunk/src/unicode/py$ ./idna2nrm.py
- ~/svn.icu/tools/trunk/src/unicode/py$ cp uts46.txt ~/svn.icu/trunk/src/source/data/unidata/norm2
* update uts46test.cpp and UTS46Test.java if there are new characters that are equivalent to
sequences with non-LDH ASCII (that is, their decompositions contain '=' or similar)
- grep IdnaMappingTable.txt or uts46.txt for "disallowed_STD3_valid" on non-ASCII characters
- Unicode 6.0: U+2260, U+226E, U+226F
* generate core properties data files
- ~/svn.icu/tools/trunk/src/unicode$ ./makeprops.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU & tools
- run makeuca.sh so that genuca picks up the new nfc.nrm:
~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU & tools
* implement new Script_Extensions property (provisional)
- parser & generator: genprops & uprops.icu
- uscript.h, uprops.h, uchar.c, uniset_props.cpp and others, plus cintltst/cucdapi.c & intltest/usettest.cpp
- UScript.java, UCharacterProperty.java, UnicodeSet.java, TestUScript.java, UnicodeSetTest.java
* switch ubidi.icu, ucase.icu and uprops.icu from UTrie to UTrie2
- (one-time change)
- genbidi/gencase/genprops tools changes
- re-run makeprops.sh (see above)
- UCharacterProperty.java, UCharacterTypeIterator.java,
UBiDiProps.java, UCaseProps.java, and several others with minor changes;
UCharacterPropertyReader.java deleted and its code folded into UCharacterProperty.java
* update Java data files
- refresh just the UCD-related files, just to be safe
- see (ICU4C)/source/data/icu4j-readme.txt
- mkdir /tmp/icu4j
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
output:
...
Unicode .icu files built to ./out/build/icudt45l
mkdir -p ./out/icu4j/com/ibm/icu/impl/data/icudt45b
echo ubidi.icu ucase.icu uprops.icu > ./out/icu4j/add.txt
LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg ./out/tmp/icudt45l.dat ./out/icu4j/icudt45b.dat -a ./out/icu4j/add.txt -s ./out/build/icudt45l -x '*' -tb -d ./out/icu4j/com/ibm/icu/impl/data/icudt45b
jar cf ./out/icu4j/icudata.jar -C ./out/icu4j com/ibm/icu/impl/data/icudt45b
mkdir -p /tmp/icu4j/main/shared/data
cp ./out/icu4j/icudata.jar /tmp/icu4j/main/shared/data
- copy the big-endian Unicode data files to another location,
separate from the other data files
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/brkitr
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt45b
~/svn.icu/trunk/bld/data/out/icu4j$ rm /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/cnvalias.icu
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/*.nrm /tmp/icu4j/com/ibm/icu/impl/data/icudt45b
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/coll/*.icu /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/brkitr/* /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/brkitr
- refresh ICU4J
~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt45b
* refresh Java test .txt files
- copy new .txt files into ICU4J's main/tests/core/src/com/ibm/icu/dev/data/unicode
* un-hardcode normalization skippable (NF*_Inert) test data
- removes one manual step from the Unicode upgrade, and removes dependency on one of Mark's tools
* copy updated break iterator test files
- now handled by early ucdcopy.py and
copying the uni60/processed/testdata files to ~/svn.icu/trunk/src/source/test/testdata
(old instructions:
copy from (Unicode 6.0)/ucd/auxiliary/*BreakTest-6....txt
to ~/svn.icu/trunk/src/source/test/testdata)
- they are not used in ICU4J
* UCA
- get output from Mark's tools; look in
http://www.unicode.org/~book/incoming/mark/uca6.0.0/
http://www.macchiato.com/unicode/utc/additional-uca-files
http://www.unicode.org/Public/UCA/6.0.0/
http://www.unicode.org/~mdavis/uca/
- update source/data/unidata/FractionalUCA.txt with FractionalUCA_SHORT.txt
- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt
- update Han-implicit ranges for new CJK extensions:
swapCJK() in ucol.cpp & ImplicitCEGenerator.java
- genuca: allow bytes 02 for U+FFFE, new merge-sort character;
do not add it into invuca so that tailoring primary-after an ignorable works
- genuca: permit space between [variable top] bytes
- ucol.cpp: treat noncharacters like unassigned rather than ignorable
- run makeuca.sh:
~/svn.icu/tools/trunk/src/unicode$ ./makeuca.sh ~/svn.icu/trunk/src ~/svn.icu/trunk/bld
- rebuild ICU4C
- refresh ICU4J collation data:
(subset of instructions above for properties data refresh, except copies all coll/*)
~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
mkdir -p /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ cp com/ibm/icu/impl/data/icudt45b/coll/* /tmp/icu4j/com/ibm/icu/impl/data/icudt45b/coll
~/svn.icu/trunk/bld/data/out/icu4j$ jar uf ~/svn.icu4j/trunk/src/main/shared/data/icudata.jar -C /tmp/icu4j com/ibm/icu/impl/data/icudt45b
- update (ICU)/source/test/testdata/CollationTest_*.txt
and (ICU4J)/main/tests/collate/src/com/ibm/icu/dev/data/CollationTest_*.txt
with output from Mark's Unicode tools
- run all tests with the *_SHORT.txt or the full files (the full ones have comments)
- note on intltest: if collate/UCAConformanceTest fails, then
utility/MultithreadTest/TestCollators will fail as well;
fix the conformance test before looking into the multi-thread test
* When refreshing all of ICU4J data from ICU4C
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=/tmp/icu4j icu4j-data-install
- cp /tmp/icu4j/main/shared/data/icudata.jar ~/svn.icu4j/trunk/src/main/shared/data
or
- ~/svn.icu/trunk/bld$ make ICU4J_ROOT=~/svn.icu4j/trunk/src icu4j-data-install
*** LayoutEngine script information
(For details see the Unicode 5.2 change log below.)
* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguages.h,
ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (It also generates
ScriptRunData.cpp, which is no longer needed.)
The generated files have a current copyright date and "@draft" statement.
* copy the above files into <icu>/source/layout, replacing the old files.
* fix mixed line endings
* review the diffs and fix incorrect @draft and missing aliases;
Unicode-derived script codes should be "born stable" like constants in uchar.h, uscript.h etc.
* manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h
---------------------------------------------------------------------------- ***
Unicode 5.2 update
*** related ICU Trac tickets
7084 Unicode 5.2
7167 verify collation bytes
7235 Java test NAME_ALIAS
7236 Java DerivedCoreProperties.txt test
7237 Java BidiTest.txt
7238 UTrie2 in core unidata
7239 test for tailoring gaps
7240 Java fix CollationMiscTest
7243 update layout engine for Unicode 5.2
*** Unicode version numbers
- makedata.mak
- uchar.h
- configure.in & configure
- update ucdVersion in gennames.c if an algorithmic range changes
*** data files & enums & parser code
* file preparation
python source\tools\genprops\misc\ucdcopy.py "C:\Documents and Settings\mscherer\My Documents\unicode\ucd\5.2.0" C:\svn\icuproj\icu\trunk\source\data\unidata
- includes finding files regardless of version numbers,
copying them, and performing the equivalent processing of the
ucdstrip and ucdmerge tools on the desired set of files
* notes on changes
- PropertyAliases.txt
moved from numeric to enumerated:
ccc ; Canonical_Combining_Class
new string properties:
NFKC_CF ; NFKC_Casefold
Name_Alias; Name_Alias
new binary properties:
Cased ; Cased
CI ; Case_Ignorable
CWCF ; Changes_When_Casefolded
CWCM ; Changes_When_Casemapped
CWKCF ; Changes_When_NFKC_Casefolded
CWL ; Changes_When_Lowercased
CWT ; Changes_When_Titlecased
CWU ; Changes_When_Uppercased
new CJK Unihan properties (not supported by ICU)
- PropertyValueAliases.txt
new block names
new scripts
one script code change:
sc ; Qaai ; Inherited
->
sc ; Zinh ; Inherited ; Qaai
new Line_Break (lb) value:
lb ; CP ; Close_Parenthesis
new Joining_Group (jg) values: Farsi_Yeh, Nya
other new values:
ccc; 214; ATA ; Attached_Above
- DerivedBidiClass.txt
new default-R range: U+1E800 - U+1EFFF
- UnicodeData.txt
all of the ISO comments are gone
new CJK block end:
9FC3;<CJK Ideograph, Last> -> 9FCB;<CJK Ideograph, Last>
new CJK block:
2A700;<CJK Ideograph Extension C, First>;Lo;0;L;;;;;N;;;;;
2B734;<CJK Ideograph Extension C, Last>;Lo;0;L;;;;;N;;;;;
* genpname
- run preparse.pl
+ cd \svn\icuproj\icu\trunk\source\tools\genpname
+ make sure that data.h is writable
+ perl preparse.pl \svn\icuproj\icu\trunk > out.txt
+ preparse.pl complains with errors like the following:
Error: sc:Egyp already set to Egyptian_Hieroglyphs, cannot set to Egyp at preparse.pl line 1322, <GEN6> line 34.
This is because ICU 4.0 had scripts from ISO 15924 which are now
added to Unicode 5.2, and the Perl script shows a conflict between SyntheticPropertyValueAliases.txt
and PropertyValueAliases.txt.
-> Removed duplicate script entries from SyntheticPropertyValueAliases.txt:
Egyp, Java, Lana, Mtei, Orkh, Armi, Avst, Kthi, Phli, Prti, Samr, Tavt
+ preparse.pl complains with errors about block names missing from uchar.h; add them
* uchar.h & uscript.h & uprops.h & uprops.c & genprops
- new block & script values
+ 26 new blocks
copy new blocks from Blocks.txt
MS VC++ 2008 regular expression:
find "^{[0-9A-F]+}\.\.{[0-9A-F]+}; {[A-Z].+}$"
replace with " UBLOCK_\3 = 172, /*[\1]*/"
+ several new script values already added in ICU 4.0 for ISO 15924 coverage
(removed from SyntheticPropertyValueAliases.txt, see genpname notes above)
+ 3 new script values added for ISO 15924 and Unicode 5.2 coverage
+ 1 new script value added for ISO 15924 coverage (not in Unicode 5.2)
(added to SyntheticPropertyValueAliases.txt)
- new Joining Group (JG) values: Farsi_Yeh, Nya
- new Line_Break (lb) value:
lb ; CP ; Close_Parenthesis
* hardcoded Unihan range end/limit
- Unihan range end moves from 9FC3 to 9FCB
search for both 9FC3 (end) and 9FC4 (limit) (regex 9FC[34], case-insensitive)
+ do change gennames.c
* Compare definitions of new binary properties with what we used to use
in algorithms, to see if the definitions changed.
- Verified that definitions for Cased and Case_Ignorable are unchanged.
The gencase tool now parses the newly public Case_Ignorable values
in case the definition changes in the future.
* uchar.c & uprops.h & uprops.c & genprops
- new numeric values that didn't exist in Unicode data before:
1/7, 1/9, 1/10, 3/10, 1/16, 3/16
the ones with denominators >9 cannot be supported by uprops.icu formatVersion 5,
therefore redesign the encoding of numeric types and values for formatVersion 6;
design for simple numbers up to at least 144 ("one gross"),
large values up to at least 10^20,
and fractions with numerators -1..17 and denominators 1..16
to cover current and expected future values
(e.g., more Han numeric values, Meroitic twelfths)
* reimplement Hangul_Syllable_Type for new Jamo characters
- the old code assumed that all Jamo characters are in the 11xx block
- Unicode 5.2 fills holes there and adds new Jamo characters in
A960..A97F; Hangul Jamo Extended-A
and in
D7B0..D7FF; Hangul Jamo Extended-B
- Hangul_Syllable_Type can be trivially derived from a subset of
Grapheme_Cluster_Break values
* build Unicode data source code for hardcoding core data
C:\svn\icuproj\icu\trunk\source\data>NMAKE /f makedata.mak ICUMAKE=\svn\icuproj\icu\trunk\source\data\ CFG=x86\release uni-core-data
ICU data make path is \svn\icuproj\icu\trunk\source\data\
ICU root path is \svn\icuproj\icu\trunk
Information: cannot find "ucmlocal.mk". Not building user-additional converter files.
Information: cannot find "brklocal.mk". Not building user-additional break iterator files.
Information: cannot find "reslocal.mk". Not building user-additional resource bundle files.
Information: cannot find "collocal.mk". Not building user-additional resource bundle files.
Information: cannot find "rbnflocal.mk". Not building user-additional resource bundle files.
Information: cannot find "trnslocal.mk". Not building user-additional transliterator files.
Information: cannot find "misclocal.mk". Not building user-additional miscellaenous files.
Information: cannot find "spreplocal.mk". Not building user-additional stringprep files.
Creating data file for Unicode Property Names
Creating data file for Unicode Character Properties
Creating data file for Unicode Case Mapping Properties
Creating data file for Unicode BiDi/Shaping Properties
Creating data file for Unicode Normalization
Unicode .icu files built to "\svn\icuproj\icu\trunk\source\data\out\build\icudt43l"
Unicode .c source files built to "\svn\icuproj\icu\trunk\source\data\out\tmp"
- copy the .c source files to C:\svn\icuproj\icu\trunk\source\common
and rebuild the common library
*** UCA
- update FractionalUCA.txt with new canonical closure (output from Mark's Unicode tools)
- update source/data/unidata/UCARules.txt with UCA_Rules_SHORT.txt from Mark's Unicode tools
- update source/test/testdata/CollationTest_*.txt with output from Mark's Unicode tools
[ Begin obsolete instructions:
Starting with UCA 5.2, we use the CollationTest_*_SHORT.txt files not the *_STUB.txt files.
- generate the source/test/testdata/CollationTest_*_STUB.txt files via source/tools/genuca/genteststub.py
on Windows:
python C:\svn\icuproj\icu\trunk\source\tools\genuca\genteststub.py CollationTest_NON_IGNORABLE_SHORT.txt CollationTest_NON_IGNORABLE_STUB.txt
python C:\svn\icuproj\icu\trunk\source\tools\genuca\genteststub.py CollationTest_SHIFTED_SHORT.txt CollationTest_SHIFTED_STUB.txt
End obsolete instructions]
- run all tests with the *_SHORT.txt or the full files (the full ones have comments)
not just the *_STUB.txt files
- note on intltest: if collate/UCAConformanceTest fails, then
utility/MultithreadTest/TestCollators will fail as well;
fix the conformance test before looking into the multi-thread test
*** Implement Cased & Case_Ignorable properties
- via UProperty; call ucase.h functions ucase_getType() and ucase_getTypeOrIgnorable()
- Problem: These properties should be disjoint, but aren't
- UTC 2009nov decision: skip all Case_Ignorable regardless of whether they are Cased or not
- change ucase.icu to be able to store any combination of Cased and Case_Ignorable
*** Implement Changes_When_Xyz properties
- without stored data
*** Implement Name_Alias property
- add it as another name field in unames.icu
- make it available via u_charName() and UCharNameChoice and
- consider it in u_charFromName()
*** Break iterators
* Update break iterator rules to new UAX versions and new property values
* Update source/test/testdata/<boundary>Test.txt files from <unicode.org ucd>/ucd/auxiliary
*** new BidiTest file
- review format and data
- copy BidiTest.txt to source/test/testdata
- write test code using this data
- fix ICU code where it fails the conformance test
*** Java
- generally, find and update code corresponding to C/C++
- UCharacter.UnicodeBlock constants:
a) add an _ID integer per new block, update COUNT
b) add a class instance per new block
Visual Studio regex:
find UBLOCK_{[^ ]+} = [0-9]+, {/.+}
replace with public static final UnicodeBlock \1 = new UnicodeBlock("\1", \1_ID); \2
- CHAR_NAME_ALIAS -> UCharacter.getNameAlias() and getCharFromNameAlias()
- port test changes to Java
*** LayoutEngine script information
(For comparison, see the Unicode 5.1 update: http://bugs.icu-project.org/trac/changeset/23833)
* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguages.h,
ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (It also generates
ScriptRunData.cpp, which is no longer needed.)
The generated files have a current copyright date and "@draft" statement.
-> Eric Mader wrote in email on 20090930:
"I think the tool has been modified to update @draft to @stable for
older scripts and to add @draft for new scripts.
(I worked with an intern on this last year.)
You should check the output after you run it."
* copy the above files into <icu>/source/layout, replacing the old files.
* fix mixed line endings
* review the diffs and fix incorrect @draft and missing aliases
* manually re-add the "Indic script xyz v.2" tags in ScriptAndLanguageTags.h
Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp
and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...)
-> Eric Mader wrote in email on 20090930:
"This is just a matter of making sure that all the per-script tables have
entries for any new scripts that were added.
If any new Indic characters were added, then the class tables in
IndicClassTables.cpp should be updated to reflect this.
John Emmons should know how to do this if it's required."
* rebuild the layout and layoutex libraries.
*** Documentation
- Update User Guide
+ Jamo_Short_Name, sfc->scf, binary property value aliases
---------------------------------------------------------------------------- ***
Unicode 5.1 update
*** related ICU Trac tickets
5696 Update to Unicode 5.1
*** Unicode version numbers
- makedata.mak
- uchar.h
- configure.in & configure
- update ucdVersion in gennames.c if an algorithmic range changes
*** data files & enums & parser code
* file preparation
- ucdstrip:
DerivedCoreProperties.txt
DerivedNormalizationProps.txt
NormalizationTest.txt
PropList.txt
Scripts.txt
GraphemeBreakProperty.txt
SentenceBreakProperty.txt
WordBreakProperty.txt
- ucdstrip and ucdmerge:
EastAsianWidth.txt
LineBreak.txt
* my ucd2unidata.bat (needs to be updated each time with UCD and file version numbers)
copy 5.1.0\ucd\BidiMirroring.txt ..\unidata\
copy 5.1.0\ucd\Blocks.txt ..\unidata\
copy 5.1.0\ucd\CaseFolding.txt ..\unidata\
copy 5.1.0\ucd\DerivedAge.txt ..\unidata\
copy 5.1.0\ucd\extracted\DerivedBidiClass.txt ..\unidata\
copy 5.1.0\ucd\extracted\DerivedJoiningGroup.txt ..\unidata\
copy 5.1.0\ucd\extracted\DerivedJoiningType.txt ..\unidata\
copy 5.1.0\ucd\extracted\DerivedNumericValues.txt ..\unidata\
copy 5.1.0\ucd\NormalizationCorrections.txt ..\unidata\
copy 5.1.0\ucd\PropertyAliases.txt ..\unidata\
copy 5.1.0\ucd\PropertyValueAliases.txt ..\unidata\
copy 5.1.0\ucd\SpecialCasing.txt ..\unidata\
copy 5.1.0\ucd\UnicodeData.txt ..\unidata\
ucdstrip < 5.1.0\ucd\DerivedCoreProperties.txt > ..\unidata\DerivedCoreProperties.txt
ucdstrip < 5.1.0\ucd\DerivedNormalizationProps.txt > ..\unidata\DerivedNormalizationProps.txt
ucdstrip < 5.1.0\ucd\NormalizationTest.txt > ..\unidata\NormalizationTest.txt
ucdstrip < 5.1.0\ucd\PropList.txt > ..\unidata\PropList.txt
ucdstrip < 5.1.0\ucd\Scripts.txt > ..\unidata\Scripts.txt
ucdstrip < 5.1.0\ucd\auxiliary\GraphemeBreakProperty.txt > ..\unidata\GraphemeBreakProperty.txt
ucdstrip < 5.1.0\ucd\auxiliary\SentenceBreakProperty.txt > ..\unidata\SentenceBreakProperty.txt
ucdstrip < 5.1.0\ucd\auxiliary\WordBreakProperty.txt > ..\unidata\WordBreakProperty.txt
ucdstrip < 5.1.0\ucd\EastAsianWidth.txt | ucdmerge > ..\unidata\EastAsianWidth.txt
ucdstrip < 5.1.0\ucd\LineBreak.txt | ucdmerge > ..\unidata\LineBreak.txt
* genpname
- run preparse.pl
+ cd \svn\icuproj\icu\uni51\source\tools\genpname
+ make sure that data.h is writable
+ perl preparse.pl \svn\icuproj\icu\uni51 > out.txt
+ preparse.pl complains with errors like the following:
Error: sc:Cari already set to Carian, cannot set to Cari at preparse.pl line 1308, <GEN6> line 30.
This is because ICU 3.8 had scripts from ISO 15924 which are now
added to Unicode 5.1, and the script shows a conflict between SyntheticPropertyValueAliases.txt
and PropertyValueAliases.txt.
-> Removed duplicate script entries from SyntheticPropertyValueAliases.txt:
Cari, Cham, Kali, Lepc, Lyci, Lydi, Olck, Rjng, Saur, Sund, Vaii
+ PropertyValueAliases.txt now explicitly contains values for boolean properties:
N/Y, No/Yes, F/T, False/True
-> Added N/No and Y/Yes to preparse.pl function read_PropertyValueAliases.
It will use further values from the file if present.
* uchar.h & uscript.h & uprops.h & uprops.c & genprops
- new block & script values
+ 17 new blocks
+ 11 new script values already added in ICU 3.8 for ISO 15924 coverage
(removed from SyntheticPropertyValueAliases.txt)
+ 14 new script values added for ISO 15924 coverage (not in Unicode 5.1)
(added to SyntheticPropertyValueAliases.txt)
- uprops.icu (uprops.h) only provides 7 bits for script codes.
In ICU 4.0 there are USCRIPT_CODE_LIMIT=130 script codes now.
There is none above 127 yet which is the script code for an
assigned Unicode character, so ICU 4.0 uprops.icu does not store any
script code values greater than 127.
However, it does need to store the maximum script value=USCRIPT_CODE_LIMIT-1=129
in a parallel bit field, and that overflows now.
Also, future values >=128 would be incompatible anyway.
uprops.h is modified to move around several of the bit fields
in the properties vector words, and now uses 8 bits for the script code.
Two other bit fields also grow to accommodate future growth:
Block (current count: 172) grows from 8 to 9 bits,
and Word_Break grows from 4 to 5 bits.
- renamed property Simple_Case_Folding (sfc->scf)
+ nothing to be done: handled as normal alias
- new property JSN Jamo_Short_Name
+ no new API: only contributes to the Name property
- new Grapheme_Cluster_Break (GCB) value: SM=SpacingMark
- new Joining Group (JG) value: Burushashki_Yeh_Barree
- new Sentence_Break (SB) values:
SB ; CR ; CR
SB ; EX ; Extend
SB ; LF ; LF
SB ; SC ; SContinue
- new Word_Break (WB) values:
WB ; CR ; CR
WB ; Extend ; Extend
WB ; LF ; LF
WB ; MB ; MidNumLet
* Further changes in the 2008-02-29 update:
- Default_Ignorable_Code_Point: The new file removes Cc, Cs, noncharacters from DICP
because they should not normally be invisible.
- new Joining Group (JG) value Burushashki_Yeh_Barree was renamed to Burushaski_Yeh_Barree (one 'h' removed)
- new Grapheme_Cluster_Break (GCB) value: PP=Prepend
- new Word_Break (WB) value: NL=Newline
* hardcoded Unihan range end/limit (see Unicode 4.1 update for comparison)
- Unihan range end moves from 9FBB to 9FC3
search for both 9FBB (end) and 9FBC (limit) (regex 9FB[BC], case-insensitive)
+ do change gennames.c
* build Unicode data source code for hardcoding core data
C:\svn\icuproj\icu\uni51\source\data>NMAKE /f makedata.mak ICUMAKE=\svn\icuproj\icu\uni51\source\data\ CFG=debug uni-core-data
ICU data make path is \svn\icuproj\icu\uni51\source\data\
ICU root path is \svn\icuproj\icu\uni51
Information: cannot find "ucmlocal.mk". Not building user-additional converter files.
Information: cannot find "brklocal.mk". Not building user-additional break iterator files.
Information: cannot find "reslocal.mk". Not building user-additional resource bundle files.
Information: cannot find "collocal.mk". Not building user-additional resource bundle files.
Information: cannot find "rbnflocal.mk". Not building user-additional resource bundle files.
Information: cannot find "trnslocal.mk". Not building user-additional transliterator files.
Information: cannot find "misclocal.mk". Not building user-additional miscellaenous files.
Creating data file for Unicode Character Properties
Creating data file for Unicode Case Mapping Properties
Creating data file for Unicode BiDi/Shaping Properties
Creating data file for Unicode Normalization
Unicode .icu files built to "\svn\icuproj\icu\uni51\source\data\out\build\icudt39l"
Unicode .c source files built to "\svn\icuproj\icu\uni51\source\data\out\tmp"
- copy the .c source files to C:\svn\icuproj\icu\uni51\source\common
and rebuild the common library
*** Break iterators
* Update break iterator rules to new UAX versions and new property values
*** UCA
* update FractionalUCA.txt and UCARules.txt with new canonical closure
*** Test suites
- Test that APIs using Unicode property value aliases (like UnicodeSet)
support all of the boolean values N/Y, No/Yes, F/T, False/True
-> TestBinaryValues() tests in both cintltst and intltest
*** LayoutEngine script information
* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguage.h,
ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (it also generates
ScriptRunData.cpp, which is no longer needed.)
The generated files have a current copyright date and "@draft" statement.
* copy the above files into <icu>/source/layout, replacing the old files.
Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp
and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...)
* rebuild the layout and layoutex libraries.
*** Documentation
- Update User Guide
+ Jamo_Short_Name, sfc->scf, binary property value aliases
---------------------------------------------------------------------------- ***
Unicode 5.0 update
*** related Jitterbugs
5084 RFE: Update to Unicode 5.0
*** data files & enums & parser code
* file preparation
- ucdstrip:
DerivedCoreProperties.txt
DerivedNormalizationProps.txt
NormalizationTest.txt
PropList.txt
Scripts.txt
GraphemeBreakProperty.txt
SentenceBreakProperty.txt
WordBreakProperty.txt
- ucdstrip and ucdmerge:
EastAsianWidth.txt
LineBreak.txt
* my ucd2unidata.bat (needs to be updated each time with UCD and file version numbers)
copy 5.0.0\ucd\BidiMirroring.txt ..\unidata\
copy 5.0.0\ucd\Blocks.txt ..\unidata\
copy 5.0.0\ucd\CaseFolding.txt ..\unidata\
copy 5.0.0\ucd\DerivedAge.txt ..\unidata\
copy 5.0.0\ucd\extracted\DerivedBidiClass.txt ..\unidata\
copy 5.0.0\ucd\extracted\DerivedJoiningGroup.txt ..\unidata\
copy 5.0.0\ucd\extracted\DerivedJoiningType.txt ..\unidata\
copy 5.0.0\ucd\extracted\DerivedNumericValues.txt ..\unidata\
copy 5.0.0\ucd\NormalizationCorrections.txt ..\unidata\
copy 5.0.0\ucd\PropertyAliases.txt ..\unidata\
copy 5.0.0\ucd\PropertyValueAliases.txt ..\unidata\
copy 5.0.0\ucd\SpecialCasing.txt ..\unidata\
copy 5.0.0\ucd\UnicodeData.txt ..\unidata\
ucdstrip < 5.0.0\ucd\DerivedCoreProperties.txt > ..\unidata\DerivedCoreProperties.txt
ucdstrip < 5.0.0\ucd\DerivedNormalizationProps.txt > ..\unidata\DerivedNormalizationProps.txt
ucdstrip < 5.0.0\ucd\NormalizationTest.txt > ..\unidata\NormalizationTest.txt
ucdstrip < 5.0.0\ucd\PropList.txt > ..\unidata\PropList.txt
ucdstrip < 5.0.0\ucd\Scripts.txt > ..\unidata\Scripts.txt
ucdstrip < 5.0.0\ucd\auxiliary\GraphemeBreakProperty.txt > ..\unidata\GraphemeBreakProperty.txt
ucdstrip < 5.0.0\ucd\auxiliary\SentenceBreakProperty.txt > ..\unidata\SentenceBreakProperty.txt
ucdstrip < 5.0.0\ucd\auxiliary\WordBreakProperty.txt > ..\unidata\WordBreakProperty.txt
ucdstrip < 5.0.0\ucd\EastAsianWidth.txt | ucdmerge > ..\unidata\EastAsianWidth.txt
ucdstrip < 5.0.0\ucd\LineBreak.txt | ucdmerge > ..\unidata\LineBreak.txt
* update FractionalUCA.txt and UCARules.txt with new canonical closure
* genpname
- run preparse.pl
+ make sure that data.h is writable
+ perl preparse.pl \cvs\oss\icu > out.txt
* uchar.h & uscript.h & uprops.h & uprops.c & genprops
- new block & script values
+ script values already added in ICU 3.6 because all of ISO 15924 is now covered
* build Unicode data source code for hardcoding core data
C:\cvs\oss\icu\source\data>NMAKE /f makedata.mak ICUMAKE=\cvs\oss\icu\source\data\ CFG=debug uni-core-data
ICU data make path is \cvs\oss\icu\source\data\
ICU root path is \cvs\oss\icu
Information: cannot find "ucmlocal.mk". Not building user-additional converter files.
[etc.]
Creating data file for Unicode Character Properties
Creating data file for Unicode Case Mapping Properties
Creating data file for Unicode BiDi/Shaping Properties
Creating data file for Unicode Normalization
Unicode .icu files built to "\cvs\oss\icu\source\data\out\build\icudt35l"
Unicode .c source files built to "\cvs\oss\icu\source\data\out\tmp"
- copy the .c source files to C:\cvs\oss\icu\source\common
and rebuild the common library
*** Unicode version numbers
- makedata.mak
- uchar.h
- configure.in
*** LayoutEngine script information
* Run ICU4J com.ibm.icu.dev.tool.layout.ScriptNameBuilder. This generates LEScripts.h, LELanguage.h,
ScriptAndLanguageTags.h and ScriptAndLanguageTags.cpp in the working directory. (it also generates
ScriptRunData.cpp, which is no longer needed.)
The generated files have a current copyright date and "@draft" statement.
* copy the above files into <icu>/source/layout, replacing the old files.
Add new default entries to the indicClassTables array in <icu>/source/layout/IndicClassTables.cpp
and the complexTable array in <icu>/source/layoutex/ParagraphLayout.cpp. (This step should be automated...)
* rebuild the layout and layoutex libraries.
---------------------------------------------------------------------------- ***
Unicode 4.1 update
*** related Jitterbugs
4332 RFE: Update to Unicode 4.1
4157 RBBI, TR29 4.1 updates
*** data files & enums & parser code
* file preparation
- ucdstrip:
DerivedCoreProperties.txt
DerivedNormalizationProps.txt
NormalizationTest.txt
GraphemeBreakProperty.txt
SentenceBreakProperty.txt
WordBreakProperty.txt
- ucdstrip and ucdmerge:
EastAsianWidth.txt
LineBreak.txt
* add new files to the repository
GraphemeBreakProperty.txt
SentenceBreakProperty.txt
WordBreakProperty.txt
* update FractionalUCA.txt and UCARules.txt with new canonical closure
* genpname
- handle new enumerated properties in sub read_uchar
- run preparse.pl
* uchar.h & uscript.h & uprops.h & uprops.c & genprops
- new binary properties
+ Pattern_Syntax
+ Pattern_White_Space
- new enumerated properties
+ Grapheme_Cluster_Break
+ Sentence_Break
+ Word_Break
- new block & script & line break values
* gencase
- case-ignorable changes
see http://www.unicode.org/versions/Unicode4.1.0/#CaseMods
now: (D47a) Word_Break=MidLetter or Mn, Me, Cf, Lm, Sk
*** Unicode version numbers
- makedata.mak
- uchar.h
- configure.in
*** tests
- verify that u_charMirror() round-trips
- test all new properties and some new values of old properties
*** other code
* hardcoded Unihan range end/limit
- Unihan range end moves from 9FA5 to 9FBB
search for both 9FA5 (end) and 9FA6 (limit) (regex 9FA[56], case-insensitive)
+ do not modify BOCU/BOCSU code because that would change the encoding
and break binary compatibility!
+ similarly, do not change the GB 18030 range data (ucnvmbcs.c),
NamePrepProfile.txt
+ ignore trietest.c: test data is arbitrary
+ ignore tstnorm.cpp: test optimization, not important
+ ignore collation: 9FA[56] only appears in comments; swapCJK() uses the whole block up to 9FFF
+ do change line_th.txt and word_th.txt
by replacing hardcoded ranges with the new property values
+ do change gennames.c
source\data\brkitr\line_th.txt(229): \u33E0-\u33FE \u3400-\u4DB5 \u4E00-\u9FA5 \uA000-\uA48C \uA490-\uA4C6
source\data\brkitr\word_th.txt(23): \u33E0-\u33FE \u3400-\u4DB5 \u4E00-\u9FA5 \uA000-\uA48C \uA490-\uA4C6
source\tools\gennames\gennames.c(971): 0x4e00, 0x9fa5,
* case mappings
- compare new special casing context conditions with previous ones
see http://www.unicode.org/versions/Unicode4.1.0/#CaseMods
* genpname
- consider storing only the short name if it is the same as the long name
*** other reviews
- UAX #29 changes (grapheme/word/sentence breaks)
- UAX #14 changes (line breaks)
- Pattern_Syntax & Pattern_White_Space
---------------------------------------------------------------------------- ***
Unicode 4.0.1 update
*** related Jitterbugs
3170 RFE: Update to Unicode 4.0.1
3171 Add new Unicode 4.0.1 properties
3520 use Unicode 4.0.1 updates for break iteration
*** data files & enums & parser code
* file preparation
- ucdstrip: DerivedNormalizationProps.txt, NormalizationTest.txt, DerivedCoreProperties.txt
- ucdstrip and ucdmerge: EastAsianWidth.txt, LineBreak.txt
* file fixes
- fix UnicodeData.txt general categories of Ethiopic digits Nd->No
according to PRI #26
http://www.unicode.org/review/resolved-pri.html#pri26
- undone again because no corrigendum in sight;
instead modified tests to not check consistency on this for Unicode 4.0.1
* ucdterms.txt
- update from http://www.unicode.org/copyright.html
formatted for plain text
* uchar.h & uprops.h & uprops.c & genprops
- add UBLOCK_CYRILLIC_SUPPLEMENT because the block is renamed
- add U_LB_INSEPARABLE due to a spelling fix
+ put short name comment only on line with new constant
for genpname perl script parser
- new binary properties
+ STerm
+ Variation_Selector
* genpname
- fix genpname perl script so that it doesn't choke on more than 2 names per property value
- perl script: correctly calculate the maximum number of fields per row
* uscript.h
- new script code Hrkt=Katakana_Or_Hiragana
* gennorm.c track changes in DerivedNormalizationProps.txt
- "FNC" -> "FC_NFKC"
- single field "NFD_NO" -> two fields "NFD_QC; N" etc.
* genprops/props2.c track changes in DerivedNumericValues.txt
- changed from 3 columns to 2, dropping the numeric type
+ assume that the type is always numeric for Han characters,
and that only those are added in addition to what UnicodeData.txt lists
*** Unicode version numbers
- makedata.mak
- uchar.h
- configure.in
*** tests
- update test of default bidi classes according to PRI #28
/tsutil/cucdtst/TestUnicodeData
http://www.unicode.org/review/resolved-pri.html#pri28
- bidi tests: change exemplar character for ES depending on Unicode version
- change hardcoded expected property values where they change
*** other code
* name matching
- read UCD.html
* scripts
- use new Hrkt=Katakana_Or_Hiragana
* ZWJ & ZWNJ
- are now part of combining character sequences
- break iteration used to assume that LB classes did not overlap; now they do for ZWJ & ZWNJ
|