Skip to content
This repository was archived by the owner on May 30, 2025. It is now read-only.

Commit 26ebcb4

Browse files
apiadapiad
committed
Updated data
1 parent 793a0fd commit 26ebcb4

8 files changed

Lines changed: 782 additions & 3 deletions

File tree

‎Readme.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,11 @@ The files and folders are organized as follows:
77
* `gold/` contains the `input_*.txt` files and the corresponding `output_*.txt` files of the training data.
88
* `dev/` is an empty folder where you are expected to output your corresponding `output_*.txt` files.
99
* `example/` contains a example input and output files, with some errors purposedly included, that you can use to understand exactly how the evaluation metric works.
10+
* `submit/` contains a sample submission (actually just the `trial` data splitted accordingly).
1011

11-
## Evaluation
12+
## Development evaluation
1213

13-
The file `evaluate.py` performs an automatic evaluation of your output files against the gold files. You can use this script to validate your technique(s). The metrics reported are exactly the same ones that will be used in the final evaluation.
14+
The file `evaluate.py` performs an automatic evaluation of your output files against the gold files. You can use this script to validate your technique(s). The metrics reported are exactly the same ones that will be used in the final evaluation. This script simply evaluates each pair of gold/dev files separately and outputs detailed information of all the mistakes.
1415

1516
To run it simply use:
1617

@@ -24,10 +25,14 @@ If the optional args `gold-folder` and `dev-folder` are provided, then the files
2425
python3 evaluate.py example/gold example/dev
2526
```
2627

28+
## Final evaluation
29+
30+
The file `score.py` performs the final evaluation exactly as described in the competition rules, i.e., according to the three evaluation scenarios presented. It assumes the gold files are in `gold` and the files to be submitted are in the `submit` folder, according to the folder structure presented there.
31+
2732
## Notes
2833

2934
The actual training data is not ready yet. Only the trial data is included in the `gold` folder now. This repository will be updated with the actual gold training files in due time.
3035

3136
You can use the trial data to see a more complex scenario than that presented in the examples, and to begin developing your ideas until the actual training data is ready.
3237

33-
**The trial data is not expected to be part of the final evaluation, just use it for your convenience now**.
38+
**The trial data is not expected to be part of the final evaluation, just use it for your convenience now.**

‎score.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/python3
2+
3+
import sys
4+
import os
5+
import random
6+
7+
with open(os.path.join(sys.argv[2], 'scores.txt'), 'wb') as fp:
8+
for label in "abc bc c".split():
9+
for val in "f1 prec rec".split():
10+
fp.write('%s_%s:%.5f\n' % (label, val, random.uniform(0,1)))
11+
12+
fp.write('macro:%.5f\n' % random.uniform(0,1))

‎submit/scenario1-ABB/output_A_trial.txt‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
1 35 46
2+
2 50 58
3+
3 3 9
4+
5 72 81
5+
6 84 91
6+
7 108 115
7+
8 121 125
8+
9 131 139
9+
10 144 149
10+
11 97 104
11+
12 63 71
12+
13 188 199
13+
15 171 180
14+
16 154 167
15+
17 216 224
16+
18 256 264
17+
19 241 252
18+
20 272 280
19+
21 229 237
20+
22 368 376
21+
24 333 346
22+
26 307 313
23+
27 292 301
24+
28 359 367
25+
29 285 291
26+
30 394 402
27+
31 412 416
28+
32 417 423
29+
33 432 438
30+
34 452 459
31+
35 446 451
32+
36 403 408
33+
37 465 471
34+
39 515 518
35+
40 525 530
36+
41 534 542
37+
42 489 494
38+
43 502 509
39+
44 658 663
40+
45 713 721
41+
46 722 731
42+
48 693 702
43+
49 750 768
44+
50 778 796
45+
51 804 809
46+
52 819 828
47+
53 810 815
48+
54 835 845
49+
55 874 877
50+
56 878 887
51+
57 917 925
52+
58 908 913
53+
59 933 941
54+
60 945 958
55+
61 961 969
56+
62 1012 1026
57+
63 987 991
58+
64 992 997
59+
65 971 980
60+
66 1037 1043
61+
67 1044 1052
62+
68 1060 1067
63+
71 1103 1109
64+
70 1127 1133
65+
72 1140 1147
66+
73 1140 1154
67+
74 1134 1139
68+
75 1203 1210
69+
76 1203 1222
70+
77 1227 1233
71+
78 1186 1196
72+
79 1364 1370
73+
80 1298 1305
74+
81 1306 1312
75+
82 1320 1323
76+
83 1324 1334
77+
84 1341 1348
78+
85 1349 1360
79+
86 1291 1297
80+
87 1421 1427
81+
88 1432 1436
82+
89 1445 1450
83+
90 1394 1401
84+
93 1475 1483
85+
94 1486 1494
86+
92 1456 1463
87+
96 1520 1523
88+
97 1524 1530
89+
98 1549 1557
90+
101 1612 1619
91+
102 1573 1580
92+
103 1624 1631
93+
108 1885 1893
94+
109 1900 1906
95+
110 1942 1946
96+
112 1927 1935
97+
113 1912 1918
98+
115 1868 1874
99+
116 1951 1963
100+
117 1970 1976
101+
118 2023 2033
102+
121 2122 2132
103+
122 2039 2047
104+
123 2053 2064
105+
124 2065 2073
106+
126 2152 2164
107+
127 2165 2172
108+
128 2173 2180
109+
129 2182 2191
110+
130 2196 2209
111+
131 2234 2241
112+
132 2242 2258
113+
133 2260 2273
114+
134 2291 2308
115+
135 2312 2322
116+
136 2326 2339
117+
137 368 388
118+
4 16 29
119+
14 188 211
120+
23 465 481
121+
38 741 746
122+
114 1868 1884
123+
111 1912 1926
124+
138 2291 2298
125+
139 2282 2287
126+
140 2242 2249
127+
141 778 785
128+
142 750 757
129+
143 333 358
130+
25 706 712
131+
69 664 673
132+
144 683 692
133+
47 733 740
134+
145 846 855
135+
91 1402 1413
136+
146 1500 1507
137+
95 1500 1516
138+
99 1588 1595
139+
147 1588 1604
140+
100 1632 1640
141+
104 2014 2022
142+
105 2086 2092
143+
106 2078 2082
144+
107 2101 2109
145+
119 2326 2333

‎submit/scenario1-ABB/output_B_trial.txt‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
1 Action
2+
2 Concept
3+
3 Concept
4+
5 Concept
5+
6 Concept
6+
7 Concept
7+
8 Concept
8+
9 Concept
9+
10 Concept
10+
11 Action
11+
12 Action
12+
13 Concept
13+
15 Action
14+
16 Concept
15+
17 Concept
16+
18 Concept
17+
19 Action
18+
20 Concept
19+
21 Concept
20+
22 Concept
21+
24 Concept
22+
26 Concept
23+
27 Action
24+
28 Action
25+
29 Action
26+
30 Concept
27+
31 Concept
28+
32 Action
29+
33 Concept
30+
34 Concept
31+
35 Concept
32+
36 Action
33+
37 Concept
34+
39 Action
35+
40 Concept
36+
41 Action
37+
42 Action
38+
43 Concept
39+
44 Concept
40+
45 Concept
41+
46 Concept
42+
48 Action
43+
49 Concept
44+
50 Concept
45+
51 Concept
46+
52 Concept
47+
53 Action
48+
54 Concept
49+
55 Action
50+
56 Concept
51+
57 Concept
52+
58 Concept
53+
59 Concept
54+
60 Concept
55+
61 Concept
56+
62 Concept
57+
63 Concept
58+
64 Action
59+
65 Action
60+
66 Concept
61+
67 Action
62+
68 Concept
63+
71 Concept
64+
70 Concept
65+
72 Concept
66+
73 Concept
67+
74 Action
68+
75 Concept
69+
76 Concept
70+
77 Action
71+
78 Action
72+
79 Action
73+
80 Concept
74+
81 Concept
75+
82 Action
76+
83 Concept
77+
84 Concept
78+
85 Concept
79+
86 Action
80+
87 Action
81+
88 Concept
82+
89 Concept
83+
90 Concept
84+
93 Concept
85+
94 Concept
86+
92 Concept
87+
96 Action
88+
97 Concept
89+
98 Concept
90+
101 Action
91+
102 Concept
92+
103 Concept
93+
108 Action
94+
109 Concept
95+
110 Concept
96+
112 Action
97+
113 Concept
98+
115 Concept
99+
116 Action
100+
117 Concept
101+
118 Concept
102+
121 Concept
103+
122 Concept
104+
123 Concept
105+
124 Action
106+
126 Concept
107+
127 Action
108+
128 Concept
109+
129 Concept
110+
130 Concept
111+
131 Action
112+
132 Concept
113+
133 Concept
114+
134 Concept
115+
135 Action
116+
136 Concept
117+
137 Concept
118+
4 Concept
119+
14 Concept
120+
23 Concept
121+
38 Action
122+
114 Concept
123+
111 Concept
124+
138 Concept
125+
139 Concept
126+
140 Concept
127+
141 Concept
128+
142 Concept
129+
143 Concept
130+
25 Concept
131+
69 Action
132+
144 Concept
133+
47 Concept
134+
145 Concept
135+
91 Concept
136+
146 Concept
137+
95 Concept
138+
99 Concept
139+
147 Concept
140+
100 Concept
141+
104 Action
142+
105 Concept
143+
106 Concept
144+
107 Action
145+
119 Concept

0 commit comments

Comments
 (0)