Synthetic supplier lines. Held-out score.
On the held-out lines, the evolved harness did not beat best-of-N. Model: gemma4:e4b. Each held-out line got 3 calls per method. Fit used 17 separate calls and those calls are not in the table.
| Method | Held-out field accuracy | Held-out calls |
|---|---|---|
| best of n | 0.875 | 24 |
| refine | 0.8958 | 24 |
| evolved | 0.5417 | 24 |
KITKAT 37G 24PK @ 720 INR
Best of N: {"brand": "KITKAT", "name": null, "size": "37G", "pack": 24, "unit_price": 720, "currency": "INR"}
Refine: {"brand": "KITKAT", "name": "KITKAT", "size": "37G", "pack": 24, "unit_price": 720, "currency": "INR"}
Evolved: {"brand": null, "name": null, "size": "37G", "pack": 24, "unit_price": 30.0, "currency": "INR"}
HORLICKS 500G 10PK RS 2100
Best of N: {"brand": "HORLICKS", "name": null, "size": "500G", "pack": 10, "unit_price": 2100, "currency": "RS"}
Refine: {"brand": "HORLICKS", "name": "RS", "size": "500G", "pack": 10, "unit_price": 2100, "currency": "RS"}
Evolved: {"brand": null, "name": null, "size": "500G", "pack": 10, "unit_price": 210, "currency": "INR"}
PEPSI 2L 6PK @ 390 INR
Best of N: {"brand": "PEPSI", "name": null, "size": "2L", "pack": 6, "unit_price": 390, "currency": "INR"}
Refine: {"brand": "PEPSI", "name": "2L", "size": "2L", "pack": 6, "unit_price": 390.0, "currency": "INR"}
Evolved: {"brand": null, "name": null, "size": "2L", "pack": 6, "unit_price": 65, "currency": "INR"}