adv-nlp-9

Exam

How long does final assignment take

• Setting up the experiment pipeline: 3 days
  • Reading the test data.
  • Running inference on both systems
  • Calculate failure rate per test
  • Store results in file
• Testing dataset
  • What is the target token?
  • Which predicate is this about?
  • Which argument label do you expect?

In an invariance test, the test passes if there’s not a change. If both before and after classifications are wrong, the test still passes. For this reason, combine an MFT with an INV test.

5 capabilities (only 1 robustness capability max) 1 test per capability easiest (to just pass) is MFT

To do:

• To semantic prototypicality
  • Make it more specific. Effect of semantic prototypicality (limited by voice, and syntactic patterns (e.g. “questions”), and argument label)
  • add two MFTs, one for the before, one for the after
  • add an INV “The man laughs at the fish.” —> “The beach laughs at the fish.”