Avirup Sil and Alexander Yates (Temple University) Ernest Cronin, Penghai Nie, Yinfei Yang (St. Joseph s University) Ana-Maria Popescu (Yahoo!

Transcription

1 Avirup Sil and Alexander Yates (Temple University) Ernest Cronin, Penghai Nie, Yinfei Yang (St. Joseph s University) Ana-Maria Popescu (Yahoo! Labs)

2 Example Text: Tennessee defeated Philadelphia thanks to Chris Johnson.. Tennessee Tennessee Titans? State_of_Tennessee? Tennessee Aquarium? Philadelphia Philadelphia Eagles? Philadelphia Phillies? City_of_Philadelphia? Chris Johnson Titans Running_Back (nfl)? Houstan_Astros Infielder (mlb)? 2

3 Example Text: Tennessee defeated Philadelphia thanks to Chris Johnson.. Goal in an NED Task: Link named-entities mentioned in text with an established catalog of entities [Bunescu and Pasca, 06, Cucerzan, 07] 3

4 Tennessee defeated Philadelphia thanks to Chris Johnson 4

5 Percentage of names having no referents in Wikipedia 57% of the Named-entities in the Text Analysis Conference s (TAC) 2009 [McNamee et al., 2009] 23% of the Named-Entities in Yahoo! News articles [Zhou et. al. 2010] 16% of the Named-Entities in MSNBC News articles [Cucerzan, 2007] Dataset: NEWS + BLOG Dataset: NEWS 5

6 Training: Tennessee defeated Philadelphia thanks to Chris Johnson.. SPORTS Database id name height position 1 Andre Smith 6 4 OT 2 Andre Smith 6 5 TE 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB Labeled Domain Corpus SPORTS Domain 5 Chris Johnson 6 1 DB Trained NED Model 6

7 Testing: Colts claim TE Andre Smith off waivers from the Bears.. Same Domain Corpus SPORTS Domain Model Trained on Sports Domain SPORTS Database id name height position 1 Andre Smith 6 4 OT 2 Andre Smith 6 5 TE 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB 5 Chris Johnson 6 1 DB 7

8 Testing: Colts claim TE Andre Smith off waivers from the Bears.. Same Domain Corpus SPORTS Domain Model Trained on Sports Domain SPORTS Database id name height position 1 Andre Smith 6 4 OT 2 Andre Smith 6 5 TE 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB 5 Chris Johnson 6 1 DB Needs manually LABELED data 8

9 Cannot use a single trained model on a different domain: NEXT DOOR by Andrew Sawyer, is a gripping ghost story.. Different Domain Corpus MOVIES Domain Model Trained on Sports Domain MOVIE Database id title year 1 Next Door Next Door

10 Current NED techniques and their drawbacks Open-Database Named Entity Disambiguation (Open-DB NED) Task Models Experiments, Results & Discussion Conclusion and Future Work 10

11 Tennessee defeated Philadelphia thanks to Chris Johnson.. SPORTS Database id name height position 1 Andre Smith 6 4 OT 2 Andre Smith 6 5 TE 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB Corpus SPORTS Domain 5 Chris Johnson 6 1 DB Goal: 1. We should not need LABELED data for each new domain 2. The same NED model should work across multiple domains 11

12 Use the same model for NED in different domains: 12

13 Use the same model for NED in different domains: NEXT DOOR by Andrew Sawyer, is a gripping ghost story.. Different Domain Corpus MOVIES Domain Model Trained on Sports Domain MOVIE Database id title year 1 Next Door Next Door

14 Databases should be in Boyce-Codd Normal Form (BCNF) Names, nicknames of the Named-Entities are functional dependencies of the entity Sufficient to use the set of super-keys for relations Result: We can link to IDs only 14

15 Distant Supervision Strategy Domain Adaptation Strategy Hybrid Approach 15

16 We should not label the data for each new source We should not manually construct features for every DB 16

17 An un-supervised approach Does not need labeled data Use automatically generated features from the DB Use the information in the database to provide uncertain labels Train a log-linear model from this probabilistically labeled data 17

18 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. Unlabeled Data from Sports Domain SPORTS id name height position 1 Carlos Lee 6 2 LF 2 Kenny Britt 6 3 WR 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB 5 Chris Johnson 6 1 DB 18

19 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive SPORTS id name height position 1 Carlos Lee 6 2 LF 2 Kenny Britt 6 3 WR 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB 5 Chris Johnson 6 1 DB 19

20 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive SPORTS id name height position 1 Carlos Lee 6 2 LF 2 Kenny Britt 6 3 WR 3 Chris Johnson 6 3 3B 4 Chris Johnson 5 11 RB 5 Chris Johnson 6 1 DB Mentions ID in DB Probability Kenny Britt ID = 2 {T. Titans} 1.0 Chris Johnson ID = 3 {H. Astros} 0.33 Chris Johnson ID = 4 {T. Titans} 0.33 Uncertain Labels Chris Johnson ID = 5 {H. Texans}

21 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans 21

22 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans 22

23 Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Automatically generated from BCNF database: 1. Attribute Counts 2. Similar Entity Counts 23

24 Count the attributes of the entity with its corresponding id in the DB Chris Johnson missed Tuesday's practice at minicamp with an excused absence.. The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans 24

25 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Feature Vector for Chris Johnson (ID=3): Labels Mention ID in DB Count <height> Attribute Count Features Count <position> Count <team> 1.0 Kenny Britt ID = 2 {T. Titans}??? 0.33 Chris Johnson 0.33 Chris Johnson 0.33 Chris Johnson ID = 3 {H. Astros} ID = 4 {T. Titans}??? ID = 5 {H. Texans}??? 25

26 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B O. Raiders.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Feature Vector for Chris Johnson (ID=4): Labels Mention ID in DB Count <height> Attribute Count Features Count <position> Count <team> 1.0 Kenny Britt ID = 2 {T. Titans}??? 0.33 Chris Johnson 0.33 Chris Johnson 0.33 Chris Johnson ID = 3 {H. Astros} ID = 4 {T. Titans} ID = 5 {H. Texans}??? 26

27 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B O. Raiders.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Feature Vector for Chris Johnson (ID=5): Labels Mention ID in DB Count <height> Attribute Count Features Count <position> Count <team> 1.0 Kenny Britt ID = 2 {T. Titans}??? 0.33 Chris Johnson 0.33 Chris Johnson 0.33 Chris Johnson ID = 3 {H. Astros} ID = 4 {T. Titans} ID = 5 {H. Texans}

28 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. SPORTS id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B O. Raiders.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Feature Vector for Kenny Britt (ID=2): Labels Mention ID in DB Count <height> Attribute Count Features Count <position> Count <team> 1.0 Kenny Britt ID = 2 {T. Titans} Chris Johnson 0.33 Chris Johnson 0.33 Chris Johnson ID = 3 {H. Astros} ID = 4 {T. Titans} ID = 5 {H. Texans}

29 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. Mention ID in DB Prob. Count <height> Count <position> Count <team> Kenny Britt ID = 2 {T. Titans} Chris Johnson ID = 3 {H. Astros} Chris Johnson ID = 4 {T. Titans} Chris Johnson ID = 5 {H. Texans} Disambiguation model: Marginal Log-likelihood: Gradient: Objective Function for training Used LBFGS for optimization 29

30 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Wide receiver Kenny Britt (Ht: 6 3, Wt: 215 lbs.) provides an explosive.. Mention ID in DB Prob. Count <height> Count <position> Count <team> Kenny Britt ID = 2 {T. Titans} Chris Johnson ID = 3 {H. Astros} Chris Johnson ID = 4 {T. Titans} Chris Johnson ID = 5 {H. Texans} After Training Phase Probability mass shifts to the correct target Chris Johnson with ID=4 {Tennessee Titans} (for this document) Concerned entity has been linked correctly 30

31 Distant Supervision Strategy Domain Adaptation Strategy Hybrid Approach 31

32 Train: Source DB Labeled Source Corpus Domain Independent NED Model Test: Target DB Unlabeled Target Corpus Trained Domain Independent NED Model Need: Labeled Data for a small set of Domains (training) Target: Generalize to All other Domains 32

33 Train: Source DB Labeled Source Corpus Domain Independent NED Model Test: Target DB Unlabeled Target Corpus Trained Domain Independent NED Model Conditional Log-likelihood: 33

34 Movies: (IMDb) Sports: (Yahoo! Sports) MOVIE Database id title year actors 1 Next Door 1975 Lee Perkins 2 Next Door 2010 Nicole Kruex... SPORTS Database id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B H. Astros.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans 34

35 NEXT DOOR (2010) by Andrew Sawyer, is a gripping ghost story that follows the tale of Anna (Nicole Kruex), a single woman.. Document with mention Next Door MOVIE Database id title year actors 1 Next Door 1975 Lee Perkins 2 Next Door 2010 Nicole Kruex IMDb Movie DB Next Door with ID=1 {1975): Count-All = Count<year> + Count<actor-name> + Count<IMDB-Rating> + = = 0 35

36 NEXT DOOR (2010) by Andrew Sawyer, is a gripping ghost story that follows the tale of Anna (Nicole Kruex), a single woman.. Document with mention Next Door MOVIE Database id title year actors 1 Next Door 1975 Lee Perkins 2 Next Door 2010 Nicole Kruex IMDB Movie DB Domain-Independent Features Feature Vector for the mention Next Door : id title year Count All Count Unique.. 1 Next Door Next Door 2010??

37 NEXT DOOR (2010) by Andrew Sawyer, is a gripping ghost story that follows the tale of Anna (Nicole Kruex), a single woman.. Document with mention Next Door MOVIE Database id title year actors 1 Next Door 1975 Lee Perkins 2 Next Door 2010 Nicole Kruex IMDb Movie DB Next Door with ID=2 {2010}: Count All = Count<year> + Count<actor-name> + Count<IMDB-Rating> + = = 2 37

38 NEXT DOOR (2010) by Andrew Sawyer, is a gripping ghost story that follows the tale of Anna (Nicole Kruex), a single woman.. Document with mention Next Door MOVIE Database id title year actors 1 Next Door 1975 Lee Perkins 2 Next Door 2010 Nicole Kruex IMDb Movie DB Domain-Independent Features Feature Vector for the mention Next Door : id title year Count All Count Unique.. 1 Next Door Next Door

39 39

40 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Document with mention Chris Johnson SPORTS Database id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B O. Raiders.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Yahoo! Sports DB Chris Johnson with ID=4 {T. Titans): Count All = Count<height> + Count<position> + Count<team-name> + = = 3 40

41 The Tennessean reports that Titans running back Chris Johnson (Ht: 5 11, Wt: 191 lbs.) wants a heavy workload in Document with mention Chris Johnson SPORTS Database id name height position team.. 1 Carlos Lee 6 2 LF H. Astros.. 2 Kenny Britt 6 3 WR T. Titans.. 3 Chris Johnson 6 3 3B O. Raiders.. 4 Chris Johnson 5 11 RB T. Titans.. 5 Chris Johnson 6 1 DB H. Texans Yahoo! Sports DB Domain-Independent Features Feature Vector for the mention Chris Johnson : id title Team Count All Count Unique.. 3 Chris Johnson O. Raiders Chris Johnson T. Titans

42 Train on Movies: Domain-Independent Features Label id title year Count All Count Unique Next Door Next Door Test on Sports: Domain-Independent Features id title Team Count All Count Unique.. 3 Chris Johnson O. Raiders Chris Johnson T. Titans

43 Distant Supervision Strategy Domain Adaptation Strategy Hybrid Approach 43

44 Example: Train on Sports and Test on Movies: Sports Movies DB s Set of Labeled Mentions (L s ) Set of Un-Labeled Mentions (M T ) DB T Features: Distant Supervision Feature Functions Training Objective: U union Domain Adaptation Feature Functions 44

45 Current NED techniques and their drawbacks Open-Database Named Entity Disambiguation (Open-DB NED) Experiments, Results, Discussion Future Work 45

46 Movies: 156 Cult movie titles from Database: IMDb Web search + retain top-5 documents Overall 770 documents Sports: Database: Yahoo! Sports [American: Major League Baseball (mlb), National Football League (nfl)] 1300 Yahoo! News articles Domain Mentions OOD Missing from Wikipedia Movies % 84% Sports 564 0% 0% 46

47 Incorporating Wikipedia referents: Allow our system to also link to Wikipedia Use cosine similarity feature Text around mention vs. Text in Wikipedia target page Evaluation Metric: Accuracy of our models Compare: Our Open-DB Model variants Baselines: All OOD Random Oracle Wikifier Modified Zhou Wikifier [Zhou et.al, 2010]: Gradient-boosted decision tree Computes the similarity between a mention s context and a Wikipedia article 47

48 Accuracy Movies Open-DB NED Test Sports Best: Hybrid (in both domains) Second best: Movies: Distant Supervision Sports: Domain Adaptation Baselines Open-DB NED 48

49 Accuracy Movies Open-DB NED Test Sports Open-DB NED outperforms all 3 non-oracle baselines Open-DB NED is significantly better Outperforms Oracle on Movies Handles any domain, like Wikifiers Baselines Open-DB NED Using relational data is better than Wikipedia alone 49

50 Traditional NED has to make way towards Open- DB NED techniques Politics Sports Movies Relational DBs No need for labeled data for each new domain Future Research Question: How can we effectively link large numbers of domainspecific DBs to text? 50

51 Dataset: Avi AVI 51