Lab: Semantic Lifting - CSV - Revision history

Sinoa at 18:56, 3 March 2024

2024-03-03T18:56:35Z

← Older revision		Revision as of 18:56, 3 March 2024
Line 2:		Line 2:
	* Reading non-semantic data tables into semantic knowledge graphs		* Reading non-semantic data tables into semantic knowledge graphs
	* Specifically, reading data in CSV format via Pandas dataframes into RDF graphs		* Specifically, reading data in CSV format via Pandas dataframes into RDF graphs
			* ''If you have more time'', reading data in CSV format via Ontotext Refine into RDF graphs

	== Useful materials ==		== Useful materials ==
Line 16:		Line 17:
	* classes/interfaces from earlier (such as Graph, Namespace, URIRef, Literal, perhaps BNode)		* classes/interfaces from earlier (such as Graph, Namespace, URIRef, Literal, perhaps BNode)
	* also vocabulary classes like RDF (e.g., type), RDFS (e.g., label) and XSD (for various datatypes)		* also vocabulary classes like RDF (e.g., type), RDFS (e.g., label) and XSD (for various datatypes)

			''If you have more time'', Ontotext Refine:
			* [https://www.ontotext.com/products/ontotext-refine/ Ontotext Refine Product page]
			* [https://platform.ontotext.com/ontorefine/ Ontotext Refine Documentation]

	== Tasks ==		== Tasks ==
Line 22:		Line 27:
	'''Task:'''		'''Task:'''
	In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].		In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

	~~''(Unfortunately, the earlier exercises used a wrong and very unfinished version of the russia_investigation_kg.txt file. It has been updated now with the correct version!)''~~

	'''Task:'''		'''Task:'''
Line 82:		Line 85:
	'''Task:'''		'''Task:'''
	If you have not done so already, you should include checks to ensure that you do not add empty columns to your graph.		If you have not done so already, you should include checks to ensure that you do not add empty columns to your graph.


	'''Task:'''		'''Task:'''
	We will be working with the same dataset as in the SPARQL exercise: [https://projects.fivethirtyeight.com/russia-investigation/ FiveThirtyEight's Russia Investigation]. It contains data about special investigations conducted by the United States from the Watergate-investigation until May 2017. [[Russian investigation KG \| This page explains]] the Russia Investigation dataset a bit more.		In the SPARQL exercise and earlier in this lab, you used data downloaded as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

	In the SPARQL exercise, you downloaded ~~the data~~ as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

	'''Task:'''		'''Task:'''
	Install Ontotext Refine on your computer, and create a ~~repostory~~ in GraphDB named ''refine''.		Install Ontotext Refine on your computer, and create a repository in GraphDB named ''refine''.

	* Windows		* Windows
	** Download the Refine .msi installer file.		** Download the Refine ''.msi'' installer file.
	** Double-click the application file and follow the on-screen installer prompts.		** Double-click the application file and follow the on-screen installer prompts.
	** You will be asked for an installation location.		** You will be asked for an installation location.
Line 100:		Line 100:

	* MacOS		* MacOS
	** Download the Refine .dmg file.		** Download the Refine ''.dmg'' file.
	** Double-click it and get a virtual disk on your desktop.		** Double-click it and get a virtual disk on your desktop.
	** Copy the program from the virtual disk to your hard disk Applications folder.		** Copy the program from the virtual disk to your hard disk Applications folder.
Line 107:		Line 107:

	* Linux		* Linux
	** Download the Refine .deb or .rpm file.		** Download the Refine ''.deb'' or ''.rpm'' file.
	** Install the package:		** Install the package:
	** Debian and derivatives: sudo dpkg -i <package-name>.deb		** Debian and derivatives: sudo dpkg -i <package-name>.deb
Line 118:		Line 118:
	Note: Alternatively you can change the ''Repository ID'' in Ontotext Refine under the Setup tab to either a preexisting repository's name, or to one that you would prefer. Ontotext Refine will not make changes to this reposity.		Note: Alternatively you can change the ''Repository ID'' in Ontotext Refine under the Setup tab to either a preexisting repository's name, or to one that you would prefer. Ontotext Refine will not make changes to this reposity.

	'''Task:'''		'''Task:''' ''Data loading''
	''Data loading''

	Load the ~~csv~~ file into Ontotext Refine, and convert te columns' types into the appropriate ones. (Dates to dates, ints to ints.)		Load the CSV-file into Ontotext Refine, and convert the columns' types into the appropriate ones. (Dates to dates, ints to ints.)

	'''Task:'''		'''Task:''' ''Setting up RDF mappings''
	''Setting up RDF mappings''

	Use the Visual RDF Mapper tool to map columns to triples. Try to recreate the structure we had in the KG up until this point. Make sure to set types for literals!		Use the Visual RDF Mapper tool to map columns to triples. Try to recreate the structure we had in the KG up until this point. Make sure to set types for literals!

	'''Task:'''		'''Task:''' ''Setting up RDF mappings''
	''Setting up RDF mappings''

	In addition to the previous structure, we would also like to store the names of presidents as properties.		In addition to the previous structure, we would also like to store the names of presidents as properties.
	Modify your mapping so each president has an foaf:surname, and a foaf:lastname property, with the appropriate values.		Modify your mapping so each president has an foaf:surname, and a foaf:lastname property, with the appropriate values.

	'''Task:'''		'''Task:''' ''Testing your KG''
	''Testing your KG''

	Load your freshly created triples into GraphDB, and try to run the queries you have created during the previous labs.		Load your freshly created triples into GraphDB, and try to run the queries you have created during the previous labs.
	If they do not work as they did before, try and update your mappings so they do!		If they do not work as they did before, try and update your mappings so they do!

	~~[[Lab: DBpedia Spotlight]]~~

Sinoa: /* Tasks */

2024-03-03T18:47:07Z

Tasks

← Older revision		Revision as of 18:47, 3 March 2024
Line 84:		Line 84:


	~~== Tasks ==~~		'''Task:'''
	We will be working with the same dataset as in the SPARQL exercise: [https://projects.fivethirtyeight.com/russia-investigation/ FiveThirtyEight's Russia Investigation]. It contains data about special investigations conducted by the United States from the Watergate-investigation until May 2017. [[Russian investigation KG \| This page explains]] the Russia Investigation dataset a bit more.		We will be working with the same dataset as in the SPARQL exercise: [https://projects.fivethirtyeight.com/russia-investigation/ FiveThirtyEight's Russia Investigation]. It contains data about special investigations conducted by the United States from the Watergate-investigation until May 2017. [[Russian investigation KG \| This page explains]] the Russia Investigation dataset a bit more.

	~~'''Task:'''~~
	In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].		In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

Line 141:		Line 140:
	If they do not work as they did before, try and update your mappings so they do!		If they do not work as they did before, try and update your mappings so they do!

	[Lab: DBpedia Spotlight]		[[Lab: DBpedia Spotlight]]

Sinoa: /* If you have more time */

2024-03-03T18:46:12Z

If you have more time

@@ Line 82: / Line 82: @@
 '''Task:'''
 If you have not done so already, you should include checks to ensure that you do not add empty columns to your graph.
 '''Task:'''
-If you have more time, you can use DBpedia Spotlight to try to link the people (and other "named entities") mentioned in the dataset to DBpedia resources.
+''Data loading''
-ENDPOINT = 'https://api.dbpedia-spotlight.org/en/annotate'
+Load the csv file into Ontotext Refine, and convert te columns' types into the appropriate ones. (Dates to dates, ints to ints.)
-CONFIDENCE = 0.5  # filter out results with lower confidence
-def annotate_entity(entity_name, filters={'types': 'DBpedia:Person'}):
+Load your freshly created triples into GraphDB, and try to run the queries you have created during the previous labs.
-    annotations = []
+If they do not work as they did before, try and update your mappings so they do!
-Useful materials:
+[Lab: DBpedia Spotlight]

Sinoa: /* Tasks */

2023-02-18T17:51:32Z

Tasks

← Older revision		Revision as of 17:51, 18 February 2023
Line 62:		Line 62:

	'''Task:'''		'''Task:'''
	Continue to extend your function to convert the non-semantic CSV dataset into a semantic RDF one. Here is an example of how the data for one investigation could look like:		Continue to extend your function to convert the non-semantic CSV dataset into a semantic RDF one. Here is an example of how the data for one investigation could look like in the end:
	muellerkg:investigation_0 a muellerkg:Indictment ;		muellerkg:investigation_0 a muellerkg:Indictment ;
	muellerkg:american true ;		muellerkg:american true ;

Sinoa: /* Tasks */

2023-02-18T17:50:33Z

Tasks

← Older revision		Revision as of 17:50, 18 February 2023
Line 23:		Line 23:
	In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].		In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

	''(Unfortunately, the earlier exercises used a wrong and very unfinished version of the russia_investigation_kg.txt . It has been updated now with the correct version!)''		''(Unfortunately, the earlier exercises used a wrong and very unfinished version of the russia_investigation_kg.txt file. It has been updated now with the correct version!)''

	'''Task:'''		'''Task:'''

Sinoa at 17:50, 18 February 2023

2023-02-18T17:50:08Z

← Older revision		Revision as of 17:50, 18 February 2023
Line 22:		Line 22:
	'''Task:'''		'''Task:'''
	In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].		In the SPARQL exercise, you downloaded the data as a Turtle file ([[File:russia_investigation_kg.txt]], which you renamed to ''.ttl''). This time you will [https://github.com/fivethirtyeight/data/tree/master/russia-investigation download the data as a CSV file from GitHub].

			''(Unfortunately, the earlier exercises used a wrong and very unfinished version of the russia_investigation_kg.txt . It has been updated now with the correct version!)''

	'''Task:'''		'''Task:'''
Line 75:		Line 77:
	muellerkg:pardoned false ;		muellerkg:pardoned false ;
	muellerkg:president muellerkg:Richard_Nixon .		muellerkg:president muellerkg:Richard_Nixon .

	~~''Unfortunately the file [[File:russia_investigation_kg.txt]] that you have used in several earlier exercises was a wrong and very unufinished version. It should be updated now!''~~

	== If you have more time ==		== If you have more time ==

Sinoa: /* Tasks */

2023-02-17T15:00:46Z

Tasks

@@ Line 60: / Line 60: @@
 '''Task:'''
-Continue to extend your function to convert the non-semantic CSV dataset into a semantic RDF one.
+Continue to extend your function to convert the non-semantic CSV dataset into a semantic RDF one. Here is an example of how the data for one investigation could look like:
 == If you have more time ==

Rbo027: Added documentation

2023-02-16T08:21:48Z

Added documentation

← Older revision		Revision as of 08:21, 16 February 2023
Line 10:		Line 10:
	Pandas:		Pandas:
	* [https://towardsdatascience.com/pandas-dataframe-playing-with-csv-files-944225d19ff Article about working with pandas.DataFrames and CSV]		* [https://towardsdatascience.com/pandas-dataframe-playing-with-csv-files-944225d19ff Article about working with pandas.DataFrames and CSV]
			* [https://pandas.pydata.org/docs/user_guide/io.html#parsing-options User Guide for Pandas]
	* class: DataFrame (methods: read_csv, set_index, apply, iterrows, astype)		* class: DataFrame (methods: read_csv, set_index, apply, iterrows, astype)

Line 28:		Line 29:

	'''Task:'''		'''Task:'''
	''(Pandas basics)'' Inspect the Pandas dataframe. If you have called your dataframe ''df'', you can check out the expressions below. Use the documentation to understand what each of them does.		''(Pandas basics)'' Inspect the Pandas dataframe. If you have called your dataframe ''df'', you can check out the expressions below. Use the [https://pandas.pydata.org/docs/user_guide/io.html#parsing-options documentation] to understand what each of them does.
	df.shape		df.shape
	df.index # ...and list(df.index)		df.index # ...and list(df.index)

Sinoa: /* If you have more time */

2023-02-13T11:36:19Z

If you have more time

← Older revision		Revision as of 11:36, 13 February 2023
Line 67:		Line 67:

	'''Task:'''		'''Task:'''
	If you have more time, you can use DBpedia Spotlight to try to link the people (and other "named entities") mentioned in the dataset to DBpedia resources. You can start with the code example below, but you will need exception-handling when DBpedia is unable to find a match. For instance:		If you have more time, you can use DBpedia Spotlight to try to link the people (and other "named entities") mentioned in the dataset to DBpedia resources.
			pip install pyspotlight
			You can start with the code example below, but you will need exception-handling when DBpedia is unable to find a match. For instance:
			<syntaxhighlight>
			import spotlight

	~~<syntaxhighlight>~~		ENDPOINT = 'https://api.dbpedia-spotlight.org/en/annotate'
	# ~~parameter given to spotlight to~~ filter out results with confidence ~~lower than this value~~		CONFIDENCE = 0.5 # filter out results with lower confidence
	~~CONFIDENCE = 0.5~~

	def annotate_entity(~~entity~~, filters={'types': 'DBpedia:Person'}):		def annotate_entity(entity_name, filters={'types': 'DBpedia:Person'}):
	annotations = []		annotations = []
	try:		try:
	annotations = spotlight.annotate(~~SERVER~~, ~~entity~~, confidence=CONFIDENCE, filters=filters)		annotations = spotlight.annotate(ENDPOINT, entity_name, confidence=CONFIDENCE, filters=filters)
	# ~~this catches errors~~ thrown from Spotlight, ~~including~~ when no resource is found ~~in DBpedia~~		except spotlight.SpotlightException as e:
	~~except SpotlightException as e:~~		# catch exceptions thrown from Spotlight, for example when no DBpedia resource is found
	print(e)		print(e)
	# handle exceptions here		# handle exceptions here
	return annotations		return annotations
	</syntaxhighlight>		</syntaxhighlight>
	~~Here we use~~ the types-filter with DBpedia:Person, as we only want it to match with people. You can choose to only implement the URIs in the response, or the types as well. ~~An issue here is that~~		The example uses the types-filter with DBpedia:Person, because we only want it to match with people. You can choose to only implement the URIs in the response, or the types as well.

	Useful materials:		Useful materials:
	* [https://www.dbpedia-spotlight.org/api Spotlight Documentation]		* [https://www.dbpedia-spotlight.org/api Spotlight Documentation]
			* [https://pypi.org/project/pyspotlight/ pyspotlight 0.7.2 at PyPi.org]

Sinoa at 11:14, 13 February 2023

2023-02-13T11:14:06Z

← Older revision		Revision as of 11:14, 13 February 2023
Line 5:		Line 5:
	== Useful materials ==		== Useful materials ==
	The textbook (Allemang, Hendler & Gandon):		The textbook (Allemang, Hendler & Gandon):
	* chapter on RDF (section on ''Distributing Data across the Web'')		* chapter 3 on RDF (from section 3.1 on ''Distributing Data across the Web'')
	* [https://github.com/fivethirtyeight/data/tree/master/russia-investigation Information about the dataset]		* [https://github.com/fivethirtyeight/data/tree/master/russia-investigation Information about the dataset]