Manage datasets
tip
Recommended tutorials for setting up the local version
1. Add a dataset
Download a Hugging Face dataset and save it on your disk. In this tutorial, the SQuAD dataset will be downloaded from the Hugging Face Hub. It is a dataset that contains questions, answers and contexts to finetune a model in the question-answering task.
from datasets import load_dataset
dataset = load_dataset("squad")
dataset.save_to_disk("./squad")
Now you can upload this dataset to the dataset registry using:
stochasticx datasets add \
--name "squad_dataset" \
--dir_path "./squad" \
--type "hf"
info
- name: will allow you to identify the dataset later.
- dir_path: directory path where your dataset is located.
- type: dataset type. Right now we are supporting Hugging Face, CSV, JSON, JSONL datasets.
2. List and inspect your datasets
You can list your datasets with the following command:
stochasticx datasets ls
CLI output
[+] Collecting all local datasets
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Id ┃ Name ┃ Directory path ┃ Type ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 62e912a598855200266c0478 │ squad_dataset │ │ hf │ True │
└──────────────────────────┴──────────────────────────────┴────────────────┴───────┴──────────┘
You can get the columns (features) of a dataset by executing the following code or command:
stochasticx datasets columns --id "62e912a598855200266c0478"
Output
['id', 'title', 'context', 'question', 'answers']
Inspect a dataset to get more details like description, splits, size in bytes and data samples:
stochasticx datasets inspect --id "62e912a598855200266c0478"
CLI output
{
"splits":{
"train":{
"size_in_bytes":79346108,
"num_examples":87599
},
"validation":{
"size_in_bytes":10472984,
"num_examples":10570
},
"total_size_in_bytes":89819092
},
"dataset_type":"hugging_face",
"name":"squad",
"description":"Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
"column_names":[
"id",
"title",
"context",
"question",
"answers"
],
"tasks":[
"question-answering-extractive"
],
"features":{
"id":{
"dtype":"string",
"id":"None",
"_type":"Value"
},
"title":{
"dtype":"string",
"id":"None",
"_type":"Value"
},
"context":{
"dtype":"string",
"id":"None",
"_type":"Value"
},
"question":{
"dtype":"string",
"id":"None",
"_type":"Value"
},
"answers":{
"feature":{
"text":{
"dtype":"string",
"id":"None",
"_type":"Value"
},
"answer_start":{
"dtype":"int32",
"id":"None",
"_type":"Value"
}
},
"length":-1,
"id":"None",
"_type":"Sequence"
}
},
"data_examples":{
"train":{
"id":[
"5733be284776f41900661182",
"5733be284776f4190066117f",
"5733be284776f41900661180",
"5733be284776f41900661181",
"5733be284776f4190066117e"
],
"title":[
"University_of_Notre_Dame",
"University_of_Notre_Dame",
"University_of_Notre_Dame",
"University_of_Notre_Dame",
"University_of_Notre_Dame"
],
"context":[
"Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
"Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
"Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
"Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
"Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary."
],
"question":[
"To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?",
"What is in front of the Notre Dame Main Building?",
"The Basilica of the Sacred heart at Notre Dame is beside to which structure?",
"What is the Grotto at Notre Dame?",
"What sits on top of the Main Building at Notre Dame?"
],
"answers":[
{
"text":[
"Saint Bernadette Soubirous"
],
"answer_start":[
515
]
},
{
"text":[
"a copper statue of Christ"
],
"answer_start":[
188
]
},
{
"text":[
"the Main Building"
],
"answer_start":[
279
]
},
{
"text":[
"a Marian place of prayer and reflection"
],
"answer_start":[
381
]
},
{
"text":[
"a golden statue of the Virgin Mary"
],
"answer_start":[
92
]
}
]
},
"validation":{
"id":[
"56be4db0acb8001400a502ec",
"56be4db0acb8001400a502ed",
"56be4db0acb8001400a502ee",
"56be4db0acb8001400a502ef",
"56be4db0acb8001400a502f0"
],
"title":[
"Super_Bowl_50",
"Super_Bowl_50",
"Super_Bowl_50",
"Super_Bowl_50",
"Super_Bowl_50"
],
"context":[
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50."
],
"question":[
"Which NFL team represented the AFC at Super Bowl 50?",
"Which NFL team represented the NFC at Super Bowl 50?",
"Where did Super Bowl 50 take place?",
"Which NFL team won Super Bowl 50?",
"What color was used to emphasize the 50th anniversary of the Super Bowl?"
],
"answers":[
{
"text":[
"Denver Broncos",
"Denver Broncos",
"Denver Broncos"
],
"answer_start":[
177,
177,
177
]
},
{
"text":[
"Carolina Panthers",
"Carolina Panthers",
"Carolina Panthers"
],
"answer_start":[
249,
249,
249
]
},
{
"text":[
"Santa Clara, California",
"Levi's Stadium",
"Levi's Stadium in the San Francisco Bay Area at Santa Clara, California."
],
"answer_start":[
403,
355,
355
]
},
{
"text":[
"Denver Broncos",
"Denver Broncos",
"Denver Broncos"
],
"answer_start":[
177,
177,
177
]
},
{
"text":[
"gold",
"gold",
"gold"
],
"answer_start":[
488,
488,
521
]
}
]
}
}
}
3. Remove a dataset
stochasticx datasets remove \
--dataset_id "62e912a598855200266c0478"
info
- dataset_id: id of the dataset to remove.