Skip to main content

Upload and download datasets

tip

Recommended tutorials before starting uploading and downloading datasets.

1. Sign in​

Before you start using the CLI, you need to log in with your username and password.

stochasticx login --username "my_email@email.com" --password "my_password"

2. Create and upload datasets​

Download a Hugging Face dataset and save it in the disk. In this tutorial, SQuAD dataset will be downloaded from the Hugging Face Hub. It is a dataset that contains questions, answers and contexts to finetune a model in the question-answering task.

from datasets import load_dataset
dataset = load_dataset("squad")
dataset.save_to_disk("./squad")

Now you can upload this dataset to the Stochastic platform:

stochasticx datasets upload \
    --name "squad_dataset" \
    --dir_path "./squad" \
    --type "hf"
Output
[+] Uploading dataset...
[+] Dataset uploaded
info
  • name: will allow you to identify the dataset later.
  • dir_path: directory path where your dataset is located.
  • type: dataset type. Right now we are supporting Hugging Face, CSV, JSON, JSONL datasets.

3. List and inspect your datasets​

You can list your datasets with the following code or command:

stochasticx datasets ls
CLI output
[+] Collecting all datasets

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Id                       ┃ Name                         ┃ Directory path ┃ Type  ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 62e912a598855200266c0478 │ squad_dataset                │                │ hf    │ True     │
└──────────────────────────┴──────────────────────────────┴────────────────┴───────┴──────────┘

You can get the columns (features) of a dataset executing the following code or command:

stochasticx datasets columns --id "62e912a598855200266c0478"
CLI output
[+] Collecting columns from the dataset

['id', 'title', 'context', 'question', 'answers']

Inspect a dataset to get more details like description, splits, size in bytes and data samples:

stochasticx datasets inspect --id "62e912a598855200266c0478"
CLI output
{
   "splits":{
      "train":{
         "size_in_bytes":79346108,
         "num_examples":87599
      },
      "validation":{
         "size_in_bytes":10472984,
         "num_examples":10570
      },
      "total_size_in_bytes":89819092
   },
   "dataset_type":"hugging_face",
   "name":"squad",
   "description":"Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
   "column_names":[
      "id",
      "title",
      "context",
      "question",
      "answers"
   ],
   "tasks":[
      "question-answering-extractive"
   ],
   "features":{
      "id":{
         "dtype":"string",
         "id":"None",
         "_type":"Value"
      },
      "title":{
         "dtype":"string",
         "id":"None",
         "_type":"Value"
      },
      "context":{
         "dtype":"string",
         "id":"None",
         "_type":"Value"
      },
      "question":{
         "dtype":"string",
         "id":"None",
         "_type":"Value"
      },
      "answers":{
         "feature":{
            "text":{
               "dtype":"string",
               "id":"None",
               "_type":"Value"
            },
            "answer_start":{
               "dtype":"int32",
               "id":"None",
               "_type":"Value"
            }
         },
         "length":-1,
         "id":"None",
         "_type":"Sequence"
      }
   },
   "data_examples":{
      "train":{
         "id":[
            "5733be284776f41900661182",
            "5733be284776f4190066117f",
            "5733be284776f41900661180",
            "5733be284776f41900661181",
            "5733be284776f4190066117e"
         ],
         "title":[
            "University_of_Notre_Dame",
            "University_of_Notre_Dame",
            "University_of_Notre_Dame",
            "University_of_Notre_Dame",
            "University_of_Notre_Dame"
         ],
         "context":[
            "Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
            "Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
            "Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
            "Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.",
            "Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary."
         ],
         "question":[
            "To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?",
            "What is in front of the Notre Dame Main Building?",
            "The Basilica of the Sacred heart at Notre Dame is beside to which structure?",
            "What is the Grotto at Notre Dame?",
            "What sits on top of the Main Building at Notre Dame?"
         ],
         "answers":[
            {
               "text":[
                  "Saint Bernadette Soubirous"
               ],
               "answer_start":[
                  515
               ]
            },
            {
               "text":[
                  "a copper statue of Christ"
               ],
               "answer_start":[
                  188
               ]
            },
            {
               "text":[
                  "the Main Building"
               ],
               "answer_start":[
                  279
               ]
            },
            {
               "text":[
                  "a Marian place of prayer and reflection"
               ],
               "answer_start":[
                  381
               ]
            },
            {
               "text":[
                  "a golden statue of the Virgin Mary"
               ],
               "answer_start":[
                  92
               ]
            }
         ]
      },
      "validation":{
         "id":[
            "56be4db0acb8001400a502ec",
            "56be4db0acb8001400a502ed",
            "56be4db0acb8001400a502ee",
            "56be4db0acb8001400a502ef",
            "56be4db0acb8001400a502f0"
         ],
         "title":[
            "Super_Bowl_50",
            "Super_Bowl_50",
            "Super_Bowl_50",
            "Super_Bowl_50",
            "Super_Bowl_50"
         ],
         "context":[
            "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
            "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
            "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
            "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
            "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50."
         ],
         "question":[
            "Which NFL team represented the AFC at Super Bowl 50?",
            "Which NFL team represented the NFC at Super Bowl 50?",
            "Where did Super Bowl 50 take place?",
            "Which NFL team won Super Bowl 50?",
            "What color was used to emphasize the 50th anniversary of the Super Bowl?"
         ],
         "answers":[
            {
               "text":[
                  "Denver Broncos",
                  "Denver Broncos",
                  "Denver Broncos"
               ],
               "answer_start":[
                  177,
                  177,
                  177
               ]
            },
            {
               "text":[
                  "Carolina Panthers",
                  "Carolina Panthers",
                  "Carolina Panthers"
               ],
               "answer_start":[
                  249,
                  249,
                  249
               ]
            },
            {
               "text":[
                  "Santa Clara, California",
                  "Levi's Stadium",
                  "Levi's Stadium in the San Francisco Bay Area at Santa Clara, California."
               ],
               "answer_start":[
                  403,
                  355,
                  355
               ]
            },
            {
               "text":[
                  "Denver Broncos",
                  "Denver Broncos",
                  "Denver Broncos"
               ],
               "answer_start":[
                  177,
                  177,
                  177
               ]
            },
            {
               "text":[
                  "gold",
                  "gold",
                  "gold"
               ],
               "answer_start":[
                  488,
                  488,
                  521
               ]
            }
         ]
      }
   }
}

4. Download a specific dataset​

You can also download these datasets in your local system:

stochasticx datasets download \
    --id "62e912a598855200266c0478" \
    --path "./downloaded_dataset.zip"
CLI output
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.0M/14.0M [00:02<00:00, 6.87MiB/s]
info
  • dataset_id: dataset ID you want to download.
  • local_path: local path where you want to save this dataset. Note that the downloaded dataset will be a zip file.