DataDrop
Sell

Add assets

Adding assets to the inventory: uploads from the browser, over SFTP and with the datadrop command.

Draft: not yet checked against the product

Add asset, at the top of the inventory, opens Add to inventory. Choose what you are adding: Codebases, Tasks or Raw data.

Raw data

Raw data is the kind you can upload today, three ways. Each one makes a new raw-data asset in your inventory as soon as the upload starts.

From the browser

Drag your files onto Drag your data here to upload it, or choose them with Browse files. To send a folder, choose it with Browse a folder (a dropped folder is refused).

  • One .zip file is kept byte for byte. A folder, or several loose files, is sent file by file.
  • Before anything is sent, the browser works out a fingerprint (SHA-256) of each file, and the platform checks every file against it when it arrives.
  • The upload tray at the bottom right of the page shows each upload: Uploading, Checking, Done, or Refused with the reason.
  • A closed tab only pauses the upload. The files are not kept in the browser, so after a reload or a new tab the upload shows Paused: choose the same files again and press Resume, and only what is still missing is sent. Files that are not the ones the upload started with are refused.
  • An upload left unfinished is kept for a week after the last part arrived, then it closes as Expired; Start again uploads into the same asset.
  • Clearing a row from the tray only takes it off this browser's tray. The asset stays in your inventory.

Your own upload has no Cancel button in the web app. A Team admin's tray also lists the colleagues' uploads in progress, each with Cancel, which asks to confirm.

Over SFTP

Upload over SFTP makes a new raw-data asset and a login for it: a host, a port, a user name, a password and the server's host key. Connect any SFTP client with them, send your files, then press Finish.

  • The password is shown this once. It is never shown again and cannot be read back; DataDrop keeps only a scrambled form of it. If you lose it, Generate new password makes a new one and the old one stops working at once.
  • The login works for five days, and the login view shows the exact time it ends. A new password starts the five days again.
  • Before you send the password, check that the server's key matches the Host key shown.
  • While the upload is open, it is listed above the inventory under SFTP uploads in progress, with what has arrived so far and three buttons: Generate new password, Revoke (the login stops working; a new password opens it again) and Finish.
  • Finish is open once a file has arrived and no file is still being sent. When a file stopped part-way, you choose Finish without them or Finish keeping them (what arrived of each file).
  • After Finish, the login stops working and the files are checked, as a browser upload's are.

If the server you use has no SFTP set up, the button says so and makes nothing.

With the datadrop command

Upload with the datadrop CLI makes the asset and a login, as Upload over SFTP does, and shows a command to run from the folder's parent:

datadrop upload ./my-data --host HOST --port PORT --user USER --fingerprint FINGERPRINT
  • The command asks for the password, or reads it from the DATADROP_SFTP_PASSWORD variable. It never takes the password as an argument.
  • It checks that the server shows the key with that fingerprint before it sends the password.
  • It sends every file of the folder (links are not followed), finishes the upload, checks that what arrived is the folder, and waits for the check's result. --no-wait stops after the finish.
  • After a dropped connection, run the same command again: it carries on from what the server already holds.
  • Its exit code says how it ended: 0 stored, 1 refused (or still being checked when it stopped waiting), 2 a usage or connection problem, or the service could not take a file just now (it says "try again": run the same command again, it carries on where it stopped), 3 what arrived is not the folder (the files are named).

datadrop upload uses the login the web app makes.

Getting the command

The datadrop command has no released download yet. Until it has one, upload over SFTP with any SFTP client (above), or from the browser.

With an API key

A Team admin makes an organization API key at Settings, API keys (the address /settings/api-keys, or API keys in the search menu): a name, the Uploads scope (the other two scopes are for the API) and a lifetime of 30, 90 or 365 days. The whole key is shown once; copy it then. Every member sees the keys and when each was last used; only a Team admin renames or revokes one.

With a key, the command makes a new upload of a folder on its own:

datadrop create ./my-data --api https://YOUR-SITE/api --org ORGANIZATION_ID --title "My data"
  • The key is read from --key, else the DATADROP_API_KEY variable, else a prompt, sent only to an https address (or this machine's), and never printed.
  • A key works only for its own organization, and datadrop create needs one with the Uploads scope. It stops at once when it is revoked, when it expires, or when the Team admin who made it is no longer one.
  • --asset ID uploads a new revision of an asset instead of making one; --no-wait stops after the finish. A stopped run is not continued: run it again for a new upload.

When an upload is refused

The check refuses an archive it cannot accept: one that is password protected, compressed in a way it does not take or far more than real data can be, holds links or entries with names that are not allowed, nests archives too deep, or unpacks to more than the limit. A file whose bytes are not the ones declared is refused too. The tray says which. Nothing of a refused upload is kept, and the asset stays, so you can upload into it again.

The limits on sizes and counts are on Fees and limits.

Codebases

The Codebases step lists the code hosts, Add in bulk via API and Upload as files. The code hosts are listed only where the site serves them; see Connect your accounts.

  • Upload as files takes one ZIP of the repository, its .git folder included so the history comes along. The codebase's row appears at once, named after the ZIP, and the upload goes on in the tray like any other. Anything but one .zip is refused before anything is sent. The details the edit form asks for are yours to fill in afterwards.
  • Add in bulk via API shows how to register many codebases in one request, POST /v2/organizations/{your organization's id}/codebases/batch: all of them are registered or none, and each shows as incomplete until its data is uploaded. It links to the Vendor API reference.
  • A personal upload link for a coding agent, good for one ZIP for two hours, is made through the API (POST .../upload-links); the web app does not offer it, and the API does not serve such links on the live site yet.

Tasks

The Tasks step's Ember Data row opens Import tasks. It needs your team's Ember Data key: with none saved, a Team admin connects one in the dialog (it is kept at Settings, Ember Data). Then choose the one type every task of the import gets, tick whole tasksets or open one to choose single tasks, and import: every task becomes its own row in your inventory, grouped by its taskset.

On this page