Historical Data Access

Downloading Historical Flat Files

To download historical flat files, you will need a set of AWS credentials that provide access to the S3 bucket where the data is stored. These credentials are separate from your API credentials and are used exclusively for backfilling historical data.

AWS Credentials

S3 Bucket Configuration

s3_bucket = "s3://the-tie-flat-files/"
AWS_ACCESS_KEY_ID = "<INSERT_YOUR_KEY>"
AWS_SECRET_ACCESS_KEY = "<INSERT_YOUR_KEY>"

Important: Replace <INSERT_YOUR_KEY> with your actual AWS credentials. Contact your Account Manager if you need access credentials.

Python Script

For your convenience, we provide a Python script to help you download and process historical sentiment files. Alternatively, you can use the S3 bucket and credentials above to retrieve the data using your own tools.

Requirements

The script requires the following Python packages:

boto3
cloudpathlib
pandas
numpy

Install them using:

pip install boto3 cloudpathlib pandas numpy

Sample Script

import numpy as np
import pandas as pd
from cloudpathlib import S3Client
import warnings
import glob

# AWS Configuration
s3_bucket = "s3://the-tie-flat-files/sentiment_historical"
AWS_ACCESS_KEY_ID = "<INSERT_YOUR_KEY>"
AWS_SECRET_ACCESS_KEY = "<INSERT_YOUR_KEY>"

# Initialize S3 Client
client = S3Client(
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)

def download_coin(coin_uid, dest_folder=".", s3_bucket=s3_bucket, client=client):
    """Download data for a specific coin"""
    s3_path = f"{s3_bucket}/coin_uid={coin_uid}"
    cp = client.CloudPath(s3_path)
    
    if not cp.exists():
        warnings.warn(
            f"Folder for {coin_uid} does not exist in the bucket."
        )
        return None
    
    cp.download_to(f"{dest_folder}/coin_uid={coin_uid}")
    return cp

def list_coins_in_bucket(s3_path, client=client):
    """List all coins available in the bucket"""
    cp = client.CloudPath(s3_path)
    
    if not cp.exists():
        warnings.warn("No coins found for this path.")
        return []
    
    return [i.key.split("coin_uid=")[-1] for i in cp.glob("*")]

def load_df(coin_uid, data_dir="."):
    """Load files into a single dataframe"""
    list_files = glob.glob(f"{data_dir}/coin_uid={coin_uid}/year=*/*.csv.gz")
    ignore_files = glob.glob(f"{data_dir}/coin_uid={coin_uid}/year=0000/*.csv.gz")
    
    df = pd.DataFrame()
    
    for f in list_files:
        if f not in ignore_files:
            try:
                df_tmp = pd.read_csv(f)
                df_tmp["datetime"] = pd.to_datetime(df_tmp["datetime"])
                df_tmp = df_tmp.set_index("datetime")
                df = pd.concat([df, df_tmp])
            except:
                pass  # Skip empty or corrupted files
    
    return df.sort_index()

# Example: Download and load Bitcoin data
coin_uid = "bitcoin"
download_coin(coin_uid, dest_folder="./data")
df = load_df(coin_uid, data_dir="./data")

print(df.head())

Data Structure

Coin Identification

The mapping between coin_uid and tickers can be obtained from the /coins reference endpoint:

https://api-thetie.io/v3/quant/coins

This endpoint also provides equivalences to CoinGecko and CoinMarketCap identifiers.

Available Coins

To get a list of all available coins in the bucket:

coin_list = list_coins_in_bucket(s3_bucket)
print(f"Found {len(coin_list)} coins")

Empty Files and Delisted Coins

Empty files indicate that no sentiment data was collected during that time period
The bucket includes coins that have been delisted to reduce selection bias in historical analysis

Data Specifications

Timezone

All timestamps are in UTC.

Granularity

Mid-2021 onwards: 1-minute granularity
Prior to mid-2021: Gradually increases from daily → hourly → 5-minute intervals

All metrics are rolling calculations with no aggregation differences between intervals.

Available Metrics

For sentiment data columns and their definitions, see the Metric Definitions documentation.

Support

If you have questions or need assistance, contact your Account Manager or [email protected].