News Data

Downloading News Flat Files

To download news flat files, use the same AWS credentials provided for sentiment data (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY).

Download Methods

Using Boto3 (Python)

import boto3

s3 = boto3.client(
    's3',
    aws_access_key_id="<INSERT_YOUR_KEY>",
    aws_secret_access_key="<INSERT_YOUR_KEY>"
)

s3.download_file('the-tie-flat-files', 'news.zip', 'news.zip')

Manual Download

Alternatively, you can download the news.zip file directly from the S3 bucket using your preferred S3 client.

Data Format

File Structure

The data consists of JSON files organized by month. Each file contains news items in the exact format returned by the API.

Complete Dataset

These files include all news items, including those that are usually filtered out.

Data Fields

Unique Identifier

link_hash - Serves as the unique identifier for each news item

Timestamps

timestamp - When the news item was scraped by The Tie

Timezone: UTC
Recommended: Use this field as the primary timestamp

date - Official publication timestamp (when available)

Provided by the original source
No timezone conversion applied (as-is from source)
May differ from timestamp

Low-Quality News Items

By default, our frontend filters out low-quality news items. However, the flat files include everything.

"Low quality" refers to any news item tagged with something in the tag_subgroup of "Low Quality", which includes:

Retweets
Technical Analysis
Customer Support tweets
And similar content

Low-Quality Tag Example

{
  "id": 553,
  "name": "Retweet",
  "tag_subgroup": {
    "id": 9,
    "name": "Low Quality"
  }
}

Metadata Synchronization

You can sync all available metadata for /tags, /sources, and /collections at their respective endpoints. This metadata is also inherently included in the nested objects for each news item.

Support

For questions or assistance with news data, contact us at [email protected].