S3Access

S3 Access

exception awsimple.s3.BucketNotFound(bucket_name)

Bases: AWSimpleException

class awsimple.s3.S3Access(bucket_name: str | None = None, **kwargs)

Bases: CacheAccess

S3 Access

Parameters:
  • bucket_name – S3 bucket name

  • kwargs – kwargs

bucket_exists() → bool

Test if S3 bucket exists

Returns:

True if bucket exists

bucket_list() → list

list out all buckets (not called list_buckets() since that’s used in boto3 but this returns a list of bucket strings not a list of dicts)

Returns:

list of buckets

create_bucket() → bool

create S3 bucket

Returns:

True if bucket created

delete_bucket() → bool

delete S3 bucket

Returns:

True if bucket deleted (False if didn’t exist in the first place)

delete_object(s3_key: str)

Delete an S3 object

Parameters:

s3_key – S3 key

dir(prefix: str = '') → Dict[str, S3ObjectMetadata]

Do a “directory” of an S3 bucket where the returned dict key is the S3 key and the value is an S3ObjectMetadata object.

Use the faster .keys() method if all you need are the keys.

Parameters:

prefix – only do a dir on objects that have this prefix in their keys (omit for all objects)

Returns:

a dict where key is the S3 key and the value is S3ObjectMetadata

download(s3_key: str, dest_path: str | Path) → bool

Download an S3 object

Parameters:
  • s3_key – S3 key

  • dest_path – destination file or directory path. If the path is a directory, the file will be downloaded to that directory with the same name as the S3 key.

Returns:

True if downloaded successfully. Raises AWSimpleException if the download fails after all retries.

download_cached(s3_key: str, dest_path: Path) → S3DownloadStatus

download from AWS S3 with caching

Parameters:
  • dest_path – destination full path or directory. If the path is a directory, the file will be downloaded to that directory with the same name as the S3 key.

  • s3_key – S3 key of source

Returns:

S3DownloadStatus instance

download_object_as_json(s3_key: str) → List | Dict
download_object_as_json_cached(s3_key: str) → List | Dict

download object from AWS S3 with caching

Parameters:

s3_key – S3 key of source

Returns:

S3DownloadStatus instance

get_s3_object_metadata(s3_key: str) → S3ObjectMetadata

Get S3 object metadata. Raises AWSimpleException if the object does not exist.

Parameters:

s3_key – S3 key

Returns:

S3ObjectMetadata

get_s3_object_url(s3_key: str) → str

Get S3 object URL

Parameters:

s3_key – S3 key

Returns:

object URL

get_s3_transfer_config() → TransferConfig
keys(prefix: str = '') → List[str]

List all the keys in this S3 Bucket.

Note that this should be faster than .dir() if all you need are the keys and not the metadata.

Parameters:

prefix – only do a dir on objects that have this prefix in their keys (omit for all objects)

Returns:

a sorted list of all the keys in this S3 Bucket (sorted for consistency)

object_exists(s3_key: str) → bool

determine if an s3 object exists

Parameters:

s3_key – the S3 object key

Returns:

True if object exists

read_lines(s3_key: str) → List[str]

Read contents of an S3 object as a list of strings

Parameters:

s3_key – S3 key

Returns:

a list of strings

read_string(s3_key: str) → str

Read contents of an S3 object as a string

Parameters:

s3_key – S3 key

Returns:

S3 object as a string

set_public_readable(public_readable: bool)
upload(file_path: str | Path, s3_key: str, force: bool = False) → bool

Upload a file to an S3 object

Parameters:
  • file_path – path to file to upload

  • s3_key – S3 key

  • force – True to force the upload, even if the file hash matches the S3 contents

Returns:

True if uploaded, False if the S3 object was already up to date. Raises AWSimpleException if the upload fails after all retries.

upload_object_as_json(json_serializable_object: List | Dict, s3_key: str, force=False) → bool

Upload a serializable Python object to an S3 object

Parameters:
  • json_serializable_object – serializable object

  • s3_key – S3 key

  • force – True to force the upload, even if the file hash matches the S3 contents

Returns:

True if uploaded

write_lines(input_lines: List[str], s3_key: str)

Write a list of strings to an S3 bucket

Parameters:
  • input_lines – a list of strings

  • s3_key – S3 key

write_string(input_str: str, s3_key: str)

Write a string to an S3 object

Parameters:
  • input_str – input string

  • s3_key – S3 key

class awsimple.s3.S3DownloadStatus(success: bool = False, cache_hit: bool | None = None, cache_write: bool | None = None)

Bases: object

cache_hit: bool | None = None
cache_write: bool | None = None
success: bool = False
class awsimple.s3.S3ObjectMetadata(bucket: str, key: str, size: int, mtime: datetime.datetime, etag: str, sha512: str | None, url: str, legacy_sha512: str | None = None, crc64nvme: str | None = None, version_id: str | None = None, checksum_type: str | None = None, checksums: Dict[str, str] = <factory>)

Bases: object

bucket: str
checksum_type: str | None = None
checksums: Dict[str, str]
crc64nvme: str | None = None
etag: str
get_content_sha512() → str | None

Get the SHA-512 of this object’s contents from S3’s metadata: S3’s native full-object SHA-512 if the object has one, otherwise the SHA-512 in awsimple’s object metadata.

Returns:

SHA-512 (as hex string), or None if the object has neither (or the object metadata value is not a SHA-512)

get_identity_sha512() → str

Get a SHA-512 of this object’s identity (bucket, key, version) and of S3’s native metadata that changes when the contents do (size, modification time, ETag and every native checksum). This is not a hash of the object’s contents. It is what is initially used to find a cached object that has no SHA-512 of its contents: awsimple never updates an S3 object’s metadata on a read, so such an object is checked against its native CRC64NVME, key, size and modification time, which are all in this hash. The key is only ever hashed, never used as a file name, so file system path irregularities (e.g. leading or doubled slashes, case-insensitive or reserved names, characters that are not allowed in a file name) can not make two keys the same or make a key unusable.

A native checksum alone is not used since a CRC (e.g. CRC64NVME, which awsimple’s uploads use) is not collision resistant: different contents with the same CRC are easy to make. With the identity in the hash, two different objects only have the same identity hash if SHA-512 itself collides.

Returns:

SHA-512 (as hex string)

get_sha512() → str

Get the SHA-512 of this object’s contents if S3’s metadata has it (see get_content_sha512()), otherwise the identity hash (see get_identity_sha512()).

Returns:

SHA-512 (as hex string)

key: str
legacy_sha512: str | None = None
mtime: datetime
sha512: str | None
size: int
url: str
version_id: str | None = None
awsimple.s3.serializable_object_to_json_as_bytes(json_serializable_object: List | Dict) → bytes