S3Access¶
S3 Access
- exception awsimple.s3.BucketNotFound(bucket_name)¶
Bases:
AWSimpleException
- class awsimple.s3.S3Access(bucket_name: str | None = None, **kwargs)¶
Bases:
CacheAccessS3 Access
- Parameters:
bucket_name – S3 bucket name
kwargs – kwargs
- bucket_exists() bool¶
Test if S3 bucket exists
- Returns:
True if bucket exists
- bucket_list() list¶
list out all buckets (not called list_buckets() since that’s used in boto3 but this returns a list of bucket strings not a list of dicts)
- Returns:
list of buckets
- create_bucket() bool¶
create S3 bucket
- Returns:
True if bucket created
- delete_bucket() bool¶
delete S3 bucket
- Returns:
True if bucket deleted (False if didn’t exist in the first place)
- delete_object(s3_key: str)¶
Delete an S3 object
- Parameters:
s3_key – S3 key
- dir(prefix: str = '') Dict[str, S3ObjectMetadata]¶
Do a “directory” of an S3 bucket where the returned dict key is the S3 key and the value is an S3ObjectMetadata object.
Use the faster .keys() method if all you need are the keys.
- Parameters:
prefix – only do a dir on objects that have this prefix in their keys (omit for all objects)
- Returns:
a dict where key is the S3 key and the value is S3ObjectMetadata
- download(s3_key: str, dest_path: str | Path) bool¶
Download an S3 object
- Parameters:
s3_key – S3 key
dest_path – destination file or directory path. If the path is a directory, the file will be downloaded to that directory with the same name as the S3 key.
- Returns:
True if downloaded successfully. Raises AWSimpleException if the download fails after all retries.
- download_cached(s3_key: str, dest_path: Path) S3DownloadStatus¶
download from AWS S3 with caching
- Parameters:
dest_path – destination full path or directory. If the path is a directory, the file will be downloaded to that directory with the same name as the S3 key.
s3_key – S3 key of source
- Returns:
S3DownloadStatus instance
- download_object_as_json(s3_key: str) List | Dict¶
- download_object_as_json_cached(s3_key: str) List | Dict¶
download object from AWS S3 with caching
- Parameters:
s3_key – S3 key of source
- Returns:
S3DownloadStatus instance
- get_s3_object_metadata(s3_key: str) S3ObjectMetadata¶
Get S3 object metadata. Raises AWSimpleException if the object does not exist.
- Parameters:
s3_key – S3 key
- Returns:
S3ObjectMetadata
- get_s3_object_url(s3_key: str) str¶
Get S3 object URL
- Parameters:
s3_key – S3 key
- Returns:
object URL
- get_s3_transfer_config() TransferConfig¶
- keys(prefix: str = '') List[str]¶
List all the keys in this S3 Bucket.
Note that this should be faster than .dir() if all you need are the keys and not the metadata.
- Parameters:
prefix – only do a dir on objects that have this prefix in their keys (omit for all objects)
- Returns:
a sorted list of all the keys in this S3 Bucket (sorted for consistency)
- object_exists(s3_key: str) bool¶
determine if an s3 object exists
- Parameters:
s3_key – the S3 object key
- Returns:
True if object exists
- read_lines(s3_key: str) List[str]¶
Read contents of an S3 object as a list of strings
- Parameters:
s3_key – S3 key
- Returns:
a list of strings
- read_string(s3_key: str) str¶
Read contents of an S3 object as a string
- Parameters:
s3_key – S3 key
- Returns:
S3 object as a string
- set_public_readable(public_readable: bool)¶
- upload(file_path: str | Path, s3_key: str, force: bool = False) bool¶
Upload a file to an S3 object
- Parameters:
file_path – path to file to upload
s3_key – S3 key
force – True to force the upload, even if the file hash matches the S3 contents
- Returns:
True if uploaded, False if the S3 object was already up to date. Raises AWSimpleException if the upload fails after all retries.
- upload_object_as_json(json_serializable_object: List | Dict, s3_key: str, force=False) bool¶
Upload a serializable Python object to an S3 object
- Parameters:
json_serializable_object – serializable object
s3_key – S3 key
force – True to force the upload, even if the file hash matches the S3 contents
- Returns:
True if uploaded
- write_lines(input_lines: List[str], s3_key: str)¶
Write a list of strings to an S3 bucket
- Parameters:
input_lines – a list of strings
s3_key – S3 key
- write_string(input_str: str, s3_key: str)¶
Write a string to an S3 object
- Parameters:
input_str – input string
s3_key – S3 key
- class awsimple.s3.S3DownloadStatus(success: bool = False, cache_hit: bool | None = None, cache_write: bool | None = None)¶
Bases:
object- cache_hit: bool | None = None¶
- cache_write: bool | None = None¶
- success: bool = False¶
- class awsimple.s3.S3ObjectMetadata(bucket: str, key: str, size: int, mtime: datetime.datetime, etag: str, sha512: str | None, url: str, legacy_sha512: str | None = None, crc64nvme: str | None = None, version_id: str | None = None, checksum_type: str | None = None, checksums: Dict[str, str] = <factory>)¶
Bases:
object- bucket: str¶
- checksum_type: str | None = None¶
- checksums: Dict[str, str]¶
- crc64nvme: str | None = None¶
- etag: str¶
- get_content_sha512() str | None¶
Get the SHA-512 of this object’s contents from S3’s metadata: S3’s native full-object SHA-512 if the object has one, otherwise the SHA-512 in awsimple’s object metadata.
- Returns:
SHA-512 (as hex string), or None if the object has neither (or the object metadata value is not a SHA-512)
- get_identity_sha512() str¶
Get a SHA-512 of this object’s identity (bucket, key, version) and of S3’s native metadata that changes when the contents do (size, modification time, ETag and every native checksum). This is not a hash of the object’s contents. It is what is initially used to find a cached object that has no SHA-512 of its contents: awsimple never updates an S3 object’s metadata on a read, so such an object is checked against its native CRC64NVME, key, size and modification time, which are all in this hash. The key is only ever hashed, never used as a file name, so file system path irregularities (e.g. leading or doubled slashes, case-insensitive or reserved names, characters that are not allowed in a file name) can not make two keys the same or make a key unusable.
A native checksum alone is not used since a CRC (e.g. CRC64NVME, which awsimple’s uploads use) is not collision resistant: different contents with the same CRC are easy to make. With the identity in the hash, two different objects only have the same identity hash if SHA-512 itself collides.
- Returns:
SHA-512 (as hex string)
- get_sha512() str¶
Get the SHA-512 of this object’s contents if S3’s metadata has it (see get_content_sha512()), otherwise the identity hash (see get_identity_sha512()).
- Returns:
SHA-512 (as hex string)
- key: str¶
- legacy_sha512: str | None = None¶
- mtime: datetime¶
- sha512: str | None¶
- size: int¶
- url: str¶
- version_id: str | None = None¶
- awsimple.s3.serializable_object_to_json_as_bytes(json_serializable_object: List | Dict) bytes¶