Context: images.nasa.gov metadata
In our work on images.nasa.gov, we extract metadata from image, video, and audio files to make them searchable. Some of these files can be quite large, especially the 4K video files we’re seeing now.
The entire site runs on Amazon Web Services (AWS), with our media files stored on S3 object storage, and metadata extraction running on EC2 autoscaling instances. We download the media files from S3 to EC2, then use ExifTool to pull out EXIF, IPTC, and XMP data including date, title, photographer, camera, geolocation, size, and other technical details.
This has worked fine for us, but presents two concerns: the copy can take nontrivial time, even over AWS internal network; the file may be too big for our currently configured EC2 EBS storage.
Talk Nerdy to Me
Although ExifTool can read data streamed from S3, that sequential access requires it to read the entire file; this is just as slow as copying it, and trades RAM for disk.
But once a file is copied to disk, ExifTool can use “random access” — instead of sequential access — to the file’s structure to skip through the parts it doesn’t need (e.g., large video segments), in order to find what it needs (metadata, in our case). The table below summarizes two posts on MP4 video file format, showing how it’s composed of multiple “boxes” with header containing box size, and type, and the content itself; below, the box sizes and sample content are illustrative only:
|
Box Header (8 bytes) |
Box Data (N bytes) |
|
|
Size (4 bytes) |
Type (4 bytes) |
|
|
2 K |
ftyp File Type |
majorBrand(int:4), minorVersion(int:4), compatibleBrands(int:32[]) |
|
4 GB |
moov
|
MvhdBox{Flags(int:4) Version(int:4), CreateTime(int:4), ModTime(int:4), TimeScale(int:4), Duration(int:4), Rate(int:4), Volume(int:2)}[] |
|
… |
[others] |
… |
|
16 K |
meta
|
Width, Height, Duration, Resolution, …; Photographers, Dates, Copyright |
With sequential access, we’d have to read the 2KB + 4GB data just to get at the metadata. With random access, we read the first 8 bytes to find that it’s “ftyp” and 2KB size, then use the standard UNIX seek() call to skip ahead 2K bytes. We do the same with the next, encountering the “moov” type, then doing a seek() 4GB ahead. We ultimately find our “meta” header then read the 16K metadata we want. The total we’ve had to read is 8 + 8 + 16KB, so just 16 bytes overhead instead of over 4GB.
All this seek()ing is great but our objects live on S3, rather than the local EC2 filesystem. What to do?
HTTP supports byte range requests and S3 Get Object supports this mechanism too. We could conceivably modify ExifTool to simulate seek() and its partners tell() and read() using HTTP byte range requests when talking to S3 objects. I’ve looked at hacking the ExifTool RandomAccess.pm code but it’s but it’s been eons since I’ve written any Perl code and now find it challenging to read. It’s certainly do-able, but it’s a matter of time and priorities, and I’ve just sat on this problem for quite a while.
AWS Mountpoint for Amazon S3
“…open source file client that makes it easy for your file-aware Linux applications to connect directly to Amazon Simple Storage Service (Amazon S3) buckets.”“… large-scale read-heavy applications: data lakes, machine learning training, image rendering, autonomous vehicle simulation, ETL, and more. It supports file-based workloads that perform sequential and random reads, sequential (append only) writes, …”“…many customers have existing applications, commands, tools, and workflows that know how to access files in UNIX style: reading directories, opening & reading existing files, and creating & writing new ones.”
It was easy try it out, we just install an RPM to our AWS Linux EC2 instance, create a mountpoint, then mount our S3 bucket; something like:
mkdir /s3
mount-s3 –read-only my-bucket-o-videos /s3
See How They Run
The release states that the S3 Mountpoint runs on EC2 and even local environments:
Mountpoint can be used from an Amazon Elastic Compute Cloud (Amazon EC2) instance, or within an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (EKS) container. It can also be installed on your existing on-premises systems, with access to S3 either directly or over an AWS Direct Connect connection via AWS PrivateLink for Amazon S3.
My first test was actually local, on my Mac. I built a Docker container based on amazonlinux, installed the S3 Mountpoint RPM and ExifTool, then issued the mount command and ran ExifTool. To run the container, I had to give it extra privileges to the FUSE filesystem, and of course supply my AWS credentials so it could access S3:
Lambda and Fargate: no soup for you!
These elevated privileges aren’t available in AWS-managed container environments like [AWS Fargate](https://aws.amazon.com/fargate/).