chmod/chown/mkdir: Operation not permitted as root

I get Operation not permitted for root user on one server but not on the other "identical" server.

Running on Amazon Linux 1.

Server 1:

[root@preprod-1 ]# chown root:root /s3mnt/outliers/
chown: changing ownership of ‘/s3mnt/outliers/’: Operation not permitted
[root@preprod-1 ]# chmod -x /s3mnt/outliers/
chmod: changing permissions of ‘/s3mnt/outliers/’: Operation not permitted
[root@preprod-1 ]# cd /s3mnt/outliers
[root@preprod-1 outliers]# mkdir test2
mkdir: cannot create directory ‘test2’: Operation not permitted
[root@preprod-1 ]# ls -ld /s3mnt/outliers
drwxrwxrwx 1 brutus brutus 0 Aug 2 2017 /s3mnt/outliers
[root@preprod-1 ]# ls -ld /s3mnt/
drwxrwxrwx 1 brutus brutus 0 Jan 1 1970 /s3mnt/
[root@preprod-1 ]# sestatus
SELinux status: disabled

Server 2:

[root@prod-1-1 via]# ls -ld /s3mnt/outliers
d--------- 1 brutus brutus 0 Jan 1 1970 /s3mnt/outliers
[root@prod-1-1 via]# ls -ld /s3mnt/
drwxrwxrwx 1 brutus brutus 0 Jan 1 1970 /s3mnt/
[root@prod-1-1 via]# cd /s3mnt/outliers
[root@prod-1-1 outliers]# mkdir test2
[root@prod-1-1 outliers]# ls -ld test2/
drwxr-xr-x 1 brutus brutus 0 Nov 25 12:45 test2/
[root@prod-1-1 outliers]# chmod -x /s3mnt/outliers/test2/
[root@prod-1-1 outliers]# ls -ld test2/
drw-r--r-- 1 brutus brutus 0 Nov 25 12:45 test2/
[root@prod-1-1 outliers]# chown root:root test2
[root@prod-1-1 outliers]# ls -ld test2/
drw-r--r-- 1 brutus brutus 0 Nov 25 12:45 test2/
[root@prod-1-1 ]# sestatus
SELinux status: disabled

Mounted as rw per mount cmd on both servers. Both servers have identical /etc/fstab.

Setup on both servers is identical:

lrwxrwxrwx 1 root root 14 May 21 2018 s3mnt -> /storage/s3mnt
drwxr-xr-x 7 brutus brutus 4096 May 21 2018 storage
# mount
s3fs on /storage/s3mnt type fuse.s3fs (rw,relatime,user_id=0,group_id=0,allow_other) 

When changing permissions on the mountpoint I get the expected changing ownership of Input/output error. So it's not that. Wish it was :(

What else can I check to find out why I am running into the above issues on one server but not the other?

2 Answers

Is that an s3 s3mntfilesystem? can you show the mounted filesystems, on both systems?

lsattr would be also usefull to check for any other attributes that would preventing you.

It turned out to be an issue with S3 bucket policy denying access to the S3 mount.