Ceph: OSD down

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

# Check cluster health
ceph health detail
# Check OSD status
ceph osd tree
# Start the OSD
sudo systemctl start ceph-osd@0
# Check OSD logs
journalctl -u ceph-osd@0 --since '10 min ago'

How to diagnose Storage errors

Storage errors are dominated by permissions and locality. S3 access denials can come from an IAM policy, a bucket policy, a Block Public Access setting, an ACL or a KMS key policy: five independent layers, all of which must allow the call. Block storage errors are usually about availability zones: a volume can only attach to an instance in the same AZ.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

1. For S3 denials, check all five layers: IAM policy, bucket policy, Block Public Access, object ACL and the KMS key policy if the object is encrypted with a customer-managed key.
2. Confirm the region and the exact bucket name. A bucket in another region returns a redirect or a permanent error that reads like a permission problem.
3. For signed URLs, check the expiry and that the clock on the signing machine is correct. A skewed clock produces URLs that are already expired.
4. For NFS stale file handles, remount rather than retrying; the file the handle referred to no longer exists on the server.
5. For volume attachment failures, verify the availability zone matches and that the volume is in the available state, not still attached elsewhere.

Tools worth reaching for

• aws s3api get-bucket-policy
• IAM Policy Simulator
• mount -v / showmount -e
• df -h / du -sh
• aws ec2 describe-volumes

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

📖 docs.ceph.com

Related Storage errors

• AWS S3: Access Denied on an object encrypted with KMSThe bucket policy allows the read but the caller has no permission on the KMS key, so S3…
• AWS S3: Bucket does not existSpecified S3 bucket doesn't exist or is in different region.
• AWS S3: SignatureDoesNotMatchThe SigV4 signature computed by the client differs from the one S3 computed. Causes: wrong…
• AWS: EBS volume not available in AZAn EBS volume cannot be attached to an EC2 instance because they are in different…
• Azure Blob: Lease conflictBlob is leased by another client. Cannot modify until lease expires or is released.
• Browser: Storage quota exceededlocalStorage or IndexedDB quota exceeded. Browser has storage limits per origin.
• Google Cloud Storage: Signed URL expiredGCS signed URL has exceeded its expiration time. Need to generate new signed URL.
• Linux: mount error(13): Permission denied (CIFS/SMB)The SMB server rejected the credentials or the requested protocol. Common causes are a domain…

See all 13 Storage errors →

Browse other categories

• HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
• JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
• Database 41Connections, deadlocks, constraints, replication and memory limits.
• AI 35Rate limits, context windows, GPU memory and model-serving failures.
• Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
• Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
• Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
• Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
• System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
• Cloud 25IAM permissions, quotas, service limits and credential failures.
• Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
• TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.