sync after mkfs?

[Mike Small]

I'm noticing places in both Slackware's installer and debian-installer
where a mkfs run is immediately followed by a sync...

---------------SeTpartitions--------------------------------------------
... other fs types ...

elif [ "$ROOT_SYS_TYPE" = "ext4" ]; then
if [ "$DOFORMAT" = "Check" ]; then
make_ext4 $ROOT_DEVICE "y"
else
make_ext4 $ROOT_DEVICE "n"
fi
elif [ "$ROOT_SYS_TYPE" = "reiserfs" ]; then

... yet more fs types ...

fi
fi # DOFORMAT?

# Sync before identifying and mounting the root device:
sync
------------------------------------------------------------------------

And debian-installer...

---packages/partman-basicfilesystems/commit.d/format_basicfilesystems---
case $filesystem in
ext2)
options=''
if [ -f $id/usage ]; then
options="$options -T $(cat $id/usage)"
fi
if [ "$(udpkg --print-os)" = hurd ]; then
options="$options -b 4096 -I 128 -o hurd"
fi
db_progress START 0 3 partman/text/formatting
db_progress INFO $template
db_progress SET 1
if log-output -t partman --pass-stdout \
mkfs.ext2 -F $device $options >/dev/null; then
sync
status=OK
else
status=failed
fi
db_progress STOP
------------------------------------------------------------------------


Is this just a known thing among those who've been around? mkfs isn't
truly done and with all bits on disk when it exits?

- Mike Sm.

[Rich]

Post by Mike Small
I'm noticing places in both Slackware's installer and
debian-installer where a mkfs run is immediately followed by a
sync...
[snip]
Is this just a known thing among those who've been around? mkfs
isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.

If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.

Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

[Richard Kettlewell]

Post by Rich

Post by Mike Small
I'm noticing places in both Slackware's installer and
debian-installer where a mkfs run is immediately followed by a
sync...
[snip]
Is this just a known thing among those who've been around? mkfs
isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.
If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.
Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
the target device. I’m not going to check other filesystems but the
situation is likely to be the same.

[Mike Small]

Post by Richard Kettlewell

Post by Rich

Post by Mike Small
I'm noticing places in both Slackware's installer and
debian-installer where a mkfs run is immediately followed by a
sync...
[snip]
Is this just a known thing among those who've been around? mkfs
isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.
If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.
Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
the target device. I’m not going to check other filesystems but the
situation is likely to be the same.

Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
i.e. two levels of device mapper. Is it possible that the fsync on the
crypt device wouldn't carry through to the raid?

My reason for looking at what Slackware and Debian do came from seeing a
script at work fail intermittently. It does the following:

1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.

This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

[Richard Kettlewell]

Post by Mike Small

Post by Richard Kettlewell

Post by Rich

Post by Mike Small
I'm noticing places in both Slackware's installer and
debian-installer where a mkfs run is immediately followed by a
sync...
[snip]
Is this just a known thing among those who've been around? mkfs
isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.
If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.
Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
the target device. I’m not going to check other filesystems but the
situation is likely to be the same.

Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
i.e. two levels of device mapper. Is it possible that the fsync on the
crypt device wouldn't carry through to the raid?

It’d certainly be a bug in whichever layer didn’t propagate it, and it
would (almost certainly) affect sync in the same way, i.e. the sync
wouldn’t help in the presence of that (hypothetical) bug.

Post by Mike Small
My reason for looking at what Slackware and Debian do came from seeing a
1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.
This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

‘Fails’ isn’t really enough information to even speculate. Looking at
the behavior, any error message, and any kernel logs, might shed some
light on the situation.

[Bit Twister]

Post by Mike Small

Post by Richard Kettlewell

Post by Rich

Post by Mike Small
I'm noticing places in both Slackware's installer and
debian-installer where a mkfs run is immediately followed by a
sync...
[snip]
Is this just a known thing among those who've been around? mkfs
isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.
If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.
Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
the target device. I’m not going to check other filesystems but the
situation is likely to be the same.

Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
i.e. two levels of device mapper. Is it possible that the fsync on the
crypt device wouldn't carry through to the raid?
My reason for looking at what Slackware and Debian do came from seeing a
1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.
This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

I suggest adding test return status after each step to help find the problem.

Example:

if [ $? -ne 0 ] ; then
_code=$?
echo "Return status $_code is not 0 for command
command_executed_here
"
exit $_code
fi

[Henrik Carlqvist]

Post by Mike Small
1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.
This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

Umount is supposed to sync the file system before unmounting and
returning.

As Bit Twister said, it might be a good idea to check the exit status of
all the commands and especially the umount commoand. Also look for error
messages from the commands and most of all, check for warnings and errors
in the output from dmesg when something like this goes wrong.

regards Henrik

[Mike Small]

Post by Henrik Carlqvist

Post by Mike Small
1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.
This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

Umount is supposed to sync the file system before unmounting and
returning.
As Bit Twister said, it might be a good idea to check the exit status of
all the commands and especially the umount commoand. Also look for error
messages from the commands and most of all, check for warnings and errors
in the output from dmesg when something like this goes wrong.

Well, we checked at least for exit code 0 both on the cp and the umount,
e.g. umount X || die...

I'm looking more closely now, but getting dmesg output is tricky cause
it runs in an initramfs and reboots on the failure. Likely I'll have to
log debugging with dmesg, etc. Just trying to narrow now the steps to
reproduce.

[Henrik Carlqvist]

Post by Mike Small
I'm looking more closely now, but getting dmesg output is tricky cause
it runs in an initramfs and reboots on the failure. Likely I'll have to
log debugging with dmesg, etc. Just trying to narrow now the steps to
reproduce.

Another thing that might be worth checking when suspecting bad hardware
is the output of something like:

smartctl --all /dev/sda

The SMART logged data in the disk is persistent between reboots.

regards Henrik